Self-Test Scores, with and without Certainty-Based Marking (CBM), on CBM Self-Tests or Moodle

This is about understanding final CBM scores on LAPT at UCL, or on Moodle**. It assumes you already know the basics of using CBM :- the why and the how.

Scores can be expressed relative to the whole quiz, or to the questions that you chose to respond to :
CBM vs Accuracy Your browser does not support the HTML5 canvas tag, so cannot show your score on the graph.
CBM vs Accuracy
This is an important distinction. In a teacher-controlled quiz (as mostly in Moodle), you will normally only get top marks (100% correct) if you answer all the questions correctly. In a self-test (as mostly in LAPT), you control what questions you work with - for example, selecting particular topics or skipping questions that seem too easy or that you haven't yet studied. You will then be more interested in how well you have done on your chosen questions. 100% means you got all the questions you responded to correct Responding "No idea" to a question gives zero marks for that question but it will count as a response, so you get feedback about the correct answer.

% Correct, or 'Accuracy' :
This is simply the percentage of the relevant questions that you got right. It doesn't depend on your certainty ratings (C=1,2,3). Correct answers all count equally (1 mark) though some partially correct answers may be given a fraction of a full mark, for example if nearly correct, or perhaps spelled wrongly. Accuracy=50% means that the total of correct and fractional marks comes to half the number of questions, either in the quiz or that you chose.

Average Certainty-Based Mark (CBM) :
This is the average of your CBM marks per question (either for the whole quiz or for the questions you chose). Maximise your CBM marks by (1) striving (of course!) for accuracy, and (2) identifying any uncertainties or grounds for confidence in each answer. The possible range of average CBM marks is from +3 (all correct at C=3) to -6 (all incorrect at C=3). Note, however, that you shouldn't ever expect to get a negative average, because even complete guesses (as long as you acknowledge that's what they are) will get you some positive marks and no penalties.

Think about this graph. One interesting thing about it is that your CBM average can't ever get above the dashed line. In practice you can't realistically expect to get even that high. So your CBM average as a percentage of the absolute maximum (3.0) is bound to be less than your Accuracy. That might seem a bit discouraging. But if you discriminate between reliable and less reliable answers using CBM, you should get a score above the heavy lines that show your CBM average if you don't discriminate (answering everything at the same C level). LAPT and Moodle** give you a positive CBM Bonus score for successfully doing better than this and getting into the green zone. This can (unless you have misconceptions and get in the pink zone with a negative bonus) enhance your Accuracy score.

CBM Bonus :
Online and exam data Online and exam data This measures how far your CBM average is into the green (or the pink) on the graph: it indicates how well (or badly!) you judged the reliability of your answers, identifying those that were more and less reliable. The bonus is added to the 'Raw Accuracy' (i.e. without CBM) to give your 'CB Accuracy'. Typically, students thinking carefully about the subject will get a positive bonus, even if they don't know vey much, so long as they know what they do and don't know. Negative bonuses mean you have misconceptions (confident incorrect ideas), or sometimes that you are underconfident about reliable answers. They are a warning that you don't really understand which bits of your knowledge are reliable. If you were always to enter the same certainty level your bonus would be either zero or (if your C level is inappropriate for your overall accuracy) negative. Nearly always you should be able to distinguish questions where you are on more and less secure ground. The bonus (as a %) is calculated as 10 times the difference, on this graph, between your average CBM mark and the mark expected with a uniform C level appropriate for your accuracy.

CB Accuracy = Raw Accuracy + Bonus :
Adding your bonus to your accuracy gives your Certainty-Based Accuracy. This better reflects the combination of your knowledge, insight and understanding of the issues. As explained above, for students practised in using CBM the CBM bonus is usually positive and CB Accuracy will be greater than simple raw accuracy (without CBM). Negative bonuses signal misconceptions or inappropriate judgements of reliability, which should benefit from practice and more careful thinking about the issues.

Exam Assessment :
If your CBM scores are being used for exam assessment, it is up to the examiners whether they use positive and/or negative bonuses. Data illustrated above show that bonuses after exam revision and CBM practice are mainly positive, though students using online self-tests commonly experience negative bonuses due to misconceptions and over-confidence before you master the subject, or at least learn which bits you understand.

Knowledge Scores (adjusted for the effect of guessing) :
When questions give you a limited number of possible answers, you can expect to get a significant percentage correct just by guessing, without any real knowledge. Your expected accuracy with guesses would be 50% for True/False questions and 20-25% if there are 4 or 5 alternatives offered for each single best answer. Examiners conveniently think about this in terms of a 'Knowledge Score' (%K) for which accuracy is rescaled so on average guesses give K=0% and all correct K=100%. If Accuracy = A and accuracy expected by chance = C, then K = (A-C) / (100% - C). A typical exam passmark might be around K=50%, similar for different question types but varying depending also on the required standard. This would correspond to 75% accuracy on a True/False test. With CBM in Moodle**, your Knowledge Score is calculated based on your CB Accuracy.

** The Moodle scores described here may require that your Moodle provider has installed the CBM enhancements provided here.
Unmodified Moodle code may implement a CBM interface without providing properly analysed scores.

Tony Gardner-Medwin (UCL) email   Home Page   CBM at UCL (LAPT open resource)