Advice on drafting multiple-choice questions:
Technical and Ethical Considerations

by

Dr Bruce Littleboy
Department of Economics
The University of Queensland

Multiple-choice questions are sometimes so poorly drafted that pointers to the right answers are supplied. Students with good "MCQ technique" but with little knowledge can do better than they deserve. Examples of common mistakes are given, and ways to avoid them are identified. Sadistic teachers can even design questions that elicit wrong answers from students relying on technique. Ethical issues also arise. Questions need to be drafted fairly as well as effectively.

 

Multiple-choice questions (MCQs) have become important as assessment instruments. Evaluation is becoming more automated and less reliant on skilled labour, under the prevailing conditions of growing student numbers and shrinking job security.

It takes effort and expertise even to write the relatively unambitious and routine multiple-choice questions that avoid assessing deeper learning. Over the years, I have written several hundred multiple-choice questions and have checked exam papers and tutorial exercises of my colleagues. Mistakes recur. This paper is prompted by work I'm now doing in adapting a large test bank associated with a well-regarded economics textbook. Going through a thousand questions would sensitise anyone to good and bad drafting techniques.

Those who draft questions face problems common across the disciplines though some examples here have an economics orientation. These commonly encountered problems have workable solutions.

Defects are commonly encountered. I do not mean ambiguities, incoherence, outright errors or ridiculously easy questions, though these are common enough. I mean nave drafting practices that help students select the right answer even if they know little about the material tested. Flagging the right answer, or narrowing the array to only a couple of selectable options, is remarkably widespread.

Here are some common drafting mistakes

 

1. What I call "naive hurdling".

    

2. One question stating the answer to another.

    

3. Failing to think of a sufficient number of plausible options.

 

1. Naive Hurdling

Often writers place several hurdles in the path of students to give them as many opportunities as possible to go wrong.

Example 1

Presumably, you can guess the right answer without even seeing the question. It's likely to be c.

There are two hurdles the student needs to jump: getting the right number and getting the right sign (knowing whether a profit or loss was made).

Here are three simple solutions.

Example 1 Solution 1

Example 1 Solution 2

Example 1 Solution 3

This is a sneaky solution involving flagging a wrong answer.

 

Those using technique alone would pick b, whereas the answer is c. The occasional use of this technique (one time in about 5?) also seems ethically justifiable to me.

 

Example 2

a  demand falls, price rises and quantities rise
b  demand falls, price rises and quantities fall
c  and rises, price rises and quantities rise
d  demand rises, price falls and quantities fall
e  demand rises, price rises and quantities fall.

The student needs to jump 3 hurdles here:

 

1. Does demand rise or fall?
2. Does price rise or fall?
3. Do quantities rise or fall?

 

Here is the problem. Frequently examiners try to put as many hurdles as possible in the student's path. Any student with good MCQ technique would narrow the options down to c or e. How? By following the longest branch to its end.

Three options have "demand rises". Only 2 have "fall". So it's probably c, d or e. Of these 2 have price rises, so it's c or e, a 50-50 bet. (Maybe e is more probable, there being 3 "quantities fall".)

Example 2 Solution 1: deception
The right answer could sometimes be one of the minority options to trick those relying on technique; i.e. demand falls is part of the right answer. But extended hurdling is too valuable a technique to forgo lightly. Here is a superior solution.

Example 2 Solution 2: cunning rearrangement
A common drafting practice is to have a string of ascending hurdles. The first barrier is low, the second is more demanding, and the last is likely to be the most challenging.

Suppose that even the most rudimentary students would probably realize that demand rises. Still, you want to include demand falling in order to identify anyone who could be deeply confused about the basics.

If one of the hurdles is rather obvious, simply move it. By carefully rearranging each of the options you can employ some disguised hurdling. Reference to demand could go last and quantities could come first in each option.

a   quantities rise and price rises as demand falls.
b   quantities fall and price rises as demand falls.
c   etc.

Example 2 Solution 3: administrative
If there are multiple versions of the paper for different meeting times or within a class to deter "roving eyes", then let it be known that you arbitrarily change "rise" and "fall" between versions of the paper. This could bluff some students into regarding any apparent hurdling structure as a random artifact.

 

2. One question states the answer to another

This can happen very easily if you use MCQ banks and instruct your computer to select randomly. I have also seen the practice in exam papers constructed by hand...

Example 3

1. The sky is
   
   1. blue
   2. green
   3. horrible
   4. all of the above

   
   

2. The sky is blue because
   
   1. ...
      
   2. ...
   3. ...

In case you hadn't noticed, 2 states the answer for 1.

Example 3 Solution 1
Be aware of this pitfall and simply stop making this mistake.

Example 3 Solution 2
Replace question 2.

2. The sky is blue because

1. ...
   
2. ...
3. ...
4. none of the above.

The answer is d. Although I have seen this technique used, I dislike it on ethical grounds. The examiner does get the pleasure of catching students who pick b for question 1, but I think a student is entitled to feel that there is a convention that the trunk of an MCQ is true. (More ethically defensible is: d. The sky is not typically green.)

 

3. Failing to devise enough plausible options

This is a common defect, but there is little excuse for it. Often the problem is that the issue involved is too thin or lacks sufficient complexity, but good drafting can still impede students who rely on mere technique to pass.

To reduce the effect of guessing, 5 options are better than 4. Reaching 5 options however, sometimes seems impossible. In fact, it is quire easy when you know how. Most of us have probably had an experience like the one shown in the next example.

Example 4

This question actually isn't too bad, provided that sometimes the right answer is d or e.

Example 4 Solution 1 (a bad but common, one)

Obviously e is wrong¹ as a matter of logic. And there can't be two answers, so b and d are out. (Having 2 essentially equivalent answers trivially rearranged is an annoyingly common and lazy practice.) It's a 50-50 for a student bet without even seeing the question.

Example 4 Solution 2

This solution involves the judicious use of "always", "never", "sometimes", "seldom" etc.

It is rare that right answers involve the extremes of always or never, and students relying on technique know not to affirm excessive claims. So try to devise a question where extreme-sounding statements are right one time in four or five.

Example 4 Solution 3

 

Expand the ambit of the question so that new options arise for students to take a position on

etc

Example 4 Solution 4

 

This solution involves bundling statements to expand the scope of the question. Try out the following question structure.

Consider the following two statements:

 

1. If blah blah, A = B
2. A does not generally equal B
   1. Both 1 and 2 are true
   2. Both 1 and 2 are false
   3. Only 1 is true
   4. Only 2 is true

Here four options are easy to generate, but you can be more creative.

For example,

 

1. 2 is true when C = D
2. 1 is true because ...
3. 1 is claimed true by Bloggs, but Eek criticises him in her article, "Bloggs Sux".

Bundling can be a handy drafting tool because, when generating multiple versions of a test for different tutorial groups, simple rearrangements or adjustments will change the right answer in a way that is not transparent to students.

Furthermore, bundling allows you to gather together several odd fragments to form a viable question. It is also obviously easier to generate 5 routine options from 3 statements than from 2. (All three are true; only 2 and 3 are true, only 1 and 3 are true...) This example also happens to raise other, more general, issues.

Example 5

 

Consider the following three statements:

 

1. The sky is blue
2. Bill Clinton is the President
3. John Lennon led the Russian revolution

 

1. Only 2 and 3 are true
2. 2 is true because the people of Australia elected him
3. 1 is never true
4. Bloggs wrongly claims 3 in his book
5. Both c and d.

This bundling technique is really a series of true-false questions. Any single T-F question yields a 50-50 chance of getting the mark, but now a guesser only has one chance in 5. Furthermore, major drawbacks of T-F questions are avoidable. If one statement is "almost true", then students are often placed in an unfair and frustratingly counter-productive position. How false does an option have to be to be judged false overall? Is the sky blue at night or when cloudy or when on Venus? These difficulties are avoided by shrewdly designing the options. In the example above, 1 is (sort of) true, 2 is (currently) true and 3 is false. None of the 5 options requires a student to judge 1 according to nuance or to try to read the question-setter's mind as to how true a true statement needs to be. The question is both effective and fair because it tests knowledge of material rather than clairvoyant ability.

Solution 5 The cunning use of apparent padding

Example6

 

... blah blah is

 

1. always true
2. usually true
3. sometimes true, but mostly false
4. intended as a joke
5. a silly confusion

The last two options look like desperate ways to pad out the list to reach five options. Most of the time such options are, but be creative and sometimes (ideally, one time in five) have an "Oh-it-couldn't-be-that!" option as the right answer.

Solution 6 Both - and -

Suppose one option is obviously true, one is obviously false and the other two would genuinely sort students out. It's easy to generate five options now.

Example 7

 

1. something obviously true
2. Something obviously false
3. Something difficult but true
4. Something difficult but false
5. Both a and d [Or "Both a and c", if you prefer.]

The fifth option is superior to the common and lazy "All of the above" or "None of the above", both of which should get no takers - except those students who are reduced to blind guessing. This option forces students to take a position on the more advanced material and it discriminates between those with three levels of knowledge (advanced, basic and none).

This is also a handy way to turn a bank of questions with four options into one with five options per question. Do ensure, however, that one time in five the "Both - and -" option is right. Otherwise students will learn over time simply to ignore this option.

 

Ethical Considerations

So far this paper has focussed mostly on how improvements in writing technique can minimise the success of students with good reading technique. By using the suggested solutions - if you aren't using some of these already - your MCQs will be both more testing and fairer to the students who have really gained the required knowledge. Avoiding being predictable is both efficient and equitable. Use any sly tricks only one time in five. Be self-aware. Make sure one-fifth of your right answers are (a) even though it is very hard psychologically to bring yourself to present the right answer straight away. Fairness matters more generally than just here. In addition to some already mentioned, there are lines I think should not be crossed.

I don't know how true this is in other disciplines, but many questions in economics not only test a student's grasp of economics, but also test the knowledge of unstated conventions relating to the interpretation of MCQs.

In economics, students are somehow supposed to know that other factors, which potentially could suddenly change in the real world, are being held constant for the purposes of the question. If government spending rises, for example, the correct answer is that total spending and output rise as a result. This assumes that private firms are not so alarmed by the government's policies that they curtail their investment.

Furthermore, economics students are deemed to know something quite subtle. Suppose they are asked to predict the effect on the system if X suddenly changes, and typically for no apparent or stated cause. ("Suppose the price of onions falls. This will cause ...") It is allegedly implicit in the question that the system was in a state of rest to begin with and now it is in disequilibrium. The question is therefore about the behaviour of the model of reality rather than about the turbulent reality itself.

Perhaps there are two counterparts in other disciplines. At very least these conventions should be expressly highlighted in tutorials or workshops. Learning in the post mortem about what the question-setter regards as implicit smacks of unethical teaching.

There are two other ethically questionable practices to reflect on. They probably reach across disciplines.

The first questionable practice relates to the misuse of "None of the above". Usually students are told to pick the relatively best answer (the most correct, or least objectionable, option). Including "None of the above" makes this difficult. How large does a slight qualification or reservation have to be before "None of the above" is the right answer? One can almost always conjure up some scenario that does cast doubt over what would otherwise clearly be the right answer. You could unintentionally penalise students who probe deeper than you expected. There is an ethical obligation to use "None of the above" with great care, and not just add it on the end to pad out the number of options.

Also be aware that choosing "None of the above" often requires courage from the student. Suppose the question involves a calculation, and the right answer is "None of the above". Students worried about making a silly arithmetic error under exam pressure are reassured if they reach an answer on the list. Otherwise, they needlessly check their working several times.

I once set a test where the first question had a right answer of "None of the above". This had a most unsettling effect on several students. Maybe I should not have done it, but I'm not sure.

The second questionable practice to reflect on relates to a series of questions where the answer to one question needs to be plugged into the information supplied in the next question to allow the latter question to be answered.

Students who make any mistake, conceptual or arithmetic, at step one are punished again and again in subsequent questions in the chain. A more ethical practice is to spend more time in drafting the questions. There are fairer ways to spread out the distribution and to discern different levels of achievement.

Example 8: the questionable way

 

1. Calculate the interest rate (i) if ...
   1. 1
   2. 2
   3. 3
   4. 4
   5. none of the above
2. Given the answer to the previous question, what is the level of investment (I) if I = 20 - 2i.
   1. 18
   2. 16
   3. 14
   4. zero
   5. None of the above

You need to get 1 right to get 2 right. Indeed, because of 2e, anyone who wrongly calculated 4 as the interest rate in question 1 does not even get a signal they got it wrong. Why not simply redraft question 2?

Example 9: the defensible way

New 2.    What would the level of investment be if i = 10 and I = 20 + 2i?

 

In conclusion, it is possible to be fair, efficient and cunning. A good writer of multiple-choice questions needs to be ... all of the above.

 

¹   Except, perhaps, in the rare case where the right answer is that sometimes A exceeds B, sometimes equals B and sometimes is less than B. Here the word "sometimes" would need to be in the trunk of the question rather than implicit.

 

Teaching and Educational Development Institute,
The University of Queensland
Brisbane, Queensland 4072 Australia
    Phone: +61 (7) 3365 2788
Authorised by: ACE Group, Teaching and Educational Development Institute
Modified: 8 March 2002
© 2002 The University of Queensland