Good Multiple-Choice Questions Measure Stated Learning Objectives from the Course

It is easy inadvertently to measure superfluous skills, such as knowledge of test-taking strategies, that do not address your learning objectives. The following strategies will help you reduce this threat to the validity of your questions:

• Avoid using obviously wrong “freebee” distractors, just for fun or because you ran out of plausible options. (If you want to use a “fun” distractor, make it an additional option, ensuring that you still have two to four distractors in which you have some confidence.)
  
• Avoid writing items that purposefully disadvantage students who know a lot about multiple-choice tests and thus are “test-wise.” Strategies used by test-wise students will be discussed more in the next section.
  
• Avoid including questions regarding material that is trivial or obscure, which is sometimes done to produce the illusion of difficulty and/or a normal distribution of scores. Your goal is to provide a valid measure of student ability, not to produce a particular distribution of scores (see Part Four, Chapter 14: Concepts in Assessment).
  
• Think carefully about the appropriate difficulty of your distractors. Most of the time distractors are not “completely” wrong, and frequently the answer is not correct to the exclusion of all other potentially correct answers; rather, answers fall on a continuum from very wrong to very right. As seen in Figure 15.2, your task is to come up with an answer that is clearly correct to prepared students, and distractors that are clearly wrong to prepared students, but attractively plausible to the unprepared. Figure 15.3 illustrates this rule with a practical example. The correct answer is the technique that most commonly produces the error, but the “kind of right” distractors are unnecessarily tricky because they are also right, but less specific than the right answer. The “wrong, but attractively plausible to the unprepared” distractors are clearly wrong, but are common errors that create problems when making blood smears, and therefore are plausible to those who did not prepare. The “very wrong” options are obviously “throw-away” distractors. Students may find them entertaining, but even the least-prepared student would not choose them. It can be very difficult to come up with three or four compelling distractors for any given item. If this occurs, take comfort in the knowledge that three options (the answer and two good distractors) are typically sufficient for appropriate statistical properties, and that few items, even professionally written and edited ones, have more than two “functional” distractors (meaning that they are chosen by more than a handful, maybe 5%, of respondents; Downing, 2006).

Effective Multiple-Choice Items Measure a Single Objective with Precision by Providing a Clear and Appropriate Stem, and Options That Address the Question Posed in the Stem

Generally such items include most of the relevant detail in the stem, which tends to be longer than the options. The options are short, clear, and address the specific question posed by the stem. The item should be answerable by covering the options and reading the stem. For this reason, questions that begin with phrases such as “Which of the following is true about …?” are generally not recommended.

Effective Multiple-Choice Items Distribute the Location of the Correct Answer Evenly among the Various Answer Positions

Test-wise students are accustomed to the correct answer occupying the extreme positions less frequently than the center positions.

1Avoid Providing Grammatical Cues Regarding Correct/Incorrect Options

Grammatical cues are quite common in selected-response item writing. In Example 1, options a, b, and d are all adjectives and align with the stem grammatically; option c, a noun, does not. In cases like this, the item writer probably initially wrote the item with all nouns or all adjectives in the options, but later made some changes (such as revising how the stem was constructed) and did not revise all of the options correctly. Your students will know that you would not be careless enough to misalign the correct answer grammatically, so they will automatically know that the misaligned answer is not right. Similarly, in Example 2, the answer cannot be “plum,” or any other fruit beginning with a consonant, because the stem ends in “an.” These kinds of errors can usually be avoided with careful proofreading.

Examples:

1. 1. The most important factor to consider when creating a test item is whether or not it is
   
   
   
   1. a. valid
      
   2. b. reliable
      
   3. c. practicality*
      
   4. d. readable.
   
   
2. 2. The fruit that, when eaten once a day will keep the doctor away, is an
   
   
   
   1. a. apple
      
   2. b. apricot
      
   3. c. orange
      
   4. d. plum.*

2Avoid Questions That Provide a Comprehensive Subset of Options

Case and Swanson (2002) subdivide this principle into two: making sure that a subset is not collectively exhaustive; and that all distractors should be homogenous (e.g., fall into the same category). In Example 3, the first three options comprise a relatively exhaustive subset (comparison between alprazolam and amitriptyline). Option d is conceptually quite different, exploring one potential side effect of alprazolam. Test-wise students will guess that the option that is unlike the comprehensive subset is wrong. Such items give the impression (usually correctly) that the item author created some options from a theme regarding the target objective (in this case, comparison between the efficacy and side effects of alprazolam and amitriptyline), ran out of ideas, and made up another option just to include a prescribed number of distractors. This item might also be considered unnecessarily tricky, depending on how an expert would judge efficacy and the importance/likelihood of various side effects.

Example:

1. 1. You are presented with Max, a 2-year-old male, neutered Labrador Retriever who, over the past six months, has been demonstrating increased anxiety when left alone by his owners. Max appears healthy on physical exam and has no history of injury or illness. Max’s owners are hoping for a medication that will alleviate his symptoms while he participates in behavior modification therapy. For a patient like Max, alprazolam
   
   
   
   1. a. is more efficacious with fewer potential serious side effects than amitriptyline.
      
   2. b. is just as efficacious, with a comparable number of potential serious side effects to amitriptyline.
      
   3. c. is less efficacious, with more potential serious side effects than amitriptyline.
      
   4. d. can be administered confidently because he does not have a history of heart disease.*

3Beware of Convergence

Convergence occurs when options that contain multiple elements overrepresent the correct elements in the distractors. When reviewing items for faculty or students, I (JD) exploit this item flaw more than any other item flaw to correctly answer veterinary-related questions about which I know nothing. This flaw happens when you write the correct answer, and then modify it to create the other options, recycling elements from the correct option. In Example 4, all of the correct elements (the body, counting area, and feathered edge) are found in at least two of the distractors, and come together nicely in option d.

1. 1. A diagnostic blood smear contains which essential elements?
   
   
   
   1. a. the body, the counting area, and the border
      
   2. b. the body, the base, and the feathered edge
      
   3. c. the droplet, the counting area, and the feathered edge
      
   4. d. the body, the counting area, and the feathered edge
      
   5. e. the droplet, the base, and the counting area.

4Make Options Similar in Length and Specificity

Your students can often tell when you have taken more care to craft the correct answer than the distractors, because the correct answer is longer, more detailed, or more nuanced than the distractors. Therefore, make all options similar in length and similarly detailed/nuanced. In Example 5, the correct answer demonstrates a level of care/specificity that is not found in the other options.

Example:

1. 1. Dr. Gutierrez has just diagnosed Buttercup, a 10-year-old German Shepherd, with hemangiosarcoma. Dr. Gutierrez knows that Buttercup’s owner, Ms. Greene, has had some difficulty understanding explanations and following discharge instructions in the past. Given that information, to what element of the SPIKES protocol should Dr. Gutierrez pay particular attention in sharing this information with Ms. Greene?
   
   
   
   1. a. Step 1: Setting up the interview.
      
   2. b. Step 2: Assessing the patient’s perception.
      
   3. c. Step 4: Giving knowledge and information (particularly taking care to give information in small chunks, pausing frequently to assess understanding).
      
   4. d. Step 5: Addressing emotions with empathetic responses.

5List Options in a Logical Order/Consistently

In the case of numerical options, you risk introducing unnecessary and irrelevant complexity when answer options within the same question vary in format or range, and when the options are not presented in sequential order. In Example 6, the answers are confusing because they are not in numerical order. In Example 7, not only are the questions in a confusing order, but one is given in the form of a range, and they vary in terms of how they employ decimals. Keep in mind that if using electronic testing with this sort of question, you will want to disable features that automatically scramble option order.

Examples:

1. 1. How many mg of Enrofloxacin are needed for a 20 kg dog if the dose is 2.5 mg/kg?
   
   
   
   1. a. 60 mg
      
   2. b. 30 mg
      
   3. c. 40 mg
      
   4. d. 50 mg.
   
   
2. 2. How much does a 58 lb dog weigh in kg?
   
   
   
   1. a. 26.3
      
   2. b. 27
      
   3. c. 20–30
      
   4. d. 0.26
      
   5. e. 2.63.

6Avoid Using None of the Above

Questions that employ none of the above as an option are problematic, because students can usually imagine some better correct answer than the best one on the list, and are left to decide whether you wish them to choose the best one on the list, or the better option you might be thinking of. If you use none of the above, be sure that all the alternatives are clearly, unambiguously, and completely incorrect. In the case of Question 8, option a is a recommended approach, but there are several other options that many would consider as good or preferable, so it is not clear whether a or e is the best answer.

1. 1. You are conducting a study to determine which of two approaches to teaching surgery produces superior learning gains. As part of the study, two blinded raters score each student’s performance. Which statistical test would you use to compare the raters’ scores in order to estimate reliability?
   
   
   
   1. a. intraclass correlation
      
   2. b. linear regression
      
   3. c. analysis of variance
      
   4. d. t-test
      
   5. e. none of the above.*

7Avoid Using All of the Above

Using all of the above is problematic for two reasons. First, students only have to rule out one of the other distractors to know that all of the above is incorrect. Second, if all of the above is the intended correct answer, all of the other options should be clearly right, and equally right. If some of the other options are more right than others, students are forced to determine whether you intend them to choose the one best answer, or all plausible answers. In Example 9, all three strategies could address the problem, although options a or c seem most promising. Prepared students will spend an inordinate amount of time trying to guess what the item writer was thinking.

1. 1. You are practicing creating blood smears that will make the faculty and staff in the pathology lab happy, but you keep producing smears that cover the entire slide and do not show a feathered edge. What would be the best strategy for solving this problem?
   
   
   
   1. a. Use a smaller drop of blood for making your smear.
      
   2. b. Make sure you are not moving your spreader slide too slowly.
      
   3. c. Make sure you are not holding your slide lower than a 30° angle.
      
   4. d. All of the above.*