- Nobody teaches doctors how to write multiple choice questions… and it shows.
- Mistake #1: Unnecessary patient vignettes and extraneous material
- Mistake #2: Negative phrasing
- Mistake #3: Lazy questions and answer choices
- Mistake #4: Obviously incorrect distractors
- Mistake #5: Misinterpreting references and failing to fact-check
- You now know more about how to write multiple choice questions than most doctors who write them
A doctor’s training and career are brimming with multiple choice questions. We face them on the MCAT, medical school quizzes and tests, the 3 steps of the USMLE, board certification exams, and CME assessments. Because they’re so ubiquitous in the medical profession, knowing how to write multiple choice questions that are clear and unflawed is a useful skill to have.
There are many opportunities for medical professionals to write multiple choice questions (MCQs). Some are a great source of income, while others are way to make a contribution to your profession. These include:
- Teaching medical school or other health-related postgraduate courses
- Writing test prep questions as a side gig
- Freelancing as a medical writer
- Volunteering to develop questions for a professional certifying board or society
- Taking a non-clinical position with a medical education or publishing company
There is a lot more to a high-quality MCQs than meets the eye. This article goes over why this is the case and how you can avoid 5 of the most common mistakes that are made in MCQ development for the health and medical professions.
Nobody teaches doctors how to write multiple choice questions… and it shows.
Have you ever gotten frustrated when taking an test or doing practice questions as you prepare for an exam, even though you thought you understood the concept being tested?
If so, you’re definitely not stupid and you probably didn’t have a brain lapse. The most likely cause of your frustration is that the question was poorly written.
I’ve spent a lot of time with MCQs over the course of my training and career, both as a learner and as a test item writer. The number of poorly questions that I come across is considerable. And these aren’t just free prep questions by some random guy online. These sub-par questions come from Q-banks offered by well-regarded professional associations, test prep companies, and medical education providers.
I’m glad that the questions are being written by medical professionals who have been treating patients and who encounter the clinical topics in their own work. However, it is clear that “Test question writing” was never a topic of study in med school or training.
Why are there so many low-quality medical MCQs out there? I can think several major reasons:
Writing MCQs is hard and there’s a science to it. Writing a high-quality test item on a complex topic is challenging. So challenging, actually, that there is an entire science to test item analysis and psychometrics. As a doctor developing test questions, you don’t need to be an expert in this, though it’s a huge help to have a general knowledge of it.
It takes time. Medical writers are often paid by the question to write test prep MCQs. Physicians serving on question-writing task forces are usually volunteering their time. Professors writing exams are busy with other things. We naturally want to get done with our work as quickly as possible. Getting to “good enough” sometimes becomes our goal instead of excellence. But when the careers of students and trainees are on the line, the bare minimum isn’t sufficient.
Medicine isn’t black and white. Medical MCQs are frequently more difficult to write than MCQs in other professions because medicine is just as much of an art as it is a science. There needs to be a “single best answer” for every MCQ, which can be difficult when there is not always a “single best decision” for every clinical situation.
People are lazy. It’s much easier to write a flawed MCQ than a perfect one. Based on my experience, I believe that the epidemic of poorly written MCQs is largely the result of laziness.
As clinicians and scientists, it would be great if we could focus on the clinical and scientific content of the questions we write. But, unfortunately, we can’t consistently rely on others to take our material and apply best item-writing practices to it. My plea to you is to do our profession a favor by avoiding some of the main mistakes made in MCQ writing.
Mistake #1: Unnecessary patient vignettes and extraneous material
Ever been faced by a question that takes up half the page? Or maybe you’ve read through a long patient case only to discover that you could have answered the actual question without it? We all have.
An MCQ can encourage learners to think about a medical topic in its clinical context through the use of patient vignettes. When used correctly, patient cases and clinical data are great additions to an MCQ. Unfortunately, they are used incorrectly far too often.
When including a vignette, the question stem should depend on the information presented in the vignette. For example, the question following a vignette might be, “What is the most likely diagnosis?” or “What is the next best step in management?” Without reading the case, the test-taker wouldn’t be able to answer these questions.
On the other hand, a lengthy vignette about a patient with acute abdominal pain followed by the question, “What is the most common cause of acute abdominal pain in patients presenting to the emergency department?” is useless.
Extraneous material and verbosity
Even when a patient case is needed in order to answer the question, it can get too verbose. Keep it succinct. Rather than listing out an entire review of systems, for instance, stating “Review of systems is noncontributory” might suffice.
Unnecessary information in an MCQ is not always in the form of a patient case. Some questions are guilty of what’s known as “teaching in the question stem.” If the question can be asked without background information, related data, or interesting factoids, then just ask the question.
Sometimes irrelevant information is ok
A challenge we face when practicing medicine in real life is that we need to figure out what information from a patient’s history or test results is relevant. As such, it is reasonable at times to include clinical data in a patient vignette that isn’t directly needed to answer the question. Nonetheless, it should contribute to the question in some way by helping the learner to rule out incorrect answers or to interpret or put into context the other clinical information.
Mistake #2: Negative phrasing
Which of the following is incorrect?
Which is false with regard to…
Which of these is not…
All of the following are true statements except…
Which is the following is least accurate with regard to…
Ugh, I just shuttered as I wrote that last example.
Negative wording such as in the examples above is unnecessarily confusing. It is used frequently by test developers because it’s often easier to come up with several true answers than several false answers.
Negative phrasing is not ideal for a two main reasons:
- A test taker who misses the negative word in the question stem will get the item wrong, even if he is knowledgeable about the topic.
- Rather than picking out a single correct answer, the test taker must essentially answer 3 or 4 true/false questions to identify the correct response.
Double and triple negatives
Negatively worded question stems are at high risk of including negatively worded answer choices, as well. This create double and sometimes even triple negatives. When this happens, you can be sure that the poor learner will spend more time trying to understand the intent of the wording than thinking through her medical knowledge.
Mistake #3: Lazy questions and answer choices
Some “techniques” that are used by MCQ writers to make a test questions more challenging are merely reflections of writer laziness. These include two main flaws: 1) overly vague question stems, and 2) combining answer options.
Which of the following is true?
When presented with a good question stem, a test taker should be able to come up with some potential answers without even reading through the multiple choice options.
For example, consider the question, “Which of the following cancer screening tests would be most appropriate to offer to a 24-year-old healthy woman?” You can probably come up several potentially correct answers. Then, when presented with 1 appropriate screening test and 3-4 others that aren’t recommended for young, healthy woman, you’d quickly be able to select the best answer option.
The systematic thought process that I just described is impossible with question stems that simply ask, “Which of the following is true?” or a similarly vague question. After reading a vague question stem, the test taker must carefully read through every answer option and decide if each one is true. To make it worse, without any contextual material, each answer option is probably lengthy and the different options may not even be related to one another.
“All of the above,” “None of the above,” and multiple correct options
It is common for one or more MCQ answer options to be “All of the above,” “None of the above,” or combinations of two or more options such as “Both A and B” or “A and B, but not C.” This is unnecessarily confusing for the student.
Moreover, these types of questions unfairly favor learners who don’t have a solid grasp of the material. In a large percentage of questions with an “All of the above” option, “All of the above” is the correct answer. So that’s the answer that someone is going to choose whether they actually know it’s the right answer or they need to make a guess.
There is a time and a place for the answer option “All of the above.” But that time and place are rare.
Ultimately, each of these types of questions are simply a way for lazy test question writers to avoid having to develop a quality question or incorrect answer options.
Mistake #4: Obviously incorrect distractors
If you’re a university professor, it’s fine to show your sense of humor from time to time with a funny answer option that is clearly wrong. Go ahead and lighten the mood in your classroom.
Aside from that, though, MCQs with extremely obvious answers are flawed. As in Mistake #3 above, obviously incorrect distractors make it more likely that a learner who doesn’t understand the topic being tested is able to guess the correct answer.
Good distractors have several qualities in common:
- They are plausible in the context of the clinical case (this one is key!)
- They have consistent grammar with other options
- They are a similar in structure and length to other answer options
- They are not considerably more detailed or more general than the other options
- They are not open to interpretation (ie, they avoid words like “often” or “usually”)
Mistake #5: Misinterpreting references and failing to fact-check
While many concepts used in medicine stand the test of time, a lot of scientific and clinical information changes over time. Studies can be debunked, guidelines can become outdated, and new treatments reach the market and become the standard of care.
Medical MCQs must be supported by references. As the question writer, you must check the original source.
I’ve seen many test prep questions with these flaws related to their cited references:
- The reference is outdated
- There are other, more relevant references than the one cited
- The study or publication that is referenced has been misinterpreted
- The reference is a secondary source of information that had already misinterpreted the initial reference or taken it out of context
- The single reference provided doesn’t give the “whole picture” as it relates to question
The main reason that clinical experts are hired or asked to write medical MCQs is because an expert-level understanding of the topic is required to write a question about it. Despite this, expert opinion is not sufficient to support a question. You must accurately interpret medical information and put it into the context of a patient case or question, and then cite the sources of this information.
The listed reference for this question doesn’t even mention the word “Thrombocytopenia.” Yet, the pregabalin FDA package insert itself states that leukopenia, anemia, myelofibrosis, and thrombocytopenia are all adverse reactions that were observed during clinical trials.
You now know more about how to write multiple choice questions than most doctors who write them
There are many more mistakes that are made when writing MCQs and many other ways that questions can be flawed. However, being aware of just the 5 common mistakes above will make you an above-average test question writer. In a profession that relies so heavily on standardized testing, you’d think that all sources of MCQs would be high quality – but this is not the case.
Simply avoiding these 5 things will prevent a lot of learners and examinees from being irritated and perplexed.
Whether you’re writing MCQs for undergraduate health sciences courses, basic medical sciences, clinical sciences, or other areas related to health and medicine, your work has the potential to help future medical professionals reach their goals.
Looking for a job or side gig writing test prep questions? Check out the Look for Zebras job board for opportunities.