A client with whom Yardstick (now Meazure Learning) is working to develop a new certification program asked me this question recently. And it’s a great question. In order to plan out exam development activities, including the recruitment of subject matter experts (SMEs) to generate multiple choice questions (MCQs), realistic expectations need to be provided as to the time commitment required for each SME. If the client wants to hold a face-to-face workshop for SMEs to develop MCQs under the guidance of a psychometrician, how many days should be scheduled to ensure the required number of questions is generated? If the item generation will take place remotely (whereby SMEs write MCQs from home using Yardstick’s secure online item development, banking and exam delivery platform), how much time should SMEs set aside for writing the number of questions assigned to them?
One way of estimating the time commitment is to consider the length of time it takes to generate a single MCQ. A few years ago, I collected data from 75 SMEs who were generating items for a licensure exam in a health care field. I asked the SMEs to record how long it took them to write a complete first draft of a MCQ. This included the time it took to (a) come up with an idea for the question, (b) develop the question stem (i.e., the question being asked or problem to solve), © create the response options (one correct answer and three distractors), (d) write a rationale for why the correct answer is correct and why each distractor is incorrect (or not the best response), and (e) document the supporting reference for the correct answer. Across the 75 SMEs, the average time to complete a first draft of an MCQ was 52 minutes .
So does this mean that, for a 100-item MCQ exam, you could expect it to take 5200 minutes or around 87 hours of SME time to come up with a first draft of 100 MCQs? Maybe, but not necessarily. There are a number of influencing factors such as the content being assessed (knowledge type questions are typically easier to create than application and critical thinking questions), the question format (independent or stand alone questions are typically generated more quickly than case-based questions), the item writing experience of SMEs, and the ability to find a supporting reference. But in my experience, I would say that it is reasonable to expect that at least a half hour is needed to create a complete MCQ.
Note that this is just the time it takes to create a solid first draft. All MCQs go through (or at least should go through) several review stages, such as psychometric review (whereby a psychometrician or test development specialist reviews the question to ensure adherence to item writing structural guidelines and blueprint parameters), industry group review (whereby content experts who work in the field review the item to ensure content accuracy, correct language and terminology, and appropriate level of difficulty), and sensitivity review (to evaluate whether the item may be offensive, or simply interpreted differently, by certain demographic populations). So, by the time an item appears on an exam, several hours would have been spent writing, reviewing and tweaking every single item to ensure they perform as they should.
*Piasentin, K. A. (2010). Exploring the optimal number of options in multiple-choice testing. Council on Licensure, Enforcement and Regulation (CLEAR) Exam Review, 21(1), 18-22.