May 25, 2012 5 Comments
A few weeks ago, a friend of mine had me look over a 100-question multiple-choice test she had just written. My friend works for a community college with a new culinary program, and she has been tasked with putting together a series of three professional wine studies classes that will ostensibly prepare students to take a certification test from one of the many groups in the U.S. that give wine people the opportunity to line the walls of their study with plaques proclaiming that the owner “knows a whole lot about wine” and lets them add a few letters following their name on a business card. She has a lot of test writing in her future.
The first question on the final exam she was writing for the introductory class was, verbatim, as follows:1. A wine region in South Africa is: a. Kirwan b. Tutuven c. Benguela d. Robertson
The answer is “D – Robertson” – but that’s about the only thing right about this question.
Much to my chagrin, the rest of her “test” was composed of 99 more questions, each very similar in style and content to the first. I knew we had a lot of work ahead of us and really needed to improve her questions, hopefully before they went out to any bright-eyed, hopeful wine students. After seriously considering how to word my comments to her, my response went something like this:1. Which of the following items is wrong with the multiple choice question quoted above? a. The question is grammatically incorrect. b. The question tests single subject, knowledge-only learning. c. The question covers trivial information. d. All of the above. The answer is “D – All of the above”!
It seems like the universe has deemed me something of an MCQ Maven, as I am tasked with reviewing tests such as these on a weekly basis. Well, I guess that makes sense for an academic director. So, I thought I’d write down a few of the tips I was about to give my wine-teaching friend. Perhaps they will help you as well!
A. When writing multiple choice questions, be grammatically correct! “A wine region in South Africa is Robertson” doesn’t sound correct even if I use my Yoda voice (which I do quite often). To be grammatically correct about being grammatically correct, the question is confusing based on subject/predicate placement and uses the passive voice unnecessarily.
Ideally, the stem of a MQC should be written as a complete sentence, and in the form of a question. This is referred to in MCQ-lingo as the “direct question format”. In the case of “question #1”, a proper direct question format would be:Which of the following is a wine region located in South Africa? Or… Which of the following wine regions is located in South Africa?
Using the incomplete statement format (also known as “fill in the blank”) is also considered appropriate, but most test writers still prefer the direct question format and use the incomplete statement format only when a direct question isn’t feasible. For the sake of illustration, a proper incomplete statement format for this question would be:
_____________ is a wine region in South Africa.
It’s grammatically correct, but I still don’t like it. You’ll see why, below.
B. When writing multiple-choice questions, ensure the validity of your test by emphasizing higher-order thinking skills. In other words, don’t write a test that can be passed by anyone who just locks themselves in a Holiday Inn for the weekend and memorizes the text, only to forget it all by Tuesday.
Writing “knowledge-only” test questions is faster and easier compared to writing questions that test for understanding. However, as in all things, you get what you give. Writing a test chock-full of single-subject knowledge-only questions is inexcusably lazy and might just get your test thrown out for being invalid and unreliable (the mortal sins of the testing world). Another sin is to quote directly from the text in your questions, unless the entire purpose of the test is to rate photographic memory recall.
Avoid these sins by writing at least half of your questions in a manner that tests understanding of the material by using memory plus application, cause-and-effect, analysis, synthesis, evaluation, or integration of subject matter. Here’s a better test question about Robertson:Which of the following wines is mostly likely to be produced in the Robertson wine region? a. A sturdy, Shiraz-based red wine made from grapes grown in South Australia. b. A dry, Riesling-based wine made from grapes grown in Western Australia. c. A high-volume, fortified wine made from grapes grown in South Africa. d. A boutique-produced Meritage blend made from grapes grown in South Africa.
Is it obvious why this test question takes longer to write than our original question? First of all, it avoids testing for trivia in that each distractor and the correct answer involves not just one but two pieces of factual information (all of which must be properly vetted, no matter how smart the test writer thinks himself/herself). It involves memory-plus application in that students need to know a bit more about the wine region rather than just where it is and integrates several types of information about the topic.
Incidentally, such a question is actually easier for a well-prepared test taker to answer correctly than a shorter question. At first glance, it would seem like a more complicated question, but in reality the test taker has more than just one chance at remembering the facts specified in the question, and can call upon a broader base of knowledge to ascertain the correct answer. This is also a good example of how to build some “validity” and “reliability” into your tests.
For much more information on writing MCQ’s that test for higher-order thinking skills, see my previous blog post here: http://bubblyprofessor.com/2012/02/26/train-the-trianer-taxonomy-of-a-wine-test/
C. When writing multiple choice questions, don’t test for trivia! If my friend was writing a 100-question test on the South African wine industry, a question about Robertson would have been appropriate. The way it stood, however, had me humming “can you guess which thing is not like the other” for days. Keep in mind that her test was intended to be a final exam covering three months worth of classes and a 653-page textbook. The kicker is that the only mention of Robertson in the massive text is a table on page 427 that lists over 75 South African regions, districts, and wards in 8-point type. The items offered as distractors are just as obscure. There is no way that any student can get that question correct unless they are just plain lucky. They could be lucky in that they just happen to remember reading “Robertson” in the midst of that periodic chart of S.A. regions, or they just took a lucky guess. Either way, this question is not a valid indicator of wine knowledge…but it would make a great “Trivial Pursuit – South African Wine Geek Edition” question.
When writing single-subject knowledge questions, ensure that the knowledge being tested is based on a student learning objective of the course, not just trivial information. You should only use your single-subject knowledge questions to test for important or significant information. When writing such questions, don’t be fooled into thinking that MCQ’s concerning broad information such as the definition of “malolactic fermentation” must by definition be “easy” questions. The ease or difficulty of such questions is totally under your command. The more similar the distractors are to each other and the correct answer, the more difficult the question will be. If malolactic fermentation is a key learning objective of your class, feel free to make it a difficult question.
One good way to manage this process is to sit down and think of the ten or twenty most important concepts you want a graduate of your program to understand. These should, of course, be reflected in your course objectives. Then, write a fair question for each of them, varying the level of difficulty across the test. For example, I tell my students on the first day of class that every day we will have a “word of the day” and it’s very important that they know them. My “words of the day” include such bon mots as Carbonic Maceration and Botrytis. (Email me if you want a complete list.) I also emphasize “Miss Jane’s Top Ten Terroirs” throughout the class and tell the class they really need to know these ten wine producing regions and what goes on there. My top ten terroirs include Burgundy, Bordeaux, Chianti, Rioja…you get the idea.
Another good way to create knowledge-based questions is to write questions that cover more than one “tidbit” of information. It is possible to write valid, knowledge-based MCQ’s if the question is integrated rather than single subject oriented. Here are a few examples of integrated questions, all of which I suggested for use in place of my friend’s ill-advised question on Robertson:
Which of the following South African wine regions are located in the interior of the country?a. Worcester and Robertson b. Stellenbosch and Paarl c. Constantia and Elgin d. Elim and Klein Karoo
(And the answer is… “A”!)Franschhoek, Constantia, and Robertson are all well-known wine producing regions in which of the following countries? a. New Zealand b. Australia c. Argentina d. South Africa (And the answer is… “D”!) You’ll note that while these questions are knowledge-based (ok for about 50% of your test), they are not trivia-based in that they include more than one piece of information, both requiring and rewarding a broad base of knowledge and application as opposed to “I’ll take South Africa for 200, Alex”!
Here’s a few more guidelines for writing MCQ’s:
- Don’t quote from the text or test for information that is specific to just one source. In other words, don’t use your textbook’s description of Chenin Blanc (“apple, straw, melon”) and assume it is a universal definition.
- Please keep in mind that there is a world of information beyond the text book that needs to be acknowledged. I recently reviewed a test that asked “which of the following wines is exclusively red” and listed Châteanuf du Pape, Hermitage, Bordeaux and Beaujolais as possible answers. You will note that none of these answers is exactly correct. The supposed “correct” answer was Beaujolais, which I happen to know produces a white wine, albeit a somewhat obscure one. My friend, who wrote the test, replied “but it’s not in the text.” Argh. By the way, mistakes such as this punish the student with broad knowledge and reward the last-minute text-memorizer. Double Argh.
- Properly vet each and every question and each and every correct answer and distractor. That means solid research (well above and beyond just the text in use) on every question and question topic. This is to ensure that’s there is one (and only one) solid correct answer and yes, this this takes a lot of time.
Good luck writing your multiple choice questions. I am sure you will soon realize that while they do take a great deal of time to develop, a valid test is worth the effort. And don’t forget the payback…they are so quick to grade, as opposed to essay questions. Stay tuned to this blog for future posts on writing and grading essay questions, by the way.
And please…help stamp out lazy test writers!