Hope that Christmas and the festive season treats you excellently. Here’s a small gift of knowledge that I’ve been pleased to share, via the e-learning industry’s advent calendar, 24 tips:
http://24tips.elearningnetwork.org/2010/12/posing-question/ [note this is now a dead link]
May your distractors find whole new levels of plausibility in 2011!
[First published in E.learning Age in June 2007, revised for Christmas 2010, and now added here because there was a dead link]
The why, when and how of effective questions.
Having encountered hundreds of multiple choice questions whilst training to become a chartered accountant, I subsequently turned trainer and e-learning developer and have designed hundreds of questions. This article is designed to help those who are designing questions or are commissioning them.
Why use questions?
Questions are a way of turning the one-way presentation of material into a two-way exchange between presenter and learner. They can be used both to build and to test the learner’s knowledge. In-course questions are known as ‘formative’ questions; these can be used by learners to help them learn by, for example, using feedback to help steer them away from popular misconceptions and towards best practice. End-of-course questions are known as ‘summative’ questions; these assess the extent of the learner’s understanding, for example whether they have met specified professional standards in a compliance module.
When to use questions?
One of the most productive uses of questions is as a pre-test. If a learner is able to meet the performance standards required without completing the learning in the module, then they have saved themselves and their employer valuable time. A pre-test is also a useful way to highlight to the learner where in particular they should focus their attention during the learning content.
However, pre-tests or diagnostics should only be used in cases where some learners already have existing knowledge of some of the course – pretesting learners who do not know the content is guaranteed to irritate!
Formative questions can be used throughout a module. As the responses are seldom tracked, the user can feel safer in experimenting with the answers that they choose.
Near the start of the module you may want to use a question to grab the learner’s attention or encourage them to recall previous knowledge. Later in the module questions can be useful for reinforcing the understanding of a concept or applying a new piece of knowledge in a practical case study scenario. As normative questions test that learning is successful, they should be at the end of the module or each section within it.
What should the questions contain?
There are many possible types of question, such as multiple choice, drag and drop, text entry. However as some authoring tools only allow you to work within the constraints of a four choice multiple choice question, this article will focus on this as a question type. The principles outlined here apply equally well to other question types.
1.Validity
The question should relate to the learning objectives and planned key points of the module. This is especially true of normative questions as the main point of asking the questions as a test is to ensure that the learning objectives have been met; only those learners who have fulfilled the learning objectives should get the questions right.
2.The question stem
The words at the beginning of the question should be unambiguous and clearly relate to content covered in the module/course. Ensure that the rubric is clear.
For summative (final assessment questions) avoid stems like “what you do think?” as from a literal point of view the learner will be correct all the time. Formative questions use this kind of stem as it can be useful in revealing underlying attitudes and thinking.
3. Distractors
A distractor is a plausible alternative to the correct answer. The better the distractors, the more assurance we can gain from the correct answer that learning has occurred. Learners are remarkably proficient at eliminating wrong answers without having knowledge of the topic – usually because the distractors are too obviously ‘wrong’. Tips to make your distractors more distracting are:
* Make sure that you don’t put the answer in the question.
* For verbal distractors make sure that they are not significantly shorter than the correct answer (it is harder to write a longer distractor so some inexperienced question writers fall into this trap).
* For computational questions make sure that the distractors are numbers that can be reached if user misapplies the principles or uses the wrong data from the question.
* Order the distractors in a logical order e.g. in alphabetical order of the first word, or in ascending order for numbers.
* Ensure that there is a mix of where the correct answer is e.g. option C tends to be the most used option for the correct answer, probably in the light of the previous point. If this means changing a few distractors, it is worth it.
* Word the options in a similar way so the learners have to think carefully about their content.
* ‘Joke’ distractors can be very appealing to the novice question writer, but it’s usually best to avoid them as they can appear amateurish.
4. Feedback
Typically the feedback to summative questions will be in the form of “you have scored X out of Y, you have scored/ not scored enough to pass this test”. If it is intended that the learner would retake a test, it is helpful if the feedback steers the learner towards the areas that they should revise most.
If a learner has got the answer right, they should be congratulated on this. The exact tone and expression of the wording should be flexed so that it’s appropriate to the learners e.g. in keeping with the organisation’s culture. Some organisations like effusive congratulations, others prefer just an acknowledgement.
In formative questions (where the learner is still learning), if a learner gets the answer wrong, the feedback needs to tell them why it’s wrong and why the right answer is right (and in the case of a calculation, how to reach the right answer). Feedback that simply tells them that they are wrong is demotivating and will detract from the overall learning experience.
In summative questions, (where the learning has finished and is being assessed) the standard is not to provide feedbacks for individual questions, but to provide an overall score. Often a ‘revision’ list of topics to revisit is also provided.
Watch out for the shortcuts
It’s human nature for people to seek shortcuts. In one organisation people discovered a bug in the testing that meant that if they got the question wrong in the pre-test they could hit “shift backspace” and then try again at the question until they got it right. As this nugget of information spread like wildfire, hundreds of learners beat the system by getting the pre-test right – thus avoiding having to do the training at all…Using randomly selected questions from a larger question bank and occasionally reversing the order of the options means that people have to answer the questions correctly rather than knowing that the answer to question 1 is D, 2 is A etc.
Are you happy that you know more about questions? If not, ask me a question!
Here’s wishing you a very happy festive season.
