The Next Revolution in Item Format

The last revolution in the format of how Americans test students may have been during and just after World War I. That’s when the multiple-choice format replaced the essay format as the most prevalent item type in educational testing. In the decades that followed, the techniques around multiple-choice testing were developed and refined.

Let me put on my futurist hat for a moment and predict that the next revolution in the format of how Americans test students will happen when the computer-delivered simulation replaces the multiple-choice format as the most prevalent item type in educational testing. I’m not going too far out on a limb because the computer-delivered simulation is already well established and widely available on the web. For example, the Rice Virtual Lab in Statistics offers a set of simulations to demonstrate various statistical concepts. Or visit the business simulations offered by Forio Business Simulations. Some nice simulations in biology are offered at Biology Labs Online, by Benjamin Cummings, an imprint of Pearson Education.

Simulations can be used as items in at least two ways. First, simulations could be used like reading passages are currently used, as the stimulus preceding multiple-choice, short- and extended-response questions. Second, simulations can be used as the test “items.” Under this use of simulations, the term item is interpreted broadly to be a situation contrived to produce student behaviors or performance that are revealing of the constructs we are trying to assess. A conventional multiple-choice item is written to produce a student marking a bubble on an answer sheet in such a way as to reveal something about the student’s understanding. A simulation would be constructed to produce a student interacting with the software in such a way as to reveal something about their declarative and procedural knowledge.


To me, the assessment possibilities are in the use of simulations as the test “items.” Using simulations similar to those found in the Rice Virtual Lab in Statistics, I could ask students to create a distribution of cases and demonstrate the influence of outliers on the mean. By capturing mouse clicks, I can collect information about students’ understandings of the concepts of distributions and means. Alternatively, CardioLab, part of Biology Labs Online, allows students to measure arterial pressure and to manipulate five variables that effect arterial pressure. Vessel radius and heart rate are two of these variables. A test question would be: “Take as many measures as you need of arterial pressure under conditions that demonstrate the interaction of vessel radius and heart rate.” I can capture mouse movements to determine the level of understanding of the concepts of variable interaction and experimental control.

Even with my futurist hat on, I can’t foresee the psychometrics that will develop and be refined around simulations. Some techniques are under development, but that's a topic for another entry.

In The News: Electronic Testing

I read with interest the claims made in Oregon recently that online testing actually caused a raise in student test scores: Online Testing Helped Raise Scores. “Oregon students of all ages showed across-the-board improvements on state tests in core subjects....” Good for Oregon and good for electronic testing! I am a big believer in online assessment. Further reading, however, caused me to pause. I began wondering about the security of the online assessments in Oregon. An Oregon official is quoted as claiming, “Web-based testing is more secure…” and “…students and teachers get immediate results…delivered automatically....” All of this raised concerns about security. Surely, someone was looking at the results before they were returned? What evidence does Oregon offer to support their claims of security? Apparently, teachers had multiple opportunities to test their students “throughout the school year!” “Often, teachers will space out the testing periods over a school year and use early test results to evaluate which material a student needs to focus on in order to reach grade-level proficiency.” Does this mean the forms are exposed the entire school year?

Being a skeptical reader, as you should be, and looking for evidence to support their claims, I went to the Oregon Department of Education website for information regarding the security of the system and perhaps some technical information. After nearly 30 minutes I gave up. It is not that I don’t believe press releases, newspapers, or other non-peer reviewed information—well, I guess it is because I don’t—but it is not that I don’t believe Oregon educators. I just wish we could have some factual evidence associated with such claims. Only this way will we truly be able to judge for ourselves what works and what does not.