Grow Your Item Bank and Make Your Exam More Secure in 8 Steps with Itematic

| Blog Articles | Share:  

What Automated Item Generation Is and Why It’s Important

There are many perks of using automated item generation. It can save you time and resources during the item development phase of the assessment life cycle. It can also expand your item bank, giving you more options to choose from and ultimately making your exam more secure.

How It Can Protect Your Exam Content

One of the biggest threats to exam security is item harvesting. In its simplest form, item harvesting happens when a test-taker memorizes content and discloses it to a peer who will be taking the exam later in the testing window. But highly organized item harvesting rings also exist. Although approaches vary, these rings commonly include test-takers who memorize a pre-determined number of questions during the session and then reconstruct the exam afterward.

The good news? There are many ways to thwart these activities. You can randomize your questions, use multiple exam forms, conduct Linear-on-the-Fly Testing (LOFT), and more. But to do so, you must have a deep pool of items to dip into. Otherwise, you simply won’t have enough content to rotate.

Think about it: If you have a larger bank of available items, you can diversify your exam content. That means people who wish to steal your content or use pre-knowledge to cheat will have trouble predicting which items will be used in the next administration. Put simply, they’ll have to work a lot harder.

How It Can Expand Your Item Bank

So how big of a bank are we talking about?

We recommend that your bank contain at least 100x to 1,000x the number of items used in an actual exam. For example, if a test is designed to include 100 items, your bank should include between 10,000 and 100,000 items. As our partner Caveon notes, this recommendation is based on the “efficiency of harvesting efforts and the [large] size of today’s test-taking audiences.”

One way to quickly expand the size of your item bank is to use an AIG tool, such as Itematic. It’s not a magic bullet though. Itematic can help you write a greater number of items in less time, but it can’t automate the whole development process. It can only add to the value created by subject-matter experts (SMEs), who provide the necessary expertise and decision-making skills.

How Itematic Works

  1. An SME creates a cognitive model by describing the knowledge, skills, and abilities they plan to assess.
  2. The SME develops a question template based on this model. To develop this template, the SME starts by writing a question stem. But unlike in traditional item writing, the SME specifies “variables.” These are a form of dynamic text that exchange different options the SME lists. For example, a question stem about state capitols may include a variable that swaps out the name of the state being asked about. By having these variables, the question template can produce multiple unique permutations (i.e., versions) of the question stem.
  3. The SME then creates a list of answer options that can appear for the resulting permutations.
  4. The SME matches these options to specify what the correct answer is for each combination of question stem variables.
  5. To help ensure each question permutation is a valid and effective assessment of test-takers’ knowledge, the SME can denote “friend” and “enemy” answer options. These are incorrect options that should always or should never appear with a given combination of question stem variables. The SME then adds any remaining question metadata to help classify what the question is assessing.
  6. The SME specifies the number of question permutations they want Itematic to generate. Itematic prioritizes creating the most unique permutations possible using a text-dissimilarity algorithm.
  7. Finally, the SME previews, validates, and rejects/approves the resulting questions before exporting them.
  8. The exported questions appear in a standard data format that can be easily imported into most item banking software, including the Meazure Exam Platform.

Where You Can Learn More

No matter the size of your testing program, your team can expand its item pool using AIG to create a more flexible and secure exam.

Dig deeper into AIG and exam security with the following resources:

View All Posts