The saying “A house is only as strong as its foundation” is as relevant to professional testing as it is to architecture. Think about it: If you build a house on a wetland, you face a long-term, uphill battle to maintain its structural integrity. The same concept can be applied to assessment. If you fail to establish security-focused psychometric solutions during test design and development processes, you can’t expect to effectively protect its content or overall program integrity. Just like it’s true for a house blueprint, strengthening your online test starts with a secure design.
Read on to learn what the term “secure test design” means and why it’s important for credentialing programs. Then, discover proven techniques to maximize security during test design and development processes.
The Importance of Secure Test Design and Development for Credentialing Programs
“Secure test design” refers to the process of designing and developing an assessment in a way that mitigates cheating and upholds content integrity. This can include psychometric measures taken during the blueprinting stage of the assessment life cycle. You can also achieve secure test design during the item writing stage or via modern delivery models that ensure unique content for each test-taker. These techniques aim to prevent test misconduct and promote accurate test-taker results.
While it’s important for all assessment programs to use secure design, development, and delivery measures, it’s especially important for organizations that deliver medium- and high-stakes assessments. Without following test security best practices during every stage of the assessment life cycle, credentialing programs are susceptible to negative consequences. These include financial, legal, and reputational ramifications.
“High-stakes credentialing is all about passing and failing the right candidates. This is what our clients, the candidates, and the public expects. Without a test security plan, we lose all assurances that the tests are doing what they’re intended to do.”
Dr. Chris Beauchamp, Vice President of Psychometrics, Meazure Learning
How Are Psychometric Security Techniques Different for Online Tests?
From a security standpoint, computer-based testing (CBT) and paper-based testing (PBT) have unique advantages and disadvantages. For example, administration and delivery processes for PBT are considered secure because a human is always physically present to verify test-taker identity and monitor test-takers during a session. On the other hand, assessment experts agree that live remote proctoring is just as secure as in-person proctoring, but computer-based test design and development processes have additional built-in security advantages. That’s because paper-and-pencil tests have static and fixed designs. Administrators can use multiple forms for PBT, but they cannot use more complex, adaptive, or personalized items. With online testing, however, test developers can build assessments so they’re psychometrically equivalent but unique to each test-taker.
Psychometric Methods for Secure Online Test Design and Development
When you build a test for online delivery, your psychometric and development team can use numerous techniques to combat cheating and collusion. The methods outlined below rely on advanced algorithms and software that make it more difficult to steal and circulate items.
Method 1: Randomizing Item Order
The simplest way to disrupt content theft (or “item harvesting”) is to randomize the order of items. When this happens, test-takers receive the same questions but in a different order. This is effective for two reasons:
- It’s more difficult for test-takers to reconstruct the test later.
- It interrupts certain types of collusion, such as answer copying.
Some organizations randomize the order of options for multiple-choice (MC) questions too. However, this approach can be problematic from a validity perspective because option order can significantly impact the difficulty of an item. Best practice says options should be presented in a logical sequence (e.g., chronologically, numerically) when necessary.
Method 2: Using Multiple Test Versions or Forms
You can reduce—or in some cases—eliminate the risk of item harvesting and exposure when you use multiple versions or forms of a test. The reason is test-takers have different test content. Psychometricians build parallel forms to the same table of specifications that are equivalent in difficulty. They may include overlapping items for statistical equating purposes, but they generally contain different content.
Psychometricians can create unique, parallel forms for test-takers using several techniques. One technique is computer-adaptive testing (CAT). With this model, items are selected for test-takers in an iterative manner based on their examination performance.
Another technique is linear-on-the-fly testing (LOFT). With LOFT, experts use complex, automated test assembly algorithms to create psychometrically equivalent forms. Like CAT, LOFT selects questions from a large, calibrated item pool. But unlike CAT, LOFT doesn’t adjust exams based on test-taker performance. Instead, LOFT builds a new fixed form for each test-taker.
What Are the Benefits of CAT and LOFT?
CAT and LOFT control item exposure and enhance test security by giving each test-taker a unique test form. The benefits of these models are numerous, despite the time commitment required to create item banks capable of supporting multiple versions or forms. They not only improve test security but also boost delivery flexibility. This allows windowed administrations to accommodate more on-demand testing. To learn how to expand your item bank quicker, read our blog article about automated item generation.
Method 3: Developing Item Types Used to Assess Higher Cognitive Functions
You can increase test security by creating items that are more difficult to memorize than traditional items. These items are often called “innovative items” and include any items that differ from the MC item type. They aim to assess test-takers’ real-world or workplace knowledge and critical-thinking skills.
In general, innovative items are longer, more complex, and harder to retain. Additionally, they help reduce unfair testing advantages because test-takers must apply critical-thinking skills to complete the assessment. Therefore, tests built with innovative items are less susceptible to item harvesting.
Common innovative items include:
- Matching questions
- Multiple-select items
- Drag-and-drop to classify
- Drag-and-drop to order
- Short-answer questions
- Constructed-response items
When Should Innovative Items Be Used?
Now and in the future, consider the following when deciding whether to use innovative items:
What is the purpose of your program?
What is the purpose and mission of your organization (e.g., protect the public, verify education)? The greater the testing stakes and competencies being evaluated, the more you should consider items that test a higher order of cognitive skill.
What does your program measure?
It’s important to determine what knowledge and skills are assessed and what the most effective and precise way to measure them is. You should know how the knowledge and skills are evaluated in practice and how they can be captured in a standardized format.
What resources are available?
Your program needs several resources—including financial, technological, and human resources—to develop and implement innovative items. You should determine which items are realistic and feasible given your organizational circumstances.
During online test development, you can also integrate media tools and other item types, such as videos, images, or PDFs. Unlike standard MC items or essay prompts, these tools and item types create content that’s more engaging and skills-based, which can also increase security against test content theft.
Conclusion
Overall, an integral part of maintaining your credential’s value is to build layers of security into your test design and development processes. This will help to ensure that the assessment results accurately reflect test-taker competence and protect your IP from being exposed. Additionally, by incorporating secure test design measures, you’ll be better prepared to address future security threats as they arise.
To learn more about security best practices for professional testing organizations, read our blog article “Comprehensive Test Security: What It Is and How to Approach It.”