How COPR Improved Exam Quality While Increasing Efficiencies with Automated Item Generation

OVERVIEW

The Canadian Organization of Paramedic Regulators (COPR) promotes and advances best practices in paramedicine nationwide. Founded in 2009, COPR is comprised of self-regulating colleges and government regulators. With the help of nine provinces, COPR develops the entry-to-practice exams for the paramedic profession and administers them in seven provinces. COPR also conducts assessments of internationally educated paramedic professionals for all Canadian jurisdictions. The organization administers three exams each year:

1. Primary Care Paramedic (PCP) Entry to Practice Examination
2. Advanced Care Paramedic (ACP) Entry to Practice Examination
3. Emergency Medical Responder (EMR) Entry to Practice Examination

THE CHALLENGE

In 2010, paramedicine was considered a non-compliant industry in Canada for labor mobility. To help move its test-takers forward, COPR needed to develop pan-Canadian exams for paramedic professionals. 

COPR had three goals for its new exams:

1. Ensure the exams are valid, reliable, and fair
2. Standardize the licensing process across the paramedicine industry
3. Develop a robust item bank as quickly as possible

The process of item development is notorious for being time- and resource-intensive, resulting in one of the costliest components of the testing process. To develop pan-Canadian exams with enough content to comprehensively assess its test-takers, COPR needed to move quickly, deliberately, and cost-effectively.

THE SOLUTION

COPR ultimately put its trust in Meazure Learning (formerly Yardstick). And before long, Meazure Learning set its sights on supplementing the invaluable work of COPR’s subject matter experts (SMEs). To do so, Meazure Learning and COPR worked closely on piloting Itematic—an automated item generation (AIG) tool. Unlike other AIG tools, Itematic allows SMEs to specify how many items they want to generate and then creates only the most unique subset. After that, a Meazure Learning psychometrician can seamlessly import approved items into an existing item bank. The goal of using Itematic was not to eliminate the need for COPR’s SMEs but to give them an extra tool to save time, increase productivity, and enjoy the item-writing process.

To get started, COPR SMEs designed the structure of new items using cognitive models that define the knowledge and skills required of test-takers. Next, the SMEs created item models, specifying which item features could be changed. After that, Itematic took over. Its dynamic algorithms matched the cognitive models to the item models, rapidly generating dozens of unique items. Because Itematic slashed the most time-consuming part of the development process, COPR SMEs were able to spend more time reviewing the items to make sure they were valid, reliable, and fair. 

THE RESULTS

With the help of Meazure Learning, COPR became the first Canadian organization to use AIG methodology. By using Itematic, COPR eased the work of its SMEs and invested in dynamic technology that worked alongside its most valuable resources, boosting overall efficiency and filling the shallow areas of its existing item pool. “It’s another tool in the toolbox,” says Tammy Leach, the examination manager at COPR. AIG tools have come along way over the past few years, but there’s a lot of work that still needs to be done. While Itematic is a resource for SMEs during the development process, it can’t automate the whole process. It can only add to the value created by human expertise and decision-making.

TAKEAWAY