OVERVIEW
The Canadian Alliance of Regulatory Bodies of Traditional Chinese Medicine Practitioners and Acupuncturists (CARB-TCMPA) is the national forum and voice of provincial regulatory authorities. The founding members comprise the regulatory bodies of Traditional Chinese Medicine Practitioners and Acupuncturists (TCM/A) of Alberta, British Columbia, Newfoundland and Labrador, Ontario, and Quebec.
CARB-TCMPA administers three exams—the Pan-Canadian Entry-Level Examination for TCM Practitioners, for Acupuncturists, and for TCM Herbalists. These exams assess the occupational competencies of entry-level TCM professionals and determine whether test-takers have the skills needed to provide safe and ethical health care.
THE GOALS
CARB-TCMPA had already decided to transition from paper-based testing (PBT) to computer-based testing (CBT) by the time it began working with Meazure Learning (formerly Yardstick). To do so, CARB-TCMPA prioritized four ambitious goals:
Strengthen the Development Processes
Before CARB-TCMPA brought Meazure Learning onboard, test-takers needed to clear two hurdles before becoming credentialed: a multiple-choice (MC) component and a clinical component with scenario-driven short-answer (SA) questions. These parts were scored separately and treated as individual exams.
Ensure Exam Defensibility
Making sure the program was defensible and complied with standards for high-stakes exams was very important to CARB-TCMPA, which placed extra emphasis on mitigating exam exposure.
Provide a Better Test-Taker Experience
Because CARB-TCMPA conducted only PBT originally, its test-takers had to take exams at brick-and-mortar testing centers. While some test-takers worked better at testing centers, others preferred the convenience of remote testing.
Streamline the Scoring Process
Before the 2020 administration, scorers were required to meet in person because CARB-TCMPA delivered only paper-based exams. Since TCM/A has been practiced for thousands of years, scorers varied in their philosophical approaches—leading to challenges in the manual scoring process.
THE SOLUTION
Developing More Innovative Items
To make the test development process easier, Meazure Learning transitioned CARB-TCMPA to the Measure Exam Platform—an intuitive solution for every stage of the exam process. Dan Garcia, Executive Director of CARB-TCMPA, says, “We’ve moved to [the Meazure Exam Platform], which has been very helpful. It’s definitely a big step forward.”
The first task involved expanding the competency profile for TCM/A professionals. CARB-TCMPA and Meazure Learning knew this process would be crucial for the success of its test-takers, helping ensure they had the knowledge and skills necessary to effectively perform TCM/A care.
Next, Meazure Learning helped CARB-TCMPA use the platform to create blueprints from scratch. Meazure Learning’s psychometricians then coached the CARB-TCMPA team on writing innovative items—including multiple select, reorder, and case-based MC—that assess higher-level cognitive processes.
With a brand-new item bank, CARB-TCMPA zeroed in on exam design. The updated design? Test-takers must take both the MC and the clinical component, with each comprising 50% of the total score.
Adding Layers of Defensibility
For CARB-TCMPA, having a more secure and psychometrically sound program meant fortifying the development processes. It wasn’t enough to produce new item types that would test higher cognitive levels. The overall process had to be more defensible. So, Meazure Learning got to work:
- Creating committees of SMEs
- Facilitating item review sessions
- Validating item keys
CARB-TCMPA avoided issues related to exam exposure by strengthening the entire development process. The items were not only more complex — which reduced the likelihood of a test-taker being able to memorize and later share them with other test-takers — but also reviewed more rigorously.
Feeling confident in its program’s defensibility, CARB-TCMPA turned its attention to delivery.
Boosting the Test-Taker Experience
CARB-TCMPA notified its administrators and test-takers in 2019 that paper-based exams would be offered for the last time in January 2020. This gave administrators a chance to update their programs. It also made sure test-takers could develop the skills required for the new competency profile.
Then COVID-19 hit.
Now CARB-TCMPA had to not only deliver a computer-based exam for the first time but also find an online proctoring partner. So, it turned to Meazure Learning’s ProctorU Proctoring Platform. The transition to remote proctoring was relatively straightforward since CARB-TCMPA was already leveraging the Meazure Exam Platform.
The resulting move to CBT and online proctoring was a big success for CARB-TCMPA. It tangibly improved the program’s security and offered test-takers a more personalized experience.
“The adoption and use of online proctoring has been a game changer for us. It’s more cost effective and arguably more secure than test centers if you look at the proctor-to-candidate ratio and the ability to record and review your sessions.”
Dan Garcia, Executive Director, CARB-TCMPA
Standardizing the Scoring Process
CARB-TCMPA eliminated the scoring variability when it switched from SA items to more innovative item types. Since the new items required less scorer judgment, there were fewer debates about which philosophical approach was right.
THE OVERALL RESULT
In the end, Meazure Learning helped CARB-TCMPA surpass its goals. Chris Beauchamp, Meazure Learning’s VP of Psychometrics, claims their efforts “made a good program even better.” Now CARB-TCMPA has peace of mind knowing its program is more defensible and secure.
THE TAKEAWAY
With the support of Meazure Learning, CARB-TCMPA turned its ideas into reality. It didn’t happen overnight. In fact, the process involved a lot of time and energy. But the reward? A secure and defensible program that helps move TCM/A professionals forward.
“Our item discrimination and our difficult values are the strongest this exam has ever seen.”
Dan Garcia, Executive Director, CARB-TCMPA