The American Council on Exercise (ACE) is a leading non-profit organization offering four NCCA-accredited certification programs. The largest of those programs, the Personal Training Certification, first launched in 1991 and has been continuously accredited since 2003. It boasts a community of over 55,000 certified professional, with more than 10,000 new candidates joining their ranks each year.
The Situation – Meeting International Interest and NCCA Standards
In 2022, ACE conducted a job task analysis that led to the development of new exam forms for their Personal Training Certification program. To serve growing interest from international candidates, these new forms needed to be available in six languages across seven exam forms, with English comprising two forms:
- English (Domestic)
- English (International)
- Simplified Chinese / 简体中文
- Thai / แบบไทย
- Finnish / Suomalainen
- Arabic / يبرع
- Korean / 한국인
While four of the languages were already available, Arabic and Korean were to be offered for the first time. Furthermore, 2021 NCCA standards presented a considerable change from the 2014 requirements: Standard 20 requires additional evidence, such as bilingual subject matter expert (SME) review or dual language item display, to validate the equivalency of translated exams. This meant that each translated exam form would have to go through a standard setting study—a time- and resource-intensive activity.
The need to set standards for multiple translated exams presented several key challenges for ACE:
- Cost: The resources associated with travel, accommodations, and time required for SMEs and psychometric consultants would be significant.
- Scheduling: Coordinating multiple sessions in a timely manner posed a logistical hurdle.
- Recruitment: An SME panel for each session would require a total of 40-60 experts.
- Equivalency: Ensuring comparable passing standards across all versions of the exam was crucial and complex.
- Burnout: The extensive workload for planning and executing separate standard setting sessions raised concerns about fatigue for all involved.
The Solution – Leveraging Bilingual SMEs for Streamlined Standard Setting
In 2023, Meazure Learning met with ACE and recommended enlisting bilingual SMEs for the translated exam forms to ensure that each translation was reviewed by SMEs who were both qualified in the profession and fluent in each language. From this strategy, an innovative approach was borne:
Instead of holding seven standard-setting sessions, ACE proposed combining five forms—International English, Thai, Finnish, Arabic, and Korean—into a single multilingual session. The remaining two forms, Domestic English and Simplified Chinese, would each have their own traditional standard setting study.
ACE and Meazure Learning worked together to coordinate the consolidated events, recruiting twelve SMEs to participate in the multilingual standard setting. The SMEs discussed the competencies of a minimally qualified candidate and were briefed on the translation process. They then reviewed the English source text alongside each translation, provided cut score ratings, and reached a consensus on the appropriate passing standard for each exam form.
The Results – Understanding Processes, Saving Resources, Reducing Stress
Upon conclusion of the standard setting process, new exam forms were successfully launched. While long-term data is still being collected for these exam forms, the overall experience was positive.
This consolidated approach to standard setting yielded several positive results:
Shared Understanding
Mutual awareness among SMEs of the standard setting process and the minimally qualified candidate.
Statistical Equivalency
SMEs reached a consensus agreement regarding the cut score for each exam.
Realized Savings
Consolidating five sessions into one study resulted in an 80% reduction in total cost.
Reduced Burnout
The streamlined process eased stress on all parties involved.
Conclusion – Collaborating for Scale and Equitability
Hosting a multilingual expert panel in a shared setting allows for a deeper understanding of potential biases and misunderstandings within and across exam forms. Collaborating with an experienced exam vendor to facilitate these large-scale standard setting studies is an effective way to create equitable exams accessible to an international audience—while meeting NCCA standards for accreditation. Ultimately, this approach empowers programs to successfully enter new markets and scale their reach.
To see how Meazure Learning can help your exam program, schedule a call with our team.