The next instalment in Chief Scientific Officer Dr. Greg Sadesky’s series on virtual and remote proctoring.
Part 1 | Part 2 | Part 3 | Part 4
Dr. Sadesky unpacks some recent research on the efficacy of remote proctoring versus live proctoring.
Paradigm shift
After having examined this issue for the past several months, I truly believe we are approaching a paradigm shift in the testing industry as automation and virtualization play a larger role. Remote proctoring is a key part of that shift. With any large change, there will be both the early adopters and those who are more reluctant to embrace the change until the uncertainty surrounding the new process is sufficiently addressed.
A couple of factors make this change within testing unique. First, remote proctoring technology and processes have been in use for some time in higher education and low-stakes testing. This means that potential adopters may already have a considerable experience and exposure with the technology so barriers to adoption may seem lower. Second, the fact that remotely proctored exam programs have already been accredited through ANSI removes much of the perceived risk of adoption. With the huge potential cost and accessibility upside of remote proctoring, widespread adoption cannot be far off.
For many in the industry, the perspective of Mickie Rops resonates with their experience and with their desires for remote proctoring to play a bigger role in the administration of high-stakes exams. Remotely proctored exams, when administered following best practice, can be as secure, standardized, and free from cheating as more traditional paper-and computer-based test administration. Just because remote proctoring is newer does not mean that it needs to be held to a higher standard.
Barriers remain to be overcome
Significant barriers still remain for many potential users of remote proctoring technology. As Isabelle Gonthier indicated in our interview, additional research and perhaps technology will provide reassurance to organizations and their members that standards consistent with high-stakes exams are at least maintained under remotely proctored conditions.
With that in mind, I was very interested in the recent research paper that appeared in the Journal of Applied Testing Technology, A Comparative Study of Online Remote Proctored versus Onsite Proctored High-Stakes Exams.
The research described in the article investigated differences in performance and perception resulting from taking exams under one of two proctoring conditions: live or remotely proctored through a kiosk. The tests were administered to over 15,000 examinees over one year, and they were not informed at time of booking in which format the test would be administered. The researchers were primarily interested in whether test scores and the perception of the testing experience would be influenced by the type of proctoring received.
The results showed that scores do not appear to be affected by proctor type, although one of the three exams showed some differences, favouring remote over live proctoring. Though the perception of both types of proctoring was favourable, remote proctoring was viewed as slightly less favourable than live proctoring. Without getting into too much detail about the study’s methodology, it could be a concern that test takers were not randomly assigned to type of proctoring, opening the door for test-takers to select the proctoring type that they preferred. The slightly improved performance on one of the tests for those taking the exam under remotely proctored conditions was not particularly well accounted for by the authors, particularly since performance differences could be indicative of cheating. Nevertheless, the large majority (89%) of candidates took one of the other two exams, for which no score differences were present.
This study is one of many that should be conducted to help determine if score differences exist between proctoring formats, and if they do, what steps could be taken to limit them. This kind of research should go a long way towards establishing remote proctoring to all stakeholders as, at least, a viable option to administer high-stakes exam.
VIRTUAL PROCTORING AND THE FUTURE OF ASSESSMENT, WITH DR. GREG SADESKY SERIES