Yardstick’s Chief Scientific Officer Dr. Greg Sadesky sat down with Isabelle Gonthier from the National Commission of Certifying Agencies to discuss virtual proctoring.
Weighing the benefits and limitations of virtual proctoring.
In a recent article, I outlined the virtues of virtual proctoring, mostly from the perspective of high-stakes testing. The American National Standards Institute (ANSI) has already certified high stakes virtually proctored testing programs through ISO 17024, but that there are those in the testing industry that have some reservations about this. This post dives deeply into those reservations, to help understand what some of the limitations of virtual proctoring are for high-stakes applications.
Isabelle Gonthier is a highly respected assessment professional and a commissioner with the National Commission of Certifying Agencies (NCCA). I had an opportunity to sit down with Isabelle and talk about her perspective on virtual proctoring, its strengths, and limitations. NCCA accreditation is also a certification standard, similar to ISO 17024 for credentialing programs. In agreeing to discuss this topic, Isabelle wanted to make it clear that she was not speaking on behalf of the NCCA, but expressing her opinion based on her experience as an assessor and assessment professional.
Greg Sadesky: First of all, thank you Isabelle for agreeing to do this interview!
Isabelle Gonthier: My pleasure!
GS: So, let me start with the recent announcement by ANSI that it has accredited a certification program that uses virtual proctoring for exam administration. What was your reaction to this news?
IG: My reaction was, “Wow!” I want to learn more and see more to understand how those standards are met. Knowing what I know about ISO 17024, it’s interesting to see that it’s opening up [to virtual proctoring], but I would want to see for myself the parameters around making that determination.
GS: Do I sense that you’re a bit skeptical that a virtually proctored program should have been accredited?
IG: Right now, I haven’t seen sufficient evidence that certification standards are being met for virtual proctoring, from a research and application standpoint. I’m not saying the standards can’t be met, but that there is yet to be evidence that shows it. One of my concerns is that much of the research that’s been conducted so far has been from vendors, so I would want to see more independent work.
GS: That’s a great point. What are the areas in which additional research or information is required?
IG: Security, cheating, and standardization. Here there are major questions.
GS: Security?
IG: Content security. Protecting exam content from theft. But, the bigger problem is how candidates could cheat. How do you ensure the environment is free from the ways that people can help themselves in dishonest ways, thus affecting the validity of the scores? I’m not saying content theft isn’t a problem, but there are ways of managing that. From a security perspective, it could be mitigated by the size of the item bank.
GS: And standardization?
IG: Standardization is mostly about technology. For example, internet connectivity could affect the experience of the test taker.
GS: What lines of evidence or information do you think it would take for you to be comfortable that virtual proctoring meets the required standard?
IG: It’s hard to say. Maybe if there is a system that is developed that can truly monitor the environment, say 360 degrees, and the likelihood of not having a blind spot is truly tested. Almost like a virtual 360-degree view and the recording and documenting of information and the session. And, on the process side, withholding the results of those candidates whose environments are suspicious.
Technology and process need to be there so that you don’t provide a score to a candidate that doesn’t reflect what they know. It’s all about the controls and ensuring that the monitoring is in place. But what evidence do we need? It’s almost a bit like show me and we’ll see if it’s compelling. I think we’re going to get there. I think we’re going to see some innovations that will allow for the monitoring that’s required. But, we have to be 300% certain, and that’s why the evidence is so critical from a standards perspective.
GS: One last question. In light of that evidence requirement, do you think that the bar might be too high for virtual proctoring? It’s not as if determined test takers are completely unable to cheat when the proctors are live.
IG: Yes, it’s almost like they need to work extra hard to prove the point. It has to be almost ironclad before people say, okay fine, we’ve established it now; now we have some standards that can be worked with. There’s kind of a parallel between this and the transition from paper-based to computer-based, around control and security. But organizations worked on this technology and met that standard and now computer-based testing is common. But I think to prove that they had to jump through higher hoops, and prove validity of the scores.
For certain parts of the virtual proctoring process, we’re at a point where technology has improved. For example, there are technologies around identity verification like keystroke and iris scans and the concerns are fading away, but that means that other concerns are now more “front and centre”.
GS: Like cheating.
IG: Like cheating, right.
GS: Isabelle, I want to thank you again for taking the time to talk with me about this important topic for our industry.
IG: Thank you for asking me!
*Since this article was published, Isabelle acted as President for the Testing Division of Yardstick for several years before accepting a new opportunity.
Virtual Proctoring and the Future of Assessment, with Dr. Greg Sadesky series
Part 1 | Part 2 | Part 3 | Part 4