Now that more than a year has passed since the onset of the pandemic, testing programs are evaluating the impact of pandemic-era changes on their security, performance, accessibility, and reach. Join four industry leaders from high-stakes testing programs in a panel discussion as they share how they’ve turned innovative pandemic changes into long-term solutions. Panelists from the Educational Testing Service (ETS), Law School Admission Council (LSAC), Global Information Assurance Certification (GIAC), and National Restaurant Association (NRA) discuss their data points and industry stories in critical areas, like exam security, modality equivalencies, validity, reliability, accessibility, equity, and user experience. Finally, they consider where they see the future of testing going, what’s here to stay, and what might come next.
Panelists:
Ray Nicosia, Executive Director, Office of Testing Integrity, ETS
Faisel Alam, Manager, Office of Test Security, LSAC
Larry Lynch, Senior Vice President of Certification and Operations, NRA
Jeff Frisk, Director, GIAC
Moderator: Jarrod Morgan, Founder and Chief Strategy Officer, Meazure Learning
Transcript
Speaker: Jarrod Morgan
Well good morning or good afternoon everybody. Can everybody hear me okay? Looks good, looks like the panelists can hear me, hopefully everybody out there can as well. Thank you for joining us today at our panel with ATP. We’re really excited to bring kind of an all-star cast here of really smart people who have gone through some really challenging times as we all have and have a lot to share about some of the things that they’ve done. So sort of a broad thank you to everyone out in in our panel now and thank you to everybody for showing up. Looks like we’ve got a great set of attendees here today. So, with that, I’d love to get started by first kind of introducing everybody that we have on our panel today. Everybody with a nice freshly branded background so thank you guys for that. We all know who they are, we’ll first start with Faisel Alam who is the manager of the Office of Test Security at the Law School Admissions Council. He’s served in that role for over 10 years and helped coordinate the council’s transition from paper and pencil to a digital tablet-based assessment and then ultimately to remotely proctored exams. The Law School Admissions Council’s mission is to advance law and justice by encouraging diverse and talented individuals to study law and by supporting their enrollment and learning journeys from pre-law to practice LSAC. They administer over 150,000 high-stakes tests a year and [are] home to the well-known law school admissions test, so thank you for joining us today, Faisel, and bringing your expertise on the call. Can you tell us a little bit about how the online proctored version of your exam came to be?
Speaker: Faisel Alam
Thanks for having me, Jarrod, and hello to my fellow panelists as well and to all of you out there. LSAC has been thought of, up until probably about three years ago, as a paper-based testing company, so we had a huge shift. When considering the move to an online test and really, you know, I think a lot of us were caught in this situation, you know, prior to the pandemic. We were exclusively an in-person testing company. We had no online testing capability whatsoever. When the pandemic made in-person testing possible, we moved really quickly. And we were in the middle of a law school application cycle, so candidates and schools were really depending upon us. And one of the best decisions we made was to actually go with live proctored and not Record and Review. That was a key component in our decision. We didn’t want to go Record and Review with AI flagging issues and things like that and the problems that are presented. So what we’ve been able to do is utilize the live proctoring, remotely proctored assessments to help us avoid some of the issues that are associated with AI bias or test-takers being blocked from testing due to something that AI flagged. Live proctoring is certainly harder and more expensive but it’s worth it for us, and that’s one of the key reasons we actually want to talk to you is for that live proctoring and it’s consistent with our commitment to diversity, equity, and access.
Speaker: Jarrod Morgan
Thank you, Faisel. Up next on our panel we have Jeff Frisk, who is the Director of the Global Information Assurance certification program or GIAC. He’s been in that role since 2005. GIAC was founded in 1999 and provides more than 45 skill-specific certifications that are directly aligned with critical info sec and cybersecurity job duties are offering a wide range of certification disciplines. They target hands-on skills required for mastery of many important cybersecurity job duties and technologies. GIAC exams validate real-world competency by testing hands-on concepts and pragmatics versus just theory. Jeff was also a SANS course author and a certified SANS instructor. Jeff has more than 25 years of experience in IT projects with computer systems and business development initiatives, centered on cybersecurity. He’s also a founding board of directors member and current officer for the cybersecurity credentials collaborative. GIAC leads the cybersecurity certification industry with 11 ISO 1702 for accredited certifications. They earned conditional and seek approval for using remote proctoring on June 3 of 2020. And so, Jeff, thank you very much for bringing your vast experience and perspective to the discussion today. Can you tell us a little bit about how that remote proctored program came about?
Speaker: Jeff Frisk
Likewise, thanks for having me. Good to see you all here as well. You know, I think for us being a technology-based company and dealing with cybersecurity type of things, a lot of our candidates for certification have high technical skills. So that was something when we came in, we and me personally, I’ve always been aware of remote proctoring but it wasn’t ever something that was high on my list to be one of the first people in doing that. We were very comfortable in our brick and mortar, you know, testing center approach. Now, our exams are all internet-based and delivered through computers. So, you know, maybe a little bit different from what Faisel had going on that we weren’t paper and pencil based where we haven’t been for a long time, but we really hadn’t been thinking about making that jump to remote proctoring. As many of us probably are here because of the, you know, pandemic being the advent of necessary remote proctoring for our organization. Anyway, I kind of got dragged kicking and screaming into the remote proctoring boat here. It is something that has enabled us right to maintain business continuity throughout the pandemic, and I think now, once the cat’s out of the bag, it’s kind of here to stay. So, we have a blended solution of brick and mortar proctored locations, as well as remote proctoring, and for us and for me personally being very security conscious and having our candidates be well versed in technology, it was a little rough when we started out, right? People were not used to having the setups that they needed and some of the security settings on computers and other things, and I guess for our organization personally, there was a lot of data overload versus brick and mortar testing locations. We had a lot more data presented to us and there’s that kind of ever-present dance or balance of exam security versus data privacy and kind of this data overload thing. So for us it wasn’t the easiest thing to do. It wasn’t my first choice but ProctorU has been a great partner and we’ve been able to work through many of the difficulties and I think, here we are, a year and a half later or so since we initially introduced remote proctoring. We were very quick to market, working with you to kind of get up to speed very, very quickly. And I think, although there were some kinks, it’s been something that our candidates have grown used to and that’s a real benefit, and it’s advantageous for our program again to remain delivering tests and having candidates take tests throughout this whole time. So it’s been a real pleasure working with ProctorU, and I think our experiences might be a little bit different, maybe than some of the other panelists here and it’s great to see such a diverse group of folks. So thanks.
Speaker: Jarrod Morgan
Thank you, Jeff. And that’s really the idea here is trying to bring a lot of different perspectives and a lot of different kind of need sets into the conversation so your perspective is valuable. Next on the panel we have Mr. Larry Lynch, who is the Senior VP of Certification and Operations at the National Restaurant Association. The National Restaurant Association represents more than 380,000 restaurant locations and is the organization behind, among many things, the ServSafe exam, a widely used food safety certification in the country. Larry oversees the organization’s credentials programs, is responsible for program accreditation, customer operations, state food safety compliance, food safety instructor quality, and overseas business operations for their multiple offices. In addition to all this, he serves as a public liaison with the food industry and regulatory bodies to ensure the industry retains that policy and science voice as it relates to food science and safety, which is obviously very important in these crazy times. Larry, thank you for bringing your decades of experience to this presentation. We’d love to hear from you on how the ServSafe exam and all the different programs that the association came to be using online proctored exams.
Speaker: Larry Lynch
Sure, Jarrod, thanks. And thanks for having me this morning. There are a couple pieces to this, and I think it differentiates us a little bit from my fellow panelists and as much as our audience is a little bit different. These are typically frontline workers and frontline managers working in restaurants in some cases, the restaurant workers themselves that we’re addressing with our credentials. But the other thing that’s a little bit different about us is our programs are in a competitive set, so it’s not just us offering a food safety manager certification exam. We have competitors. So for me, what was very important was trying to figure out what differentiates us in terms of giving an exam. It’s much like what other companies give, and it really came down to how can we be convenient to our customers because if you think about restaurant workers, they don’t have the kind of daytime hours of the rest of the staff. They can’t sit down and take an exam. We needed to be where they were, when they were. And so, several years ago, and I know I didn’t put this in my bio, but the National Registry of Food Safety Professionals still exists as a wholly owned subsidiary of the National Restaurant Association now. Early on, we used a remote kiosk system, and we managed to get approved by entities several years ago with the same idea in mind that we could put those kiosks in places and that would start. So it gave us that head start in doing it. So, when we were acquired by the Restaurant Association, that same mindset was “Okay, now we’ve got a bigger program, a bigger footprint. How can we make sure that we’re more places now we have an academic audience?” We have the frontline workers have we make this play, and so early on we started talking with your group. Of course we talked to other companies as well. It really came down to the integrity and the support and the work that that you guys did, the fact that we were addressing things like data security. So we understood that’s the GDPR and those questions and issues that we have to address with our candidate data is critical to making sure that our program is safe. And much like the LSATs, and by the way I did take LSATs and I plan on never having to take them again. But I never had the advantage of doing them remotely. But for us, we were also heavy paper and pencil. And part of that just addresses the broad ways that we were offering exams we had online. But everything was live proctored, and later on we’ll talk a little bit about accreditation and some of the headaches that that we faced with that. But putting that aside for a moment, that ability to get to be convenient, be where people were, and begin to shift to a more technologically-driven solution makes sense for us. And so again, you know, try working with your team, making sure that we’re tying in the top technology components, making sure we’re addressing security. It all fell together perfectly. Then of course the pandemic took us from “let’s begin to work through this slowly” to accelerating that process pretty quickly.
Speaker: Jarrod Morgan
Thanks very much, Larry. And finally we have Mr. Ray Nicosia, the Executive Director of the Office of Test Integrity for Educational Testing Service, or ETS. ETS was founded in 1947 and is the world’s largest private nonprofit educational testing and assessment organization and is home to the GRE and TOEFL exams as well as a numerous other well-known high-stakes examination programs. He oversees a staff of 50 that is responsible for the security of all high-stakes paper- and computer-based testing programs for ETS across 192 countries. In the 26 years since assuming his current responsibilities, Mr. Nicosia has been a leader in the industry, implementing multiple prevention and detection test security techniques to better serve ETS and its clients. His investigations have resulted in actions taken by federal and local law enforcement agencies, civil prosecutions, and the closing of test centers. Ray is frequently quoted by the media as an expert in academic integrity issues and has represented ETS and business development matters on six continents. Ray, thank you for bringing your 30-plus years of experience to our discussion today. I know 2020 was quite a year for ETS as it was for everybody. Can you tell us a little bit about how ETS came to use online proctoring for its high-stakes programs?
Speaker: Ray Nicosia
Sure, Jarrod, thanks for having me here today. My experience with remote proctoring goes back about 20 years, where I testified against the idea of testing at home in a military tribunal case. At the time, it was not that developed or a secure way to give tests. I attended ATP conferences, got to see the industry grow, got to see the changes in remote proctoring. About five years ago, ETS picked up a low-stakes test, put it in a remote proctoring environment, got to see how that works, and dip our toes in the water a little bit. And with the very talented colleagues at ETS, we started looking at different options for, could we do this with our high-stakes testing programs. We had areas of concern where there were security issues. Some of our test centers we thought, “You know, could remote proctoring be better for us?” We had to ask those questions. We had customer service issues where people were located. Could remote proctoring better serve our testers? We were looking at those issues. In 2019, we began interviewing different vendors to look at the option of potentially moving some high-stakes programs into the at-home testing model, and next thing you know, 2020 hits and COVID starts creeping up, and I found myself in Asia, back in January and I asked if I could go visit the ProctorU facility in Manila. And that gave me the bird’s eye view and how it operates. It’s one thing to sit in a meeting room and get a presentation about how things operate, but it’s another thing to walk in the door of the proctoring facility out there in the Philippines and see the operations for a day and observe everything and ask them questions that gave us a sense of security and, you know, we feel comfortable going forward. As COVID began closing more and more of our test centers, ETS was able to hit the ground running. So we did sign on with ProctorU and we were the first high-stakes program out the door on March 28 last year, with our TOEFL in theory and practice programs. That’s sort of our journey that got us to where we are today.
Speaker: Jarrod Morgan
Yeah, not a lot of people call me and say, “Hey I’m in the neighborhood,” when the neighborhood is in Manila. So, we’re glad you got to see one of our facilities and glad I could share some of that story. Okay, so we’ve gotten to know our panel here. We’ve got some questions that we’ve queued up for everybody. We want to keep this very kind of conversational, very fact based, pros and cons, and the real story of what’s going on. I’ve already had a couple of people jump into the chat function down there. If you’d like to use that chat function to send in some specific questions, we’ll try to get through as many of those as we can as we’re going through our discussion. So feel free to chime in there. We have our first question coming up. The first question is:
You were all delivering exams pre-COVID in different ways, and today you’re using online proctoring for some or all of your exams. What results have you seen from an exam performance perspective since introducing online exam delivery?
So we’ll start with Larry from the National Restaurant Association. Larry, what kind of results have you seen from a candidate performance perspective?
Speaker: Larry Lynch
The good news is one of the things we have to do as part of our standards and accreditation requirement is comparability studies. We just did complete a comparability study with remote exams against our paper-and-pencil exams and found them to be perfectly comparable, and that’s clearly something that was a concern to us and of concern to our accrediting body. So for us, that was just mission number one, to make sure that we had an exam program that we felt comfortable with being delivered in the right way and it’s proving that.
Speaker: Jarrod Morgan
That’s great. And I’ve just been, you know, I don’t do Zoom presentations as often as I should. So you can ask people questions into the Q&A or the chat. Either would be fine. I know we’ve got some Q&A in there so thank you for keeping me on the straight and narrow, my friends from designing events. I’d like to ask Faisel to sort of chime in there. What kind of performance differences have you guys seen from LSAC and your users there when they started using the online version?
Speaker: Faisel Alam
I think we’ve seen some scoring differences. You know, I’m not a psychometrician to really jump into a bunch of the scoring in the performance indicators. I can give you a couple of data points about this. Overall, if we compare a full year of the pre-COVID LSAT to a full year of LSAT-Flex, which was our online three-section test that we were offering, we saw an average score increase of about .9 points. And there are a lot of theories out there why that might have happened and a lot of it has to do with the way that test-takers were able to prepare in the middle of the pandemic where. What else were you really going to do? So they had a lot more time to spend preparing for it. We have a lot of low cost or free options for prep out there like Khan Academy. We have our own law hub prep test that they can practice on. We have a score preview program that allows people to see what their score is before deciding whether to cancel it. I think a lot of those things played into helping reduce stress and the anxiety that goes into preparing for a test and what may be a completely different modality. If they tested prior to COVID, I think all those played into it. People are comfortable testing at home. I think those all sort of helped and it’s pretty interesting. But you know what we found from our research was that people during the pandemic actually ended up spending 25 to 30% more time preparing for the LSAT. And they’d spent months preparing for it before but now that they were actually able to focus on it rather than having to worry about other things, I think we saw some positive results in that. But I would like to tackle another part of this. The big change for us is we’ve only done in-person tests. So, we talked about performance. I think it was Jeff that talked about business continuity and that was a huge issue for us because our business continuity really refers to law school dreams, people that are dreaming of a legal education. If we’re not able to continue that business, we’re not providing the service to our member organizations of law schools and people that want to pursue a career in law. Not to minimize anyone else, but that is a huge responsibility that we carry and we try to fulfill as faithfully as possible. When the pandemic made it really impossible to test in person, we had to cancel three exam administrations. So, March 2020, April 2020, June 2020 – and that represents about 40,000 candidates that have signed up to take those three tests. Huge numbers. So you can imagine. Law schools were freaking out. Candidates were freaking out. People were predicting as much as a 10% drop in law school applicants, but because we were able to pivot very quickly and go into an online testing model. And that wasn’t without a lot of work. People across the organization pitched in and when we made the decision, we really flipped the switch in a little over two months to made it possible, which is just an incredible effort in my organization, if I can toot our own horn for a second. What we were able to do with the online tests for May, June, July, and August, we ended up seeing instead of a 10% drop in applicants, we actually saw a 1.5% increase overall. And in traditionally underrepresented groups, big increases as well. We saw African American applicants were up by 2.5%. Hispanic/Latino applicants were up 4%. And I’d love to place that in the lap of moving to a super convenient and accessible test modality. But I think there are a lot of other factors that are involved and not the least of which is the national conversation we’re having about systemic injustice inequities felt by marginalized identities, so I think all that played into it and saw some real positive outcomes, not just in scoring but also overall for our industry’s climate.
Speaker: Jarrod Morgan
I was actually hoping you were going to tell that story because I think it’s so interesting to watch, not just how remote proctoring is changing but just how changes in society are affecting what the candidate pool looks like and how we’re all reacting to that and making sure that we’re welcoming as many people as we can. So thank you for sharing that perspective. Now our next question is from a security standpoint. We’ve got a lot of questions coming into the Q&A chat about security and stopping different things, so we’re trying to roll all that up into this question:
From a security standpoint, online proctoring changes the way people cheat while introducing new tools to detect cheating at the same time. Can you guys share some of your thoughts on exam security and examples of how you or we have identified cheaters in this modality?
We’ll start with Ray since he sort of lives and breathes these types of things. Ray from ETS, talk a little bit about security and how you guys have caught some cheaters this way.
Speaker: Ray Nicosia
Sure. Well, one of the things I advocated for remote practicing is that it really helps eliminate some of the forms of cheating, and some of the rare forms and some of the more ugly forms. For instance, you cannot bribe a proctor in remote proctoring, so people are familiar with the Varsity Blues case. Most cases are very, very rare when a testing staff member would take a bribe to help someone cheat – you can’t do that in this environment. We also have rare situations in some parts of the world where our testing staff are physically confronted and physically intimidated to help people cheat or turn over test materials, things of that nature. You can’t do that, that’s not a concern in the remote proctoring environment, because you can’t get to the person proctoring the test. So that’s a really big upgrade, a really good preventive technique with the at-home testing model. Also impersonation. It’s really helped because we are now getting videos of the test-takers throughout the test. So, it is not just a matter of checking an ID and taking a photo, but we’re actually getting the video throughout the actual exam process. So on rare occasions, you might have someone do what we call a seat switch, where they would have someone check into a test center and then someone else would come in to take a test. In this environment with the artificial intelligence, just for the testing at-home model, we’re making sure that it’s the same person who checks in. If I check in to take it and it’s me sitting there for the next three hours, it’s not someone who kind of looks like me to come in and sit there and take the test for me. So these are some of the really big enhancements at-home proctoring gives us that we use and we can see a nice effect on how we’re preventing and stopping some of these forms of cheating from actually taking place.
Speaker: Jarrod Morgan
Thank you, Ray. I’ll call in Jeff from GIAC now. Jeff, what kind of things have you guys seen in your programs as it pertains to cheating?
Speaker: Jeff Frisk
First of all, we don’t really like to use the word “cheater.” More like, I don’t know, an anomalous exam data creator or something. Sorry that that’s kind of a joke. But you have to be careful about accusing people of cheating versus having anomalous results, and as Ray mentioned, I think the advent of remote proctoring brings in new types of challenges, but it also helps us with some of the things that you talked about. Like our exams are a little bit longer. And if people take a break in the middle of the exam, they must get re-checked in afterward, their IDs checked again, that the room is scanned. I also feel a little bit more confident in that just being able to compare. Now another thing is, you know, as I mentioned before, we’re inundated with data, when we started doing this and having recorded videos of the sessions and the room pans and things like that has been a lot of data for us, but also I think it really helps with some of the security aspects as well there, to be able to review sessions and see the things that are happening. And for us to, and I know a number of people mentioned this, having live proctors versus AI or some computer-based algorithm or after-the-fact proctors, that’s really where we’re centered as well as having a live human proctored exam, so that if something weird is going on they can stop the candidate and say, “Hey, why are you looking over here so much or what just happened over here that something changed in your condition?” I’m seeing a couple of the questions that are rolling in the Q&A chat room about high-stakes testing and performance-based things. Our program has those integrated in our exams, actual virtual machine-based performance, and I haven’t seen any material difference at all. And echoing I think what Larry had mentioned, for our accreditation we also have to do comparisons between live/remote proctoring and brick-and-mortar proctoring, and we haven’t seen any material difference at all in scoring. This is not a challenge to anybody, for cheaters out there. But we’re heavily focused on security in the high-stakes business, and I haven’t really seen any difference in types of cheating or different things. There’s always going to be a bad egg here or there regardless of what happens, but I have not seen any systemic differences. Maybe some different types of things as you all mentioned, instead of someone colluding at a test center to pay someone to get a copy of that test, maybe there are different things that happen with remote proctoring. Some of the modalities of cheating have changed, potentially, but I guess for us, I haven’t seen any big difference in huge breaches or issues. Even as far as candidate acceptance as well. I know we’re going to get to that a little bit later, Jarrod, but I haven’t seen any big differences in feedback from how different candidates are accepting the transition that we made to remote proctoring. Larry, did you have something you wanted to chime in on?
Speaker: Larry Lynch
I think one of the things that’s unique about our program is the fact that trainers are also the proctors for the exam in many cases. Most of us operate in paper-or-pencil world. We’re shipping thousands of exams around the country during the year. We don’t know what happens at the location and quite frankly at the end of the day, if your business is training people in food safety, there is an incentive whether it’s right or wrong for people to pass the exam. That’s your basis for success. For us to monitor all those locations is extraordinarily difficult. Obviously we invest a lot of money in cheating detection when exams come back and try and make a determination, but over the years – I’ve been doing this for almost 20 years – we’ve clearly seen all kinds of different ways that people find to cheat in a paper-and-pencil environment like that with that level of incentive. And what we found with remote, every day, almost every day, I’ll get two or three videos and our numbers have gone up dramatically in terms of the number of people who take remote exams. And so to me, getting an opportunity to see that assessment, from a proctor who actually has no incentive to cheat, the integrity of the proctor can’t be compromised. I know that the integrity of our exam isn’t compromised, so we feel very comfortable right now that what we’re seeing in terms of the exam takers is legitimate. And while it hasn’t eliminated, our goal was to be convenient to try and be where test-takers need to be. But the fact that we can get data back now that really positively identifies issues of questionable integrity, how’s that rather than cheating questionable integrity, allows us to do the due diligence that we have to do to make sure that the candidates are getting the right due process that they need for follow-up. We have all kinds of follow-up that we can’t always do in a paper-and-pencil environment.
Speaker: Jarrod Morgan
Thank you, Larry. The next question I have is specifically for Faisel at LSAC. I know that you guys have made some changes to your program to mitigate cheating. Several questions kind of coming in about specific ways to mitigate different things, item theft, and whatnot. Can you share a little bit about what you guys did with your program?
Speaker: Faisel Alam
First of all, I don’t want to give away all the secrets because I don’t know who’s posting the questions. I don’t know if we’ve got any moles in the audience, per se, so let’s just talk in some generalities and I’ll bring in a couple of specifics. We’re always trying to stay one step ahead of certain people. One thing that I’m proud of and you probably got this from my previous responses: it’s just a concerted group effort, how we manage the change from what was our single-day event testing model that was given whether it was on a tablet or it was paper-based and in-person. And then, shifting that to one where it’s a testing window. I noticed quickly trying to look at some of the chat there, Jarrod, and looking at some of the questions about item exposure and harvesting and things like that. Let me touch on that because I think that’s one of the biggest issues that comes with possible remote proctoring. So going from one of the best things we can do. If you have one of the most secure tests in the world and always protect your items, then we’ll test one person at a time in a secure room who’s got to be naked when we control the computer, right? But that’s not realistic. That’s not scalable. Well, maybe it’s scalable for some people. I don’t know, certainly not in my world. Our exam is given in one sitting to over 30,000 people at a time, and there’s simply not enough computer-based testing centers to be able to handle that with upgraded security. So remote proctoring was the way to go. We still couldn’t just do one event because, even with no proctor and you have 35,000 people at one time on a Saturday morning, and all the bandwidth for one organization to be able to handle all that became untenable. So we had to start looking at multiple testing windows, and in fact, we went to multiple testing days. With the in-person testing model we gave one or two forms per admin so it’s difficult for anyone to get any pre-knowledge unless you were manipulating time zones or had an in with the TV with online testing. It required us to move to that multiple-day model, and so we’re using so many more forms now and we have incorporated variable sections as well. So if anyone’s trying to game the system, they’re most likely going to be guessing incorrectly and fall flat on their face on their variable section or what they’ve heard from somebody else that’s going to be on their test. So it’s not using either bank, and our development folks have worked their tails off to get us in a great position to use these tools against individual cheaters. And so that form rotation, introducing the variables section, has changed what any one person is going to see on our test prep. When people in test prep talked about our test and he said it’s all but impossible to cheat by trying to gather information on what forms might be used due to the number of forms and the way that we mix and match them. But of course, as I mentioned previously, what really keeps me up at night is these item harvesters right and that’s sort of been the Achilles heel, if you will, of remotely proctored tests. But with a little bit of innovation and in some investigation, we are able to catch these people. Video was huge for us, being able to have that when we were paper-and-pencil or a tablet-based administration where we were in person, we were able to go back and review video and stop and figure out what kind of suspicious behaviors were happening. Certain AI flags are helpful. They can’t make the determination, but they can point us in the right direction. Exam item response time, things like that are helpful to us, and one of the biggest things is because we are able to stop a session if we see suspicious activity. Before when it was a paper- or tablet-based exam, people that were harvesting could harvest an entire form, especially in paper-based. You lose one paper-based paper booklet, the form is gone. Here, we’re able to limit the exposure of those items. If something is compromised, it’s on a minimal basis. We can throw those items out. We can rotate things. So a lot of innovative ways to look at things. Risk mitigation, you know. And so we’ve incorporated those things. Again I can’t get into too many specifics about what we’ve really done, just for fear of divulging too much.
Speaker: Jarrod Morgan
Really, really helpful. Thank you. Our next question is one that comes up a lot, and I think is one of the key drivers of the growth of online programs, which is about candidate satisfaction.
So, you know, Larry. We’ll start with you on this one.
What have you guys seen from a candidate test-taker satisfaction perspective, as you’ve moved and had an online format available?
Speaker: Larry Lynch
And maybe a little history with this, too, I think helps, Jarrod, because we were going down this path with you and trying to figure out where do we tie in our existing systems with ProctorU’s systems and of course with the pandemic last year, it kind of forced us to rethink that model, simply because, as the pandemic was hitting, the academic community was going into test-taking time, particularly for us. High school students take our ProStart Program. ProStart requires a service AP exam, and high schools had no way of face-to-face proctored exams, and so we came up with an innovative way to move faster. Our intention of course was always to offer this to our broader audience and to me it was that acceptance, once we got past the academic piece and then offered it in general to our total population was the shift, we’ve seen high double-digit growth in our remote proctored exams. So that’s incremental, which is great. And there’s some of that as a shift as we watch our paper-and-pencil numbers drop but then equally, our remote proctoring numbers increase. The adoption has been extremely strong on the satisfaction side. You know what we’re seeing is good. The technology works. Obviously, our customer service teams on my end and your end communicate well, so if there are technological issues, we can resolve them pretty quickly. I think the most important thing that we hear from test-takers is the convenience part. Now I haven’t seen anybody worry about the integrity of the exam or the concerns there. I mean obviously the proctors do a fantastic job of holding them accountable, and we certainly see that in the videos that you send us when there is something questionable that takes place during an exam. But our customer feedback is positive. I don’t have exact numbers on that, but like I said that high double-digit growth for us in the remote proctoring by itself, so that being the shift from paper and pencil, but some of that being incremental, tells us that this is a modality that’s going to work for us.
Speaker: Jarrod Morgan
Thank you, Larry. Ray, I’d love to get your perspective on that too. You guys have got a global audience. ETS has different programs. How have you seen the kind of feedback when you introduce this option? It’s remained an option for people and then sort of what’s the sense of how many people are using that versus the old way?
Speaker: Ray Nicosia
Well, obviously we were happy to be able to launch it last March, we’ve seen a lot of our test-takers throughout our 192 countries have been taking it online at home. The satisfaction is very high. We’ve actually been able to offer the test more frequently. We have one of our testing programs that used to be 10 times a year paper-based and then maybe three times a month in the computer-based model, but now we’re 24/5. So you could take a test 24 hours a day, five days a week whenever. Other testing programs is 24/7. We have people who are students who would like to take tests late at night, where people don’t want to take off work during the day. We have people with issues getting into a crowded test center on a Saturday for instance. So it’s usually a prime time to get into a test center but people could take the test at their leisure around their schedule. They could take it at their home, dorm room, an office. So we don’t have the physical limitations of the tests and are we are still operating our test centers. Over three-quarters of our test centers are still open right now and testing, but that we are going to this modality and we’re giving the students a choice – where do you want to go: at home or go to one of the test centers? We are in that mode right now and will continue to do that for the foreseeable future.
Speaker: Jeff Frisk
Hey, Jarrod. Do you mind if I add something? And I think this will address a couple of the questions that I see rolling in in the chat window as well about, you know, technology differences with different people’s computers and settings and things like that. Bridging into the satisfaction or do people dig it or not dig it. The 24/7 thing raises huge flags for us as well. We have people that are all across the world taking tests at different times and remote proctoring really helps with that. And I think for us, there were definitely some technology or adoption hurdles at first when we went out of the gate. And what I found is really made a difference is getting information out to the candidates in advance. Folks have to know what their computer setup is going to be, what they have to do with their desk and clear things off and not have a bunch of other video cameras and things in the room taking pictures of the test questions. So I think we struggled with that a little bit at first, from a satisfaction or technology point of view, but we found really by getting the right information out to people and making sure they could be aware of what the conditions needed to be both from a computer setup in a room and what they needed to do in advance. That’s really helped us and although when we first started, there were more complaints and people not being familiar in our program with remote testing, that’s really evened out for us. And we found by getting the right information in advance, we’ve taken care of almost all of those issues. So I think that bridges the gap between a couple of the questions I saw rolling in the chat window too, Jarrod.
Speaker: Larry Lynch
We’ve had kind of same situation with ours where restaurant workers, you know don’t have the nine-to-five hours that many people do to go down and sit down and take a test after hours is important. Some people are already well-versed in food safety, so they don’t need to get the training. So it’s a great option for them but even if you’re not, again that multiple modality – that to us is important. You can still take a course online if you want to and then turn around and take the exam at your convenience. Again, I think to me more than anything else, the convenience is most important but convenience locked into a secure environment, just adds to the value of what we’re doing.
Speaker: Ray Nicosia
And if I can jump in a little bit. It’s kind of crossover. Jeff brought up some of the questions that are coming on the security side. As we were the first ones out there with ProctorU in the high stakes world last March, and there were things we did at our test center that ProctorU was not doing at the time. So we worked with ProctorU. We said, “This is how we think secure administration should take place. There were some changes to the flight path, as it’s called for the check-in for the test-takers. We’ve heard you’re cleaning up the test but also just looking to see, you know, is someone trying to remove test content during the test?” There are certain things that are done before the test is even launched to check for that option and then, ProctorU, you did develop a ETS-specific proctored exam, that the staff had to take in order to give the proctor one of our ETS tests. It’s our tests tend to be more unique with different sections and speaking components. So there was a special ProctorU test you did develop for us. And the proctors have done a much better job there now picking up the test-takers who are trying to bring in the cheat sheets or are trying to, you know, have an answer key written down on something or trying to sneak in a cell phone. So, the staff does a good job picking up these little cheating techniques and are able to confront the test-taker online and stop a test.
Speaker: Jarrod Morgan
Yeah, for sure. This is a really good, I mean that’s a key question right? What’s the reaction been to and I think most of the people that were attending here today wanted to hear that. Now in the interest of time, I’m actually going to skip over the next question. And we had some information about accessibility we’ll share with all the questions that are coming in that are not getting answered, we’re going to gather them together and try to send out some of these things digitally afterward so we’ll put some stuff in for accessibility there. But I wanted to jump to one that I think is a really important question and it’s about accreditation. It’s very important for programs and online proctoring, you know, to consider that if you’ve got a program that’s accredited and how does that work. We’re lucky enough on the panel here we’ve got two different organizations that have gone through that process. So I’ll start with Jeff from GIAC. Can you share a little bit about what that experience was like to get accreditors to see this and understand it and allow you to use it?
Speaker: Jeff Fisk
Yeah, and you know it’s very timely for better or worse. This is our week that all of our ISO NC 17024 annual audit paperwork is due. So, I am neck-deep currently in finalizing some of the thousands of pages of paperwork that we turn in for the soon-to-be 13 accredited exams that we have, but that’s a huge thing for us right? Our program is high-stakes as well, probably a different way than other people’s programs where a lot of times to have a specific job in cybersecurity you need to hold a credential in the US government or other places. And so it’s a big deal. I’m not only the ISO NC 17024 standards, you know, focused on validity and fairness, and all different kinds of things. And that’s regardless of if it’s a paper-based or computer-based or remote-proctored exam, but as part of that, the standard has developed very quickly with the advent of our pandemic state. A special remote proctoring additional section that you have to fill out or whatnot and so that also bridges some of the stuff that we do anyway. When we talk about our scores going up or down, are there more or less cheaters or are we seeing item leak or be looking at data forensics at incorrect answer option patterns? For us, we kind of look at that with blinders on, but then we also specifically look at it with brick and mortar proctoring versus remote proctoring. And for us, having run all the numbers and I think Larry mentioned some of this stuff too. We haven’t seen any material difference in scores or, you know, cheaters that we’ve caught one way or the other. I do think it really helps to be able to stop an exam midstream or real-time to have a proctor ask a candidate to pan the room again or do this or let’s check out why, you know, whatever is happening. But for us, accreditation wise, there were some additional things that we had to do, and we had to make sure we were comfortable with the security of ProctorU’s proctors and the processes, but as I think Ray mentioned as well, updating the flight path so it’s called or the steps that a proctor would go through for a specific exam. We found some of those things had to be updated after our initial kind of beta launch, but I haven’t seen any real material differences in our accreditation aspect and I think most of that has been centered for us on: are we delivering exams in a fair and reliable and valid way, and then do we stand behind the results? The same way, when we dissect things and look at it as a whole, and then separated out for brick and mortar, or live in-person testing, or remote-proctored testing, we haven’t really seen any noticeable difference in any of the metrics that we use to look at our exams – the same fair, valid, and reliable kind of things.
Speaker: Jarrod Morgan
And of course, you know, Jeff’s got that perspective, and you guys have got 11 different programs that go through the 17024 process. Larry has the distinct pleasure of having two different accreditation standards to adhere to not just 17024 but the food safety standards. Larry, talk a little bit about what that was like.
Speaker: Larry Lynch
I mean obviously, you know, the good news is that the standard that we operate under the most which is developed by the conference with food protection. We tried to build to at least get close to mirroring ISO 17024 and that’s been something we’ve been working on for a lot of years. You know with that said, however, we always have to remember that when it comes to the accreditation part, they’re approving us to use ProctorU, they’re approving us to use another vendor. So we have to go through a lot in terms of what we report at the outset, and application process is pretty onerous and it was relatively new because the conference with Food Protection standards was never designed for remote proctoring early on. Just paper and pencil because that was the modality that was created many years ago. It shifted a little bit to the online delivery, but never addressed some of the other security concerns they had to worry about. So I do have to give kudos to answer, they pretty much pivoted fast when they recognize that the early stages of the pandemic that a lot of us are going to have to find a solution that we needed. But clearly, I think, what it does is it also reassures the public with that accreditation, that we’re still held accountable to the security and the integrity of our exam programs, the fundamental fairness to our candidates is the thing that we should be most concerned about. They’re holding us accountable to so it’s not as though we woke up one day and say that we’re just going to throw out this new remote prospering program and life is going to be good. We do have to do those studies like Jeff, we’re right now in the midst of our assessment accreditation process which is why we have this data on comparability. But it was great news when our exam team called me and said, “Hey, great news! We just discovered that there are no comparability issues, that these exams work well together.” And I think it continues to build the case for the integrity of the program. To me, the most important thing of all is that public perception that we are doing the right things for our programs, are doing the right things for our candidates, are we seeing data that says that there was a problem, and we’re not seeing any of that. And so, that accreditation process to me more than anything else, there’s a lot more data work that we have to do the applications, but to me, it just speaks volumes on the integrity of exactly what it is that we’re doing.
Speaker: Jarrod Morgan
Thank you, guys, and I want to jump to the next question in the interest of time. This next question was about, you know, reaching a diverse candidate pool and kind of the feedback you’ve gotten. I think we got some really good feedback from Ray and Larry about the candidate reaction. And you guys talk a little bit about how to open up a new pool, specifically Faisel with LSAC, can you talk a little bit about, you know, you’ve talked about its kind of core to your mission to open up your exams to a diverse candidate base. How does this process helps in that effort?
Speaker: Faisel Alam
We’ve roughly administered over 170,000 tests, and before you know that’s sort of reaching our high-water marks when back in 2000, lagging on the years now, but it was like late-2000s, like 2008/2009 cycle. And so, you know, we’re starting to see some applications really in this current cycle. Overall, application applicants are up 12.6% compared to last year, which was trending to be a decrease but it actually increased. African American applicants are up 11.5%. Hispanic/LatinX applicants are up close to 13%. So, you know, we wanted to make sure that we were reaching . . . core to our mission was making sure that we were accessible to anybody who wants to take this and making the test available. Reducing the barriers to get to testing, rather than driving to a physical center and worrying about the other things that come with in-person testing, we really focused hard when we were going to pivot to online that we can create new barriers in the shift. So one of the things that I think we’ll talk about it, I don’t know how much we’re pressed for time, but I mentioned one of the big things for us was about the technology. I see some comments about technology that’s being required. So, you know, look, if you didn’t have a tablet, if you didn’t have a computer or a laptop to take the test, it didn’t matter if you didn’t have it. We wanted to make sure that you had one that worked. So we shipped you a tablet if you requested it. We didn’t check to see if you had a laptop or anything, we would ship your tablet. And to date we’ve shipped out nearly 4,000 of them to test-takers so they’ll be able to take the test. If your house is anything like mine, it’s not a distraction-free environment. It’s kind of like that frictionless plane in physics class, right? Doesn’t exist. Well, hey, then let’s take you out of your house, let’s give you a place where you can have a quiet space. So, a lot of test-takers face this. People with children, things like that. So what we did was we said, “Well, you know, what if they didn’t have reliable as well you know other, there’s not just distractions, but we can be reimbursed for rental fees for quiet space.” And today we’ve reimbursed hotel costs for up to about 2,000 test-takers, so we’re starting to see a lot of this. So we want to make sure that there are no new barriers created by going to this modality. And in fact, in core to our mission about accessibility, you know, 70% of the test-takers who have taken us up on these forms of assistance are people of color, so that is core to our mission of diversity, inclusion, and equity.
Speaker: Jarrod Morgan
Yes, pretty unique to you guys and a really cool effort that you guys do to tear down any barriers that pop up and it’s actually a great segue into the next question which we have, which is the one that comes up a lot. Technology issues people are concerned about it. They’re concerned about how it sort of, you know, affects the candidates and what they say. So we’re gonna have Jeff and Larry talk about these two very different candidate populations. Jeff, what’s your take on technical issues and how they’ve affected the candidates?
Speaker: Jeff Frisk
I think it’s interesting, you know, you probably look at the dichotomy of Larry’s typical candidate and then our typical candidate. Someone, and again not painting any broad brushstrokes, but just maybe not, you know, as computer savvy as some of our people who that’s all that they do is computer circuits, security stuff. So very different. And I think from our side, I saw one question in the chat window as well about the people using virtual machines. I think, as far as I understand, no, you can’t be on a virtual machine and take a test with ProctorU, and there are just security concerns with that so that was different for some of our candidates. They say, “Well, I have 14 virtual machines running all the time.” You say, “Well, you need to turn all those off before you go take one of these tests.” And I think we did a couple of kind of loaner programs, but a lot of times, the way that we’ve talked to our candidates, and it’s not from a “you don’t have a computer,” it’s the “my computer has top-secret data on it or my company won’t allow me to do this.” So it took us a little while to work with. Well, let’s get a guest profile set up, and you can have a corporate solution where you can push out from the security team or operations center, a guest profile that you would use only to take a test and then burn it down afterward. And clearly, having some of the intrusive types of things that you need to do with remote proxy, I need to take over. Not me but the ProctorU proctors need to take over your webcam, they need to take over your keyboard to shut down applications that are running like whatever instant messenger and stop, things that you shouldn’t have. So for us, that was a challenge just insofar as people weren’t used to it, but I think once we had that right out front, that this is going to be different, you probably shouldn’t be taking the test from your top-secret work computer, or whatever else, right? And I often jokingly say, “If you have children, what computer do your children use to go online or, you know, check their email or watch Netflix, right?” You probably aren’t using your corporate-sponsored, security-enabled device. For us, it was a learning curve, but we found when we get the right information out to people and we let people know in advance what the requirements are going to be both with the ProctorU test-it-out kind of program or showing up in advance the day before to make sure you check bandwidth right and other things. That’s really been a game-changer for us. The first time someone went and tried it without doing those pre-checks, yeah, there were issues. But we’ve kind of brought that problem forward. I’d rather you have connectivity or technical issues a couple days before you test so we can work that out versus on test day. So that’s been the biggest thing for us is just moving the issues forward in time so that we can deal with them.
Speaker: Jarrod Morgan
To Larry in about, you know, 30 seconds. Give me an idea of like what’s your … That was super good stuff from everybody.
Speaker: Larry Lynch
Just ditto, Jeff. No, you know I think for us, it’s really a couple of things. We recognize economically and geographically we may have candidates where this might not be the best choice and so for us the solution was, make sure we have other modalities available. Oftentimes people do need the training. We do have live classes around the country so there were options for people. Where we have done the remote practice, we haven’t seen too many technological challenges and, more often than not, they can be addressed. And so for us, again, the constant concern is just making sure that we’re making our test as accessible as we can, recognize that our population is broad-based both geographically and demographically, and we have to be able to address that. And we can address it with remote proctoring, at least have options for people to select otherwise.
Speaker: Jarrod Morgan
All right, so final question. This one’s for everybody on the panel and we’ll go kind of in order here. Everybody wants to know what the future of testing looks like, right? So I’m not asking for a full crystal ball, you know, analysis here but, pick a date in the future at five years from now. What does testing look like? Is it hybrid? Do you still see online? Do we somehow snap back to, you know, 2005? What is the future? I’ll start with Ray since you’ve spent a lot of time watching the industry evolve. What do you think about the future of testing?
Speaker: Ray Nicosia
I don’t think we could put the toothpaste back in the tube. I think you know the remote proctoring is going to be here to stay. ETS is committed to keeping some physical test centers open for people who need to go to a test center, whether you have something at your house that prohibits you from taking the test there, but I can see both models being in play for a long time. We launched computer-based testing in 1993 with TOEFL and GRE to computer, but it took decades to totally eliminate the paper-based world, so I can see using that modality. I can see the test center is still being used in some locations where they have to be, people choose to be. And remote proctoring, that will definitely be offered as well.
Speaker: Jarrod Morgan
Faisel, what’s your take on the future?
Speaker: Faisel Alam
I feel like I have to echo Ray and I can’t bring any new information into this, but I think that’s exactly right. You go where your customers are essentially and to make sure that we are serving our populations. Like I said, aspiring people that are aspiring for legal education and specifically for our tests, but the industry in general, you know they’re going to make sure that they serve this industry and this community. Those that are seeking licensure credentialing, they’re going to be able to serve them, and we’re always going to have a contingent, you know, we still have people that send faxes today, right? Not really necessary but they do it. We have a condition with folks that want to take our tests that want to only take it in person, you know, so what do we do for those people? We can’t just leave them behind. We have to make sure that we’re serving anyone that wants to pursue legal education. And I think that’s going to be true for everyone. So I think it’s not earth-shattering, that it’s going to be a dual, that mixed modality solution. I think that’s pretty fair to say in the next, you know, five to ten years. The biggest asterisk I would say is making sure that those two are equivalent. And that’s something that our organization is working very hard on. Our test is the gold standard of what we do, and we had a paper-based model for a long time. We started studying computer for much longer before that and never made the change because we had a reliable assessment tool in place for our law schools to use. So not going down the path of Coca-Cola and adjusting new Coke, we stuck with what worked, and so will always stick with what works. So making sure that we’re going mixed modality, that those two things are equivalent, and the score that it produces is relevant to those that need to use it.
Speaker: Larry Lynch
Yeah, I’ll be a little more predictive in mine. If you think about the speed of change, Ray was talking about how long it took to get ETS to where it is today. I think exponentially things are changing faster. And if we’re going to want to be where our customers are, I think we’re going to see – and security still being parallel and everything – that we do focus on the security of our exams and the opportunity for our candidates. I think we’re going to start to see things shift more mobile. You know I heard Faisel talk a lot about tablets and we think about where we could be five years from now, I think that convenience and security are both going to be drivers for us going forward.
Speaker: Jarrod Morgan
And finally, we’re in overtime but we need to hear from our friend Jeff. Jeff, what do you think?
Speaker: Jeff Frisk
I just have one thing and this is probably counterintuitive. I can get behind what all y’all said. I think for us, one of the big differentiators working with ProctorU, instead of relying on AI and robots proctoring, although that might be futuristic, I think having humans that are proctors is a huge benefit for us. So if I look into my future, I don’t see it changing to AI is ruling the world for proctoring. And I think humans are going to be involved because I get a lot better feel for the test security when there’s a human there. So, anyway, thanks I really appreciate it and great job, Jarrod, for facilitating this great discussion.
Speaker: Jarrod Morgan
Thank you guys very much. We at Meazure Learning also agree with that position on AI, got some information out there on that. I want to say thank you to everybody for attending. Larry, Ray, Faisel, Jeff, thank you for lending your expertise here today. It’s been great to hear from you. We’ll follow up with some information on your Q&A that you guys sent in if we didn’t get to your question and thank you guys very much. Thanks, everybody.