The higher education and professional assessment industries are no strangers to the transformative power of artificial intelligence (AI). In the past decade alone, we’ve seen advancements in personalized learning and tutoring systems that can adapt to individual learning styles, improvements in assistive technology such as text-to-speech and speech-to-text that enable people with visual impairment or hearing loss to access material, developments in adaptive assessment systems that can tailor exams based on performance, and much more. Generative AI—a subset of AI that can generate human-like text and responses—has taken center stage lately, with programs like Google’s Bard or Microsoft’s updated Bing search engine. One program in particular, ChatGPT, has captured widespread attention for its user-friendly interface, availability, and versatility.
As with any disruptive technology, concerns and questions have emerged about the use of ChatGPT. The higher-ed and professional credentialing industries have begun to question how it will impact the future of assessment—for both test-takers and program owners and administrators. We’ve heard many of those questions—some about Meazure Learning specifically and some broader, more philosophical questions about ChatGPT in general. In this article, we’ve gathered the most common questions from program owners, psychometrician teams, and higher education administrators and faculty. Let’s take a look at a few of them.
Common Questions and Concerns
Question #1: How is Meazure Learning detecting ChatGPT and/or preventing its use?
When you break it down to its basic elements, ChatGPT is a web and mobile application. It requires users to navigate to a website (or mobile app) to log in and access previous interactions or to start a new chat. Our exam integrity solutions are already well equipped to detect and prevent the use of ChatGPT—just like we would detect and prevent any other internet-based resource. Our proctoring and secure browser technology allows test owners to restrict URLs allowed, block the ability to open any new browser tabs, detect and deny virtual machines, and restrict applications and processes from running in the background during an exam. Our technology can also disable screenshots, print functions, and copy/paste functionality, limiting the potential for test item harvesting or unauthorized sharing of exam content—with other test-takers or with ChatGPT itself. Beyond the detection and prevention methods facilitated through our technology, all our online proctoring services include a screen recording of every exam session as well as either a professional proctor review or live proctor monitoring and intervention.
Question #2: Will my long-form essay assessments be at risk?
It’s too early to say how or if ChatGPT will impact the integrity of long-form assessment formats. But if you’re concerned about test-takers using the tool to gain an unfair advantage or potentially plagiarize responses on your assessments, there are a number of security measures to consider. Implementing remote proctoring or using a secure browser can help protect essay assignments. Plagiarism detection tools are also a common way to identify academic misconduct, but it’s unclear how effective these tools will be in detecting the use of ChatGPT. Additionally, if you haven’t already, you might consider adding a section to your syllabi or candidate handbooks about ChatGPT—and other generative AI. This is a good way to remind test-takers of any integrity policy that might apply to using AI and reiterate that AI results are not always accurate.
When it comes down to it, in its current iteration, ChatGPT can’t generate the types of nuanced answers that test-takers could use in assessments that require critical-thinking, such as long-form essays. As technology continues to evolve, however, it is essential to monitor its impact on assessment formats and adapt strategies accordingly.
“If critical-thinking skills, performance skills, and observations play a role in determining one’s competency to practice, then it’s likely traditional assessment strategies will be around for a while.”
Dr. Ashley Norris, SVP of Strategic Growth Initiatives & Compliance, Meazure Learning
Question #3: How are test-takers, program owners, and test administrators using ChatGPT today?
In educational settings, both test-takers and administrators have begun exploring the possibilities offered by ChatGPT. In computer science classes as well as language or writing classes, for instance, some instructors use ChatGPT to teach students how to interact with and analyze AI models, providing assignments that require the use of ChatGPT and a log of both inputted prompts and the responses they generated. Much like Wikipedia—and with many of the same caveats about reliability and sources—ChatGPT may be a helpful springboard into further research on a topic. It could also serve as a tool for generating supplemental review materials and practice content to help test-takers prepare for their exams.
In the higher education industry, instructors have been adopting ChatGPT into their curriculum in innovative ways. Some are encouraging test-takers to discuss learning objectives with ChatGPT—asking it questions and requesting clarification on more complicated topics—as a way to deepen their understanding of the material.
In the realm of professional testing, ChatGPT may be able to assist with outlining competency profiles or job-task analyses. For example, if you’re developing competency profiles for Certified Public Accountants (CPAs), you could input prompts such as, “What are the primary responsibilities of a CPA?” or “What are the typical daily tasks of a CPA?” You can then use ChatGPT’s response as a foundation prior to the validation study.
However, test administrators, program owners, and test-takers must be cautious. ChatGPT lacks the ability to distinguish between reliable and unreliable sources, potentially returning incorrect or irrelevant information. Administrators, program owners, and test-takers must be aware of this limitation when incorporating content generated by ChatGPT into their endeavors.
Question #4: How can ChatGPT be used to create test items?
ChatGPT opens new possibilities for creating test items, but those possibilities come with risks. For instance, if you provide the right criteria and specifications, such as key concepts and cognitive levels, ChatGPT can help generate drafts of test items that align with specific learning objectives or competency requirements. These items may not be as sophisticated as those created by trained, professional humans—and should always be fact-checked for accuracy—but they can be of use in building a very large item bank, for instance. Test developers can also use ChatGPT to help revise and refresh old or retired items, rephrase questions, and update outdated scenarios or examples for use in practice exams.
When considering using ChatGPT for item development, there are some caveats to keep in mind. Because ChatGPT learns by analyzing whatever text it encounters, any information—including proprietary exam content—that is entered into ChatGPT will be incorporated into its system. In other words, ChatGPT will “learn” from your content. There is also some debate over who owns content generated by ChatGPT. If you use ChatGPT’s content in your exam, you may not own the copyright to that content—it may even be public domain!
Conclusion
The rise of ChatGPT in the higher education and professional assessment industries presents both opportunities and challenges. While concerns exist, we are actively implementing measures to detect and prevent unauthorized use during exams. It is crucial for assessment program owners or faculty and instructors to use a vendor that adapts its best practices to the growing prevalence of ChatGPT and other generative AI tools like it. Although we recommend proceeding with caution, by embracing ChatGPT’s potential, program owners or faculty and instructors can use these technologies to enhance assessment experiences.
To learn more about the ethical use of AI in the higher education and professional assessment industries, read “5 Principles of Ethical Online Exam Proctoring.”