A group of University of Minnesota Law School instructors gave the AI bot ChatGPT four exams alongside some real students. The bot, which is considered the world’s most advanced form of generative AI, was found to be a C student.
The professors were curious about how ChatGPT might be used to help students cheat or aid teachers in teaching. Jon Choi, an associate law school professor, ran exam questions through the AI, reformatted them to fix the exam and then mixed the exams in with student’s tests.
The bot earned a C+ average in Constitutional Law, Employee Benefits, Taxation and Torts, which is considered a low but passing grade.
The bot came 36th out of 40 students in Constitutional Law, 18th out of 19 in Employee Benefits, 66th out of 67 in Tax, and last out of 75 students in Torts.
Choi said that the bot could recite legal rules and correctly describe cases without citations. However, it proved ineffective at spotting issues and providing deep reasoning for analysis.
ChatGPT performed better on essays than on multiple choice questions, and performed particularly poor at multiple choice questions which involved mathematics.
Of the experiment’s three human grades, who graded tests blindly, at least two told Choi they suspected which exams were from AI. They were correct. For now, spotting an AI generated exam result is easy, but a ChatGPT generated rough draft which was revised by a student will prove much more difficult, Choi said.
#MaltaDaily