AI Taking the Bar Exam... Scored Over Half Correct

Loen Company BigCaseGPT Accuracy Rate 53.3%

Legal tech company 'Law&Company' announced on the 18th that BigCaseGPT achieved a 53.3% accuracy rate on the multiple-choice questions of the 12th Korean Bar Examination.

BigCaseGPT is an AI-based integrated legal information service developed by BigCase and Law&Company's proprietary AI system. Earlier in March, Law&Company had used OpenAI's latest language models, GPT-3.5 and GPT-4, to solve the multiple-choice questions of the bar exam. According to Law&Company, the accuracy rates were confirmed to be 28.7% and 34%, respectively.

The bar exam multiple-choice questions consist of a total of 150 questions: 40 in public law, 70 in civil law, and 40 in criminal law. BigCaseGPT correctly answered 80 questions in total, including 19 in public law, 40 in civil law, and 21 in criminal law.

Law&Company applied a Retrieval-Augmented Generation (RAG) model based on GPT-4 and their own prompt engineering to BigCaseGPT. Additionally, they built a vectorized database focusing on high-information-value precedents among the 3.29 million precedents held by BigCase to provide optimized answers for legal queries.

Law&Company also trained BigCaseGPT with a total of 160,000 legal information items, including 145,000 statutes, 7,000 rulings, and 7,000 official interpretations held by BigCase, in addition to precedents. According to Law&Company, "BigCaseGPT features a near-zero control of hallucination, a major limitation of generative AI, significantly enhancing data reliability. Through repeated testing, we have verified that BigCaseGPT does not present fake precedents or statutes, and we plan to completely eliminate such errors through continuous quality assurance (QA) in the future."

Going forward, Law&Company plans to enhance the system by applying multiple language models and their own optimally developed language model to BigCaseGPT. Through this, they aim to launch a new AI-based B2B SaaS service for lawyers called 'Super Lawyer' in the first half of next year. The service will provide legal memo generation, legal document summarization, and legal Q&A to contribute to improving lawyers' work efficiency.

Kim Bonhwan, CEO of Law&Company, stated, "Super Lawyer, developed as a language model optimized for Korean and Korean law, will operate like an AI legal assistant helping lawyers and will play a role in maximizing lawyers' work efficiency."