Name
Leveraging LLMs for Item Generation in Specialized Assessment Domains
Description

Generative AI methods using Large Language Models (LLMs) have emerged as tools to alleviate the resource intensive demands of generating item content. However, the use of LLMs for Automated Item Generation (AIG) may underperform for highly domain-specific exams (e.g., medicine). In this presentation, two national medical certifying organizations discuss their implementation of various generative AI approaches to AIG. Specifically, we discuss the impact of different LLMs, prompting strategies, and the use of Retrieval Augmented Generation (RAG) on content accuracy, content relevance, and item development efficiency gains. Additionally, we discuss how Subject Matter Experts interacted with AIG tools to generate items, and their review and revisions of generated content. Attendees interested in: 1) optimizing resource allocation in specialized assessment domains through generative AIG approaches, and 2) increasing the involvement of SMEs in AIG processes will find value in this session.

Date & Time
Tuesday, March 3, 2026, 4:55 PM - 5:40 PM
Location Name
Celestin F - 3rd Fl