Name
Artificially Generated Items: Shepherding the Next Generation of Subject-Matter Experts
Speakers
Heath Kincaid, American Board of Obstetrics and Gynecology
Pooja Shivraj, American Board of Obstetrics and Gynecology
Pooja Shivraj, American Board of Obstetrics and Gynecology
Description
Each year, the American Board of Obstetrics and Gynecology (ABOG) relies on over 250 physician volunteers to create items for certification content. However, the efforts in volunteer management, item editing, and field testing proves costly, involving services from volunteer physicians and ABOG staff. To supplement volunteer-created items, ABOG used a large language model (LLM) to create multiple-choice questions by utilizing proprietary content from oral board certification items.
An oral board certification item consists of a vignette, discussion prompt, and an examiner guide. In the Artificially Generated Item (AGI) process, a private LLM converts oral certification items into multiple-choice items for initial and continuing certification. We create AGIs by transforming discussion prompts into lead-ins, followed by the transition of the examiner guide into a key and 2-3 distractors. This AGI process allows for scalable, rapid item generation and efficient use of human services.
Session Type
Snapshots
Session Area
Certification/Licensure, Education, Industrial/Organizational
Primary Topic
Test Development and Psychometrics