Authors: Kaitlyn Healy, PharmD; Siu Yan (Amy) Yeung, PharmD, BCCCP; Clement Ng, PharmD, CAHIMS; Brian Grover, PharmD, BCPS; Hyunuk Seung, MS
Learning Objective: Describe the feasibility of utilizing a large language model (LLM) to analyze clinical interventions completed by pharmacists and assign appropriate categorization.
Background/Objective: The goal of this study is to use a LLM software to sort pharmacist interventions into predefined categories determined by the research team and compare the interrater reliability of sorting completed by the LLM compared to that of a pharmacist.
Methods: This retrospective study evaluated pharmacist interventions documented within the electronic health record (EHR). Completed pharmacist interventions include both structured fields and free-text documentation. Interventions were extracted from the EHR, de-identified, and a random sampling was selected for inclusion. Selected interventions were categorized by both the research team and a large language model (LLM). The LLM used in this study, GPT-5 accessed via the Microsoft Copilot™ interface, was adapted using structured prompt engineering to simulate pharmacist clinical reasoning. Retrieval-augmented generation (RAG) was incorporated to support predefined categorization of interventions. Inter-rater agreement between the research team and the LLM will be assessed using weighted Cohen’s kappa analysis. A total of 852 interventions will be analyzed based on a power calculation targeting a kappa of 0.70.
Results: Overall, 889 interventions were included. All 13 categories were used by both the LLM and the pharmacist reference standard. Interventions received a mean of 1.23 labels and a median of 1, though 185 pharmacist-reviewed and 168 LLM-reviewed interventions were assigned multiple labels. Primary category agreement between the LLM and the reference standard was substantial (κ=0.70; 95% CI: 0.67–0.74), closely aligning with inter‑pharmacist agreement (κ=0.72; 95% CI: 0.69–0.76). Multi‑label agreement was similarly strong, with Jaccard similarity of 0.76 for LLM vs reference and 0.77 for pharmacist vs pharmacist, alongside low Hamming loss and no consistent pattern found regarding category‑level variability.
Conclusions: LLM performance in classifying pharmacist interventions aligned with human classification, with substantial agreement seen on primary categorization and multilabel alignment. These findings support the potential use of LLMs for pharmacist intervention classification tasks.
Self-Assessment Question:
- Which of the following describes the purpose of using a large language model in this study?
- To replace pharmacist clinical decision making.
- To generate new pharmacist interventions.
- To categorize completed pharmacist interventions.
- To standardize documentation of pharmacist interventions.
