P1769 - Navigating the Future: Evaluating Artificial Intelligence Recommendations for Colonoscopy Screening Against Standardized Guidelines

Mohammed S. Al-Zakwani, MD, MS¹, Shuhaib Ali, MD¹, Pir Shah, MD¹, Juan Echavarria, MD², Omer Shahab, MD³, Muhammad Haris, MD¹, Bara El Kurdi, MD^1
1UT Health San Antonio, San Antonio, TX; ²University of Texas Health Science Center at San Antonio, San Antonio, TX; ³Virginia Hospital Center, McLean, VA

Introduction: The application of Artificial Intelligence (AI) in healthcare has seen rapid progression, with Large Language Models (LLMs) emerging as essential tools due to their proficiency in assimilating unstructured clinical data & generating meaningful inferences. LLMs have demonstrated potential in executing routine clinical tasks. We aim to assess the aptitude of widely accessible LLMs to perform a specific clinical function: providing recommendations for polyp surveillance colonoscopy consistent with society guidelines.

Methods: A cohort of 100 patients who underwent the most recent screening/surveillance colonoscopies at our institution were evaluated. LLMs, including GPT 3.5 & GPT 4 (chatGPT) & PaLM-1 & PaLM-2 (Google-BARD), were employed to determine the recommended interval for the subsequent surveillance exam following the 2020 multi-society task force guidelines for colorectal cancer (CRC). Patient age, sex, family history of CRC, colonoscopy findings, and pathology reports, were inputted into the respective LLM. Model-generated recommendations were then compared to those provided by a human expert. Inaccurate responses prompted further input until the correct answer was reached. The most efficacious model was additionally tested using an engineered prompt that necessitated quoting guidelines for each advisory.

Results: GPT 3.5 achieved a 26% success rate, mandating 135 additional total prompts to attain 100% accuracy. Conversely, GPT 4 displayed 72% accuracy, requiring 44 extra prompts to reach 100%. PaLM-1 demonstrated a 27% accuracy rate, needing 109 supplementary prompts to achieve 100%, while PaLM-2 garnered a 36% accuracy rate with 105 additional prompts to reach 100%. Employing the engineered prompt, GPT 4 accomplished an 89% accuracy rate.

Discussion: Our findings indicate that LLMs, particularly GPT 4, possess substantial potential in performing specific clinical tasks, boasting a 72% success rate in a zero-shot approach (single direct prompt) for determining the recommended timeframe for surveillance colonoscopy. The efficacy of these models can be optimized through prompt-engineering, resulting in an enhanced accuracy rate of 89%. Furthermore, both advanced versions of GPT and PaLM surpassed their predecessors. As these models continue to evolve, we anticipate that combining future iterations, engineered prompts, and fine-tuning based on clinical guidelines will considerably elevate overall performance to clinically acceptable standards.


Disclosures:


Mohammed S. Al-Zakwani, MD, MS¹, Shuhaib Ali, MD¹, Pir Shah, MD¹, Juan Echavarria, MD², Omer Shahab, MD³, Muhammad Haris, MD¹, Bara El Kurdi, MD¹. P1769 - Navigating the Future: Evaluating Artificial Intelligence Recommendations for Colonoscopy Screening Against Standardized Guidelines, ACG 2023 Annual Scientific Meeting Abstracts. Vancouver, BC, Canada: American College of Gastroenterology.