Chatbots Show Ability to Accurately Diagnose, Triage Ophthalmic Conditions

Chatbots shows promise for diagnosing ophthalmic conditions and accurately triaging patients without providing grossly inaccurate statements, according to a study published in the Canadian Journal of Ophthalmology. The software demonstrated similar performance to ophthalmology trainees in its triaging and diagnostic ability, the report shows.

Researchers developed 44 vignettes describing common ophthalmic conditions and presented them to the ophthalmologists in training (n=8) and 2 chatbots. The team also entered symptoms into a free online tool that provides a differential diagnosis. The primary outcome was a response with the correct diagnosis listed in the top 3 possible diagnoses and correct determination of the condition’s urgency. Secondary outcomes included grossly inaccurate statements, mean reading grade level, attributions, and most common sources cited.  

Overall, the correct diagnosis was identified in 42 (95%), 41 (93%), 34 (77%), and 8 (33%) cases for the ophthalmologists in training, 2 chatbots, and online diagnostic tool, respectively. Acceptable triage urgency was determined in 86%, 98%, and 84% of cases for the ophthalmology trainees and 2 chatbots, respectively. However, both artificial intelligence (AI) conversation entities showed a greater tendency to overstate urgency for non urgent cases compared with the ophthalmology trainees (26% and 61% vs 9%), the report shows.

The online diagnostic tool provided grossly inaccurate information in more cases compared with the chatbots (50% vs 14% and 0%). The 2 interactive agents varied in their frequency for providing disclaimers stating their limitations — 1 provided disclaimers in 100% of cases vs 5% of cases for the other chatbot.

“[A]lthough there are potential shortcomings to AI-based medical triage, these readily accessible tools may address existing flaws within health care systems,” according to the researchers. “Ophthalmologists should be prepared for a new paradigm in health care delivery as the lay public turns to AI chatbots to address personal health needs.”