POST-DISCHARGE ADVICE AFTER SPONTANEOUS PNEUMOTHORAX FROM FREELY ACCESSIBLE CONSUMER AI CHATBOTS: A SCENARIO-BASED STUDY OF GUIDELINE CONCORDANCE AND CLINICAL SAFETY

Abstract

Aim: Large language models (LLMs) are increasingly accessible to patients seeking health-related information, including practical advice after treatment for spontaneous pneumothorax. The clinical usefulness of such responses depends not only on overall guideline concordance but also on the avoidance of potentially unsafe recommendations. In the present study, the evaluated advice reflected the models assigned by default through each provider’s free consumer interface rather than investigator-selected model configurations.

Methodology: Thirty standardized synthetic scenarios representing routine discharge, lifestyle and travel, recurrence and high-risk situations, secondary spontaneous pneumothorax, and post-discharge emergencies were evaluated using three freely accessible consumer-facing LLMs: ChatGPT (GPT-5.6 Luna), Claude (Sonnet 5 Medium), and Gemini (Flash Lite 3.5). All 90 queries were submitted on a single day (30 August 2026) from Antalya, Türkiye. Responses were assessed against a prespecified scenario-specific reference framework based on contemporary international pneumothorax guidelines. Applicable counselling domains were scored from 0 to 2, and a Guideline Concordance Score (GCS) was calculated for each response. Factual errors, unsupported claims, and clinically significant errors (CSEs) were assessed separately.

Results: Ninety responses were evaluated. Mean GCS was 89.0% for Claude, 88.5% for ChatGPT, and 72.7% for Gemini. Overall differences among the models were significant (Friedman statistic=31.19, P<0.001). ChatGPT and Claude did not differ significantly (Holm-adjusted P=0.736), whereas both achieved significantly higher GCS values than Gemini (both P<0.001). All three models achieved a mean and median GCS of 100% in emergency safety scenarios. Factual errors occurred in 0, 7, and 4 responses and unsupported claims in 1, 19, and 10 responses for ChatGPT, Claude, and Gemini, respectively. One CSE was identified, occurring in a Gemini air-travel response.

Conclusion: Post-discharge advice obtained from freely accessible consumer chatbot interfaces was heterogeneous in its concordance with the prespecified pneumothorax guideline framework. All interfaces reliably identified overt emergencies, but they differed substantially in factual accuracy, unsupported recommendations, and safety-sensitive counselling such as air travel. Because these differences arose between interfaces that appear equivalent to the user and depend on provider-assigned default configurations rather than matched model selection, the quality of advice may vary at the point of use. Clinicians should anticipate that patients may act on such advice, and should address safety-critical items explicitly at discharge rather than assume they will be covered.