Artificial Intelligence in Emergency Response: Which Model Offers More Reliable Solutions?

Abstract

Background: The increasing integration of artificial intelligence (AI) and large language models (LLMs) into medical education highlights the need to evaluate their performance in specialized fields like emergency medicine. This study addresses a gap in the literature by comparing the performance of ChatGPT and GEMINI models in a clinical context.

Objective: This study aimed to compare the readability and interobserver consistency of texts generated by Open AI (GPT) and Google GEMINI in response to emergency medicine cardiovascular questions.

Materials and Methods: Text samples were generated by both models (n=20 per model) in response to a predefined set of questions. The interobserver consistency was analyzed by two independent observers using an Intraclass Correlation Coefficient (ICC). Readability was assessed using multiple quantitative formulas, including Flesch Reading Ease, Fog Scale, and Flesch-Kincaid Grade Level, and the results were statistically compared.

Results: The interobserver correlation for the GEMINI group (ICC = 0.636) was notably higher than for the GPT group (ICC = 0.048), indicating better consistency in GEMINI's output, although neither correlation was statistically significant. Readability analysis revealed that GEMINI texts were significantly more readable than GPT texts across most formulas (e.g., Flesch Reading Ease: GEMINI M=57.00 vs. GPT M=47.00, p<0.001). This suggests GEMINI's output is characterized by simpler sentence structures and more accessible language.

Conclusion: The findings suggest that the GEMINI model outperforms GPT in both interobserver consistency and text readability. These results have important implications for the selection of AI tools in medical education and clinical practice, where the clarity and reliability of information are critical.