Readability Analysis of AI-Generated Medical Content: Comparison of ChatGPT and Gemini in Diverticulitis
DOI:
https://doi.org/10.66288/actamedi.2026.80Keywords:
Diverticulitis, Readability, Large Language Models, ChatGPT, GeminiAbstract
Background: Diverticulitis is a common gastrointestinal condition for which patients increasingly seek information. Large language models (LLMs), such as ChatGPT and Gemini, are widely used to generate medical texts. However, the readability of these outputs remains a critical concern for effective patient education.
Objective: This study aimed to evaluate and compare the readability of responses generated by ChatGPT and Gemini for commonly asked questions about diverticulitis.
Methods: A set of standardized diverticulitis-related questions reflecting common patient concerns was developed and submitted to both models for evaluation. The generated responses were analyzed using multiple validated readability indices, including the Automated Readability Index (ARI), Flesch Reading Ease (FRE), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman–Liau Index (CLI), SMOG Index, and Linsear Write Formula. An aggregate measure, the Average Reading Level Consensus (ARLC), was also calculated. Statistical comparisons were performed, with p < 0.05 considered significant.
Results: Both ChatGPT and Gemini produced responses with readability levels exceeding the recommended standards for patient education. The ARLC was higher for ChatGPT than Gemini (11.45 ± 3.1 vs. 10.5 ± 1.6; p = 0.322), although the difference was not statistically significant. Gemini demonstrated significantly better readability based on the FRE score (32.4 ± 10.2 vs. 28.22 ± 10.4; p = 0.027). No significant differences were observed in ARI, GFI, FKGL, CLI, SMOG, or the original Linsear Write Formula (p > 0.05). However, the Linsear Write Grade Level was significantly higher for ChatGPT (20.01 ± 8.9 vs. 7.8 ± 2.1, p < 0.001).
Conclusions: Both LLMs generated diverticulitis-related content at readability levels above the recommended thresholds, potentially limiting patient comprehension. Gemini generally produced slightly more accessible text than ChatGPT. These findings highlight the need to optimize AI-generated medical content to better align with health literacy standards and improve patient understanding.
References
1. McNaught AD, Wilkinson A. IUPAC Compendium of Chemical Terminology (the “Gold Book”). 2nd ed. Oxford: Blackwell Scientific Publications; 1997.
2. Flesch R. A new readability yardstick. J Appl Psychol. 1948;32(3):221–233. DOI: https://doi.org/10.1037/h0057532
3. Kincaid JP, Fishburne RP Jr, Rogers RL, Chissom BS. Derivation of new readability formulas for Navy enlisted personnel. Res Branch Rep. 1975;8–75.
4. Smith EA, Senter RJ. Automated Readability Index. Aerospace Medical Research Laboratories Report. 1967.
5. Coleman M, Liau TL. A computer readability formula designed for machine scoring. J Appl Psychol. 1975;60(2):283–284. DOI: https://doi.org/10.1037/h0076540
6. McLaughlin GH. SMOG grading: a new readability formula. J Read. 1969;12(8):639–646.
7. Gunning R. The Technique of Clear Writing. New York: McGraw-Hill; 1952.
8. Weiss BD. Health literacy and patient safety: Help patients understand. AMA Foundation. 2007.
9. Kutner M, Greenberg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults. Washington, DC: National Center for Education Statistics; 2006.
10. Baddeley A, Hitch G. Working memory. In: Bower GH, editor. The Psychology of Learning and Motivation. New York: Academic Press; 1974. DOI: https://doi.org/10.1016/S0079-7421(08)60452-1
11. Strate LL, Morris AM. Epidemiology, pathophysiology, and treatment of diverticulitis. Gastroenterology. 2019;156(5):1282–1298. DOI: https://doi.org/10.1053/j.gastro.2018.12.033
12. Sartelli M, et al. 2020 update of the WSES guidelines for the management of acute colonic diverticulitis. World J Emerg Surg. 2020;15:32. DOI: https://doi.org/10.1186/s13017-020-00318-z
13. Regenbogen SE, Hardiman KM. Surgery for diverticulitis in the 21st century: a systematic review. JAMA Surg. 2014;149(3):292–303. DOI: https://doi.org/10.1001/jamasurg.2013.5477
14. van de Wall BJM, et al. The value of CRP in the diagnosis of acute diverticulitis. Int J Colorectal Dis. 2013;28(6):775–781.
15. Weizman AV, Nguyen GC. Diverticular disease: epidemiology and management. Can J Gastroenterol. 2011;25:385–389. doi: 10.1155/2011/795241. - DOI - PMC - PubMed DOI: https://doi.org/10.1155/2011/795241
16. Schoetz DJ. Diverticular disease of the colon: a century-old problem. Dis Colon Rectum. 1999;42:703–709. doi: 10.1007/BF02236921. - DOI - PubMed DOI: https://doi.org/10.1007/BF02236921
17. Shahedi K, Fuller G, Bolus R, Cohen E, Vu M, Shah R, et al. Long-term risk of acute diverticulitis among patients with incidental diverticulosis found during colonoscopy. Clin Gastroenterol Hepatol. 2013;11:1609–1613. doi: 10.1016/j.cgh.2013.06.020. - DOI - PMC - PubMed DOI: https://doi.org/10.1016/j.cgh.2013.06.020
18. Collins D, Winter DC. Modern concepts in diverticular disease. J Clin Gastroenterol. 2015;49:358–369. doi: 10.1097/MCG.0000000000000308. - DOI - PubMe DOI: https://doi.org/10.1097/MCG.0000000000000308
19. Warner E, Crighton EJ, Moineddin R, Mamdani M, Upshur R. Fourteen-year study of hospital admissions for diverticular disease in Ontario. Can J Gastroenterol. 2007;21:97–99. doi: 10.1155/2007/943460. - DOI: https://doi.org/10.1155/2007/943460
20. Etzioni DA, Mack TM, Beart RW Jr, Kaiser AM. Diverticulitis in the United States: 1998-2005: changing patterns of disease and treatment. Ann Surg. 2009;249(2):210-217 DOI: https://doi.org/10.1097/SLA.0b013e3181952888
21. Ricciardi R, Baxter NN, Read TE, Marcello PW, Hall J, Roberts PL. Is the decline in the surgical treatment for diverticulitis associated with an increase in complicated diverticulitis? Dis Colon Rectum. 2009;52(9):1558-1563 DOI: https://doi.org/10.1007/DCR.0b013e3181a90a5b
22. Centers for Disease Control and Prevention; National Center for Health Statistics. CDC/NCHS National Hospital Discharge Survey, 2010. http://www.cdc.gov/nchs/. Accessed April 5, 2013.
23. Yen L, Davis KL, Hodgkins P, Loftus EV Jr, Erder MH. Direct costs of diverticulitis in a US managed care population. Am J Pharm Benefits. 2012;4(5):e118-e129.Google Scholar
24. Ambrosetti P. Value of CT for acute left-colonic diverticulitis: the surgeon’s view. Dig Dis. 2012;30(1):51-55. DOI: https://doi.org/10.1159/000335717
25. Ambrosetti P, Becker C, Terrier F. Colonic diverticulitis: impact of imaging on surgical management: a prospective study of 542 patients. Eur Radiol. 2002;12(5):1145-1149. DOI: https://doi.org/10.1007/s00330-001-1143-y
26. Janes S, Meagher A, Frizelle FA. Elective surgery after acute diverticulitis. Br J Surg. 2005;92(2):133-142.PubMedGoogle ScholarCrossref DOI: https://doi.org/10.1002/bjs.4873
27. Rafferty J, Shellito P, Hyman NH, Buie WD; Standards Committee of American Society of Colon and Rectal Surgeons. Practice parameters for sigmoid diverticulitis. Dis Colon Rectum. 2006;49(7):939-944. DOI: https://doi.org/10.1007/s10350-006-0578-2
28. Etzioni DA, Cannom RR, Ault GT, Beart RW Jr, Kaiser AM. Diverticulitis in California from 1995 to 2006: increased rates of treatment for younger patients. Am Surg. 2009;75(10):981-985. DOI: https://doi.org/10.1177/000313480907501026
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Remziye Nur Okudan
This work is licensed under a Creative Commons Attribution 4.0 International License.
