Evaluation of Readability Indices Of Chatgpt-4 and Google Gemini in Cervical Disc Herniation: Evaluation of Readability Indices Of Chatgpt-4 and Google Gemini in Cervical Disc Herniation

Umut Ogün Mutlucan

doi:10.5281/zenodo.17156411

Authors

Umut Ogün Mutlucan Antalya Research and Training Hospital https://orcid.org/0000-0002-5052-7244

DOI:

https://doi.org/10.5281/zenodo.17156411

Keywords:

Artificial intelligence, Cervical disc herniation, Readability

Abstract

Background: Cervical disc herniation (CDH) is a condition that can result in severe symptoms by exerting pressure on the nerve root or spinal cord. A multitude of prior studies have demonstrated that, despite ChatGPT's high accuracy rate, its readability level has been classified as "very difficult," which may impede patients' comprehension. The objective of this study is to assess the readability levels of texts generated by ChatGPT-5 and Gemini-2.5 Pro in response to inquiries posed by patients with CDH.

Methods: A compendium of frequently asked questions was meticulously curated from a plethora of online forums and medical websites. A total of twenty text samples were obtained from both ChatGPT-5 and Gemini-2.5 Pro models. The readability levels of these texts were measured using nine standard formulas, including Flesch Reading Ease, Flesch–Kincaid Grade Level, Gunning Fog Index, and SMOG Index.

Results: The analysis indicates that texts generated by Gemini exhibit higher readability scores (i.e., are more easily understood) than those generated by OpenAI in most readability assessments, with the exception of the Flesch Reading Ease score. The disparities in the Average Reading Level Consensus, ARI, Flesch Reading Ease, Fesch-Kincaid Grade Level, Coleman-Liau Index, and Forecast Readability Formula outcomes were found to be statistically significant. However, a statistical analysis revealed that the differences between the SMOG index and the Linsear Write Formula values were not statistically significant.

Conculusion: In the majority of readability assessments, the text generated by Gemini was determined to be statistically less challenging in comparison to that of OpenAI.

References

1. Margetis K, Al Khalili Y. Cervical Disc Herniation. 2025 Aug 2. In: StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing; 2025 Jan–. PMID: 31536225.

2. Daentzer D, Welke B, Baseem Ismail A, Daentzer J, Plagge J. Zervikaler Bandscheibenvorfall : Symptomatik, Diagnostik, Therapie [Cervical disc herniation : Symptomatology, diagnostics, therapy]. Orthopadie (Heidelb). 2025 Jan;54(1):26-39. German. doi: 10.1007/s00132-024-04582-8. Epub 2024 Nov 19. PMID: 39560707.

3. Dursun Ö, Dincer E, Reyhanlı G. Scapular asymmetry in patients with cervical disc herniation. J Back Musculoskelet Rehabil. 2025 Jul;38(4):750-758. doi: 10.1177/10538127241312781. Epub 2025 Feb 12. PMID: 39973253.

4. Wang S, Wang Y, Jiang L, Chang Y, Zhang S, Zhao K, Chen L, Gao C. Assessing the clinical support capabilities of ChatGPT 4o and ChatGPT 4o mini in managing lumbar disc herniation. Eur J Med Res. 2025 Jan 22;30(1):45. doi: 10.1186/s40001-025-02296-x. PMID: 39844276; PMCID: PMC11753088.

5. Yang S, Chang MC. The assessment of the validity, safety, and utility of ChatGPT for patients with herniated lumbar disc: A preliminary study. Medicine (Baltimore). 2024 Jun 7;103(23):e38445. doi: 10.1097/MD.0000000000038445. PMID: 38847711; PMCID: PMC11155576.

6. Wang S, Wang Y, Jiang L, Chang Y, Zhang S, Zhao K, Chen L, Gao C. Assessing the clinical support capabilities of ChatGPT 4o and ChatGPT 4o mini in managing lumbar disc herniation. Eur J Med Res. 2025 Jan 22;30(1):45. doi: 10.1186/s40001-025-02296-x. PMID: 39844276; PMCID: PMC11753088.

7. Liu X, Shi S, Zhang X, Gao Q, Wang W. The role of ChatGPT-4o in differential diagnosis and management of vertigo-related disorders. Sci Rep. 2025 May 28;15(1):18688. doi: 10.1038/s41598-025-96309-8. PMID: 40437044; PMCID: PMC12119837.

8. Scaff SPS, Reis FJJ, Ferreira GE, Jacob MF, Saragiotto BT. Assessing the performance of AI chatbots in answering patients' common questions about low back pain. Ann Rheum Dis. 2025 Jan;84(1):143-149. doi: 10.1136/ard-2024-226202. Epub 2025 Jan 2. PMID: 39874229.

9. Ali A, Kumar RP, Polavarapu H, Lavadi RS, Mahavadi A, Legarreta AD, Hudson JS, Shah M, Paul D, Mooney J, Dietz N, Fields DP, Hamilton DK, Agarwal N. Bridging the Gap: Can Large Language Models Match Human Expertise in Writing Neurosurgical Operative Notes? World Neurosurg. 2024 Dec;192:e34-e41. doi: 10.1016/j.wneu.2024.08.062. Epub 2024 Aug 15. PMID: 39153569.

10. Rodler S, Cei F, Ganjavi C, Checcucci E, De Backer P, Rivero Belenchon I, Taratkin M, Puliatti S, Veccia A, Piazza P, Baekelandt L, Kowalewski KF, Gómez Rivas J, Fankhauser CD, Moschini M, Gandaglia G, Campi R, De Castro Abreu A, Russo GI, Cocci A, Maruccia S, Cacciamani GE; YAU Collaborators. GPT-4 generates accurate and readable patient education materials aligned with current oncological guidelines: A randomized assessment. PLoS One. 2025 Jun 4;20(6):e0324175. doi: 10.1371/journal.pone.0324175. PMID: 40465696; PMCID: PMC12136319.

11. Yassa A, Ayad O, Cohen DA, Patel AM, Vengsarkar VA, Hegazin MS, Filimonov A, Hsueh WD, Eloy JA. Search for medical information for chronic rhinosinusitis through an artificial intelligence ChatBot. Laryngoscope Investig Otolaryngol. 2024 Sep 9;9(5):e70009. doi: 10.1002/lio2.70009. PMID: 39257728; PMCID: PMC11382362.

12. Ayad O, Yassa A, Patel AM, Vengsarkar VA, Ayad S, Ayad S, Mikhael M. Artificial intelligence in patient care: evaluating artificial intelligence's accuracy and accessibility in addressing blepharoplasty concerns. Int Ophthalmol. 2025 Jun 12;45(1):244. doi: 10.1007/s10792-025-03611-5. PMID: 40504405.

13. Nian PP, Saleet J, Magruder M, Wellington IJ, Choueka J, Houten JK, Saleh A, Razi AE, Ng MK. ChatGPT as a Source of Patient Information for Lumbar Spinal Fusion and Laminectomy: A Comparative Analysis Against Google Web Search. Clin Spine Surg. 2024 Dec 1;37(10):E394-E403. doi: 10.1097/BSD.0000000000001582. Epub 2024 Feb 20. PMID: 38409676.

14. Roberts H, Zhang D, Dyer GS. The Readability of AAOS Patient Education Materials: Evaluating the Progress Since 2008. J Bone Joint Surg Am. 2016 Sep 7;98(17):e70. doi: 10.2106/JBJS.15.00658. PMID: 27605695.

15. Huang G, Fang CH, Agarwal N, Bhagat N, Eloy JA, Langer PD. Assessment of online patient education materials from major ophthalmologic associations. JAMA Ophthalmol. 2015 Apr;133(4):449-54. doi: 10.1001/jamaophthalmol.2014.6104. PMID: 25654639.

16. Fu S, Zhang C, Yan X, Li D, Wang Y, Dong C, Cao Z, Ning Y, Shao C, Yang T. A New Automated AI-Assisted System to Assess Cervical Disc Herniation. Spine (Phila Pa 1976). 2022 Aug 15;47(16):E536-E544. doi: 10.1097/BRS.0000000000004376. Epub 2022 Jul 15. PMID: 35867605.

17. Fu S, Zhang C, Yan X, Li D, Wang Y, Dong C, Cao Z, Ning Y, Shao C, Yang T. Volumetric Changes in Cervical Disc Herniation: Comparison of Cervical Expansive Open-door Laminoplasty and Cervical Microendoscopic Laminoplasty. Spine (Phila Pa 1976). 2022 Apr 1;47(7):E296-E303. doi: 10.1097/BRS.0000000000004197. PMID: 34381000.

18. Fu S, Zhang C, Yan X, Li D, Wang Y, Dong C, Cao Z, Ning Y, Shao C, Yang T. A New Automated AI-Assisted System to Assess Cervical Disc Herniation. Spine (Phila Pa 1976). 2022 Aug 15;47(16):E536-E544. doi: 10.1097/BRS.0000000000004376. Epub 2022 Jul 15. PMID: 35867605.

19. He Y, He Z, Qiu Y, Liu Z, Huang A, Chen C, Bian J. Deep Learning for Lumbar Disc Herniation Diagnosis and Treatment Decision-Making Using Magnetic Resonance Imagings: A Retrospective Study. World Neurosurg. 2025 Mar;195:123728. doi: 10.1016/j.wneu.2025.123728. Epub 2025 Feb 26. PMID: 39880078.

20. Cui YM, Liu FG. [Artificial disc replacement for the treatment of cervical disc herniation]. Zhongguo Gu Shang. 2024 Nov 25;37(11):1051-5. Chinese. doi: 10.12200/j.issn.1003-0034.20230459. PMID: 39604334.

21. Tabla Cendra D, Gao TP, HoSang KM, Gao TJ, Wu J, Pronovost MT, Williams AD, Kuo LE. A Bilingual Readability Assessment of Online Breast Cancer Screening and Treatment Information. J Surg Res. 2024 Oct;302:200-207. doi: 10.1016/j.jss.2024.07.026. Epub 2024 Aug 3. PMID: 39098118.

22. Piñero-López MÁ, Modamio P, Lastra CF, Mariño EL. Readability Analysis of the Package Leaflets for Biological Medicines Available on the Internet Between 2007 and 2013: An Analytical Longitudinal Study. J Med Internet Res. 2016 May 25;18(5):e100. doi: 10.2196/jmir.5145. PMID: 27226241; PMCID: PMC4899622.

23. Piñero-López MÁ, Modamio P, Lastra CF, Mariño EL. Readability assessment of package inserts of biological medicinal products from the European medicines agency website. Drug Saf. 2014 Jul;37(7):543-54. doi: 10.1007/s40264-014-0188-1. PMID: 24942755.

24. Singh S, Jamal A, Qureshi F. Readability Metrics in Patient Education: Where Do We Innovate? Clin Pract. 2024 Nov 4;14(6):2341-2349. doi: 10.3390/clinpract14060183. PMID: 39585011; PMCID: PMC11586978.

Evaluation of Readability Indices Of Chatgpt-4 and Google Gemini in Cervical Disc Herniation