Readability Assessment of Large Language Model Responses to Common Postnatal Questions Among Migrant Parents: A Comparative Analysis of ChatGPT and Gemini

Abstract

Language and communication barriers in migrant communities can substantially hinder access to health information, particularly regarding frequently asked questions in the neonatal period. Large language models (LLMs) have emerged as potential tools to address this information gap; however, the readability of their outputs is critically important from a public health perspective. This study comparatively evaluated the readability of responses generated by ChatGPT and Google Gemini to ten core postnatal questions commonly posed by migrant parents to pediatricians. The ten questions were translated into English and repeatedly submitted to both models on different days and in varying orders. The resulting texts were analyzed using multiple readability tests, including the Automated Readability Index, Flesch Reading Ease, Flesch–Kincaid Grade Level, SMOG Index, and Linsear Write Formula. The findings indicate that Gemini systematically produced more complex texts requiring a higher literacy level in most indices. In contrast, ChatGPT yielded more easily understandable content on scales closely related to health literacy, such as Flesch Reading Ease and SMOG. Overall, both models show potential to support access to health information among migrant populations; however, the complexity of their outputs remains above recommended public health literacy levels, highlighting the need for further optimization and text simplification, particularly for groups with low health literacy.