Comparison of ChatGPT Models in Patient Education on Obstructive Sleep Apnea

Abstract

The objective of this study is to evaluate and compare the accuracy, comprehensiveness, and readability of responses to common patient questions regarding obstructive sleep apnea syndrome provided by ChatGPT-3.5 and ChatGPT-4. With the increasing use of artificial ıntelligence–powered tools for patient education, understanding the reliability of these models is crucial. Fifty potential patient questions were generated using guidelines from the American Academy of Sleep Medicine and the American Thoracic Society. These questions were presented to both ChatGPT-3.5 and ChatGPT-4 twice, with a 45-day interval between evaluations. The responses were rated by five ENT specialists and three residents. The responses were graded for accuracy using a 4-point scale (1 = comprehensive and correct, 4 = completely incorrect) and assessed for readability using the Flesch‐Kincaid Grade Level and Flesch Reading Ease scores. ChatGPT-4 responses were found to be more accurate and comprehensive compared to ChatGPT-3.5, with 88% of ChatGPT-4 responses rated as comprehensive and accurate versus 79% for ChatGPT-3.5. However, both models produced responses that required a university-level reading proficiency, with no significant difference in readability between ChatGPT-3.5 and ChatGPT-4. ChatGPT-4 demonstrated improved accuracy over ChatGPT-3.5 in generating responses to obstructive sleep apnea syndrome–related patient questions. However, both models’ responses were difficult to read for the general population.