Analysis of the capability of artificial intelligence to construct patient information for screening mammography

Objectives
ChatGPT3.5 was released publicly in late November 2022 and has emerged as a convenient source of patient information, including for screening mammography. Offering enhanced capability, GPT4 was released as a paid user subscription in March 2023. The purported benefits of ChatGPT for supporting patient education and information need to be objectively evaluated.
Methods
ChatGPT3.5 and GPT4 were used to generate breast screening patient information sheets, which were independently evaluated using the Patient Education Materials Assessment Tool for printed materials ¹ (PEMAT-P) and the CDC Clear Communication Index ² (CDC Index) Score Sheet together with a gold standard BreastScreen NSW exemplar. The mean scores of both tools were reported for comparison.
Results
The BreastScreen NSW information sheet scored highly across all evaluations: PEMAT-P understandability 94% SD=3.46; actionability 100% SD=0; CDC Index 100% SD=0. GPT4 was deemed superior to GPT3.5 but included several key omissions: PEMAT-P understandability 75.0% SD=17 and actionability 53.3% SD=11.54; CDC Index 66.0% SD=4.1. GPT3.5 provided appropriate tone and currency of information but lacked accuracy, omitting key insights: PEMAT-P understandability 68.0% SD=6.56 and actionability 36.7% SD=20.4; CDC Index 58.8% SD=15.3.
Discussion/Conclusion
Both ChatGPT versions exhibited poor understandability and actionability and were unclear in their messaging. ChatGPT is an unreliable and inaccurate source of information concerning breast screening that may undermine screening participation and the informed consent process. Those with poor health literacy will not benefit from accessing the current versions of ChatGPT. ChatGPT may increase the demands on healthcare staff, including radiographers, to rectify educational gaps and misinformation.