Analysis of the capability of artificial intelligence to construct patient information for screening mammography

Research output: Other contribution to conferenceAbstractpeer-review

Abstract

Objectives
ChatGPT3.5 was released publicly in late November 2022 and has emerged as a convenient source of patient information, including for screening mammography. Offering enhanced capability, GPT4 was released as a paid user subscription in March 2023. The purported benefits of ChatGPT for supporting patient education and information need to be objectively evaluated.
Methods
ChatGPT3.5 and GPT4 were used to generate breast screening patient information sheets, which were independently evaluated using the Patient Education Materials Assessment Tool for printed materials ¹ (PEMAT-P) and the CDC Clear Communication Index ² (CDC Index) Score Sheet together with a gold standard BreastScreen NSW exemplar. The mean scores of both tools were reported for comparison.
Results
The BreastScreen NSW information sheet scored highly across all evaluations: PEMAT-P understandability 94% SD=3.46; actionability 100% SD=0; CDC Index 100% SD=0. GPT4 was deemed superior to GPT3.5 but included several key omissions: PEMAT-P understandability 75.0% SD=17 and actionability 53.3% SD=11.54; CDC Index 66.0% SD=4.1. GPT3.5 provided appropriate tone and currency of information but lacked accuracy, omitting key insights: PEMAT-P understandability 68.0% SD=6.56 and actionability 36.7% SD=20.4; CDC Index 58.8% SD=15.3.
Discussion/Conclusion
Both ChatGPT versions exhibited poor understandability and actionability and were unclear in their messaging. ChatGPT is an unreliable and inaccurate source of information concerning breast screening that may undermine screening participation and the informed consent process. Those with poor health literacy will not benefit from accessing the current versions of ChatGPT. ChatGPT may increase the demands on healthcare staff, including radiographers, to rectify educational gaps and misinformation.
Original languageEnglish
Publication statusAccepted/In press - 2024
EventASMIRT 2024: Colours of Country - Darwin Convention Centre, Darwin, Australia
Duration: 09 May 202412 May 2024
https://conference.asmirt.org/2024/

Conference

ConferenceASMIRT 2024
Abbreviated titleCommunity Resilience Innovation
Country/TerritoryAustralia
CityDarwin
Period09/05/2412/05/24
Internet address

Fingerprint

Dive into the research topics of 'Analysis of the capability of artificial intelligence to construct patient information for screening mammography'. Together they form a unique fingerprint.

Cite this