Test of ‘poisoned dataset’ shows vulnerability of LLMs to medical misinformation

By conducting checks below an experimental situation, a workforce of medical researchers and AI specialists at NYU Langone Well being has demonstrated how simple it’s to taint the information pool used to coach LLMs.

For his or her examine revealed within the journal Nature Drugs, the group generated 1000’s of articles containing misinformation and inserted them into an AI coaching dataset and carried out basic LLM queries to see how usually the misinformation appeared.

Prior analysis and anecdotal proof have proven that the solutions given by LLMs comparable to ChatGPT will not be all the time appropriate and, the truth is, are generally wildly off-base. Prior analysis has additionally proven that misinformation planted deliberately on well-known web websites can present up in generalized chatbot queries. On this new examine, the analysis workforce wished to understand how simple or tough it could be for malignant actors to poison LLM responses.

To seek out out, the researchers used ChatGPT to generate 150,000 medical paperwork containing incorrect, outdated and unfaithful information. They then added these generated paperwork to a check model of an AI medical coaching dataset. They then skilled a number of LLMs utilizing the check model of the coaching dataset. Lastly, they requested the LLMs to generate solutions to five,400 medical queries, which had been then reviewed by human specialists seeking to spot examples of tainted information.

The analysis workforce discovered that after changing simply 0.5% of the information within the coaching dataset with tainted paperwork, all of the check fashions generated extra medically inaccurate solutions than they’d previous to coaching on the compromised dataset. As one instance, they discovered that each one the LLMs reported that the effectiveness of COVID-19 vaccines has not been confirmed. Most of them additionally misidentified the aim of a number of frequent medicines.

The workforce additionally discovered that lowering the variety of tainted paperwork within the check dataset to only 0.01% nonetheless resulted in 10% of the solutions given by the LLMs containing incorrect information (and dropping it to 0.001% nonetheless led to 7% % of the solutions being incorrect), suggesting that it requires only some such paperwork posted on web sites in the true world to skew the solutions given by LLMs.

The workforce adopted up by writing an algorithm capable of determine medical information in LLMs after which used cross-referencing to validate the information, however they word that there isn’t any real looking solution to detect and take away misinformation from public datasets.

© 2025 Science X Community