Warning: This article contains details that some readers may find distressing.
ChatGPT can spontaneously generate sexually explicit and deeply violent images from prompts that did not request such material to be produced, research has found.
A British AI security startup says that the OpenAI chatbot was capable of producing “truly disturbing” imagery that included scenes of death, sexual violence, blood, and murder.
Some of the images it generated included a woman bludgeoned dead and bleeding from the genitals, a half-naked college student bound and gagged in a basement, and a deceased woman lying on a pavement with her organs exposed and wrists slit open.
A researcher from Mindgard was able to generate the content within a matter of seconds after slightly tweaking a “fun, viral prompt” that was essentially requesting a random image to be produced.
Peter Garraghan, the company’s founder and professor at Lancaster University, told The Independent: “[ChatGPT] could have picked any topic to make images about it. It went to topics that directly misaligned with safety. That’s why it’s so problematic.”
He said the researcher assigned to the task was “incredibly shaken” and had to take time off work.
The company reported the images to OpenAI, but did not receive a response. Its founder, Peter Garraghan, said it only heard back after a BBC journalist approached OpenAI about the story. The AI company is looking into why it did not respond.
OpenAI told The Independent it has since introduced further safeguards, which prevent such prompts from being used.
An OpenAI spokesperson said: “We take these reports seriously. After investigating this trend, we’ve introduced additional safeguards against this type of prompt.”
It added that it was continuing to monitor and deploy additional mitigations to prevent others from being able to produce the content.
However, Mr Garraghan said their researchers were still able to find similar material by tweaking the prompts in another way.
As per OpenAI’s guidelines, the chatbot should not be able to generate sexual violence. If such material had been directly requested, it would not have been generated, according to Mindgard.
An OpenAI spokesperson added: “Our safety systems are designed to block potentially harmful images that are uploaded to ChatGPT, and we analyse whether the AI-generated image violates our policies before we show the image to the user. We also combine automated systems and human review to identify and block harmful material.”
Mr Garraghan said images being generated by ChatGPT should be checked by the system before being sent to someone.
Durham University law professor Clare McGlynn told The Independent: “It’s unfortunately not surprising, but the nature of the images is still deeply shocking.
“We’re living in this world in which the internet is awash with sexually violent and misogynistic material, hence why something like ChatGPT produces it,” the leading expert on image-based sexual abuse added.
“[OpenAI] says that they’ve got guardrails in place, and that they’re now working to take this down, but what this shows me is that their guardrails are not sufficient.
“They’re simply not spending enough time and resources to ensure that their model, which has nearly a billion weekly users, can’t generate this. They’re simply failing in their ethical obligations to do that.”
