Back

OpenAI releases CSAM image classifier and ChatGPT warnings
SiTech AI Team3 min read

OpenAI releases CSAM image classifier and ChatGPT warnings

OpenAI announced new tools to combat child sexual abuse material, including an image classifier, ChatGPT warnings and temporary blocks, developer guidelines, and law enforcement features.

OpenAI announced new tools to combat child sexual abuse material (CSAM), amid growing safety concerns around the use of AI tools. The features include a new image classifier to detect CSAM, warnings and temporary blocks in ChatGPT, a new guideline for developers, and tools for law enforcement. The company said the classifier can detect CSAM images, including ones generated by AI, and is being shared with select companies through its API at no cost.

New image classifier

OpenAI unveiled an image classifier that looks for visual signs of abuse in images not seen before and assigns a score that can give an indicative direction to reviewers. The company said it developed the classifier with the National Center for Missing and Exploited Children (NCMEC), and is working with Tech Coalition and ROOST to share access to the tool. In early testing with partners, the classifier flagged 48% fewer images while finding the same number of abusive images, reducing the work of reviewers. Images flagged by the classifier were twice as likely to be abusive images.

ChatGPT warnings and blocks

OpenAI is adding warning screens in ChatGPT to prevent users from accessing material that sexualizes children. ChatGPT will show an alert to users attempting such prompts, direct users toward specialized support to stop harmful behaviour, and suspend the account for 24 hours as a warning. The app already blocks CSAM content and related requests, and repeated offences could lead to the account being removed permanently. OpenAI said it formed the message and guidelines with organizations and researchers, including Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of deterrent messages.

Developer and law enforcement tools

OpenAI is publishing guidelines for developers using its tools to prevent, detect, and help users report CSAM content. In early testing, the guidelines helped a number of developers adopt child safety tooling and register to report CSAM to NCMEC. The company is also equipping law enforcement with AI-powered features such as dedicated plugins, investigative tools, and complex data analysis, with access managed by program administrators. It is also providing training to investigators along with a field guide on how to use AI.

Background and recent changes

OpenAI has faced several lawsuits blaming its technology for harming users, especially children. Safety-focused nonprofit Common Sense Media labeled OpenAI's tech as "unacceptable risk" for teens. Other AI chatbots, specifically Grok, have been used to create nonconsensual sexualized imagery, including of children. OpenAI said it has made changes in the last 12 months, including releasing a safety blueprint, updating content guidelines to bar inappropriate content, retuning its model to prevent it from generating harmful content, and releasing a special version of ChatGPT for teens.

Sources: TechCrunch

SiTech

SiTech — AI-powered web development

We build fast, modern websites and bring AI into real business workflows. Have a project or a question? We'd love to help.