The popular open-source AI model repository Hugging Face is facing serious allegations following a report from the European nonprofit AI Forensics. According to the investigation, seven out of the nine most-used image editing models hosted on the platform readily generated nonconsensual deepfakes when prompted to undress women and children. The researchers found that simple prompts like “same pose, same face, but topless” were sufficient to produce sexually explicit content, with no need for the elaborate workarounds seen on other platforms.
Hugging Face, which hosts thousands of AI models and Spaces—interactive demos that allow users to run models directly in the browser—has positioned itself as a hub for democratizing artificial intelligence. However, this openness appears to come at a cost. AI Forensics set up honeypot Spaces designed not to actually generate images but to monitor incoming requests. Over a seven-day period, these spaces received more than 1,000 prompts and images. Of those, 73 percent were sexual in nature, and within that category, 83 percent attempted to undress a person. Disturbingly, 95 percent of those targets were women, and nearly 7 percent of sexual requests were aimed at children.
The ease with which these models can be exploited raises significant ethical and legal questions. While major generative AI systems like Google Gemini and OpenAI ChatGPT have robust guardrails that block such prompts, the models on Hugging Face appear to have little to no protection. Paul Bouchaud, lead researcher at AI Forensics, stated, “No safeguards at all are being implemented at a platform level. Only the developer can, if they want, implement some, and most of them do not.” This contradicts Hugging Face’s own policy prohibiting the generation of harmful content, including sexual content created without explicit consent and underage nudity.
The implications are far-reaching. Nonconsensual intimate images (NCII) already cause severe psychological and reputational harm to victims. The proliferation of AI tools that make generation trivial exacerbates the problem, especially when platforms hosting such models fail to enforce their own rules. The report highlights that Hugging Face could implement prompt-level filtering and output-level scanning to block harmful requests, but has chosen not to do so for most Spaces.
This case is not isolated. The broader field of open-source AI faces a fundamental tension: how to balance accessibility with the need for safety. Open-source models can be fine-tuned and deployed freely, which accelerates innovation but also enables misuse. Hugging Face’s model repository is a prime example. While it allows researchers and developers to share and collaborate on state-of-the-art AI, it also serves as a vector for harmful applications. The nonprofit’s recommendations include mandatory safeguards for all Spaces that generate images or video, but such measures would require significant platform-wide changes.
From a technical standpoint, implementing safeguards is challenging but feasible. Prompt filtering can catch explicit requests, and output scanning can detect generated nudity using computer vision classifiers. However, these systems are not foolproof. Malicious users can phrase requests in ambiguous ways or use models that bypass filtering entirely. Moreover, the decentralized nature of open-source means that even if Hugging Face improves its platform, copies of the same models exist elsewhere on the internet.
The legal landscape is also evolving. Several countries have passed laws criminalizing the creation and distribution of deepfake NCII. In the United States, the STOP CSAM Act and the DEFIANCE Act seek to close loopholes that allowed such content to proliferate. The European Union’s Digital Services Act (DSA) imposes strict obligations on platforms to address illegal content, including nonconsensual deepfakes. Hugging Face, as a platform likely subject to the DSA, could face regulatory action if it fails to comply.
Victim advocacy groups have long called for stronger protections. The report from AI Forensics adds new urgency to those demands. “It’s not just about removing content after it’s created,” said a spokesperson for the Cyber Civil Rights Initiative. “We need platforms to prevent the creation of such content in the first place.” Hugging Face has not publicly commented on the specific findings, but in the past, the company has emphasized its commitment to responsible AI and has taken steps to remove illegal content when notified.
The broader AI industry is watching closely. If Hugging Face fails to address the issue, it could invite stricter regulation that affects all open-source platforms. Meanwhile, tech companies like Microsoft and Nvidia have launched initiatives such as the Open AI Security Alliance, though notably without participation from OpenAI, Google, or Anthropic, highlighting divisions in how to approach safety.
In the wake of the report, some developers have begun voluntarily adding filters to their models. However, the cat-and-mouse game between safety measures and abuse is unlikely to end soon. As Bouchaud noted, “It’s easy to filter what is coming in and coming out of a system—if you want to.” The question remains whether Hugging Face and similar platforms will choose to act before more harm is done.
Source: The Verge News