Deepfakes—AI-generated images, audio, or videos that realistically portray individuals saying or doing things they never did, have moved beyond a niche technological development to become a growing social concern. While the technology itself is not new, its increasing misuse, particularly in the creation of non-consensual explicit content targeting women and girls, has raised serious ethical and legal challenges. In many countries, regulatory frameworks have yet to fully address this issue, leaving victims uncertain about their rights and the possibility of holding perpetrators accountable. In the case of South Korea, although its technological infrastructure supports innovation and connectivity, it also creates new avenues for exploitation. Combined with persistent gender norms, this digital environment has contributed to an increasingly unsafe online space for women and girls. A global analysis by the U.S.-based cybersecurity firm Security Hero found that more than half of all deepfake explicit content online features South Korean women, which highlights the scale and urgency of the issue, raising questions on where improvements can be made—legal design, enforcement, governance, or social norms.

Photo: Women in South Korea protesting regarding prevalence of deepfakes (Source: PBS News)

The Rapid Expansion and Normalization of Deepfake Abuse

Although the term “deepfake” feels relatively new, the underlying technology dates back to the mid-2010s, when researchers and online communities began using advanced machine learning techniques, such as generative adversarial networks (GANs), to create realistic manipulated videos. The phenomenon gained widespread attention in 2017, when online communities started creating synthetic explicit content using celebrity images. Since then, deepfake creation has become increasingly accessible, with easy-to-use tools and apps enabling users with little technical expertise to generate convincing altered images and videos. Reports indicate a 550% annual increase in deepfake content since 2019. Even mainstream social media platforms such as Snapchat and TikTok now incorporate similar AI-driven features, further normalizing the technology. Alongside this accessibility, the scale of misuse has grown rapidly. In South Korea, the impact is particularly stark. Since 2019, digital sex crimes involving children and adolescents in South Korea have become a major concern, especially after the “Nth Room” case. The scandal involved hundreds of victims, many minors, and around 260,000 participants who circulated exploitative content. It revealed how encrypted and anonymous online platforms can enable and hide large-scale, organized sexual abuse. 

Reports involving deepfake images have increased sharply since then, and regulatory bodies are handling a rapidly growing volume of cases, highlighting how quickly this form of abuse is expanding. According to the Women’s Human Rights Institute of Korea, 8,983 individuals sought support for digital sexual violence in 2023, marking a 12.6 percent increase from the previous year. Nearly three-quarters of those affected were women under the age of 30, and incidents involving deepfake images more than doubled over the same period. Moreover, the Korea Communications Standards Commission reviewed over 15,800 deepfake images between January and July 2024 alone, a sharp rise from 1,900 cases in 2021 and exceeding 23,000 reported cases in 2024. This broader trend is reflected in a major 2024 case, where coordinated deepfake abuse affected over 500 schools and universities. In this instance, perpetrators, often classmates, used Telegram to create “Humiliation Rooms” to collect victims’ photos and personal data, which were then used to produce and circulate AI-manipulated explicit images of students. Police detained 387 individuals for alleged deepfake-related crimes in that year, with over 80% of those involved being teenagers.

Photo: Statistics about deepfake crimes in South Korea (Source: The Chosun)

Fragmented Governance of Deepfakes: Legal Progress and Persistent Loopholes

Deepfake creators often evade accountability due to significant legal gaps. Fewer than half of the world’s countries regulate online abuse, and even fewer address AI-generated content, leaving loopholes in outdated laws. Many existing laws on image-based abuse were introduced before deepfakes, leaving loopholes for perpetrators to exploit. As a result, such content often falls into legal grey areas, creating uncertainty for survivors about whether it is punishable. Although regulatory efforts are emerging, they remain fragmented and insufficient. Measures like the “EU AI Act”, the U.S. “Take It Down Act,” which mandates removal of AI-generated intimate content within 48 hours; and the 2025 legal amendment increasing penalties for AI-enabled psychological violence exemplify recent regulation efforts. However, gaps persist, as seen in the UK’s “Online Safety Act”, which bans the sharing of manipulated explicit images but does not fully address their creation or cases where intent is difficult to prove. In South Korea, the “Nth Room” case sparked public outrage and led to stronger provisions in the 2020 Act on Special Cases Concerning the Punishment of Sexual Crimes. For example, the March 2020 amendment to the Act on the Punishment of Sexual Crimes criminalizes the production and distribution of deepfakes intended for circulation, while updated internet regulations require service providers to take measures to prevent the spread of illegal content. In 2022, a Seoul Metropolitan Government support center introduced an automated system to detect, monitor, and remove deepfake content, reducing identification time from two hours to just three minutes and earning the 2024 UN Public Administration Prize. However, while such measures help limit further harm, they are not comprehensive solutions, as the impact on victims often persists. In October 2024, South Korea further strengthened its legal framework by criminalizing the production of deepfake pornography even without intent to distribute and increasing penalties for both production and distribution. The law also extended criminal liability to the possession, purchase, storage, and viewing of such content, while assigning the state responsibility for content removal and victim support. Additionally, the legislation authorized investigative agencies to carry out covert operations in digital sex crime cases.

Photo: Screenshots from Telegram show an automated deepfake bot generating images (Source: The conversation)

Structural Background of Deepfake Sex Crimes in South Korea

However, the persistence of deepfake abuse cannot be attributed solely to legal and regulatory gaps but also reflects underlying structural and social origins. Deepfake-related sexual crimes in South Korea are influenced by a combination of socio-economic pressures, shifting gender perceptions, and weak enforcement mechanisms. Rising economic pressures among young men such as unemployment and housing challenges, combined with lingering patriarchal expectations, have contributed to growing gender tensions. However, these factors alone do not fully explain the rise of such crimes, especially since many perpetrators are teenagers. Survey data highlights more immediate drivers: among adolescents, 26.1 percent identified weak punishment, and 22.3 percent cited the anonymity of the internet as key reasons for the persistence of digital sex crimes. Among adults, 31.5 percent pointed to financial motives, while 30.5 percent also emphasized weak punishment. Notably, very few respondents in either group considered these crimes insignificant, underscoring a broader awareness of their severity despite their continued prevalence. A comprehensive review of Supreme Court rulings following the introduction of the 2020 law aimed at strengthening penalties for deepfake sexual exploitation materials reveals that 40% of cases resulted in probation. This indicates that, despite the legal framework intended to impose stricter punishment on the production and distribution of such materials, those prosecuted have often not faced adequately severe consequences in practice. Building on this gap between stricter laws and their limited enforcement, growing concerns over the rise of deepfake sex crimes have led to renewed calls for harsher penalties. Authorities are now considering expanding criminal liability beyond producers and distributors to also include those who view or possess such content. In this context, the first comprehensive analysis of Supreme Court rulings under Article 14-2 of the Act on Special Cases Concerning the Punishment of Sexual Crimes provides important insight. Conducted using court decisions from June 2020 to June 2024, the analysis highlights how, despite stronger legislation, many offenders have continued to receive inadequate punishment. The analysis found that out of 87 prosecuted defendants, 39% received suspended sentences (probation). In comparison, only 27.5% were given custodial sentences, while 16% were fined.  Enforcement is often weak due to limited resources, lack of digital expertise, and difficulties in cross-border coordination. The use of encrypted platforms like Telegram, along with VPNs and the international nature of deepfake content, further complicates identifying perpetrators and creates jurisdictional barriers to accountability. Underreporting is another major barrier. Survivors face stigma, exposure, legal risks, and victim-blaming within the justice system, discouraging them from coming forward. 

Digital sex crimes are a serious violation of women’s fundamental human rights, causing both psychological trauma and physical health impacts. They also carry broader social costs, limiting women’s participation in public and professional life. Research further shows the severity of the issue, with more than half of deepfake victims in the United States reporting suicidal thoughts. Far from being a marginal online issue, this represents a global crisis. While South Korea has taken important steps to regulate deepfake technology, legal reforms alone are not enough to shift the culture of sexual exploitation among South Korean youth. In the end, addressing deepfake sexual exploitation requires more than stronger laws. It calls for greater accountability from digital platforms, sustained efforts to raise awareness of gender-based violence, and a shift toward societal change for prevention through education and forward-looking policies, alongside continued support for victims. Until then, the line between victim and bystander remains fragile.

Sadia Masharuf
Latest posts by Sadia Masharuf (see all)
facebookShare on Facebook
TwitterTweet
FollowFollow us