Inside the deepfake crisis plaguing Korean schools

Last Saturday, a Telegram message appeared on Heejin’s phone from an anonymous sender. “Your photos and personal information have been leaked. Let’s discuss it.”

When the student entered the chat room to read the message, she was shown a photo of herself taken a few years ago when she was still in school. This was followed by a second image of the same photo, only this one was sexually explicit and fake.

Terrified, Heejin, which is not her real name, did not respond, but the images kept coming. In all the images, her face was attached to a body engaged in a sex act, using advanced deepfake technology.

Deepfakes, which usually combine the face of a real person with a sexually explicit fake body, are increasingly being generated using artificial intelligence.

“I was terrified, I felt so alone,” Heejin told the BBC.

But she was not alone.

Two days earlier, South Korean journalist Ko Narin had published what would become the biggest scoop of her career. It had recently emerged that police were investigating deepfake porn networks at two of the region’s largest universities, and Ms. Ko was convinced there had to be more.

She began searching on social media and discovered dozens of chat groups on the messaging app Telegram, where users were sharing photos of women they knew and, using AI software, turning those photos into fake pornographic images in seconds.

“Every minute people were uploading photos of girls they knew and asking to make deepfakes of them,” Ms. Ko told us.

Ms Ko found that these groups didn’t just target university students. There were rooms dedicated to specific high schools and even middle schools. If a lot of content was created featuring images of a particular student, she could even be given her own room. They are commonly referred to as “humiliation rooms” or “friends of friends rooms” and often have strict entry requirements.

Ms Ko’s report in the Hankyoreh newspaper has shocked South Korea. On Monday, police announced they were considering launching an investigation into Telegram, following the lead of authorities in France, who recently charged Telegram’s Russian founder with crimes related to the app. The government has promised tougher punishments for those involved, and the president has called for better education for young men.

The BBC has contacted Telegram for comment and while it has not yet responded to this specific case, it has previously told the BBC that it proactively searches for illegal activity, including child sexual abuse, on its site. It said that in August alone, 45,000 groups worldwide had been subjected to covert action.

‘A systematic and organized process’

The BBC has reviewed descriptions of a number of these chat rooms, one of which asks members to post more than four photos of someone, along with their name, age and the area where they live.

“I was shocked at how systematic and organized the process was,” Ms. Ko said. “The most horrific thing I discovered was a group for underage students at a school with over 2,000 members.”

In the days after Ms. Ko’s article was published, women’s rights activists also began scouring Telegram and following tips.

By the end of the week, more than 500 schools and universities had been identified as targets. The actual number of people affected and the number of victims has yet to be determined, but many are believed to be under 16, the age of consent in South Korea. Many of the suspected perpetrators are themselves teenagers.

Heejin said learning about the scale of the crisis had made her anxiety worse, as she now worried about how many people had viewed her deepfakes. At first, she blamed herself. “I couldn’t stop thinking, did this happen because I uploaded my photos to social media? Should I have been more careful?”

Since then, dozens of women and teens across the country have deleted their photos from social media or deactivated their accounts altogether, fearing they would be subjected to abuse again.

“We are frustrated and angry that we have to censor our behavior and use of social media when we have done nothing wrong,” said Ah-eun, a university student whose fellow students have been the target of attacks.

Ah-eun said a victim at her university was told by police that she didn’t need to pursue her case because it would be too difficult to catch the perpetrator, and that it was “not really a crime” because “the photos were fake.”

At the center of this scandal is the messaging app Telegram. Unlike public websites, which authorities can easily access and then request that images be removed, Telegram is a private, encrypted messaging app.

Users are often anonymous, rooms can be set to “secret” mode, and their contents can be quickly deleted without a trace. This has made it a prime location for criminal behavior to flourish.

Last week, politicians and police responded forcefully, promising to investigate the crimes and bring the perpetrators to justice.

On Monday, Seoul National Police announced they would investigate Telegram over the company’s role in distributing fake pornographic images of children.

The app’s founder, Pavel Durov, was charged in France last week with complicity in a number of crimes related to the app, including facilitating the sharing of child pornography.

But women’s rights activists accuse South Korean authorities of letting sexual abuse on Telegram fester with impunity for too long, saying Korea has dealt with this crisis before. In 2019, it was revealed that a sex ring was using Telegram to coerce women and children into taking and sharing sexually explicit images of themselves.

Police at the time asked Telegram for help with their investigation, but the app ignored all seven requests. Although the leader was eventually sentenced to more than 40 years in prison, no action was taken against the platform, due to fears of censorship.

“They have condemned the main actors, but otherwise ignored the situation. I think this has only made the situation worse,” said Ms Ko.

Park Jihyun, who discovered the Nth Room sex ring as a young student journalist in 2019, has since become a political advocate for victims of digital sex crimes. She said that since the deepfake scandal broke, students and parents have called her several times a day, crying.

“They have seen their school listed on social media and are terrified.”

Ms Park has called on the government to regulate or even ban the app in South Korea. “If these tech companies are not willing to cooperate with law enforcement agencies, then the state should regulate them to protect its citizens,” she said.

Before this recent crisis broke out, the South Korean Advocacy Center for Online Sexual Abuse Victims (ACOSAV) already noticed a sharp increase in the number of underage victims of deepfake pornography.

In 2023, they assisted 86 victims. That jumped to 238 in the first eight months of this year, excluding the last seven days.

One of the center’s leaders, Park Seonghye, said her staff had been inundated with calls over the past week and were working around the clock. “It was a large-scale emergency for us, like a war situation,” she said.

“Thanks to the latest deepfake technology, there is now much more footage than before, and we fear that this will only increase.”

In addition to counseling victims, the center tracks harmful content and works with online platforms to have it removed. Ms. Park said there were cases where Telegram had removed content at their request. “So it’s not impossible,” she noted.

Women’s rights groups acknowledge that new AI technology makes it easier to exploit victims, but they argue that this is just the latest form of misogyny being propagated online in South Korea.

First, women were subjected to waves of verbal abuse online. Then came the spycam epidemic, where they were secretly filmed using public toilets and changing rooms.

“The root cause of this is structural sexism and the solution is gender equality,” said a statement signed by 84 women’s groups.

This is a direct criticism of the country’s President Yoon Suk Yeol, who denies the existence of structural sexism, has cut funding to victim support organizations and disbanded the government’s Ministry of Gender Equality.

Lee Myung-hwa, who treats young sex offenders, agreed that while the outbreak of deepfake abuse may seem sudden, it has been simmering beneath the surface for a long time. “For teenagers, deepfakes have become part of their culture, seen as a game or a joke,” said the counselor, who runs the Aha Seoul Youth Cultural Center.

Ms Lee said it was vital to educate young men, citing research that shows that telling offenders exactly what they did wrong makes them more aware of what sexual abuse is, which prevents them from repeating the crime.

Meanwhile, the government has announced that it will increase penalties for those who create and share deepfake images. Those who watch such pornography will also be punished.

It follows criticism that not enough offenders were being punished. One problem is that the majority of offenders are teenagers, who are typically tried in juvenile courts, where they receive more lenient sentences.

Since the chat rooms were exposed, many have been closed down, but new ones will almost certainly open in their place. A humiliation room has already been created to target the journalists who covered the story. Ms Ko, who reported the news, said she had lost sleep over it. “I keep checking the room to see if my photo has been uploaded,” she said.

Such fear has spread to nearly every teenage girl and young woman in South Korea. Ah-eun, the university student, said it has made her suspicious of her male acquaintances.

“I can no longer be certain that people aren’t going to commit these crimes behind my back without my knowledge,” she said. “I’ve become hypervigilant in all of my interactions with people, which can’t be good.”

Additional reporting by Hosu Lee and Sunwook Lee