Cybercrime Might Be the One Job AI Isn’t Taking, Study Suggests

TL;DR

A study reveals that AI is primarily used in cybercrime for SEO spam and scams, rather than advanced hacking. Researchers analyzed nearly 98,000 cybercrime forum threads post-ChatGPT launch.

Key points

Researchers analyzed 97,895 cybercrime forum threads
AI is mainly used for SEO spam and romance scams
Dark AI tools like WormGPT produced little effective malware
Jailbroken chatbots are hard to maintain for long periods
The study aims to understand AI adoption in cybercrime

In brief

Cambridge, Edinburgh, and Strathclyde researchers analyzed 97,895 cybercrime forum threads posted after ChatGPT’s launch.
“Dark AI” tools like WormGPT generated cultural buzz but produced almost no working malware, while jailbroken chatbots are increasingly hard to keep working for more than a few days.
The biggest measurable AI-driven crime is not hacking. It is mass-produced SEO spam, romance scams, and AI-generated nudes sold for a dollar each.

For three years, cybersecurity firms, governments, and AI labs have warned that generative AI would unleash a new generation of supercharged hackers. According to a new academic paper that actually went and looked, the supercharged hackers are mostly using ChatGPT to write spam and generate nudes for fun.

The study, titled Stand-Alone Complex or Vibercrime?, was published on arXiv by researchers from Cambridge and other universities and aims to understand how the cybercrime underground is actually adopting AI, not how cybersecurity vendors say it is.

"We present here one of the first attempts at a mixed-methods empirical study of early patterns of GenAI adoption in the cybercrime underground," researchers wrote.

The team analyzed 97,895 forum threads posted after ChatGPT launched in November 2022, drawn from the Cambridge Cybercrime Centre's CrimeBB dataset of underground and dark web forums. They ran topic models, manually read more than 3,200 threads, and ethnographically immersed themselves in the scene.

The conclusion is unflattering for the AI doom community: 97.3% of threads in the sample were classified as “other,” meaning not actually about using AI for crime at all. Only 1.9% involved someone using vibe coding tools.

‘Nothing more than an unrestricted ChatGPT’

Remember WormGPT, FraudGPT, and the wave of supposedly malicious chatbots that flooded headlines in 2023? The forum data tells a different story.

Most posts about “Dark AI” products, the researchers found, were people begging for free access, idle speculation, and complaints that the tools didn’t actually work. One developer of a popular Dark AI service eventually admitted to forum members that the product was a marketing exercise.

“At the end of the day, [CybercrimeAI] is nothing more than an unrestricted ChatGPT,” the developer wrote, before the project shut down. “Anyone on the Internet can use a well-known jailbreak technique and achieve the same, if not better, results.”

By late 2024, the researchers say, jailbreaks for mainstream models had become disposable. Most stop working in a week or less. Open-source models can be jailbroken indefinitely, but they are slow, resource-heavy, and frozen in time.

Q&A

What did the study on cybercrime and AI find?

The study found that AI is mainly used for generating SEO spam, romance scams, and low-cost AI-generated nudes, rather than sophisticated hacking.

How many cybercrime forum threads were analyzed in the research?

The researchers analyzed 97,895 cybercrime forum threads posted after the launch of ChatGPT.

What are 'Dark AI' tools and how effective are they in cybercrime?

'Dark AI' tools like WormGPT have generated buzz but have produced very little effective malware, with jailbroken chatbots often failing to work for long.

What is the title of the study that examined AI adoption in cybercrime?

The study is titled 'Stand-Alone Complex or Vibercrime?' and was published on arXiv by researchers from Cambridge and other universities.