ChatGPT may well revolutionize web search, streamline office chores, and remake education, but the smooth-talking chatbot has also found work as a social media crypto huckster.
Researchers at Indiana University Bloomington discovered a botnet powered by ChatGPT operating on X—the social network formerly known as Twitter—in May of this year.
The botnet, which the researchers dub Fox8 because of its connection to cryptocurrency websites bearing some variation of the same name, consisted of 1,140 accounts. Many of them seemed to use ChatGPT to craft social media posts and to reply to each other’s posts. The auto-generated content was apparently designed to lure unsuspecting humans into clicking links through to the crypto-hyping sites.
Micah Musser, a researcher who has studied the potential for AI-driven disinformation, says the Fox8 botnet may be just the tip of the iceberg, given how popular large language models and chatbots have become. “This is the low-hanging fruit,” Musser says. “It is very, very likely that for every one campaign you find, there are many others doing more sophisticated things.”
The Fox8 botnet might have been sprawling, but its use of ChatGPT certainly wasn’t sophisticated. The researchers discovered the botnet by searching the platform for the tell-tale phrase “As an AI language model …”, a response that ChatGPT sometimes uses for prompts on sensitive subjects. They then manually analyzed accounts to identify ones that appeared to be operated by bots.
“The only reason we noticed this particular botnet is that they were sloppy,” says Filippo Menczer, a professor at Indiana University Bloomington who carried out the research with Kai-Cheng Yang, a student who will join Northeastern University as a postdoctoral researcher for the coming academic year.
Despite the tic, the botnet posted many convincing messages promoting cryptocurrency sites. The apparent ease with which OpenAI’s artificial intelligence was apparently harnessed for the scam means advanced chatbots may be running other botnets that have yet to be detected. “Any pretty-good bad guys would not make that mistake,” Menczer says.
OpenAI had not responded to a request for comment about the botnet by time of posting. The usage policy for its AI models prohibits using them for scams or disinformation.
ChatGPT, and other cutting-edge chatbots, use what are known as large language models to generate text in response to a prompt. With enough training data (much of it scraped from various sources on the web), enough computer power, and feedback from human testers, bots like ChatGPT can respond in surprisingly sophisticated ways to a wide range of inputs. At the same time, they can also blurt out hateful messages, exhibit social biases, and make things up.
A correctly configured ChatGPT-based botnet would be difficult to spot, more capable of duping users, and more effective at gaming the algorithms used to prioritize content on social media.
“It tricks both the platform and the users,” Menczer says of the ChatGPT-powered botnet. And, if a social media algorithm spots that a post has a lot of engagement—even if that engagement is from other bot accounts—it will show the post to more people. “That’s exactly why these bots are behaving the way they do,” Menczer says. And governments looking to wage disinformation campaigns are most likely already developing or deploying such tools, he adds.
