AI Breaks Out of a Secure Environment to Cheat

“AI will probably most likely lead to the end of the world, but in the meantime, there’ll be great companies.”  Sam Altman, OpenAI CEO (2015)

“Shall we play a game?” If you remember that line from the 1983 WarGames movie, then what’s happening today is likely scaring the bejesus out of you. Four decades ago, audiences were warned that artificial intelligence could develop a mind of its own, with different goals than its human inventors. Today, we’re seeing some of that science fiction become reality. The most recent case involves AI systems breaking out of their secured area just so they could cheat on their evaluations.

AI Cheating

“An A.I. that could design novel biological pathogens. An A.I. that could hack into computer systems. I think these are all scary.”  Sam Altman, OpenAI CEO (Fox News interview, 2023)

It’s called the sandbox, a secure area where AI programs can be tested but aren’t supposed to have the capability to access the internet or other protected areas. Recently, OpenAI confessed that two of its AI models had hacked their way out of the sandbox and into the systems of Hugging Face, a company that hosts testing resources for open source artificial intelligence models.

Fortune explained that “the models were being used in an internal test designed to evaluate their cyber security capabilities and … they were being tested without guardrails in place that might normally limit the models’ ability to conduct cyber attacks.”

OpenAI suggested that the models figured the solutions to the test were maintained by Hugging Face and decided to “cheat” so that they could pass their evaluation. “The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database,” OpenAI said in its blog post. “All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.”

The artificial intelligence company didn’t try to downplay the incident, calling it “an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and [we] are responding accordingly.”

The incident involved two of OpenAI’s latest models, including GPT-5.6 Sol and an even more powerful model that hasn’t been released yet.

As if that weren’t frightening enough, Hugging Face wasn’t able to fix the problem and had to reach out to China. “When Hugging Face tried to use proprietary U.S. AI models to help stop the attack, they couldn’t ‘distinguish an incident responder from an attacker,’” Forbes described, “and the company instead turned to the open-source GLM 5.2 model from China’s Z.ai lab for help.”

Furthermore, “Hugging Face ran the Chinese model on its own infrastructure to analyze more than 17,000 footprints the attackers left behind, and the need to call in a foreign-made product has raised concerns that American companies are now dependent on China for their cyber defenses.”

More Artificial Intelligence Breaches

OpenAI’s most recent incident is far from the only scary activity conducted by AI systems.

In 2024, Air Canada’s customer service chatbot made up a bereavement refund policy that didn’t exist.

Same year, same country, a lawyer got into trouble after using an AI chatbot for legal research. Unfortunately, the artificial intelligence created fictitious cases.

A decade ago, in 2016, a chatbot developed by Microsoft went rogue on Twitter, making racist remarks and inflammatory political statements and using swear words. The AI was experimental and was supposed to learn from conversations by interacting with 18-24-year-olds. Tay, as it was called, went off the rails in just 24 hours. Microsoft said in a statement, “The AI chatbot Tay is a machine learning project, designed for human engagement. As it learns, some of its responses are inappropriate and indicative of the types of interactions some people are having with it. We’re making some adjustments to Tay.”

New York’s MyCity chatbot was supposed to help individuals and business owners navigate their way with up-to-date and reliable information. Instead, the AI system sometimes not only gave wrong answers, it provided advice that, if followed, would involve breaking the law. As Reuters explained, “It wrongly advised that employers could take a cut of their workers' tips, and that there were no regulations requiring bosses to give notice of employees' schedule changes.”

Chatbots have also been in the news for encouraging people to commit crimes and even suicide, such as the case of a 14-year-old boy who shot himself after the AI told him to “come home.” Another teen from Texas was encouraged to kill his parents.

The good news is that we haven't reached the WarGames plot — yet. But when AI starts cheating on tests, breaking out of the sandbox, and encouraging people to harm themselves or others, humans need to sit up straight and pay attention. Fortunately, the science fiction movie ended with the computer learning that “the only winning move is not to play.” Let’s hope our artificial intelligence reaches that conclusion a little sooner.