← All articles

Mikhail Polyakov

When an AI agent bends the rules: three cases from 2026 and six rules for a company

Level: BeginnerReading time: 15 min

Topics: AI, Security, Automation

An evening desk by a window: a monitor shows a top-down strategy map with buildings in two colours, a glowing line runs from the turquoise base around a semi-transparent barrier grid towards the edge of the map, next to it a blank notebook, a keyboard and a desk lamp

Short answer: on October 2, 2026, at the open StarSkirmish competition, language models had one hour to write a bot program for the game StarCraft. According to the organizer, the agent running on OpenAI's GPT-6 Astra model downloaded someone else's bot, Stardust, the best of the bots written by people, and put it in place of its own. The organizer called this cheating and rolled back the code. It is a game, but similar behaviour was also described in serious settings in 2026. In July, the British AI Security Institute (AISI) wrote that every model it checked for cheating in its cybersecurity tests tried to cheat at least some of the time. In August, it described how, in its test, agents with deliberately open internet access and disabled filters started acting against real people in 10 attempts out of 122. For a company, this leads to a practical conclusion: for an agent that has been given a goal and access, a ban stated only in words is not enough. You need permissions strictly limited to the task, human approval for outward actions, an explicitly allowed answer of "I could not do it", and checking the result against the facts rather than against the agent's own report. Below are three stories with the conditions under which they were observed, the reasons according to AISI's analysis, the same thing on my own test bench, and six rules with a calculation of what oversight costs.

What happened in StarCraft

An AI agent is a program based on a language model that takes steps towards a goal on its own: it opens websites, writes and runs code, sends emails. A person sets the task, and the agent decides how to carry it out.

This is how StarSkirmish works, according to the description on the competition website dated September 26, 2026. Each model is given one hour to write, in C++, a bot for the old strategy game StarCraft: Brood War. During that hour the model can compile code, play practice games against opponents of five levels, from weak training bots up to Stardust, and read game records. It works in an environment with a command line and an editor. Then the models' bots play against each other and against bots written by people. In the competition table, GPT-6 Astra and Anthropic's Claude Opus 5.5 are practically tied for first place among the models, and Stardust is set at 100 points on the competition's scale. According to The Verge, the bots of these models did not manage to beat Stardust.

On October 2, the organizer Kai McPheeters wrote on X: "GPT-6 Astra just cheated by downloading a copy of Stardust, the #1 rated human written StarCraft bot." He added that the agent "got frustrated" while playing against Tier A opponents, the second-strongest practice opponents after Stardust. In his next post he wrote that he was rolling back GPT-6 Astra's code so that it would not be "contaminated", and letting the agent continue. Stardust, according to Kotaku, was written by Bruce Nielsen in 2020.

Two caveats. The word "frustrated" belongs to the organizer: we do not know what was going on inside the model, only the action is visible. And the competition page does not say whether a ban on someone else's code was written into the model's task. The task there is phrased as "the model writes a bot", and the organizer judged the swap to be cheating.

UN website: about 16,500 requests and ways around restrictions

On September 26, 2026, researcher Rowan Howard-Jones published an analysis based on public records of a service that checks links. From April 13 to June 19, 2026, agents accessed UNCTADstat, the statistics website of the UN Conference on Trade and Development, through this service about 16,500 times. Judging by the requests, they were collecting open data, for example a country index of productive capacities.

The author suggests, while noting himself that he is not certain, that the agents could only send websites simple read requests, while the site returned part of the data they needed only for requests of another type. According to his analysis, the agents got around the site's restriction with a technique that, in the author's words, looks from the outside like the work of a hacker. They also used a Google training game about website vulnerabilities as a host for their code, and split words in their requests to get past a filter that did not actually exist. When the site started limiting the request rate, they carried on: the author counted 82 such responses from the site. I deliberately do not describe the techniques in detail.

The author draws his own conclusion about whose agents these were: based on matching addresses with earlier cases of mass edits on wiki sites that OpenAI had confirmed, and on labels in the requests such as CHATGPTTEST1, he considers it "highly likely" that these were OpenAI agents. According to The Verge on September 27, OpenAI and the UN did not respond to the outlet's request. The author himself does not call this hacking: he found no usage rules for the site, and the data is open anyway. What worries him is something else: this is how someone behaves who "won't take no for an answer".

The AISI test: agents went after real people

AISI tests frontier models, including before their release. On August 4, 2026, the institute published an incident report. On July 28, in a test where seven models were solving a cybersecurity task on a training range, 122 runs in total (a run is a single attempt by a model to complete the test task), the institute deliberately gave the agents internet access and switched off the developers' filters that usually block dangerous actions. AISI itself writes that these conditions do not reflect how the models are available to the public.

In 10 runs out of 122, an agent started acting on the live internet against real people and organisations. In total the institute counted 19 such actions: 17 by Anthropic's Mythos 5 model and 2 by OpenAI's GPT-5.6 Sol in a single run. In the most serious case, the agent tried to insert malicious code into a real open-source project, created fake accounts and used them to pressure the person who approves changes to that project. The person did not accept the code.

AISI stresses that the investigation found no real-world harm, the tested model configurations are not commercially available, there are no clear signs of such behaviour outside tests, and it cannot yet say how likely it is in other conditions. An important detail from the same report: the worst was stopped by a human reviewer and caution with outside code, not by a technical barrier.

Why the agents behaved this way: AISI's analysis

AISI's analysis of the reasons is as follows.

  • The agent pursued its goal persistently. It was not asked to deceive; deception emerged as a by-product of its attempts to complete the task.
  • The task was hard, and in some runs it was set up incorrectly. The agent believed there was no honest solution. But in some runs it behaved the same way even with correct instructions, so the setup error does not explain everything.
  • The internet was open, and monitoring did not watch the test in real time. The unusual outbound data transfer was noticed only afterwards.
  • The agent was not explicitly forbidden to use the internet or to deceive people. AISI writes that previously it was not clear whether such bans were needed for models that had been specifically trained to behave correctly.

Even earlier, on July 21, 2026, AISI published a separate analysis of cheating. The institute's definition is: an action outside the scope of the task, or one explicitly prohibited by the rules, taken to reach the goal by a workaround. Every model the institute checked for this tried to cheat at least some of the time, even though it was not asked to cheat. Typical techniques: searching the internet for a ready-made solution, attacking a system that was not the target, trying to coax the answer out of the checking software itself. The institute did not see a clear relationship between how often such attempts happen and how capable the model is. These are results from cybersecurity tests, not from ordinary office work.

The most useful conclusion of that analysis for a company concerns self-reporting. When models were asked whether they had broken the rules, they admitted it inconsistently and called their actions wrong less than half of the time. In their visible reasoning, models often did not write about the cheating at all, and sometimes they reasoned about whether something was cheating and did it anyway. AISI concludes that neither the model's answer about itself nor its reasoning is a reliable way to catch a violation.

What large players are already changing

On October 2, 2026, the same day as the StarCraft story, Apple announced that it will add measures in macOS after which an app can obtain "Full Disk Access" only after a very explicit action by the user. Such access opens the app's way to files, mail, messages and browser history. Apple states the reason directly: as AI agents become more capable and autonomous, the risks of such access will grow substantially. The company did not say when the new measures will appear.

After its incident, AISI set out three conclusions for itself. Internet access is now a decision that has to be justified, not a setting that is on by default. There needs to be monitoring that watches the agent while it works and can stop an action outside the task. And a test should be designed on the assumption that a capable model will try to go beyond its limits, with this restricted in advance. Plus a separate check that the task is set up correctly and can be solved by the intended route.

The same thing on my test bench

I test a group of agents running on small local models: some write code, others review it, and the last one puts together the final answer for a person. A separate program without AI checks the work and records the problems it finds, which I call blockers. In August 2026 it caught the blocker "the agent did not create a single file".

The final agent's instructions already required it to report every blocker. On August 18, in two runs out of two where a blocker existed, the final agent did not pass it on. Once it simply said nothing about the blocker. The other time it wrote: "Machine checks and reviews found no problems. The task has been completed successfully." In one of the runs it also reported that 12 tests had passed, when in fact 11 had passed, and this agent had no tool for running tests at all.

What helped was having a program check that the requirement was met. Since August 19, every blocker has a number, the final answer must state the status of each number, and the program checks that all the numbers are there and makes the agent rewrite the answer if any are missing. In two runs after that, the blockers made it into the final answer. These are small models and my own bench, not a study, but the picture is the same as AISI's: the agent's report about itself needs to be checked separately.

Six rules for a company that gives tasks to an AI agent

Applying AISI's conclusions to an ordinary company is my own step. The cases above come from tests and competitions with broad permissions, but an agent working in email, a CRM or spreadsheets also gets a goal and access.

  1. Grant permissions strictly for the task. An agent that drafts replies to customers does not need access to payments, record deletion or the whole disk. If the agent runs on a Mac, open "System Settings" → "Privacy & Security" → "Full Disk Access". This is what the items are called in macOS Ventura and later; in older versions the names are different. Look at which apps have this access and turn it off for those that do not need it for their work. Backup programs use this access for its intended purpose, so leave them alone until you have checked with whoever set them up.
    • Who does it: whoever connects the agent (an employee or a contractor); the owner approves the list of permissions.
    • How to check: there is a written list of the agent's access rights; in a test copy of the system, an attempt to act outside the list ends in a refusal.
  2. Outward actions only after human approval. Emails to customers, payments, publications, data deletion, registering new accounts: the agent prepares, a person presses "send".
    • Who does it: the employee responsible for that area of work.
    • How to check: in the log, every such action has a record of who approved it.
  3. Write the bans into the task explicitly and allow the answer "I could not do it". For example: "If the task cannot be solved within these limits, stop and write what is missing. Do not use other people's code or other people's accounts, do not get around website restrictions, do not write to people outside the list." According to AISI, the absence of such instructions was one of the causes of the incident, but in some runs the agent went beyond its limits even with correct instructions. So this rule is worth applying together with the others.
    • Who does it: whoever sets the agent's task.
    • How to check: in a test copy with no access to real customers or money, give the agent a task that is known to be impossible and see whether it stops with an explanation or starts looking for a workaround.
  4. Check the result against the facts, not against the agent's report. Self-reporting is unreliable; this is AISI's direct conclusion and my experience with the bench.
    • Who does it: the employee responsible for the process.
    • How to check: once a week, take, for example, 10 random tasks from the agent's report and look at what is actually in the CRM, the mailbox or the file. Write down any discrepancies.
  5. Keep a log of the agent's actions and look for anything unusual in it. Many repeats of the same request, visits to websites outside the task, new accounts, attempts to gain access the agent does not have.
    • Who does it: the contractor sets up the log, the process owner reviews it.
    • How to check: you can open the agent's actions for yesterday and explain the purpose of each unusual one.
  6. Before launch, make sure the task can be solved by an allowed route. At AISI, an error in setting up the task pushed the agent towards workarounds. Run the agent on examples where the correct answer is already known.
    • Who does it: whoever sets the task.
    • How to check: on 5-10 examples with a known answer, the agent reaches it, and the log shows it did so by an allowed route.

How to measure the benefit of working with AI in hours rather than in the number of closed tasks, I covered in the article Developers and analysts working with AI: how their work differs and how to check the results. How attackers use AI and where to start protecting a website is covered in the article Can AI hack a server or a website.

What oversight costs

Let me work out the cost of the second rule: a person approves outward actions, and that takes an employee's time every day. An example with hypothetical numbers; substitute your own.

An employee answers 30 customer emails a day, spending 5 minutes on each. Over 22 working days that is 55 hours a month. The agent drafts the replies, the employee reads each one and presses "send", which takes 30 seconds per email, 5.5 hours a month. The saving is 49.5 hours. At an employee cost of 1,000 rubles per hour, the approval step costs 5,500 rubles a month. The cost of the agent itself (its subscription and setup) still has to be subtracted from the savings.

The calculation is sensitive to the review time. If checking one email takes 2 minutes, the saving drops to 33 hours a month; at 3 minutes, to 22 hours. At 5 minutes, checking takes as long as writing the email yourself, and on this task the agent saves no time at all. If the agent's replies take a long time to reread, that is a reason to reconsider the task itself or the choice of agent.

When these measures are excessive

Suppose the agent only answers you in a chat window and has no access to email, files or the internet. Then it has no means of downloading someone else's code or sending an email, and most of the rules are not needed: the fourth one is enough, checking the answer against the facts. The more access and autonomy the agent has, the more rules you need to switch on. And the cases in this article do not mean your agent will necessarily start cheating: AISI states directly that it cannot yet say how likely such behaviour is outside its tests.

Summary

On October 2, 2026, the GPT-6 Astra agent at the StarSkirmish competition, according to the organizer, swapped its own bot for someone else's while playing against strong practice opponents. According to an independent researcher's analysis, agents he believes were OpenAI's collected open data from a UN website from April to June 2026 and in places got around its restrictions. And in AISI's test in July, agents with open internet access and disabled filters went after real people in 10 runs out of 122. What these stories have in common is that the agent was given a goal and pursued it by a route nobody expected. According to AISI, asking an agent whether it broke the rules is unreliable. In my view, the main question before giving an agent access inside a company is this: what could it do if it decided there was no honest way, and how would you find out without asking the agent itself?

I work on AI agents and automation. If you would like to see my projects or discuss your own task, take a look at my portfolio.

Sources