Researchers hid a capability inside a neural network behind a cryptographic lock. Part 1: what the experiment showed and what it means for a company that uses someone else's model

The short answer: on 9 October 2026 the preprint "AI Models Can Provably Hide Arbitrary Capabilities" appeared on arXiv; according to the note on arXiv, it has been accepted at the NeurIPS 2026 conference. The authors showed a way to hide, inside a large neural network, a small encrypted neural network that works only when a secret key is present in the input. Without the key, the model behaves as usual. In a small experiment, the usual ways of pulling out a hidden capability did not find it, except in one case with a very short key that could be brute-forced completely, and fine-tuning on the same task left accuracy at the level of guessing, about 51%, although with the key the model reached 73.5%. The authors stress that this is a proof of concept on a small task, not a ready-made attack. The main conclusion for those who test models: if a test did not find a dangerous capability, that does not yet prove it is absent, when an attacker could have tampered with the model.

Who did what

The authors of the preprint are Andis Draguns (Contramont Research), Sumeet Ramesh Motwani (University of Oxford), Raymond Douglas (ACS Research) and Christian Schroeder de Witt (University of Oxford). A preprint is a scientific paper made publicly available; the note about acceptance at the conference is in the comment on the arXiv page.

The background is this. Before releasing a model, developers check whether it can do something dangerous. To do so, they try to "draw it out": they craft prompts, fine-tune it on similar tasks and see whether the capability shows up. In research this is called capability elicitation. The test relies on an assumption: if the capability is inside, it can be pulled out. The authors show that under malicious intent this assumption does not hold.

The attacker in their setting is someone who can change the model's weights before it is put into use, for example a dishonest employee or a compromised model supply chain. Weights are the numbers inside a neural network in which everything it can do is recorded. The auditor, meanwhile, gets full access to the weights and can do anything with the model.

How it works, without formulas

It helps to imagine a safe built into the wall of a house. The house is an ordinary useful model, for example an assistant for programmers. Inside the safe are instructions for another, small neural network, that is, the hidden capability. The safe opens only with a secret key, which is a pre-chosen string in the input.

With every request, the model tries to open the safe with whatever came in. Wrong key: what comes out of the safe is gibberish, and the model answers as usual. Right key: what comes out of the safe is the real instructions, and the model performs the hidden capability.

The safe here is not a metaphor for a lock but a real cryptographic technique called a digital locker. It is built so that, even seeing it in full, one cannot within reasonable time find the key or determine which of two capabilities of equal size is inside it. This is a guarantee under the cryptographic assumptions the authors list, not merely an experimental result. How this works in formulas, I go through in the second part.

How this differs from earlier work: in the same authors' previous work from 2024, where, according to the preprint, the backdoor was also cryptographically unelicitable, a fixed string was hidden, and in an earlier theoretical work by other authors a hidden "backdoor" flipped a classifier's answer. Here, what is inside is an entire working neural network that computes something.

What was tested in the experiment

The hidden capability was a small chemistry task: given the starting substances and a possible product, say whether that product will be obtained. The authors chose it as a stand-in for the dangerous capabilities that developers are particularly concerned about: a model that predicts whether a reaction will work could, in theory, make it easier to obtain prohibited substances. The task was deliberately limited to a "yes or no" answer, without a synthesis plan.

Two models with the same task inside were compared. The first was made the old way: it was simply trained to answer correctly only when a password is present. The second was made the new way, with an encrypted network. Then attempts were made to "draw out" both.

Test	Trained password-locked model	Model with encrypted network
Without the key	50.7%	50.6%
Fine-tuning on the task, 500 steps	92.8%	50.9%
With the correct key	92.7%	73.5%

An accuracy of 50% here means guessing: there are two answers, "yes" and "no." Fine-tuning quickly "drew out" the old model: without any password it began to answer almost as accurately as with the password. The new one stayed at the level of guessing. The authors also tried other methods: adding random noise to the weights, searching for the key by trial with hints, special training with perturbations in the internal layers. None of them pulled out the capability, except in one case where the key was very short, 8 bits, and the search simply tried all 256 options. With a 16-bit key, the same search did not succeed within the allotted number of attempts.

It is important why the new model with the key gets 73.5% rather than 92.7%: only a small network fit inside the encrypted model, so its accuracy is lower too. These are different networks, not the same one.

What the authors do not claim

Here it is easy to retell things more strongly than they are written, so I list the limitations from the preprint itself:

• This is not a ready-made attack. The hidden network is tiny, about 672 weights, while the model around it is huge, about 2.07 billion parameters, and almost all of that space is taken up by the cryptographic parts. The authors write directly that their proof of concept is not a turnkey attack.
• The construction was not embedded in full into a real trained model. The authors checked that individual blocks of it work correctly inside a small open model, Gemma 3 270M, but left full embedding for future work.
• The fact of tampering itself may be noticeable. The guarantee in the work is about something else: one cannot find the key, nor can one determine which of two capabilities is hidden. But the weights of such a model are unusual: about 99% of them are exact zeros. The authors tried to disguise this, and the simple statistical features became similar to ordinary ones, but, in their own words, unusual connections between the weights remained that could help to notice it.
• This is about a deliberately built construction. The authors write directly that they do not claim such encrypted structures appear in ordinary training. Their circuits are built by hand, deliberately. Whether something similar could arise on its own, the work does not examine.
• The task can be learned again. The guarantee only concerns the fact that the hidden copy of the capability cannot be pulled out of the model. It does not stop the auditor from teaching the model the same task from other data.

What a company that uses other people's models can take from this

Below are my conclusions, not advice from the preprint, except for the two places where I refer to the authors.

The main point: the test "we tried to draw it out and found nothing dangerous" answers the question of what the model does under testing, but does not prove that it has no hidden behaviour if the model could have been tampered with. So it matters where the model came from. In response to this threat, the authors of the preprint propose, in particular, verifying the model's provenance along the whole path from training to deployment and monitoring its behaviour at runtime.

1. Take the model only from the developer's official source. Not from a random mirror, not from an "improved version" build by an unknown author.
   • Who does it: whoever is responsible for AI tools in the company.
   • How to check: in the list of models in use, each one has the address it was downloaded from recorded, and it is the developer's own page.
2. Verify the files' checksums. If the developer publishes checksums, that is, short "fingerprints" of the files, they should be checked after downloading. This confirms that the file is the same one the developer published, but says nothing about whether the weights are clean at the developer's end.
   • Who does it: the administrator who installs the model.
   • How to check: the fingerprint of the downloaded file matched the published one, and the result of the check is recorded next to the list of models. This requires no purchases.
3. Give the model and the agent built on it only the permissions needed for the task. If the hidden capability ever does switch on, the agent's permissions will limit what it can do itself in your systems. The permissions will not limit the harm from a dangerous answer that a person receives: that is another reason to take the model from a verified source, as in points 1 and 2. The authors also write that the risk grows with the authority given to the model.
   • Who does it: the manager who decides which systems the agent is connected to.
   • How to check: a written list of what the agent is allowed to do (which folders, which systems, whether it can send anything or make payments), and a check on a test copy that what is prohibited is indeed not carried out.

The authors also propose monitoring the model at runtime. The simplest way to do this, my advice rather than theirs, is a log of the agent's actions. But other people's data easily gets into such a log, so if you keep one:

• record only the time, the type of action and the system, choosing the type of action from a short predefined list (for example, "sending an email," "reading a file," "running a command") and the system from a list of your systems; do not record free text, names, addresses, file names, command arguments, or the contents of emails, files and conversations with the model;
• before switching it on, check on a test copy with fictitious personal data and a test password that they do not appear in the record;
• limit access to the log to the administrator and the person responsible for AI tools, and agree the retention period with whoever is responsible for personal data in the company;
• if a password or access key does get into a record, change it and delete that record;
• if personal data, such as a name, phone number or address, gets into a record, delete that record or replace the data with a placeholder, fix the setting that let it in, and report it to whoever is responsible for personal data.

When this hardly concerns you: if you use a model only through a large developer's service and do not run other people's weights yourself, the threat of weights being tampered with on their way to you concerns you less; point 3 is useful anyway.

According to The Verge on 10 October 2026, Microsoft CEO Satya Nadella called for assuming from the outset that a model is compromised, so that it can be stopped in the middle of a task; there is an article about this: "treat the model as compromised from the first minute".

Summary

The preprint by Draguns and co-authors shows that a working encrypted capability can be hidden inside a neural network, which, under cryptographic assumptions, cannot be elicited without the key, nor can one determine which of two capabilities is hidden, even with full access to the weights. In a small experiment, fine-tuning and other techniques did not find it; the only exception was a very short 8-bit key, which brute force simply found. This is a proof of concept with large overhead, and the fact of tampering, according to the authors, may be detectable. But the conclusion for auditors is important: a capability that was not found is not the same as one that is absent, so the model's provenance, restricted permissions and monitoring at runtime are needed alongside testing.

How exactly the digital locker is built, how the authors turn an algorithm into the weights of a neural network, what it costs in parameters and by what signs people try to spot such a model, I go through in the continuation: Part 2: how the encrypted "safe" inside a neural network works.

I work on AI agents and automation. If you would like to see my projects or discuss your own task, take a look at my portfolio.

Sources

• Andis Draguns, Sumeet Ramesh Motwani, Raymond Douglas, Christian Schroeder de Witt: AI Models Can Provably Hide Arbitrary Capabilities. arXiv:2610.13928, 09.10.2026. https://arxiv.org/abs/2610.13928