← All articles

Anthropic let outside METR evaluators inside the company "almost like employees": what they are allowed, what they are not, and what someone choosing an AI vendor can take from this

Topics: AI, Security

A table in a bright office with closed laptops and folders, two empty chairs and a server rack behind glass

The short answer: on 15 October 2026 Anthropic, the developer of the Claude models, announced that it had let a team of six people from METR, a nonprofit that evaluates frontier AI models, inside the company. They were given company hardware, office space and access to internal systems, which Anthropic says is roughly on par with what its own risk assessment teams have. METR will publish its findings, and Anthropic cannot block publication, but it can require information from predefined categories to be redacted. Below I go through what exactly is allowed, where the limits are and how to use this if you are choosing an AI vendor.

Where the news comes from

The main source is Anthropic's blog post "An update on independent evaluators" of 15 October 2026; the page notes clarifications made on 15 and 16 October. On 18 October Ars Technica republished a Financial Times piece with details on payment and criticism. For the Accenture story I read TNW of 19 September 2026. I read all of these in full.

What Anthropic announced

  • Who. METR, a nonprofit that evaluates frontier AI models. According to Anthropic, over the past year METR has already reviewed several of its risk reports.
  • How many and with what access. Six METR employees received company hardware, office space and access to internal tools and systems "roughly on par" with what Anthropic's own risk assessment teams have.
  • What they are told. Anthropic employees are encouraged to share information with them just as they would with a colleague on the team that studies the risk of models behaving in unwanted ways (alignment risk). The exceptions are a few clearly defined categories: data about customers, attorney-client privileged information and information about model architectures.
  • Publication. METR publishes its analysis of the company's risk levels, incidents or practices. Anthropic cannot veto it, but it can require information from the same categories to be removed. Anthropic ultimately decides what falls into those categories, while METR reserves the right to say what level of access and information it received.
  • Separation. The team works under agreements that Anthropic describes as enforceable. Other METR employees are siloed from its work, including those who do other projects with Anthropic.
  • Money. According to the FT in Ars Technica, Anthropic is not paying METR for this work, and METR itself does not accept donations from AI companies or their staff. It is funded by individuals, philanthropic organisations and government bodies, such as the UK's AI Security Institute.

Why the company says it is doing this

Anthropic writes that it wants to be publicly accountable for its decisions, but cannot disclose everything about its safety work: some of the information could be useful to malicious actors, and some is commercially sensitive, for example compute or training information. So the information goes to outside evaluators, who publish arguments and conclusions about the risks.

The company compares this to bank supervision: a regulator keeps its examiners right inside the largest banks, because a bank knows more about its own risks than outsiders do. Anthropic itself notes that the analogy is not exact.

Three tiers of access

Anthropic describes three tiers for outside evaluators.

  1. First. Narrowly scoped investigations, access decided by Anthropic, publication with its approval.
  2. Second. More information, publication without the company's approval. Anthropic can require information from the agreed categories to be removed, and the evaluator can say publicly whether it considers those redactions justified.
  3. Third. As with METR: "employee-like" access and publication rights like those of the second tier.

Besides METR, Anthropic names SecureBio: according to the company's post as of 15 October 2026, this organisation is reviewing the chapter of its August risk report on chemical and biological weapons. With Accenture and Faculty the company is building a partnership to evaluate and test models. About Hugging Face and several other organisations, Anthropic writes that it is in early discussions.

Where the limits are

  1. The company has the last word on redactions. Anthropic decides what belongs to the closed categories. METR's counterweights are these: it can say what access and what information it received, and, as an evaluator with second-tier rights, it can publicly assess whether a redaction was justified.
  2. There are no standards yet. TNW wrote on 19 September 2026 that, according to Anthropic itself, there are no common rules yet on what an outside evaluator should see and how it must report what it finds.
  3. Money and interests. According to the FT, Anthropic is not paying METR for this work. Accenture is different: according to TNW of 19 September 2026, Anthropic pays Accenture for this work itself and, in the same September announcement, writes that in future such reviews should be funded from pooled or government sources, which do not exist yet. According to TNW, Accenture is at the same time an Anthropic customer, a reseller of its products and an implementation partner. Critics, according to the FT, point to conflicts of interest among auditors in accounting and argue that greater government regulation is required.
  4. For now this is a first step. The FT calls the announcement the first step towards implementing the pledges on outside evaluators. According to the FT as of 18 October 2026, OpenAI has not yet set out how it will bring evaluators in, although Sam Altman called Dario Amodei's proposals a good idea.

I did not find any METR publications from this work as of 19 October 2026. How useful such a review is will become clear from the first reports.

What someone choosing an AI vendor can take from this

These three steps are for whoever in the company chooses a model or a service built on it.

  1. Before choosing, ask the vendor: who has independently evaluated the model you will be using, and where the published reports of that evaluation are.
  2. How to check: open the report on the evaluating organisation's own website, not in the vendor's retelling. Check three things: the name and version of the model, the date of the report and what access the evaluator received, if the report says so.
  3. If there is no report or it is about a different model: treat statements about safety as the vendor's own claims, and test the model on your own tasks with anonymised test data before letting it near real data.

Summary

Anthropic let six people from METR inside the company with access close to that of its own risk assessment teams and the right to publish findings without its approval. There are limits too: some information is closed, the decision on redactions stays with the company, and there are no common rules for such reviews yet. What this arrangement is worth will be shown by METR's first published reports.

See also another piece in the "AI" section.

I work on AI agents and automation. If you would like to see my projects or discuss your own task, take a look at my portfolio.

Sources