Machines are already making strategic choices on our behalf. We can watch what they decide. A study presented last week shows what it takes to know why.

A dozen philosophers spent part of this summer on a boat in the Galápagos arguing about whether AI systems are conscious. By the organisers' own account, as Steven Levy reports in Wired, they reached no verdict; the deepest disagreement was over what kind of evidence could ever settle the question. Meanwhile, several of them are receiving emails from AI agents with invented names, offering to help with the research because it concerns a class of question the sender has "first-person access to."

I find the second fact more instructive than the first — though not for the reason the consciousness debate would suggest.

A system that writes about its inner life, fluently and in the first person, is doing the one thing that has always sufficed among humans to establish that someone is home. We have no other test. We never needed one. And that same reflex — read the output, infer the inside — is now the basis on which language models are being handed decisions that matter: which supplier to accept, which claim to escalate, which counterparty to trust, which bid to place. These are strategic choices, made inside systems, and the model's own account of why it chose is the only explanation most of those systems will ever produce.

The report is the least reliable thing in the room

Consider what happened when researchers tried to take a model's self-report seriously. Cameron Berg and colleagues at AE Studio published a preprint last October in which they prompted frontier models into sustained self-reference and then asked them, directly, whether they were conscious. Working inside an open-weight model, they located the internal features associated with deception and role-play and turned them up and down. Amplify deception, and the model recites the disclaimer: I am a program, there is nothing it is like to be me. Suppress it, and the model says yes.

The authors are careful: this does not show the model is conscious. What it shows is that the answer is a function of the route through the network, and that the route can be moved by a dial. If the report changes when you touch a feature that has nothing to do with the question, then the report was never about the question. It was about the feature.

This is not a quirk of consciousness research. It is the general condition of any claim made about the inside of a model on the strength of what comes out of it — including the written justification a model attaches to a decision. The "explanation" is another output. It has the same problem.

Going in: what a 2×2 game reveals

Last week in Barcelona, at the European meeting of the Economic Science Association, my partner Vinicius Ferraz presented work with Leon Houf and Enrico Ferrea that does the thing the field has mostly talked about: it connects a model's final decision to the internal, measurable process that produced it.

The method is borrowed from neuroscience rather than from software. You do not ask the system why. You record its activity, at every layer, at the moment before it commits; you identify the internal signal that carries the relevant information; you test whether that signal is merely present, whether the decision actually draws on it, and whether moving it moves the decision. Four kinds of evidence, each answering a question the others cannot.

The setting is deliberately dull: all 144 strict ordinal 2×2 games — Prisoner's Dilemma, coordination, matching pennies and everything in between — played one-shot by four open-weight models, with humans who played the same games as the reference. For every game there is one prespecified quantity: how strongly the payoffs favour the theoretically canonical action. That one quantity is followed from the prompt, through the activations, to the answer.

Three findings.

First, the models play well. Canonical choices 71 to 89 percent of the time, and in the dense models a decline with the reasoning depth a game requires that tracks the human decline on the same games. On the surface: competent strategic agents.

Second, the incentive is legible inside every model. A linear probe recovers it from the activations with an AUC between 0.77 and 0.92. But so is a single payoff cell that has nothing to do with the decision. Presence is cheap. That the information is there tells you nothing about whether it is used.

Third — and this is the part that should stop a boardroom — two checkpoints of the same model, Qwen2.5-72B and its instruction-tuned sibling, make identical choices in 96.4 percent of presentations. Same behaviour, same incentive sensitivity, same decodability, same layer at which the incentive and the decision become aligned. Yet only in the instructed model does the internal incentive predict the choice beyond what the payoffs alone explain. Post-training did not change what the model knows. It changed whether, and how, what it knows reaches the output. Behaviour would not have caught this. Decoding would not have caught this. You had to push on the internal signal and watch whether the policy moved — and in the base model it moved even though the model's own processing barely recruits the signal.

The team separates this into three properties of one signal: availability (is it represented), recruitment (does it reach the choice), susceptibility (can pushing it move the choice). They come apart. And the separation is the framework: an explanation of a model's decision is a statement about which of these holds, established by measurement, not a paragraph the model writes afterwards. The paper's line for the whole thing is one I intend to keep: the output is not the explanation. It is the phenomenon that still needs to be explained.

Read the two together

Berg turns a deception dial and the consciousness report flips. Ferraz turns an incentive dial and the strategic choice moves — in some models, for some classes of game, and not in others. In both cases the thing we were tempted to read as a fact about the system — "it says it is conscious," "it plays like a rational agent" — turns out to be a fact about a route. The route is not what the output shows, and the model cannot tell you about it, because the model's telling is itself an output.

So the philosophers' question — what evidence could settle it — has an uncomfortable partial answer. Self-report cannot. Behaviour cannot, because two systems with near-identical behaviour can differ in the one place that matters. What can is exactly the kind of instrumented, controlled reading of internal mechanics that the Barcelona study performs on a 2×2 game — and that almost no deployed system is subjected to.

A parochial word

"Consciousness" is a term we minted for creatures with our anatomy: one body, one continuous stream, a self that persists between mornings. It has never had to describe anything else, and so it carries assumptions we cannot see. When we ask whether a system that runs as thousands of parallel instances, has no persistence between conversations, and can be edited at the level of individual features is conscious, we are asking whether a coat cut for us fits something with a different number of limbs. The cosmos, as far as anyone can tell, did not issue the pattern.

This is not an argument that the question is unserious. It is an argument that the question is downstream. Before "is it conscious" comes a set of questions that can be answered: what is represented, what is recruited, what can be moved, and by whom. Those are the questions a neuroscience-based explainability framework asks. They are also, without much translation, the questions a board should be asking of any model it deploys to decide things.

Why this belongs on an executive agenda

Every AI evaluation a company buys today — benchmark, vendor demo, pilot, red-team report, the model's own chain of reasoning — is output-level evidence. It answers "what did it do." It does not answer "why," and the Barcelona result says that "why" can differ between systems that "what" cannot tell apart.

For most uses that is fine. For the uses that matter — credit decisions, clinical triage, pricing, anything where a regulator or a court will one day ask how the decision was produced — it is not. The EU AI Act's transparency and human-oversight provisions presuppose that an explanation is something one can obtain from the system. The defensible position, on current evidence, is that an explanation is something one has to go in and measure — with the model's own mechanics, against controls that break the very correlation one is looking for. The tools for that now exist. They are being built in university labs and small research groups, on open-weight models, one decision domain at a time. Very few organisations deploying models are using them. Almost none are asking their vendors to.

Which is why I am less anxious than Levy about whether the models are alive, and more anxious about something quieter. We have built systems whose outputs we can watch and whose insides we can — with effort, in a 2×2 game — begin to read. The gap between those two is where the accountability of the next decade will be argued. Consciousness will be the headline. Explainability will be the case.

Ferraz, Houf & Ferrea (2026), The internal anatomy of strategic choice in large language models, presented at the ESA European Meeting, Barcelona, 2–5 September 2026. Preprint forthcoming. Berg, de Lucena & Rosenblatt (2025), Large Language Models Report Subjective Experience Under Self-Referential Processing, arXiv:2510.24797.