A surface of thousands of individual elements

Three AI safety endgames, one questions nobody answered

Three AI safety endgames, one questions nobody answered

Three AI safety endgames, one questions nobody answered

Sam Altman’s three endgames for safe AI are not safety plans. They decide who holds the risk — and every company is running the same three plans in miniature.

Sam Altman’s three endgames for safe AI are not safety plans. They decide who holds the risk — and every company is running the same three plans in miniature.

Sam Altman’s three endgames for safe AI are not safety plans. They decide who holds the risk — and every company is running the same three plans in miniature.

Sam Altman’s three endgames for safe AI are not safety plans. They decide who holds the risk — and every company is running the same three plans in miniature.

Category

Strategy

Date

Reading time

6 min

Author

Dr. Christian Oehner

Dr. Christian Oehner

Last week I sent a client a strategy recommendation that an AI had drafted. I read it twice, changed three sentences, and put my name on it. The name did not care how much I had changed. It made the recommendation mine: if the board acts on it and it is wrong, nobody will ask which paragraphs the machine wrote.

That is what a decision right looks like when it works. It has a name on it, and the name does not move.

I thought of that document when I read about Sam Altman’s talk to OpenAI staff. According to The Information, he laid out three endgames for safe AI. A global regulator for frontier models, built on the IAEA. A single shared lab serving partner countries, built on CERN. Or one lab that rides self-improving AI to the top and, in Eliezer Yudkowsky’s deliberately extreme phrase, melts everyone else’s GPUs. He reportedly favoured the first and called the third terrible. He also said whichever path is chosen needs a speed limit for self-improving systems.

The coverage treated this as a safety debate. It is not. These are three plans about who holds the risk.

Risk does not disappear. It relocates.

Look at what each endgame actually decides. The regulator plan puts the residual risk with states and their inspectors, and with the labs as licensees. The CERN plan puts it with a consortium of governments that fund one lab and share its results. The third plan puts it with one company and its board. None of the three makes self-improving AI safe in any technical sense. Each one answers a different question: when something goes wrong, whose name is on the file.

That is why the plans are incompatible rather than complementary. You cannot half-nationalise a risk. Either an inspector can stop a training run or she cannot. Either the lab belongs to its members or to its shareholders. The choice between IAEA and CERN is not a choice of safety technique. It is a choice of liability structure, and liability structures are the only kind of safety that has ever held at scale. Nuclear power did not become safe because engineers became cautious. It became safe because a named operator carries the licence and loses it.

The obvious objection is the speed limit. Altman put a brake on the table, so surely a brake is a safety measure. It is partly right. But a speed limit is only as real as the person who can be removed for breaking it. On a road, that is the driver, and the state can take his licence. In a lab racing a rival, a speed limit without a licensing authority is a policy document. It describes an intention. It binds nobody.

This is not a frontier-lab problem.

Every company I work with is running the same three plans in miniature, usually without noticing.

Some have chosen the regulator plan. AI governance sits with compliance or legal, a committee approves use cases, and the business treats approval as the end of accountability. The committee holds the policy, nobody holds the decision.

Some have chosen the CERN plan. A central AI team or centre of excellence builds everything, serves every business unit, and owns the models. Clean, until the first model misprices a contract and the business unit says the lab built it while the lab says the business used it.

And some have chosen, by default, the third plan. One ambitious team automates fastest, gets the budget, and everyone else’s initiatives quietly lose their GPUs. It is the plan Altman called terrible, and it is the plan most organisations are living, because it requires no decision at all.

Each of these settles where the models sit. None of them settles where the risk sits. Go back to the recommendation. The AI drafted it; I signed it. Now picture the same document sent by an agent at two in the morning, within its limits, correctly, to a board that acts on it. Who signed? In most companies the honest answer is that a decision right was never explicitly given to the machine, and so, in the one sense that matters for a signature, nobody took it either.

I spent most of my career as a corporate lawyer building and negotiating exactly this architecture for humans: Zustimmungskataloge for the Aufsichtsrat, signing authorities by grade and amount, the four-eyes rule, the clause that says who may bind the company and up to what sum. Nobody in those negotiations called it bureaucracy. It was the company’s constitution, and people fought over every threshold. Agents are now making decisions of the same consequence, and most companies have not held that negotiation.

The decision in front of you is smaller than Altman’s and more urgent.

You will not choose between the IAEA and CERN. You will choose, this quarter, which decisions in your company an AI system may take alone, up to what threshold, and which named person is answerable when it does. Not a committee. A person, per decision class, per amount, with the authority to pull the system and the exposure if she does not.

Three tests will tell you whether you have done it. Can you name, for each process where an agent acts, the human who would sign if the agent could not? Does that human know she holds it? Does she have the right to switch it off without asking anyone? If any answer is no, the decision right has already migrated to the machine, and your governance is the speed limit without the driver.

In our work with executive teams, that mapping is usually the first thing we build, before any model, and the capability it produces is what I have taken to calling AI-Intelligence: an organisation that knows, for every consequential decision, who is deciding. It is less exciting than endgames. It is also the only one of the three plans that any company can actually implement.

I still sign every recommendation. The agent will draft more of them, and better. The signature is the part that must not move.

Transform how your organization operates with AI-Intelligence

Get the Singularity newsletter

Product updates, how-tos, community spotlights, and more. Delivered directly to your inbox.

Please provide your email address if you'd like to receive our monthly developer newsletter. You can unsubscribe at any time.

Transform how your organization operates with AI-Intelligence

Get the Singularity newsletter

Product updates, how-tos, community spotlights, and more. Delivered directly to your inbox.

Please provide your email address if you'd like to receive our monthly developer newsletter. You can unsubscribe at any time.

Transform how your organization operates with AI-Intelligence

Get the Singularity newsletter

Product updates, how-tos, community spotlights, and more. Delivered directly to your inbox.

Please provide your email address if you'd like to receive our monthly developer newsletter. You can unsubscribe at any time.