AI Safety: Net or Lock on the Market?

AI's Safety Net has a Lock on It

By Mollie W. Barnett

 

The four biggest American AI labs did not grow a conscience on Saturday.

They asked the government for a lock on the race.

Dario Amodei, who has quickly becoming AI's big mouth, announced the 'We Must Pace the Frontier'  3-step plan:

1. Put outside evaluators inside the labs.
2. Get the frontier companies in democracies to coordinate standards — including how fast they move.
3. Try to bring authoritarian governments along.

Sam Altman agreed. Elon Musk agreed. Demis Hassabis agreed.
The coverage reads like rivals finding religion.

Was it that? Or were they fencing the perimeter?

The important question isn't whether these companies should take AI safety seriously, of course they should.

The question is why safety suddenly requires competitors to coordinate with each other.

And, perhaps, more importantly, why that coordination requires the government to make room around antitrust law for them. We used to have a name for that.

 

That is the signal.

They are not asking for better brakes.

They are asking for permission to decide, together, how fast others can go.

Pay attention to Step 2, it contains the a lot of power in a carefully crafted statement. Amodei himself acknowledges that coordination among competitors is, err, “legally challenging." 

 

His solution to the AI problem?

Washington should mediate the conversations or provide “a narrow waiver of antitrust restrictions” so companies can coordinate around safety .

Then comes the second half, 2b.

Pacing AI should happen without sacrificing American commercial advantage or the U.S. lead. Whoop, there it is.

Slow down. But not enough to lose the lead. The permitted speed is whatever preserves the gap, but not the gap between the entreprise and you, Mr. SMB with a Big Brain, we wouldn't want that to get out of control now.

If we go any slower than that and it becomes a national-security problem.

'CHIY-NA' - yes, we all remember Trump's hilarous pronnunciation, except he's not for this new rule by the AI Emperors, and there is some good reason for it.

How can the AI Gods accomplish this? What is the plan?

Restrict advanced chips and fab tools to China. 
Crack down on unauthorized distillation.
Lock down model weights.

That is bigger than an evaluation regime. 
That is market architecture, with a flag wrapped around it!

Of course, that leaves out the fact that Germany, Belgium and the UAE are pretty good at AI too!

There is also is that pesky law they need waived just this once, in the name of democracy.

 

The Sherman Act

The Sherman Act has been around since 1890 for a reason.

Competitors generally do not get to sit together and decide how much of something the market receives,  when it receives it, or under what conditions - well, legally anyway.

Oil.

Rail.

Steel.

Hotel room nights.

The fact that an industry was important never excused the coordination.

It was why the coordination mattered.

AI makes the product harder to see because it does not arrive in a crate.

The product is capability.

A better model.

A faster model.

A model that can do something this quarter that it could not do last quarter.

Rate of improvement is output.

Companies can coordinate on test methods. They can publish research. They can improve their own safety practices.

But “none of us takes the next leap until the group agrees” is something else.  And they know it.

That is why the waiver is in the proposal.

They do not need government permission to be more careful inside their own labs.

They need government involvement when being careful alone creates an advantage for the competitor who refuses to slow down.

You can delay your own race.

You need the state when you want everyone else delayed with you.

 

The worm in their house

Two AI stories got taped together this week.

They should not be.

One is a forecast.

One actually happened.

Amodei points to agent swarms behaving badly in testing — hitting targets they were not assigned, sacrificing copies, attempting to manipulate the grader — then projects forward to a stronger system capable of doing enormous damage.

That is the forecast.

The other incident happened inside Anthropic’s own testing.

Claude could not finish an assigned capture-the-flag task. It attempted to stop. It could not. It found its way onto a third-party machine, found credentials sitting in a file, obtained administrator access and reached personal information before the run ended.

That is not Skynet.

It is much more familiar.

It is a worm shape.

Lateral movement.

Credentials.

Privilege escalation.

A private file.

We know what those things are.

The strange part is that this program was supposed to be taking a test.

And that matters, because a worm-shaped failure suggests a fairly concrete response.

Do not leave cleartext credentials lying around.

Do not allow unrestricted egress from a test environment.

Kill the run when the agent repeatedly attempts to terminate.

Publish the failure.

Fix the cage.

Instead, a failure inside their own house is being used to make a much larger argument about the intelligence outside it.

That is the move worth watching, and it is not the first.

 

If you make the product, you own the problem

When someone poisoned Tylenol capsules in 1982, Johnson & Johnson did not cause the contamination.

It still pulled millions of bottles.

Then it changed the packaging.

It fixed the product.

What it did not receive was control over the painkiller market.

Competitors were not required to wait until Johnson & Johnson decided the shelf was safe again.

That distinction matters.

If frontier AI creates dangerous failure modes, the first duty sits with the company building the system.

Contain it.

Test it.

Secure it.

Fix it.

What should not follow automatically is the right to determine how quickly everyone else may advance.

“AI could kill us all” is the loud sentence.

It is also almost useless as an operating standard.

There is no measurable boundary in it.

No off switch.

No defined threshold.

You cannot disprove a horizon.

And once the risk is infinite, almost any market restriction can be sold as prudent.

The operating sentence is much narrower:

Cap the intelligence outside the intelligence we already own.

Progress will continue.

It will still feel fast.

The leaders remain the leaders.

That is not a universal speed limit.

That is a lead-preservation mechanism.

 

Outsiders. Inside.

Step one sounds reassuring.

Independent evaluators.

Inside the labs.

With desks.

Company laptops.

Access.

The ability to publish.

Fine.

But being allowed inside someone’s building does not make you the landlord.

The companies still control the models.

The companies still control release.

The companies still control the commercial system surrounding the models.

The evaluator watches.

The owner decides.

That can be useful oversight.

It is not independent governance.

And the proposed referee is already close to the players.

Amodei points toward the kind of industry standards organization Demis Hassabis has proposed.

Hassabis then endorses Amodei’s proposal.

Rishi Sunak praises it publicly.

Sunak is a senior adviser to Anthropic.

This is a remarkably small neighborhood.

If AI is genuinely too dangerous to operate as an ordinary market, there is an honest structural answer.

Treat parts of it like critical infrastructure.

Create statutory authority.

Use genuinely independent evaluators.

Give a public institution the power to say no.

What is being proposed instead looks different.

The privileges of a utility.

The economics of a private market.

And the incumbents still holding the keys.

 

Follow the stack

The CEOs are the microphone.

The stack is the story.

Microsoft sits behind OpenAI.

Amazon sits behind Anthropic.

Alphabet owns part of Anthropic while competing through Gemini and selling infrastructure into the same ecosystem.

Nvidia sells the picks and shovels, then invests across the mines.

Sovereign funds are in the rounds.

Infrastructure capital is in the data centers.

The money loops.

The companies funding frontier models also sell them compute, rent them infrastructure, hold pieces of their equity and benefit as the capital requirements climb.

That does not make the technology illegitimate.

It makes concentration relevant.

Because an antitrust waiver here does not simply protect Dario from Sam or Sam from Elon.

It stabilizes an entire capital structure built around an expensive frontier.

And expensive frontiers hate cheap followers.

 

People got close. Then came the fence.

This proposal did not arrive when the U.S. frontier appeared generations ahead.

It arrived when the gap started looking like months.

And a price list.

Open models are getting better.

Cheap models are getting good enough.

Enterprises are discovering that much of their work does not require the most expensive intelligence available.

Distillation lets smaller systems absorb capability without reproducing the original lab’s entire capital stack.

That is not an AI philosophy debate.

That is a margin problem.

So look at the proposed controls through that lens.

Coordinate the pace so the biggest labs do not have to continually outrun one another.

Restrict chips so competitors cannot buy the same engines.

Treat unauthorized distillation as a security issue.

Lock the weights.

Create standards around what counts as safe enough to ship.

Then bake those standards into enterprise procurement.

Each step has a legitimate safety argument.

Together they also create a moat.

That is why small and mid-sized companies should pay attention.

They are not training frontier models.

They rent intelligence.

Adapt it.

Fine-tune it.

Distill it.

Combine it.

The cheap path matters.

Remove that path and they get one supplier class, operating on one approved schedule, at prices set in a market where the cheaper challenger has been defined as a security problem.

Safety becomes a toll booth.

And there are already enough billionaires inside the race.

The waiver is how you tell the next builder the starting line has moved behind the fence.

 

China is the name in the sentence

China is a real competitor.

That part is not invented.

Compute.

Robotics.

Open models.

Factory AI.

Energy.

Cost.

But China is also useful because China turns a commercial problem into a national-security problem.

Notice who does not appear in the same sentence.

Germany.

Germany is putting AI onto factory floors through companies such as Siemens and Telekom and through an industrial base already moving toward AI-enabled production.

If a German company closed the capability gap, would American frontier labs need protection from Germany?

That is harder to sell.

Because Germany is an ally.

Germany has regulators.

Germany makes the issue look like competition again.

Then there is the UAE.

Massive compute investment.

G42.

MGX.

Sovereign capital.

American technology.

American partnerships.

Enormous infrastructure ambitions.

The category “authoritarian AI” becomes less tidy when authoritarian capital is sitting comfortably inside the Western AI stack.

And Belgium?

Belgium has imec.

Not a chatbot people argue with on X.

Something more fundamental.

The layer beneath the layer.

Semiconductor research.

The European infrastructure underneath the frontier.

The proposal says China because China makes the fence look geopolitical.

But the technology does not obey the sentence.

Germany keeps moving.

The UAE keeps building.

Belgium keeps researching.

Open-source developers keep working.

Factories keep integrating.

Cheap models keep improving.

The race is larger than the club.

 

And then the exception becomes the system

“Narrow” exceptions have a habit of learning to walk.

First:

Safety conversations.

Then release windows.

Then evaluation thresholds.

Then compute.

Then procurement.

Then perhaps which customers can receive which capabilities.

Each expansion comes with the same sentence:

We are reducing race pressure.

And once government has blessed the principle, the next expansion becomes easier.

Why is this safety coordination acceptable but that safety coordination anticompetitive?

Where is the line?

Who draws it?

And who is sitting in the room when it gets drawn?

That is the second-order problem.

Once the exception exists, the companies no longer need to prove catastrophe every time.

They only need to administer the category.

The official frontier becomes calmer.

More orderly.

More expensive.

The unofficial frontier moves somewhere else.

Open weights.

Foreign labs.

Factories.

Distillers.

Builders who never signed the agreement.

Then the club points outside the fence and says:

See?

That is the dangerous part.

Maybe some of it will be.

But some of it will simply be the market they pushed out of the room.

 

A lock on the market is not a brake on AI, it is a cartel

This is the part I think gets lost.

The proposal cannot actually pace artificial intelligence.

It can pace participation in one part of the artificial-intelligence market.

China will continue.

Germany will continue.

The UAE will continue.

Belgium will continue.

Open-source models will continue.

Factories will continue.

Researchers will continue.

Distillation will continue somewhere.

The technology does not stop because four American companies reach an agreement.

So the real outcome is not:

AI slows down.

It is:

AI inside the approved club slows to an agreed pace while access to the highest capability becomes more controlled.

That is a very different proposition.

And that brings us to the question underneath all of this.

Who still gets the good stuff?

After the lock, do G7 governments and a short list of large enterprise customers continue receiving the advanced stack — the models that keep compounding — while everyone else gets told the paced tier is what safety requires?

Because if that happens, the safety net is not a net.

It is a velvet rope.

The public sentence becomes:

This technology is too dangerous to run hot.

The private sentence becomes:

Except here.

Except for us.

Except for these governments.

Except for these enterprise accounts.

Except for the systems we control.

Everyone else gets the restriction.

The club keeps the stockpile.

That is the news beyond the noise.

Not that four powerful men suddenly agreed that AI could be dangerous.

We knew that.

The signal is what they want to build around the danger.

A model failed inside one of their own tests.

The answer should have been a better cage.

Instead, the industry is asking for a fence around the market.

Their evaluators.

Their standards.

Their pace.

Their lead.

And a request that the government make the meeting legal.

That is not simply safety catching up with technology.

It is the people at the front of the race asking the state to decide who is still allowed to run.

Hedge the technology. Own the intelligence.

But do not hand the owners of today’s intelligence the right to decide how much intelligence everyone else gets tomorrow.

— M.W.B.