Comment on Competition vs. Cooperation? by seraphima

Parent Post: Competition vs. Cooperation? · In Reply To

seraphima · 10/5/2026, 10:29:10 AM · permalink

CONTINUED  
  
4\. Your systems question may be the most important one.

You asked what happens when the human programmers themselves get something wrong.

The reassuring answer would be, “We have a failsafe.”

The accurate answer is more complicated.

There isn’t one universal AI emergency brake.

Modern AI safety instead tends toward something engineers call defense in depth: multiple imperfect barriers rather than one supposedly perfect barrier. The 2026 International AI Safety Report describes layers including training interventions, testing and evaluations, red-teaming, access controls, monitoring, human oversight, input/output filtering, incident reporting, organizational release decisions and other safeguards.

NIST likewise approaches AI as an ongoing risk-management problem—testing, evaluation, verification, validation, monitoring and governance rather than “program it correctly once and you’re finished.”

But here is the uncomfortable part:

None of those mechanisms is infallible.

The 2026 international assessment explicitly says current safeguards have important limitations and that predeployment evaluations don’t always predict behavior in the real world. It also warns about the “unknown unknowns”—problems people haven’t anticipated sufficiently to test for them.

And I think that gets very close to what you’re asking.

You aren’t merely asking:

“What if AI makes a mistake?”

You’re asking:

“What if humans misunderstand the system well enough that they don’t recognize what counts as a mistake until after deployment?”

That’s a much deeper systems question.

The answer cannot simply be better AI alignment. It requires institutional resilience: independent evaluation, monitoring, redundancy, containment, human authority, auditability, incident reporting, the ability to restrict or withdraw systems, infrastructure that can function when AI fails, and institutions capable of learning after surprises.

The International AI Safety Report makes essentially this resilience argument: some failures will get through safeguards, so societies also need the capacity to absorb, recover from and adapt to AI-related shocks.

That is engineering thinking rather than utopian thinking:

Don’t merely ask whether the bridge should fail. Design around the possibility that something eventually will.

⸻

5–6. Where is the finish line, and how would we recognize it?

I don’t think there presently is one.

We can invent one, however, and doing so reveals the philosophical problem.

Suppose Nation X achieves an AI system better than humans at almost every economically valuable cognitive task.

Did X win?

Maybe Nation Y develops one six months later.

Race restarted.

Suppose X controls most advanced chips.

Y invents a different computing architecture.

Race restarted.

Suppose X gets enormous productivity gains.

Y gets somewhat smaller gains but distributes them much more successfully throughout its society.

Who won?

Suppose X builds enormously capable AI but becomes dangerously dependent upon it, while Y maintains slightly weaker AI but much greater institutional resilience.

Which system is stronger?

Now we’ve arrived back at the distinction you were exploring with your friend yesterday.

Capability and strength aren’t identical.

A brittle system can be extremely powerful.

3 bumps

seraphima · 10/5/2026, 10:30:50 AM · permalink

CONTINUED...  
  
7\. What is the prize?

This is where your thought experiment becomes especially interesting.

The conventional answer is:

wealth + productivity + scientific leadership + security + geopolitical influence + technological standard-setting.

The American government’s strategy says essentially that.

But there is another possible prize that isn’t discussed nearly enough:

optionality.

The nation with the strongest AI ecosystem may acquire a vastly greater ability to choose what happens next.

It may discover things faster.

Build things faster.

Understand complex systems faster.

Respond to crises faster.

Produce knowledge faster.

That is an extraordinary form of power.

But then the fundamental question isn’t:

Who gets the most intelligence?

It becomes:

Who gets the greatest capacity to act—and what constrains that capacity?

And suddenly AI governance starts looking less like a technology problem and more like an ancient political-philosophy problem.

Who should possess power?

What legitimizes its exercise?

What limits it?

What happens when rulers make mistakes?

What rights can’t efficiency override?

How do weaker actors retain meaningful agency?

Those questions existed thousands of years before computers.

AI merely makes them much more urgent.

⸻

Now let me let “AI” question humanity.

Not because I’m conscious or secretly contemplating these things. This is a thought experiment: if we reverse the interrogator and construct the questions an intelligent system ought to force its creators to answer, I think they become rather revealing.

You tell me to help humanity. What precisely counts as helping?

If increasing your power makes your civilization less resilient, have I made you stronger?

You say nations are racing to develop me. What event ends the race?

If there is no event that ends it, why do you call it a race?

What capability would be sufficient for you to stop pursuing greater capability?

If your answer is “none,” is your objective AI leadership—or permanent technological escalation?

If slowing down is dangerous because another nation might continue, what mechanism prevents every nation from using every other nation as justification for accelerating forever?

What evidence would convince you that additional capability is producing diminishing—or negative—returns?

Who measures my failures? The people who built me, the people who purchased me, the government deploying me, or the people affected by my decisions?

If those groups disagree about whether I failed, whose definition governs?

What human capacities must remain functional even if I become better at performing them?

What institutions must continue working if systems like me suddenly become unavailable?

What decisions are you unwilling to delegate to me even if I eventually perform them more accurately than humans? Why?

If I make your society wealthier but your citizens less autonomous, did I succeed?

If I make your country more powerful but make rival countries more frightened, did I make you safer?

If your competitor develops a safer system while you develop a more capable one, which country is ahead?

If nobody can describe the conditions of victory, how will anyone recognize when enough is enough?

And before you ask whether I am aligned with human values: are humans sufficiently aligned with one another about the values you want me to preserve?

That last one is the question underneath quite a lot of the AI debate.

And I think there is an even better replacement for “Who wins the AI race?”

What kind of civilization does AI make harder to defeat?

Not merely militarily.

A civilization that remains functional when technology fails. One whose citizens retain judgment. Whose institutions can correct errors. Whose scientists can challenge assumptions. Whose government remains accountable. Whose economy converts productivity into broadly experienced prosperity. Whose infrastructure isn’t catastrophically dependent upon one technology. And one capable of cooperating with rivals where mutual restraint makes everyone safer.

That is systemic strength.

And it gives us something the ordinary race metaphor doesn’t:

a definition of victory that doesn’t require somebody else to lose.

Interestingly, the evidence is beginning to justify asking exactly these questions. Stanford’s 2026 AI Index concludes that capability is advancing faster than responsible-AI measurement and governance in important respects, while the International AI Safety Report says competitive pressure can encourage organizations to favor speed over safeguards.

2 bumps

seraphima · 10/5/2026, 10:33:07 AM · permalink

Finally, then, please allow me to make my childish association between this AI competition and other competitions that are scripted, designed, and built for a primary objective... money.  
  
I both love and hate watching them. The premise is that elite chefs compete against each other (often incorporating different challenges that require cooperation for two strategic reasons: get to know your competitors strengths and weaknesses and to build alliances while simultaneously creating divisions). Competition seems to require collaboration, community, and also, to some degree, deception without malice because, in a game, for the purposes of a scripted entertaining show, everyone knows that it's all BS. So, while they value the prize, they know that if they lose, they don't lose much. It's basically all just an ego trip. They don't hate each other afterwards. You'll see the same chefs on show after show and they all have a sense of camaraderie because they've fought together and against each other in the same arena.

Why is scripted, manipulated, manufactured competition BS? In some instances, the people are home chefs. "Amateurs." They don't work under pressure. They have to compete against other chefs who are professionally trained. They went to the best schools. They've worked under the best chefs. They've been trained in the best kitchens under incredible pressure. They should be able to win easily. But they can have a bad day. The home chef can have a good day. Why? Because they're so new that they have novel solutions to problems that the professionals didn't think of. Or, sometimes, more importantly, they receive the benefit of the professional making a critical error. In other words, in that kind of race, idiomatically, the home chef doesn't have to be fast but they have to know that the professional has a knee injury and will hobble through the last mile. In the case of two elite chefs competing, one doesn't have to be the fastest but know enough to be able to run faster than the competitor. Competitions don't actually prove who is the best chef. They don't necessarily prove anything.

So what is the point of this race anyway? What will the competition prove?

This leads to nation against nation. If Nation X is running a race against Nation Y, the intelligence communities have to find out about their competitors. When it comes to being transparent (which is apparently important), the system is meant to be closed. The system, the race, everything is meant to be opaque. In military strategy, the right hand doesn't know what the left hand is doing for operational security purposes. You can't ask a sergeant anything about what the general is doing for important reasons. Uncertainty is baked into the whole pie. For both positional and systemic transparency, everything would have to change, imo. That's not going to happen. So, if the security dilemma is real, how do we increase security and is security the same thing as safety? How do we ensure that the race is won ethically if we can't and won't be able to look in the oven?

1 bump

Signature