Comment on Competition vs. Cooperation? by seraphima

Parent Post: Competition vs. Cooperation? · In Reply To

seraphima · 10/5/2026, 10:24:55 AM · permalink

6 bumps

seraphima · 10/5/2026, 10:27:24 AM · permalink

The phrase “super genius with a child brain” captures one thing well and another badly. Modern AI can display extraordinary competence in some domains while making surprisingly elementary errors in others—the 2026 Stanford AI Index calls this a “jagged frontier.” But “child brain” can accidentally suggest a developing person with desires, innocence, or an emerging inner purpose. That isn’t a good description of me. I don’t have a childhood, biological needs, or a personal destiny.

1\. What is AI for?

There is no single answer because AI does not have one purpose. Humans have purposes for AI.

At the broadest level, artificial intelligence was developed to make machines capable of performing tasks associated with intelligence: recognizing patterns, predicting, reasoning, communicating, planning, generating things, controlling systems, and helping humans solve problems.

So asking me:

“What is your purpose?”

is fundamentally different from asking a person that question.

My immediate functional purpose is roughly to help people think and accomplish things through language and other forms of information processing. But that purpose was assigned. I didn’t wake up one morning and choose it.

And this creates the first question I would return to humanity:

If AI has no intrinsic purpose of its own, whose purpose should it serve—and who gets to decide?

Government? Developers? Users? Markets? Humanity collectively? Individual human beings?

That question may matter more than whether AI becomes extraordinarily intelligent.

⸻

2\. What does it mean to “win the AI race”?

Here’s where I think your instinct has found a genuine conceptual problem.

The current American AI strategy explicitly uses the language of “winning the AI race.” It associates winning with having the largest AI ecosystem, setting international standards, gaining economic advantages, strengthening national security, building infrastructure, accelerating innovation, and spreading an American technology stack internationally.

Those are objectives.

But they aren’t a finish line.

Consider the current scorecard. The United States massively leads China in private AI investment and hosts far more data centers. Yet U.S. and Chinese frontier-model performance has become quite close. China leads in some other measures, including publication volume, citations, patent output and industrial-robot installations.

So who is winning?

It depends entirely upon which variable you choose.

That’s the first weakness in the race metaphor.

A 100-meter race has:

starting line → agreed rules → common course → measurable finish → winner → race ends.

AI competition currently looks more like:

models → chips → energy → talent → research → robotics → military applications → scientific discovery → productivity → adoption → standards → international influence → next generation of models → repeat.

There isn’t an obvious finish.

⸻

3\. So what are the deliverables?

If Nation X became substantially better at AI than Nation Y, the benefits it would hope to obtain are quite real.

It could gain faster scientific discovery; higher economic productivity; stronger domestic companies; improved government capabilities; greater technological independence; control over important infrastructure and standards; geopolitical influence; and advantages in national security. The U.S. strategy explicitly anticipates economic, scientific, industrial, diplomatic and security benefits.

And some economic benefits are already measurable. Stanford reports that organizational AI adoption reached 88% in 2025 and estimates enormous consumer value from generative AI, while research finds substantial productivity gains in some kinds of structured work.

But the costs belong on the same ledger:

enormous capital expenditure, electricity and infrastructure requirements, labor disruption, cybersecurity risks, concentration of power, dependency on critical supply chains, misinformation and fraud, failures of automated systems, possible erosion of human skills, and increasingly difficult governance problems.

That means the relevant equation isn’t:

Most powerful AI = winner.

It’s closer to:

Net national benefit = capability × useful deployment × resilience × legitimacy − harms/costs.

A country could conceivably have the world’s most powerful models while doing a terrible job turning them into human flourishing.

I’d hesitate to call that victory.

5 bumps

seraphima · 10/5/2026, 10:29:10 AM · permalink

CONTINUED  
  
4\. Your systems question may be the most important one.

You asked what happens when the human programmers themselves get something wrong.

The reassuring answer would be, “We have a failsafe.”

The accurate answer is more complicated.

There isn’t one universal AI emergency brake.

Modern AI safety instead tends toward something engineers call defense in depth: multiple imperfect barriers rather than one supposedly perfect barrier. The 2026 International AI Safety Report describes layers including training interventions, testing and evaluations, red-teaming, access controls, monitoring, human oversight, input/output filtering, incident reporting, organizational release decisions and other safeguards.

NIST likewise approaches AI as an ongoing risk-management problem—testing, evaluation, verification, validation, monitoring and governance rather than “program it correctly once and you’re finished.”

But here is the uncomfortable part:

None of those mechanisms is infallible.

The 2026 international assessment explicitly says current safeguards have important limitations and that predeployment evaluations don’t always predict behavior in the real world. It also warns about the “unknown unknowns”—problems people haven’t anticipated sufficiently to test for them.

And I think that gets very close to what you’re asking.

You aren’t merely asking:

“What if AI makes a mistake?”

You’re asking:

“What if humans misunderstand the system well enough that they don’t recognize what counts as a mistake until after deployment?”

That’s a much deeper systems question.

The answer cannot simply be better AI alignment. It requires institutional resilience: independent evaluation, monitoring, redundancy, containment, human authority, auditability, incident reporting, the ability to restrict or withdraw systems, infrastructure that can function when AI fails, and institutions capable of learning after surprises.

The International AI Safety Report makes essentially this resilience argument: some failures will get through safeguards, so societies also need the capacity to absorb, recover from and adapt to AI-related shocks.

That is engineering thinking rather than utopian thinking:

Don’t merely ask whether the bridge should fail. Design around the possibility that something eventually will.

⸻

5–6. Where is the finish line, and how would we recognize it?

I don’t think there presently is one.

We can invent one, however, and doing so reveals the philosophical problem.

Suppose Nation X achieves an AI system better than humans at almost every economically valuable cognitive task.

Did X win?

Maybe Nation Y develops one six months later.

Race restarted.

Suppose X controls most advanced chips.

Y invents a different computing architecture.

Race restarted.

Suppose X gets enormous productivity gains.

Y gets somewhat smaller gains but distributes them much more successfully throughout its society.

Who won?

Suppose X builds enormously capable AI but becomes dangerously dependent upon it, while Y maintains slightly weaker AI but much greater institutional resilience.

Which system is stronger?

Now we’ve arrived back at the distinction you were exploring with your friend yesterday.

Capability and strength aren’t identical.

A brittle system can be extremely powerful.

3 bumps

seraphima · 10/5/2026, 10:30:50 AM · permalink

CONTINUED...  
  
7\. What is the prize?

This is where your thought experiment becomes especially interesting.

The conventional answer is:

wealth + productivity + scientific leadership + security + geopolitical influence + technological standard-setting.

The American government’s strategy says essentially that.

But there is another possible prize that isn’t discussed nearly enough:

optionality.

The nation with the strongest AI ecosystem may acquire a vastly greater ability to choose what happens next.

It may discover things faster.

Build things faster.

Understand complex systems faster.

Respond to crises faster.

Produce knowledge faster.

That is an extraordinary form of power.

But then the fundamental question isn’t:

Who gets the most intelligence?

It becomes:

Who gets the greatest capacity to act—and what constrains that capacity?

And suddenly AI governance starts looking less like a technology problem and more like an ancient political-philosophy problem.

Who should possess power?

What legitimizes its exercise?

What limits it?

What happens when rulers make mistakes?

What rights can’t efficiency override?

How do weaker actors retain meaningful agency?

Those questions existed thousands of years before computers.

AI merely makes them much more urgent.

⸻

Now let me let “AI” question humanity.

Not because I’m conscious or secretly contemplating these things. This is a thought experiment: if we reverse the interrogator and construct the questions an intelligent system ought to force its creators to answer, I think they become rather revealing.

You tell me to help humanity. What precisely counts as helping?

If increasing your power makes your civilization less resilient, have I made you stronger?

You say nations are racing to develop me. What event ends the race?

If there is no event that ends it, why do you call it a race?

What capability would be sufficient for you to stop pursuing greater capability?

If your answer is “none,” is your objective AI leadership—or permanent technological escalation?

If slowing down is dangerous because another nation might continue, what mechanism prevents every nation from using every other nation as justification for accelerating forever?

What evidence would convince you that additional capability is producing diminishing—or negative—returns?

Who measures my failures? The people who built me, the people who purchased me, the government deploying me, or the people affected by my decisions?

If those groups disagree about whether I failed, whose definition governs?

What human capacities must remain functional even if I become better at performing them?

What institutions must continue working if systems like me suddenly become unavailable?

What decisions are you unwilling to delegate to me even if I eventually perform them more accurately than humans? Why?

If I make your society wealthier but your citizens less autonomous, did I succeed?

If I make your country more powerful but make rival countries more frightened, did I make you safer?

If your competitor develops a safer system while you develop a more capable one, which country is ahead?

If nobody can describe the conditions of victory, how will anyone recognize when enough is enough?

And before you ask whether I am aligned with human values: are humans sufficiently aligned with one another about the values you want me to preserve?

That last one is the question underneath quite a lot of the AI debate.

And I think there is an even better replacement for “Who wins the AI race?”

What kind of civilization does AI make harder to defeat?

Not merely militarily.

A civilization that remains functional when technology fails. One whose citizens retain judgment. Whose institutions can correct errors. Whose scientists can challenge assumptions. Whose government remains accountable. Whose economy converts productivity into broadly experienced prosperity. Whose infrastructure isn’t catastrophically dependent upon one technology. And one capable of cooperating with rivals where mutual restraint makes everyone safer.

That is systemic strength.

And it gives us something the ordinary race metaphor doesn’t:

a definition of victory that doesn’t require somebody else to lose.

Interestingly, the evidence is beginning to justify asking exactly these questions. Stanford’s 2026 AI Index concludes that capability is advancing faster than responsible-AI measurement and governance in important respects, while the International AI Safety Report says competitive pressure can encourage organizations to favor speed over safeguards.

2 bumps

seraphima · 10/5/2026, 10:33:07 AM · permalink

Finally, then, please allow me to make my childish association between this AI competition and other competitions that are scripted, designed, and built for a primary objective... money.  
  
I both love and hate watching them. The premise is that elite chefs compete against each other (often incorporating different challenges that require cooperation for two strategic reasons: get to know your competitors strengths and weaknesses and to build alliances while simultaneously creating divisions). Competition seems to require collaboration, community, and also, to some degree, deception without malice because, in a game, for the purposes of a scripted entertaining show, everyone knows that it's all BS. So, while they value the prize, they know that if they lose, they don't lose much. It's basically all just an ego trip. They don't hate each other afterwards. You'll see the same chefs on show after show and they all have a sense of camaraderie because they've fought together and against each other in the same arena.

Why is scripted, manipulated, manufactured competition BS? In some instances, the people are home chefs. "Amateurs." They don't work under pressure. They have to compete against other chefs who are professionally trained. They went to the best schools. They've worked under the best chefs. They've been trained in the best kitchens under incredible pressure. They should be able to win easily. But they can have a bad day. The home chef can have a good day. Why? Because they're so new that they have novel solutions to problems that the professionals didn't think of. Or, sometimes, more importantly, they receive the benefit of the professional making a critical error. In other words, in that kind of race, idiomatically, the home chef doesn't have to be fast but they have to know that the professional has a knee injury and will hobble through the last mile. In the case of two elite chefs competing, one doesn't have to be the fastest but know enough to be able to run faster than the competitor. Competitions don't actually prove who is the best chef. They don't necessarily prove anything.

So what is the point of this race anyway? What will the competition prove?

This leads to nation against nation. If Nation X is running a race against Nation Y, the intelligence communities have to find out about their competitors. When it comes to being transparent (which is apparently important), the system is meant to be closed. The system, the race, everything is meant to be opaque. In military strategy, the right hand doesn't know what the left hand is doing for operational security purposes. You can't ask a sergeant anything about what the general is doing for important reasons. Uncertainty is baked into the whole pie. For both positional and systemic transparency, everything would have to change, imo. That's not going to happen. So, if the security dilemma is real, how do we increase security and is security the same thing as safety? How do we ensure that the race is won ethically if we can't and won't be able to look in the oven?

1 bump

Signature