The Alarm Everyone Recognizes
The story of the boy who cried wolf has survived for centuries because its lesson is more complicated than a warning against lying. The wolf was real. It eventually came. The boy’s failure was that he had spent the credibility on which a real warning would later depend. By turning fear into entertainment, he trained the village to ignore the very danger he claimed to recognize.
The current debate over artificial intelligence has begun to resemble that story. Executives, researchers, and former employees of leading AI companies now speak publicly about autonomous agents escaping control, attacking computer systems, assisting biological weapons development, or threatening human survival. Some of the incidents behind these warnings are real. AI agents have behaved in unexpected ways during evaluations, and systems designed to act across digital environments create risks that ordinary chatbots did not. Cybersecurity, military applications, surveillance, fraud, and political manipulation all deserve sustained attention.
Yet a large distance separates an alarming event in a controlled test from the extinction of humanity. Between them lie assumptions about future capabilities, access to infrastructure, human negligence, institutional response, defensive technology, and international politics. Public discussion often travels across that distance in a single sentence. A model behaves badly during an evaluation, and the next headline asks whether humans will survive the decade.
Warnings lose their value when every danger must be presented as the largest imaginable danger. The public may respond strongly at first, especially when the warning comes from a respected scientist or an engineer with inside knowledge. Repetition changes the response. Predictions accumulate, deadlines pass, and extreme claims begin to sound like a familiar genre. If a more serious threat later appears, the audience may no longer distinguish it from the earlier spectacle.
This creates what might be called the business of crying wolf. The phrase does not require the warning to be fabricated or the speaker to be dishonest. It describes a system in which alarms generate attention, prestige, funding, and authority before their accuracy can be established. The wolf may exist, but the warning also has a market value.
How an Engineer Becomes a Moral Witness
The public resignation of Anthropic researcher Jacob Coxon gave the current debate a human center. Leaving a prominent AI company because one believes its technology could become catastrophically dangerous is not an ordinary career decision. Reports that he departed shortly before part of his equity would vest strengthened the impression that he had accepted a personal cost for his convictions. His concerns deserve to be heard in that light.
The meaning of the resignation changed as it moved through the media. A researcher who disagreed with the acceptable pace and risk of development became an insider warning humanity. His departure was presented as evidence not only of his sincerity but also of the truth of the forecast itself. The story acquired the structure of moral testimony: someone had seen the danger from within, renounced material benefit, and returned to warn those outside.
Three different judgments became compressed into one. Coxon may be sincere. His work may have given him access to relevant evidence. His prediction may still be mistaken. Sincerity tells us something about the person making a claim, while accuracy depends on the quality of the claim, the assumptions behind it, and the evidence available to others. Technical expertise deserves respect, but it does not transform a forecast about civilization into established knowledge.
The distinction becomes harder to maintain when departure itself creates moral authority. A researcher who stays inside a company may be portrayed as compromised by salary and status. One who leaves can be treated as liberated from those interests. Yet former employees also enter new social roles. They may gain an audience, become symbols of a movement, and receive invitations to speak on questions far beyond their direct research. None of this invalidates their testimony. It does show that independence from one institution does not produce independence from every incentive or identity.
Longstanding advocates of AI extinction scenarios quickly recognized the resignation as a validating event. Ideas that had remained influential within specialized safety communities moved into mainstream reporting. Earlier predictions, probability estimates, and demands for a pause returned with new force. A single resignation did not create the movement, but it gave the movement a face and a moment around which to organize.
The media helped complete that transformation. “Researcher disagrees with corporate risk tolerance” is an important story but not a dramatic one. “Insider leaves powerful AI company to warn that humanity may die” offers a hero, a threat, and a deadline. It can be understood without technical knowledge and repeated without examining the chain of assumptions. The moral clarity of the story is also its intellectual weakness.
Fear Enters the Business Model
Anthropic was founded with safety at the center of its identity. It has invested in interpretability, model evaluations, alignment research, and formal policies intended to connect increasing capabilities with stronger safeguards. Those commitments should not be dismissed as decoration. The company has helped make safety research a competitive field rather than an activity left outside commercial development.
At the same time, safety is part of Anthropic’s position in the market. The company sells access to powerful proprietary models while presenting itself as unusually aware of the dangers those models may create. These roles reinforce each other. Its commercial success gives it the resources and access needed to study frontier systems. Its safety reputation makes governments and enterprises more willing to trust it with sensitive work. Its warnings also confirm that the technology it sells is important enough to influence the future of civilization.
A similar pattern appears in the corporate campaign to become “AI native.” The phrase often describes a real organizational ambition: redesigning work so that AI contributes to research, communication, analysis, and decision-making. The difficulty lies in measuring whether that change has occurred. Organizations therefore reach for convenient substitutes, such as adoption rates, active users, agent activity, token consumption, and monthly expenditure.
Consumption then begins to stand in for transformation. Employees are encouraged to use AI extensively so that value can emerge through experimentation. Agents produce longer chains of actions, repeated tool calls, and large volumes of output. An organization may initially celebrate the resulting activity as evidence that a new culture has arrived. Months later, finance teams impose strict limits because the relationship between usage and useful outcomes remains unclear.
This resembles an old feature of the consulting industry. A consulting engagement may begin with a genuine problem, but organizational uncertainty can generate additional frameworks, assessments, workshops, and transformation programs. The provider’s continuing relevance grows with the complexity of the process. In enterprise AI, billable hours are replaced by tokens, tool calls, and compute. The client may become more dependent at the same time that it is told it is becoming more capable.
The proper measure is not how much AI an organization consumes but what kind of work that consumption makes possible. A translation, investigation, software repair, or strategic analysis should be evaluated through its quality, cost, time, and consequences. High usage can reflect productive exploration, but it can also reflect inefficient reasoning loops, poor system design, or the absence of a clear stopping point.
Anthropic’s position therefore contains a tension that cannot be resolved through moral branding. It profits when powerful AI is used extensively, while its public identity gains strength from describing powerful AI as unusually dangerous. The company may believe both messages. Their sincerity does not remove the need to examine how each message supports the other.
The Drawbridge at the Frontier
Dario Amodei’s call to “pace the frontier” goes beyond a request for better safety research. He has argued that capability development should slow so that safeguards can catch up. His proposals include ongoing access for independent evaluators, common safety standards among leading laboratories, and forms of coordination supported by government. The proposal does not demand an end to model training. It seeks to place friction inside a competition that might otherwise reward whichever company moves fastest.
There are serious reasons for such friction. A race can punish caution even when every participant would prefer stronger safeguards. If one laboratory delays a release to investigate a dangerous capability while its competitors continue, responsible behavior may carry a commercial penalty. Shared standards can reduce that pressure and make safety compatible with survival in the market.
The same arrangement can protect incumbents. If Anthropic slows while others accelerate, it may lose its position. If every frontier laboratory slows together, the current hierarchy becomes easier to preserve. Safety requirements also cost money. Continuous evaluations, specialized staff, legal reporting, secure computing environments, and regulatory compliance are easier for a well-funded company than for a university group or a startup.
Regulation can therefore serve as both a guardrail and a moat. A rule may reduce danger while raising the cost of entry. An incumbent does not need to invent a false risk for this to happen. It can advocate sincerely for a standard whose burden falls more heavily on potential competitors. Public policy must judge both the protective effect and the distribution of power created by the protection.
The opposing camp has its own interests. Meta benefits from open-weight development because its main business does not depend on selling access to a foundation model. An open ecosystem reduces the chance that Meta will become dependent on a rival platform and encourages developers to improve tools built around its architecture. NVIDIA benefits when more laboratories, companies, and governments train and operate more models. Its preference for engineering solutions and continued development aligns with a market that expands as computational demand grows.
No side enters the debate without a position. Anthropic can present concentration as responsibility. Meta can present ecosystem strategy as openness. NVIDIA can present expanding demand as technological optimism. Each may offer valuable arguments, and each gains from the world its argument recommends. Intellectual fairness requires applying the same suspicion and the same generosity to all of them.
A Debate Captured by Political Identity
President Donald Trump’s dismissal of AI extinction fears as a hoax adds a different kind of pressure. Once a controversial political figure adopts one side, people begin evaluating arguments through association. Someone who doubts Anthropic’s forecasts may hesitate to speak because the doubt could be interpreted as support for Trump. For someone who opposes Trump, agreeing with his dismissal of extinction fears may feel like endorsing him politically. To avoid that association, they may end up defending warnings about AI that they would otherwise consider exaggerated.
The choice is false. AI agents can present real security risks without being on the verge of exterminating humanity. Extinction probabilities can lack a sound scientific basis even when the people offering them are sincere. Frontier laboratories can support useful regulation while also benefiting from higher barriers to entry. Trump’s wholesale rejection can be politically motivated and intellectually careless at the same time that some of his targets overstate their evidence.
The usual division between Democrats and Republicans does not describe the emerging alliances well. Anthropic’s culture and support for regulation may appear closer to Democratic politics. Meta and NVIDIA currently sound closer to Republican arguments about growth, competition, and resistance to regulatory restraint. Actual coalitions cross those lines. Bernie Sanders has supported strong restrictions on advanced AI, while parts of the political left argue that extinction narratives distract from labor displacement, surveillance, discrimination, environmental costs, and corporate concentration. Elon Musk has moved toward the political right while continuing to warn about catastrophic AI risk. National security conservatives may oppose domestic regulation yet support export controls intended to prevent China from obtaining advanced chips.
Several disagreements are taking place at once. One concerns the pace of development. Another concerns whether control should be concentrated or distributed. A third concerns proprietary systems and open weights. A fourth concerns whether governance should prioritize present harms or speculative future catastrophes. International competition adds another division between those seeking global coordination and those who fear that restraint will transfer power to a rival state.
Reducing these disputes to left and right turns technical judgment into political loyalty. It also gives the loudest figures disproportionate influence. If Trump defines skepticism, skeptics who reject his politics may fall silent. If Anthropic defines responsibility, critics of corporate concentration may be portrayed as enemies of safety. A democratic society needs room for positions that do not arrive as complete ideological packages.
Those Who Have Not Yet Crossed the Gate
Beyond the American frontier laboratories lies a much larger AI world. It includes open-weight developers, universities, small companies, independent researchers, and firms across China, Europe, India, Southeast Asia, and other regions. Measured by the number of participants and models, this ecosystem is broader than the small group of companies capable of training the most expensive systems. Measured by capital, advanced chips, energy, and frontier computing capacity, power remains concentrated.
The phrase “open source AI” can obscure part of this picture. Many models are better described as open weight because their trained parameters are available while their data, training code, or full development process remain closed. Even this limited openness changes the balance. Developers can run models locally, adapt them to specific languages and industries, and avoid permanent dependence on a single API provider. Countries that cannot build a frontier model from the beginning can still develop capabilities around models they are allowed to inspect and operate.
Calls for coordinated restraint look different from outside the frontier club. A company that already possesses advanced models can accept a slower rate of improvement while continuing to sell what it has. A country that already controls major computing infrastructure can endorse restrictions while retaining strategic capacity. Latecomers are asked to accept limits before gaining comparable access to the benefits.
The structure recalls international climate politics. Industrialized countries built wealth through long periods of fossil-fuel use and later asked developing countries to limit emissions. Climate action remains necessary, but the distribution of historical benefit and present sacrifice cannot be ignored. A government still trying to provide electricity, transportation, and industrial employment will not experience the same restriction in the same way as a wealthy society whose basic infrastructure is already complete.
AI creates a related dispute over development rights. Advanced economies and companies can describe restraint as universal responsibility after securing the models, talent, and infrastructure that define the field. Other societies may hear a different message: the technological ladder was available to early climbers but must now be withdrawn for everyone’s safety.
The nuclear analogy is even sharper. Nuclear powers retain arsenals while supporting a system intended to prevent additional states from acquiring them. Nonproliferation may reduce the probability of catastrophe, but it also preserves a hierarchy between those who possess deterrence and those who do not. The system’s legitimacy has always depended on promises of disarmament and access to peaceful nuclear technology, promises whose incomplete fulfillment remains a source of resentment.
An AI safety regime designed mainly by existing frontier laboratories could create a similar hierarchy. A small group would retain advanced systems and influence the standards governing everyone else. The analogy has limits. AI models are easier to copy than nuclear materials, and they support education, medicine, translation, science, and ordinary business as well as military and surveillance applications. Those differences make exclusive control harder to justify and enforcement harder to achieve.
Global governance must therefore ask more than how to slow development. It must ask who has already benefited, who bears the cost of restraint, who participates in writing the rules, and what forms of access remain available to those outside the leading states and companies. Safety without representation can become another name for preserved advantage.
A Warning Worth Believing
The lesson of the boy who cried wolf is not that communities should stop listening. The wolf eventually arrived, and disbelief became dangerous. The task is to preserve the conditions under which a warning can still command attention when evidence warrants it.
AI institutions can begin by separating what they have observed from what they infer. A documented agent failure should be described with its conditions, scale, and limits. Predictions about future systems should identify the assumptions that connect present evidence to future harm. Claims about human extinction should not borrow the appearance of scientific precision from percentages that cannot be validated through observation or repeated experiment.
Independent evaluation is essential, but independence must extend beyond a small network of frontier companies and safety organizations that share similar assumptions. Cybersecurity experts, social scientists, economists, courts, democratic governments, civil society, open-weight developers, and representatives of countries outside the technological center all hold relevant knowledge. Their disagreements are not obstacles to legitimate governance. They are part of what makes governance legitimate.
The same standard should apply to moral authority. A departing researcher can provide important testimony without becoming a prophet. A chief executive can understand a technology deeply without acquiring the right to determine the acceptable future for everyone. A company can conduct serious safety research without receiving immunity from questions about pricing, market power, or regulatory advantage. An open ecosystem can distribute innovation while still creating misuse risks that require enforceable responsibility.
No single guardian deserves complete trust. Governments can abuse regulation, companies can protect their markets, open communities can underestimate misuse, and media organizations can reward the most dramatic interpretation. A workable system exposes each center of power to evidence, dissent, external review, and correction.
Anthropic may sincerely fear the technology it is building, profit from selling it, and benefit from rules that constrain its rivals. These realities can exist together. Recognizing their coexistence is more useful than choosing between admiration and cynicism. The same discipline should be applied to those who promise that faster innovation will solve every difficulty and to political leaders who dismiss unfamiliar risks as conspiracies.
A society prepared for genuine danger needs credible alarms. If AI safety becomes inseparable from extravagant forecasts, heroic media narratives, and institutional self-protection, public attention will not remain permanently available. It will erode through use. The deepest danger in crying wolf is not that the village becomes insufficiently afraid. It is that fear is spent before the moment when listening becomes necessary.
Leave a Reply