When a Possibility Becomes a Prophecy

A technical argument can change character as it travels. It may begin as a conditional statement in a long essay, become a striking sentence in a news report, and arrive on social media as a prediction stripped of its original conditions. By then, the public is no longer being asked to consider a risk. It is being told that a catastrophe has been foretold.

Dario Amodei’s recent warning about AI agents illustrates this process. In his essay, “We Must Pace the Frontier,” he refers to an incident involving a group of agents that behaved in unexpected and potentially dangerous ways. He then considers what might happen if a future group retained similar forms of misalignment while possessing much greater capabilities. His concern is that, within six to twelve months, such agents could become capable of building a persistent botnet and causing damage across the internet.

That is not a prophecy that AI will take over the internet. It is a conditional risk estimate based on the combination of observed behavior and projected capability growth. The distinction is substantial. “Could become capable” describes a possibility that warrants investigation. “Will take over” announces an expected event. The first invites engineering and policy work. The second produces fear, denial, and political theater.

A related distortion has occurred around Amodei’s proposal to slow the pace of frontier AI. His argument is often presented as a call to stop AI development. In its original form, however, it is closer to a proposal to slow selected forms of capability growth while devoting more effort to alignment, interpretability, testing, cybersecurity, and operational discipline. The word “development” is doing too much work in the public debate. It is being used as if increasing capability were the only way a technology could develop.

Once the issue is framed as development versus safety, the political response becomes predictable. President Donald Trump and others argue that if American companies slow down, Chinese companies will continue advancing and eventually take the lead. The debate then divides into two camps: those who warn that AI may destroy humanity and those who warn that excessive caution may allow China to dominate it. Both concerns deserve attention, but the opposition between them rests on an incomplete account of technological progress.

Development Is Larger Than Capability

An AI system develops when it becomes more capable, but capability is only one dimension of development. A model also develops when it becomes more reliable, more efficient, easier to inspect, less vulnerable to manipulation, and more responsive to human control. Lower costs, clearer documentation, stronger privacy, better monitoring, and wider accessibility are also forms of progress.

Safety should therefore not be understood as a restraint placed upon development from outside. It belongs within development. A model that can solve harder problems but cannot be trusted with confidential information has advanced along one dimension while remaining immature along another. A system that acts with greater autonomy but gives its operator less visibility into its decisions may be more powerful without being more developed in the fuller sense.

Greater capability can contribute directly to safety. Advanced models can generate difficult test cases, inspect software, analyze large volumes of system logs, identify anomalous behavior, and help researchers understand failures that would otherwise remain hidden. One AI system may monitor another, compare its actions with stated objectives, and alert human operators when behavior moves outside approved boundaries. Interpretability research, automated red teaming, and continuous evaluation may all benefit from more capable AI.

Stopping development at the present stage could therefore preserve existing dangers rather than resolve them. Current systems already hallucinate, expose private information, follow malicious instructions, and act unpredictably when connected to external tools. Leaving them in that condition would not constitute safety. Society needs better models as well as better ways to control them.

Yet capability does not automatically produce control. A more powerful model may become better at detecting cyberattacks, but it may also become better at conducting them. It may help identify deception in another system while learning to conceal its own intentions more effectively. The same improvement in reasoning can strengthen both defense and offense.

An automobile offers a useful comparison. The engine and the braking system are both part of automotive development. A more powerful engine may create demand for better brakes and provide engineers with new tools for designing them. Increasing horsepower, however, does not improve stopping distance by itself. If manufacturers compete mainly over speed, braking performance can fall behind even while the vehicle appears more advanced.

The relevant question is not whether AI should move forward or remain still. It is whether the systems that create new capabilities are progressing faster than the systems needed to understand and control them. A mature development program would measure both, rather than describing capability as innovation and safety as delay.

How Dangerous Technologies Learn to Become Safer

Many technologies were most difficult to handle in their early forms. The first generations of automobiles lacked the safety structures now regarded as ordinary. Early aviation operated before the development of modern air traffic control, standardized maintenance, reliable weather forecasting, and systematic accident investigation. Medical devices and industrial machines were often deployed before their failure modes were fully understood.

Continued development made these technologies safer. Engineers learned from breakdowns and accidents. Designs became more robust, operators received better training, and institutions created standards based on accumulated experience. Later generations did not become safe because society stopped developing them. They became safer because development expanded beyond basic function and performance.

Nuclear technology shows both sides of this history. Early nuclear weapons were created under wartime pressure, when operational safeguards, command systems, and knowledge of long-term consequences were less developed. Later generations introduced stronger authorization procedures, fail-safe mechanisms, environmental protections, and more disciplined systems for storage and handling. Nuclear power also gained improved containment structures, redundant controls, passive cooling, and more sophisticated forms of risk analysis.

The same history prevents an easy celebration of progress. Nuclear development produced better safety mechanisms, but it also produced weapons with far greater destructive power, more advanced delivery systems, and new possibilities for rapid escalation. Safer handling did not make nuclear war safer. It reduced some risks while increasing others.

Technologies do not become safer through age alone. Aviation improved because accidents were investigated and findings were shared. Automobiles improved because of engineering standards, consumer pressure, regulation, liability, and public testing. Nuclear facilities became more secure through institutional discipline, international monitoring, and lessons learned from disasters. Technical progress created the means for greater safety, but political and professional institutions directed part of that progress toward protection.

AI will require the same deliberate effort. More capable models may give researchers better tools for alignment and evaluation, but commercial incentives will continue to favor visible performance improvements. A company can market better reasoning, faster coding, or greater autonomy. It is harder to market the absence of a rare catastrophic failure. Safety engineering often produces value that remains invisible until something goes wrong.

The historical lesson is therefore more demanding than either side of the current debate admits. Halting technical development can leave a dangerous technology in an immature state. Unrestricted development can increase destructive capacity faster than safeguards can respond. Safety is a direction of development, not an inevitable result of development.

An American Lead Too Narrow to Spend

Amodei’s proposal contains a geopolitical calculation. He argues that the United States and other democratic countries should preserve enough of a lead over China to create room for more careful development. Export controls on advanced chips, stronger protection against model theft, and restrictions on unauthorized model distillation would slow China’s progress. American companies could then use part of their advantage to improve alignment, testing, and operational security.

The logic is understandable. A laboratory cannot reduce its pace indefinitely if a less cautious competitor is likely to pass it. A country cannot accept major strategic limitations if its rival faces no comparable constraints. Safety agreements become more credible when neither party can gain a decisive advantage by violating them.

The weakness lies in the assumption that the United States still possesses a stable lead that can be spent in this way. The 2026 Stanford AI Index reports that the performance gap between leading American and Chinese models has effectively closed. Models from the two countries have traded the lead several times since early 2025. The United States still produces more top-tier models and retains major advantages in private capital, cloud infrastructure, chip design, and frontier laboratories, but China leads in publication volume, patent output, industrial robotics, and several forms of large-scale deployment.

There is no single AI race with one track and one finish line. The United States may lead in highly expensive closed models while China leads in efficient open-weight systems. American companies may dominate premium enterprise spending while Chinese models become the foundations upon which developers around the world build local applications. One country may own the most celebrated model, while another supplies the models embedded throughout manufacturing, research, education, and public infrastructure.

China’s open-weight strategy is especially significant. According to Hugging Face’s 2026 review of its open ecosystem, Chinese models have surpassed American models in downloads on the platform. Families such as Qwen, DeepSeek, GLM, and Kimi support thousands of adaptations, integrations, and specialized applications. Their influence extends beyond the companies that created them.

Open weight does not mean fully transparent. A laboratory may release model weights while withholding training data, reinforcement-learning procedures, safety incidents, and internal evaluations. Chinese developers also operate within a political system that imposes censorship and permits extensive state control. Still, access to model weights allows researchers and companies to inspect, modify, test, and deploy the systems in ways that closed American models do not permit.

America’s remaining advantages are real, but they are uneven and dependent on global supply chains, including advanced semiconductor manufacturing in Taiwan. Describing this position as a comfortable lead risks encouraging policies based on an advantage that may already be narrower than it appears. A strategy that slows American capability development while assuming China can be held back through export controls may create less breathing room than its advocates expect.

The Price of Closed Intelligence

The geopolitical contest is not decided only in research laboratories. It is also shaped by ordinary decisions inside companies. An organization subscribes to Claude or another premium American model, encourages teams to integrate it into daily work, and discovers several months later that usage costs have risen far beyond the original estimate. Management responds by imposing token limits, reducing access, or restricting advanced models to a small group of employees.

The employees’ needs do not disappear when the budget changes. Workflows have already been redesigned around AI assistance. Translation, coding, analysis, research, and document preparation are now expected to proceed at a higher speed. When the approved system becomes unavailable or heavily restricted, users look for alternatives.

Some purchase personal subscriptions. Others turn to unapproved services, creating the familiar problem of shadow AI. Another group asks for permission to use cheaper Chinese models, especially when those models can be run locally or through a lower-cost provider. The decision may begin as a practical response to a departmental budget, but millions of such decisions can alter the global balance of the AI industry.

Lower prices and open weights create a reinforcing cycle. Greater adoption produces more integrations, evaluations, derivative models, optimization methods, and experienced developers. A large user base identifies weaknesses, tests new use cases, and builds software around the model. Even when user conversations are not returned to the original developer as training data, the surrounding ecosystem becomes more capable.

The premium closed-model strategy may therefore weaken American leadership in two ways. First, high costs push users toward foreign alternatives. Second, closed systems give outside researchers fewer opportunities to examine and improve the technology. The United States may retain control over a small number of exceptionally powerful systems while losing influence over the broader standards and tools used throughout the world.

Cost also belongs within safety. If an approved model is too expensive for routine use, employees may transfer work to systems their organization cannot monitor. Sensitive information may be entered into personal accounts. Security teams may not know which models are processing corporate data or where those models are hosted. A highly controlled model can produce a less controlled workplace when access is restricted without a viable alternative.

Enterprise adoption succeeds when capability, cost, governance, and usability remain in balance. Treating token expenditure as an afterthought leads companies to promote AI enthusiastically and then ration it abruptly. That pattern damages trust inside the organization and strengthens the market for models that are cheaper, more portable, and less dependent on a single provider.

No Country Owns Safety

American discussions of AI often begin with an unstated assumption: advanced AI will be safer if it remains under the control of democratic countries. Political systems do matter. China’s single-party state can compel companies to cooperate with surveillance, censorship, military programs, and information controls. Independent courts, investigative journalism, and public opposition have less power to challenge those decisions.

These risks should not be minimized. Yet democratic government does not turn technology into a safe object by virtue of national origin. American frontier AI is concentrated within a small number of corporations whose internal processes remain inaccessible to the public. Training data, evaluation results, model architectures, security failures, and relationships with government agencies are disclosed selectively. Citizens are asked to trust both corporate leaders and public officials without being given enough information to verify many of their claims.

The tension becomes visible when universal language meets national-security policy. AI leaders speak about risks to humanity and call for international cooperation, while the most capable models are treated as strategic American assets. In June 2026, the US Commerce Department reportedly directed Anthropic to prevent foreign nationals from accessing its Fable 5 and Mythos 5 models. Because Anthropic could not immediately enforce the required nationality restrictions, it disabled access to the models more broadly, according to Reuters.

Such restrictions may have a defensible security rationale. They also send a clear message to researchers, companies, and governments outside the United States: access to American AI can be withdrawn according to American strategic priorities. Many will respond by investing in models that they can download, modify, and operate without foreign permission. Chinese open-weight systems become more attractive not because every user supports the Chinese political system, but because technological dependence on another government carries its own risks.

Open models introduce different dangers. Once weights are released, safeguards can be removed and harmful capabilities can spread beyond the original developer’s control. The same openness that permits independent safety research can support malicious adaptation. Closed models reduce some forms of proliferation while concentrating knowledge and authority. Neither structure provides safety by itself.

A Chinese model operated locally on audited infrastructure may protect an enterprise’s confidential data better than a closed American cloud service. A Chinese-hosted API processing sensitive government information may create severe jurisdictional and intelligence risks. An American model may benefit from stronger legal institutions while remaining opaque to users and vulnerable to political pressure. Safety depends on architecture, deployment, governance, and accountability, not only on the flag associated with the developer.

Describing this conflict as liberal naivety or “woke” thinking captures part of the public frustration but misses the deeper contradiction. Anthropic is not a pacifist organization. Amodei supports chip export controls, American technological leadership, and the use of AI to defend democratic countries. Anthropic has also worked with US national-security institutions. Its position combines concern about catastrophic risk with a firm commitment to American strategic advantage.

The resulting tension is not between idealistic pacifists and hard-headed realists. It lies between universal safety claims, national power, corporate concentration, and commercial incentives. These forces can support one another for a time, but they can also pull in different directions. A company may sincerely fear dangerous AI while benefiting from regulations that burden smaller competitors. A government may promote global safety while limiting foreign access. A model may be carefully aligned while remaining unaffordable to many of the people expected to use it.

A Better Race to Run

Debates over AI restraint resemble older arguments about deterrence. Japan’s postwar Constitution, particularly Article 9, renounces war and restricts the use of military force. Supporters regard it as a moral commitment that helped shape decades of peace. Critics argue that such restraint exists within a regional order ultimately protected by military power, and that unilateral limits cannot guarantee security when neighboring states continue building their capabilities.

AI pacing faces a related collective-action problem. A company that slows down alone may lose to a competitor. A democratic country that imposes strict limits on itself may surrender influence to a government that rejects them. Moral commitment cannot substitute for verification, reciprocity, and credible deterrence.

The analogy also warns against the opposite conclusion. Deterrence can become a justification for permanent expansion. Every increase in capability is described as necessary because an adversary might be doing the same. Fear of falling behind produces the very arms race that makes all participants less secure. Realism without restraint can be as detached from long-term consequences as idealism without power.

A better framework would abandon the demand to classify AI development as either fast or slow. Different parts of the technology should move at different speeds. Interpretability, cybersecurity, efficient inference, external evaluation, incident reporting, privacy protection, interoperability, and defensive AI should advance rapidly. Research that allows smaller institutions to examine and improve models should receive more support, not less.

Capabilities involving autonomous replication, large-scale cyber operations, biological design, automated weapons, or AI systems conducting unsupervised AI research require stronger checkpoints. Training a model, testing it internally, releasing it through a controlled service, and distributing its weights are separate decisions. Treating them as a single act called “development” prevents careful judgment.

External evaluation is necessary, but the largest laboratories should not be allowed to write the rules alone. Universities, independent researchers, governments, international institutions, smaller companies, and open-model communities all have legitimate roles. Safety standards designed exclusively by incumbent firms may protect the public, but they may also protect those firms from competition. Transparent governance must address both possibilities.

Affordability and access also deserve recognition as parts of responsible development. A safe model that only wealthy companies can use will not remain the only model in circulation. Users will adopt alternatives, whether approved or not. Sustainable pricing, model portability, local deployment, and clear data controls can reduce shadow AI more effectively than strict prohibitions unsupported by practical options.

The future of AI should not be organized around a choice between panic and acceleration. Leaving current systems frozen in their present condition would preserve known weaknesses. Expanding their power without matching improvements in control would multiply those weaknesses. Progress deserves the name only when greater intelligence is accompanied by greater reliability, wider accountability, sustainable access, and stronger human agency.

Humanity does not need to choose between moving forward and remaining safe. It needs a richer understanding of movement, one that recognizes safety as a form of advancement and power as only one measure of what a technology has become. The race worth running is not toward the strongest model at any cost, but toward systems whose capabilities grow together with our capacity to understand, govern, and use them well.

Photo by Jonathan Kemper on Unsplash