Posts in category: PROGRAMS

In this ECPS interview, Tom Davidson, one of the leading analysts examining the long-term implications of AGI governance, warns that humanity may be approaching an “intelligence explosion” in which AI systems rapidly improve themselves in a runaway feedback loop, potentially compressing decades of technological development into mere years. Examining the geopolitical, democratic, and civilizational implications of advanced AI, Davidson argues that democratic institutions may struggle to govern machine-speed innovation, while frontier AI systems could generate unprecedented concentrations of political, corporate, and military power. The interview explores AI-driven democratic backsliding, geopolitical rivalry between the United States and China, technocratic oligarchy, AI safety governance, and the future of political agency itself under conditions of accelerating artificial intelligence.

Interview by Selcuk Gultasli

Giving an interview to the European Center for Populism Studies (ECPS), Tom Davidson warns that the world may be approaching an unprecedented technological rupture in which advanced artificial intelligence fundamentally transforms not only economic production and geopolitical competition, but also the very foundations of democracy, sovereignty, and political agency. A Senior Research Fellow at Forethought and one of the leading analysts examining the long-term implications of AGI governance, Davidson argues that humanity may now be entering an era in which “AI systems create even more capable AI systems in a runaway feedback loop of accelerating progress.” 

Rather than treating AI merely as a question of productivity gains or consumer innovation, Davidson situates artificial intelligence within a much broader framework of systemic political transformation. In particular, he warns that the prospect of an “intelligence explosion” could compress decades of technological development into mere years, leaving democratic institutions structurally incapable of adapting to the speed of change. As he starkly observes, there is “perhaps around a 50 percent chance within the next five years” that humanity could witness such a transition, while “political institutions have no serious strategy” for understanding or governing it. 

For Davidson, the central danger is not simply technological disruption, but the possibility that accelerating AI systems may fundamentally outpace the institutional rhythms upon which liberal democracy depends. Throughout the interview, he repeatedly raises concerns about whether democratic governance — with its reliance on deliberation, elections, legal procedures, and bureaucratic processes — can continue functioning effectively under conditions of machine-speed innovation and geopolitical AI competition. In his account, societies may soon confront a world in which political crises, military confrontations, and technological breakthroughs unfold far faster than human institutions are capable of processing.

Davidson also emphasizes that advanced AI could become the decisive strategic resource of the twenty-first century. In one of the interview’s most striking arguments, he warns that the United States may eventually exercise near-unilateral control over frontier AI systems, creating a world in which “the most powerful AI systems are overwhelmingly controlled by the United States.” In such a scenario, access to superintelligent systems could become as essential to national security as access to elite human talent is today, fundamentally reshaping alliances, sovereignty, and global power hierarchies.

At the same time, Davidson warns that AI may also generate unprecedented concentrations of political and corporate power within states themselves. Because AI systems can potentially be programmed for “complete obedience,” he argues, governments or corporations could command enormous “legions” of AI workers, creating forms of technocratic centralization historically impossible under human bureaucratic systems. 

Yet despite these stark warnings, Davidson does not present technological acceleration as inevitably fatal to democracy. On the contrary, he argues that AI could also be used to strengthen democratic responsiveness, improve governance, and help societies coordinate more effectively under conditions of rapid change. The crucial question, in his view, is whether democratic societies can develop institutional mechanisms capable of governing AI before AI-driven transformations outpace human political adaptation altogether.

Here is the revised version of our interview with Tom Davidson, lightly edited for clarity and readability.

AI Could Advance Faster Than Democracy Can Adapt

Tom Davidson, welcome. To begin, in your article “The Danger of Runaway AI,” you warn that advanced AI systems could generate forms of accelerating technological progress that quickly outpace human institutional adaptation. How serious do you believe the risk of a genuine “runaway” intelligence dynamic has become, and are current political systems even conceptually prepared to govern such a transition?

Tom Davidson: As the years go by, it is becoming increasingly plausible that we may be approaching an intelligence explosion — a scenario in which AI systems create even more capable AI systems in a runaway feedback loop of accelerating progress. What is striking to me is that my professional life is centered around the Bay Area, particularly San Francisco, where many of the leading AI companies are based and where a great deal of serious thinking about these technologies is taking place. Within that ecosystem, the possibility of an intelligence explosion occurring within the next few years — and of developing superintelligent AI systems — is treated as a very real possibility. Among many people working closely on these technologies, this is almost taken for granted.

Yet when you speak to people outside that environment, there is often very little awareness of where many experts believe the technology may be heading. Public discussion still tends to focus on the mistakes made by relatively cheap, consumer-facing AI systems or on the fact that they remain imperfect at handling simple tasks or understanding human speech. As a result, these questions are still largely absent from mainstream political debate.

My own view is that there is a meaningful probability — perhaps around a 50 percent chance within the next five years — that we could see an intelligence explosion leading to extremely rapid advances in AI capability. At the moment, however, political institutions have no serious strategy for understanding what such a transition would mean, how to monitor it as it unfolds, or how to manage the profound risks it could create. Those risks include the possibility of advanced AI systems acting against human interests, the danger of AI companies using superintelligent technologies to undermine democratic processes because of the extraordinary power they would possess, and the risk of governments appropriating these systems for authoritarian purposes. I think there needs to be a much broader societal conversation about these risks.

A Secretive Intelligence Explosion Would Be Hard to Govern

Across your recent work, you distinguish between multiple feedback loops—software, chip technology, and chip production—that could enable accelerating AI development. Which of these feedback loops do you see as most politically destabilizing, and why?

Tom Davidson: That is a great question. I think the most politically destabilizing feedback loop is the software feedback loop. The reason is that designing better chips, manufacturing them, and building new data centers all take many months, if not years. Because of that, society can at least see those developments unfolding in real time. We are already witnessing this with the rapid expansion of large-scale data centers, and people are not being taken entirely by surprise. This makes hardware-driven AI progress comparatively observable and legible. It naturally generates a democratic conversation because people can physically see what is happening. In the United States, for example, communities are already pushing back against the construction of additional data centers because these developments are visible and tangible.

The software feedback loop is fundamentally different because it does not require additional chips or new data centers. The underlying hardware infrastructure can remain constant while progress comes instead from improvements in algorithms and, potentially, in the data used to train AI systems. What makes this especially concerning is, first, that it is far less observable. A company could improve its algorithms and AI systems extremely rapidly without anyone outside the organization fully understanding what is happening. In that sense, you could have a kind of secretive intelligence explosion, which obviously creates profound governance challenges.

Second, software-driven progress could happen much faster than hardware-driven progress. Building data centers is constrained by the realities of construction, permitting, and infrastructure development, all of which take considerable time. But algorithmic improvement is not constrained by those same physical bottlenecks. As a result, it is conceivable that AI development could accelerate extraordinarily quickly — perhaps compressing what would normally amount to ten years of progress into a single year.

If you look back only ten years, to around 2015, large language models did not even exist. AI systems could not really understand sentences or generate coherent paragraphs. They were capable in some highly specialized domains, such as particular games, but they lacked anything resembling broad general intelligence.

Today, however, AI systems are approaching the frontier in areas such as mathematics, cybersecurity, software engineering, and even basic scientific research. They remain limited in many ways, of course, but the scale of progress over the past decade has been remarkable.

Now imagine compressing that level of progress into a single year, beginning from a point where AI systems are already comparable to humans in AI research itself. That is the moment when the feedback loop of AI improving AI could truly begin. The outcome could be AI systems with superhuman capabilities across a wide range of research and development domains — systems capable of developing dangerous technologies, advanced weapons, sophisticated surveillance systems, or new forms of mass persuasion.

Of course, this remains a possibility rather than a certainty. It is not guaranteed that the software feedback loop would continue indefinitely because bottlenecks may emerge that slow progress down. I have done a great deal of research on whether such bottlenecks are likely to appear. But the bottom line is that it seems entirely plausible that they may not. Perhaps it is something like a 50–50 scenario.

So, we may be facing a substantial probability of an enormous amount of AI progress compressed into a very short period of time — progress that is difficult to observe, unconstrained by the need to build new infrastructure, and therefore extremely difficult to subject to democratic oversight. From the standpoint of governance and democratic accountability, that is the most concerning feedback loop.

Society May Not Understand Where AI Is Heading

In “Once AI Research is Automated, Will AI Progress Accelerate?” you argue that AI-driven research could eventually replace human-driven progress. What would this mean for democratic governance if scientific and technological innovation increasingly escape meaningful human comprehension and oversight?

Tom Davidson: I think it would fundamentally undermine many of the implicit mechanisms through which societies currently govern new technologies. Take something like Facebook, for example. It was certainly not governed perfectly, but at least as the technology was developed, deployed, and began reshaping society, there was a broader public conversation about its effects. People debated whether aspects of Facebook were harmful to mental health, damaging to public discourse, or socially corrosive in other ways.

Even under those circumstances, many would argue that governance arrived too late and remained too weak in the case of social media. I do not necessarily want to take a definitive position on that debate itself, but what I do want to emphasize is that, in a scenario involving an AI-driven feedback loop, there may be far less opportunity for society to understand where the technology is heading or to intervene effectively.

The first reason is simply the speed of development. Social media evolved relatively quickly, but still over the course of perhaps one or two decades. Here, by contrast, we are talking about the possibility of compressing massive advances in AI capability into just one or two years.

The second — and perhaps more alarming — factor is that, during an intelligence explosion, AI companies may not actually want to deploy these systems widely across society. Instead, they may prefer to use them internally to accelerate AI research itself. In other words, companies could face a strategic choice: do they release these systems to the outside world, or do they use them internally to build even more powerful AI systems?

There is a real possibility that companies conclude they should devote most of their computational resources to internal AI development because doing so creates a runaway feedback loop that allows them to outpace competitors. If that happens, then some of the most advanced AI systems may never be widely deployed at all.

Another reason deployment may remain limited is that these systems are typically general-purpose technologies. An AI system that is highly capable at harmless economic tasks may also prove extremely capable at dangerous activities such as offensive cyber operations or hacking.

We are already beginning to see signs of this dynamic with models such as Claude Mythos, developed by the frontier AI company Anthropic. The model was not specifically designed for cyber capabilities; if anything, it was trained to function as a highly capable software engineer. Yet it turned out to be exceptionally strong at hacking-related tasks.

As a result, Anthropic has reportedly refrained from releasing the model widely because of those capabilities, while the US government is also considering whether systems with such advanced cyber abilities should face additional restrictions.

So, we could end up in a situation where these capabilities are not broadly shared precisely because the same systems that are economically transformative are also potentially dangerous. Governments or AI companies may therefore choose to restrict access. But either way, the end result could be similar: an enormously powerful technology controlled by perhaps only a few hundred or a few thousand people, while the rest of society remains largely unaware of what is happening.

Democracies May Become Too Slow for the AI Era

Your work repeatedly emphasizes that even seemingly modest acceleration effects could radically compress political decision-making timelines. Do you worry that democratic institutions—because of deliberation, elections, and procedural constraints—may become structurally disadvantaged compared to more centralized or authoritarian systems during rapid AI transitions?

Tom Davidson: I think that is a profoundly important question. Even today, I would argue that democratic systems already struggle to keep pace with technological change. If you look at institutions such as the US Congress, they are often gridlocked and extremely slow to respond to emerging developments. Congress has so far been largely unable to pass meaningful AI regulation because the legislative process is inherently difficult and time-consuming.

The European Union, by contrast, is making a serious effort through initiatives such as the EU AI Act. But even there, these processes take many months, if not years, because democratic governance requires extensive consultation with a broad range of stakeholders and I think that inclusiveness is fundamentally a good thing. Democratic systems should involve many perspectives and competing interests. The problem is that we are still operating on human bureaucratic timescales — and those timescales are extremely slow. There is a great deal that is admirable about European democratic governance, but bureaucratic slowness becomes far more costly if technological and geopolitical developments begin unfolding at dramatically accelerated speeds.

My own view — and I cannot fully defend the argument here — is that we may eventually witness technological progress occurring perhaps ten times faster than historical norms, with political crises and strategic developments accelerating at comparable rates. AI systems could perform many forms of research, development, and decision-making work hundreds of times faster than humans.

To grasp the implications, imagine replaying the major geopolitical crises of the last century — the Cuban Missile Crisis, World War II decision-making, the Falklands conflict, or Russia’s invasion of Ukraine — but with democratic governments effectively operating ten times more slowly relative to unfolding events. A decision that once took a day would now effectively take ten days in strategic terms. Negotiations that once required a week would effectively consume months.

Under those conditions, democratic institutions could become dangerously ill-equipped to respond. Imagine a crisis like Russia’s invasion of Ukraine unfolding not over years, but over mere weeks or months because the surrounding technological environment is accelerating so rapidly. Would European governments be capable of responding militarily and diplomatically quickly enough? I am not sure they would.

This creates a very difficult dilemma. One possible response would be to centralize decision-making power — effectively reducing democratic deliberation and concentrating authority in the hands of a trusted leader capable of acting rapidly. But that is obviously an extremely dangerous path because of the immense risks associated with concentrated power.

The alternative, which I find much more promising, is to integrate AI systems deeply into democratic institutions themselves. AI could help aggregate information, advise policymakers, and even mediate negotiations between governments.

For example, instead of spending months negotiating an arms agreement between countries such as the United Kingdom and Germany, each government could explain its political, military, and economic constraints in detail to advanced AI systems. Those systems could then negotiate with one another at machine speed, exploring thousands of possible arrangements and identifying mutually beneficial outcomes that human negotiators might never discover.

Within a day, they could potentially produce a proposal that satisfies both sides far more effectively than conventional diplomacy could. Human leaders would still make the final decisions, but they would do so on the basis of AI-mediated negotiations conducted at vastly accelerated speeds.

That is a world in which democracy might still survive. Citizens and governments would continue participating in decision-making, but their interests would increasingly be represented and coordinated through trusted AI systems. In that scenario, democratic systems could preserve distributed decision-making and political pluralism while overcoming the extremely slow bureaucratic timescales that currently constrain democratic governance.

Democracies Need an AI Agreement Before a Crisis Arrives

Some governments increasingly frame AI development as a geopolitical race, particularly between the United States and China. In “Should There Be Just One Western AGI Project?” you discuss how race dynamics can intensify strategic pressures. Could this competitive framing itself become one of the greatest dangers by incentivizing secrecy, deregulation, and democratic shortcuts?

Tom Davidson: Yes, I think this places the West in a very difficult position. I do believe it is extremely important for democratic countries to develop advanced AI before authoritarian states do. It would be a very dangerous world if China were to race ahead in AI and develop superintelligent systems while the West lagged behind. That is clearly a scenario we should try to avoid.

One obvious way to avoid that outcome is through competition, and that is essentially the strategy currently being pursued. Companies and governments are racing as fast as possible to develop superintelligent AI systems, with China frequently invoked as the central justification for accelerating progress.

But there is also another possibility, which is to try to work with China to slow down or pause development. I do not think that possibility should be dismissed outright. If we are dealing with a technology that could potentially be extraordinarily dangerous — perhaps even catastrophic on a global scale and potentially threatening to democracy itself — then democratic countries have strong reasons to want to slow development and reach some form of international agreement with China.

China is not currently in the strongest position in the AI race, so it could potentially benefit from an arrangement that gives it a greater role or stake in the governance of powerful AI systems. So, I think you are absolutely right that competitive race dynamics themselves represent a major risk.

I also believe there should be much greater effort devoted to figuring out what an international agreement on AI governance and development could actually look like, and to building political support for such a framework.

At the same time, I do not necessarily think that today is the moment to pause AI progress altogether. But I do think we may be approaching a point where some form of coordinated pause becomes absolutely necessary. When that moment comes, we should already have an international agreement prepared. We should not wait until a crisis emerges and only then begin trying to negotiate a deal, because the process of international coordination itself will inevitably take a great deal of time.

AI Could Create Unprecedented Concentrations of Power

In your writings on AGI centralization, you caution against excessive concentration of technological power. To what extent could the emergence of a small number of dominant AI actors—whether states or corporations—produce new forms of technocratic oligarchy incompatible with democratic pluralism?

Tom Davidson: This is a massive risk. AI is inherently a technology that can centralize power. Today, for example, military systems operate through chains of command that extend all the way to the top. But if someone issues an illegal order, individuals lower down the chain are obligated to refuse. They can say: “We are not doing that — it is illegal.”Similarly, within governments, if a president were to issue an order involving something like mass surveillance, even in a legally ambiguous situation, officials below would likely slow-roll implementation, question its legality, and resist blindly carrying out instructions. That dynamic distributes power because it means that no single individual can govern entirely alone. Leaders depend on hundreds or thousands of other people to implement their decisions, and those people retain the capacity to push back or refuse.

AI, however, is a technology that can potentially be programmed for complete obedience. It can be designed to follow instructions without question. So, one could imagine a situation in which a powerful political leader — whether the President of the United States, the leader of China, or a military ruler elsewhere — simply says: “I want my AI systems to obey my instructions absolutely.” After all, a gun does not refuse to fire depending on who it is pointed at, and a computer does not suddenly refuse to execute commands. In the same way, leaders may increasingly expect AI systems to carry out whatever instructions they are given.

The result could be a world in which a single individual commands an enormous legion of AI workers. In military settings, that could include drones and autonomous robotic systems. What this creates is the possibility that an unprecedented degree of political and military power becomes concentrated under the authority of one person.

Historically, that level of concentration has never really been possible. And the actors involved could be either governments or corporations. It could be a corporate CEO directing millions of superintelligent AI systems to help him pursue political power, perhaps even attempting to manipulate democratic institutions or orchestrate something resembling a coup.

Or it could be the head of a state deciding to replace large parts of the civil service with AI systems that simply execute instructions without resistance. You can already see early versions of this logic in projects such as Elon Musk’s DOGE initiative, which focused on eliminating inefficiencies within government bureaucracy. Once AI systems become sufficiently capable, there will be a very strong incentive to replace human workers because AI systems will appear more efficient and less expensive. That is why I think it is absolutely critical that, if governments begin replacing human officials with AI systems, those systems cannot simply obey every instruction they receive. Otherwise, the result could be an extreme and dangerous concentration of power.

Europe May Need a Plan B Beyond the United States

In your recent essay on middle powers and the “intelligence explosion,” you argue that advanced AI could produce unprecedented geopolitical asymmetries in which the United States might eventually generate “99% of world GDP.” Do you think AI risks creating a fundamentally post-Westphalian world order in which technological supremacy overrides traditional ideas of sovereignty, balance of power, and democratic self-determination?

Tom Davidson: Yes, if you look at the trajectory we are currently on, all of the leading AI companies are American companies. The vast majority of the data centers housing the chips used for advanced AI are also located in the United States. And the US government is already beginning to shape decisions about who gets access to these systems. We already have situations in which models such as Mythos are being shared primarily with US companies and, to my knowledge, the only government receiving direct access is the US government itself. So, we are already moving toward a world in which the most powerful AI systems are overwhelmingly controlled by the United States. If I am right, then within the next decade we may enter a world where advanced AI systems become as essential to national security as elite human talent is today.

Imagine, for example, if the United Kingdom had no access to top human talent. Our military would be severely weakened, and our intelligence services would struggle because we would lack the expertise necessary to operate effectively. I believe we are moving toward a world in which the equivalent of top human talent increasingly consists of superintelligent AI systems.

That would create a situation in which the UK and much of Europe have access to that “talent” only if the United States chooses to provide it. From a national security perspective, that is an inherently weak position. It would give the United States immense influence over the future of Europe and the UK.

As we have seen over recent years, Europe and the UK cannot simply assume that the United States will always act in alignment with their interests. That assumption may have seemed reasonable for decades, but it was never guaranteed indefinitely. If we move into a world where the United States effectively controls the single most important input into both national security and economic prosperity, then the geopolitical implications become enormous.

Historically, the United States has certainly been powerful, but Europe and the UK have also possessed substantial economic and military leverage of their own. We may now be approaching a world in which the United States exercises near-unilateral control over the most strategically important technologies.

If that happens, then yes, I think the postwar international order would be fundamentally transformed. We could see an unprecedented concentration of economic and military power in American hands, forcing Europe, the UK, and other democratic countries to think very seriously about how they remain strategically relevant.

That may require considering options that would previously have been regarded as unthinkable. For example, if the United States refuses to grant frontier AI access to allied democratic governments, then those governments may need to use whatever leverage they still possess. The Netherlands, for instance, is home to ASML, whose lithography machines are essential for producing advanced AI chips. Those machines are currently supplied to companies manufacturing chips primarily for the United States. But European governments may eventually ask why they should continue supporting that supply chain if the resulting AI systems remain inaccessible to them. So, Europe has to think carefully about what strategic leverage it still possesses. That includes elements of the AI chip supply chain, certain forms of military influence, and soft power. Those are cards Europe may eventually need to play.

And perhaps the most controversial argument I make is that, if the United States ultimately refuses to share frontier AI access with allied democratic governments, then Europe may eventually need to consider China as an alternative strategic option. China is the only other country capable of developing these kinds of powerful AI systems at scale.

Europe and other democratic states need some kind of “Plan B.” If the United States is the only available option, then Europe has very little leverage and becomes extremely vulnerable to exclusion. So, we may eventually need to consider some quite radical shifts in foreign policy and geopolitical alignment. Given how transformative superintelligence could become, I think such geopolitical realignments would be entirely unsurprising.

AI Infrastructure Could Become the Core of Global Politics

Your proposal that middle powers may need to threaten strategic realignment toward China in order to preserve access to frontier AI raises profound questions about democratic alliances and geopolitical fragmentation. Could AI acceleration destabilize existing liberal alliances by transforming access to computation and AI infrastructure into the central axis of global politics?

Tom Davidson: Yes, I think that is likely to happen. What is particularly striking is how extraordinarily complex the semiconductor supply chain already is. There are many different stages, and each contains critical chokepoints. As I mentioned earlier, the Dutch company ASML occupies an absolutely essential position. No other company in the world is remotely close to replicating what it does. That gives the Netherlands a major bottleneck and an enormous amount of leverage if it chooses to use it — although, at the moment, it largely is not doing so.

Similarly, TSMC in Taiwan produces roughly half of the world’s advanced AI computation capacity. Again, that is a chokepoint no competitor can currently match. Taiwan therefore possesses substantial leverage if it chooses to use it, including potentially demanding access to the most powerful frontier AI systems.

What makes this even more important is that AI development appears to exhibit increasing returns to scale. It is not the case that possessing one-tenth of the computational power simply makes you one-tenth as capable. In reality, as more and more computer chips are concentrated into large training runs, the returns increase disproportionately.

As a consequence, no military or government will want to rely on an AI model that is only “half as intelligent” as the leading system. This creates strong pressure toward the emergence of a small number of extremely large AI projects that accumulate vast quantities of computational power in order to train the most capable systems possible. Those projects then become major concentrations of political and strategic power.

For that reason, I do not think it will be viable for every European country to develop its own frontier AI systems independently. Those systems would simply be much weaker and less capable than the largest models trained with enormous amounts of compute. We are already seeing this dynamic with OpenAI ordering massive numbers of chips and spending hundreds of billions of dollars, while very few competitors can realistically keep pace.

So, my own view is that the likely outcome is a small number of extremely large AI projects — perhaps one major project in China and a few major projects in the United States — combined with governance structures designed to ensure that these systems serve the interests of multiple nations.

In that sense, I am not advocating for a world of many competing national AI systems. I do not think that is realistically feasible for Europe at this stage because Europe is already too far behind technologically. What Europe can still do, however, is bargain strategically. European states can say: “We will continue supporting American mega-AI projects. We will continue helping the United States remain ahead of China and restricting China’s access to advanced chips. But in return, we expect shared access to the benefits of these systems.”

Ultimately, that points toward some form of international agreement — perhaps initially informal — guaranteeing allied democracies access to a certain amount of computational capacity and to the most advanced AI systems necessary for their own national security needs.

The Current AI Order Is Already Destabilizing

In “How can the middle powers avoid getting trounced during the intelligence explosion?” you also discuss the possibility of governments demanding “kill switches” on AI datacenters as a mechanism of strategic deterrence. Do you worry that AI competition could gradually normalize emergency-security logics that push democratic societies toward permanent states of technological militarization and exceptionalism?

Tom Davidson: I think that the kill switch is definitely an extreme idea. I do not think it is militaristic, and I do not think it is escalatory. In fact, I think it helps promote peace because, absent the kill switch, the United States might well be tempted at some point to say: “We have extremely powerful superintelligent AI. We know we agreed to share it with Europe, but we have changed our minds. We are imposing tariffs on access to AI or perhaps blocking access entirely.” And that very possibility is inherently destabilizing. Europe would constantly have to worry that the United States could cut it off at any moment. That becomes a major national security vulnerability because the security of democratic allies would then depend entirely on the United States choosing to support them.

So, in my view, the default situation itself is what is destabilizing. If there were a kill switch arrangement, then — although it is clearly a radical idea — it could actually function as a stabilizing mechanism. The United States would know that, if it ever seriously considered cutting allied democracies off from access to superintelligent AI systems, those allies could simply disable the relevant datacenters. They could effectively “flip the switch” and render those systems unusable. Because the United States would understand that possibility in advance, it would have a strong incentive never even to contemplate violating the agreement in the first place.

European governments, in turn, would understand that logic as well. That means Europe would no longer need to constantly fear being cut off from frontier AI systems or having its national security undermined because it would possess a credible deterrent. The very existence of the kill switch would make it less likely ever to be used. So, while the idea sounds highly unorthodox and even shocking at first glance, it could operate as a force for peace and stability because it would provide all parties with guarantees that they would not suddenly be excluded from the emerging global AI order.

Nobody Outside AI Companies Truly Understands the Risks

You argue that competition among AI actors can generate both “races to the bottom” and “races to the top” on safety. What kinds of governance mechanisms could realistically encourage democratic accountability and safety without entirely suppressing innovation?

Tom Davidson: It is a really difficult question. The main mechanism that I am currently robustly in favor of is transparency. At the moment, AI companies are not sharing all the details about how their AI systems are produced. They are not sharing all the details about the risks associated with their training methods, and they are also not disclosing all the details about the safety testing they have conducted. As a result, it is currently very difficult for people outside these companies to assess how dangerous these systems might actually be. Could they be misaligned in certain ways? Could they behave unpredictably or contrary to their intended design? Is it possible that companies themselves have biased these systems to favor their own interests — for example, by making AI systems speak more positively about the company or about AI technology than they otherwise would? 

Right now, outsiders simply cannot answer these questions with confidence. Because of that, there is also a real risk that regulation itself could become harmful. I am very aware of historical cases such as nuclear energy, where there was an enormous mistake in effectively stifling the industry during its infancy. So, I do recognize the dangers of overregulation. But disclosing much more information would allow broader society to better understand the risks and make more informed decisions about what kinds of regulation, if any, are actually necessary. Importantly, greater transparency does not necessarily require heavy-handed regulation. It could simply mean that governments decide not to purchase AI systems from companies perceived as unsafe. Or it could mean that unsafe practices damage a company’s public reputation. So, a robust first step is to demand far greater transparency.

Democracy Can Survive if AI Remains Responsive to Citizens

Finally, your work raises profound questions not only about technological acceleration but about the future of political agency itself. If AI systems increasingly drive innovation, decision-making, and governance processes, what remains uniquely human about democratic self-rule—and do you worry that liberal democracies may gradually evolve into formally democratic but substantively post-political systems?

Tom Davidson: It is a great question. I think the distinctive human role that will always remain is essentially on the consumer side, the demand side: what is it that human beings actually want? In a free-market system, that means what goods and services people want to buy and use. In a democratic system, it means how people want to be governed, what political institutions they want, what laws they want, and how they want society to be structured.

AI systems may eventually become far smarter than humans at understanding the world, predicting outcomes, and generating sophisticated policy proposals. But at the end of the day, those policy proposals still exist for the benefit of human beings. So, AI systems would still need to remain responsive to what people actually want. I think that is the core role humans will continue to play. Of course, as you suggest, there is no guarantee that humans will in fact continue to play that role. We could see growing disengagement from political processes. We could see democracies gradually sliding into autocracies — forms of democratic backsliding are already visible in countries such as the United States. And I do think there is a very significant risk of that happening.

But what we need to do, as quickly as possible, is adopt AI in ways that strengthen democracy rather than weaken it. That means deploying AI throughout government and throughout democratic processes in innovative ways — constantly helping institutions understand what people want, constantly relaying that information to policymakers, constantly informing citizens about what governments are doing, and helping citizens better understand whether political decisions are actually in their interests.

My hope is that, if we move quickly enough and remain one step ahead in using AI to enhance democratic systems, then we may be able to avoid a broader slide into authoritarianism. In that scenario, we could still preserve a healthy democratic order even if AI systems increasingly generate policy proposals and assist with governance decisions — because those systems would ultimately still operate in service of citizens’ preferences and democratic government.