TL;DR: AI can make the hacker better. To cause a catastrophe, it still has to defeat the plant, and the plant has decades of engineering, safety and physical controls the AI can't rewrite. The real risk isn't a superhuman AI ending civilisation; it's the poorly engineered environments where those protections were never independent in the first place.
AI changes the attack, not the physics
There is an increasingly common argument that an advanced AI, given access to cyber capabilities, could compromise critical infrastructure and cause catastrophic or even civilisation-ending damage.
I have spent much of my career doing the human version of part of this problem: finding ways into critical environments, and then working out what an attacker could actually do once inside. Breaking into some of these environments is often surprisingly easy, while making them do something catastrophically dangerous is frequently much, much harder.
The internet thinks AI is about to turn the lights off
Let me be clear up front: AI-enabled and AI-native attacks are credible. The offensive security industry has been using large language models to do genuinely nasty things to computers since the first models became useful enough to do so.
That might be as simple as using an LLM to help build offensive tooling, modify payloads to avoid simplistic detection, understand unfamiliar code, make use of libraries already available on a target, or just apply existing tradecraft much faster than a human could alone. I have seen and heard of LLMs and AI agents wired up to cyber tooling and used to find and exploit specific vulnerabilities on the systems they encounter. More often than not these are controlled tests. Offensive security teams can sometimes be given much broader latitude where the scope and permissions allow it, while real threat actors obviously operate without those contractual constraints, but carte-blanche, do-anything hacking is not the default for legitimate offensive security work.
This article is not about how likely an AI doomsday scenario is, what an advanced AI might ultimately want, or the politics behind why people may want you to believe the threat is imminent. There are also plenty of proposed AI risk scenarios that have nothing to do with cybersecurity or critical infrastructure, and I am not trying to address those here. I am interested in something much narrower: if an extraordinarily capable AI became very good at hacking critical infrastructure, how easily would that access translate into catastrophic physical consequence? Cyber access and physical consequence are not the same thing.
Cyber access is not physical consequence
An analogy is in order. I like to think of hacking as making keys for locks and then using those keys to get into rooms behind locked doors.
Sometimes the "key" is an exploit sent to a vulnerable web service, which gives you control over the service, the system it is running on and potentially whatever that system can reach. Other times it is ringing the help desk and convincing the operator that you are who you say you are and that you really do need your password reset so you can log in.
AI is proving to be very good at helping offensive teams, and criminals, find and create those keys. Sometimes that includes keys for doors nobody knew could be opened by anything other than the one the architects intended. We call those zero-days. There is a lot more nuance to this, but in the main the concept holds.
Now let's take zero-days and public exploits out of the picture entirely. Using nothing more than our simple scam-call technique, we get somebody's password and log in to something critical as them. We do this kind of thing often as part of an impact assessment: we've got control of your critical system, now what?
Just because you can log in to the engineering workstation that has access to the critical PLC and the safety system does not mean you can force the turbine to explode. You might be able to turn it off and give the people on-site a genuinely busy day while they work out what went wrong, but you very quickly discover that hardware, engineering, safety and physical limits stand between having software access and producing whatever physical consequence you choose.
What would "an AI" actually have to accomplish?
To bring about the collapse of civilisation, a great many things would need to happen in parallel, and it is worth remembering just how much disruption modern societies already absorb without collapsing.
When a large part of the north-eastern United States and Canada lost power in 2003, around fifty million people were affected. When Texas froze in 2021 its grid came dangerously close to a much broader and longer collapse. These were serious events with real economic consequences and real risk to life. Society continued to function. Operators recovered systems, governments responded and eventually the lights came back on.
But a widespread, lasting blackout that operators cannot recover from is a different problem altogether. The attacker (or AI) has to know which systems truly matter, break into enough of them, and stay in control while engineers and field crews fight to isolate the damage and restore power, often by hand. That is a much taller order than simply "hacking the grid".
This is one of the reasons states continue to use missiles, drones and explosives when they want reliable physical destruction in a war. Cyber can absolutely produce physical effects, and there are some very good examples of that which I will get to shortly, but if your actual objective is to destroy a turbine, a missile remains considerably more predictable than trying to discover some combination of software commands that will cause the machine to destroy itself.
Even in the comparatively simple case where an attacker shuts down a turbine and prevents remote access to it, there is also the inconvenient problem that people may still be physically standing beside it. Depending on the plant and what has happened, those people may be able to isolate systems, operate equipment locally, restore known-good configurations and start bringing the process back online without relying on the same systems the attacker compromised. It would be a serious incident, but it is still a long way from the end of civilisation.
There is also a slightly amusing circular problem with the most extreme version of this scenario: if an AI somehow succeeds in turning off enormous amounts of electricity, at some point it starts turning off the datacentres, communications infrastructure and GPU farms it presumably depends on to continue doing whatever it is doing. Human Battery as a Service may solve that one, but I will leave that for another article.
Layer 1: Cybersecurity controls
At their core, cybersecurity controls exist to enable trust. Is this person who they claim to be? Are they entitled to reach the thing they are trying to reach? Should this machine be talking to that one? Should this software be able to execute here at all?
As you move from loosely controlled environments to tightly controlled ones, actually operating the thing usually gets harder. There is natural friction between doing something with a computer conveniently and doing it securely, and there are always trade-offs. Well-designed controls make that friction less painful, but they never eliminate it completely.
Network segmentation, identity and access management, privileged access management, firewalls, application allow-listing, anti-virus, jump hosts, monitoring and secure remote access all exist to constrain what a person or system is able to do once it reaches a particular point.
Some of the controls I like best are the ones where even I cannot get to my own data without possessing the thing required to access it. Forgot the password but left an unencrypted backup lying around? Not as secure as you thought. Lost the authenticator application but kept SMS as a fallback? Again, weaker than you thought. Don't know the PIN but the service is vulnerable to some public exploit? The attacker may not need the PIN at all.
If there is no hardware key, no backup key and no alternative recovery path, the attacker now needs another route. They might compromise an already authenticated endpoint, steal a session, attack something on the trusted side of the authentication boundary or find some completely different weakness, but being smarter does not make the hardware-bound control disappear. This is relevant when we talk about AI because AI may become exceptionally good at finding paths around controls, but that is different from the controls somehow ceasing to exist because the attacker has become more intelligent.
Layer 2: Engineering and safety controls
Much as we in cybersecurity talk about defence-in-depth, engineering and safety controls provide another set of protection layers, and in my view these are the most important ones when we are talking about catastrophic cyber-physical effects. Many of them are intended to be independent of the normal control system and of the failures they exist to protect against, although how independent they really are varies significantly from plant to plant.
Let's use a power station as a simple example. Power stations commonly use turbines to drive a generator, which generates electricity. Turbines come in several types, but two of the most common are gas turbines and steam turbines.
For a steam turbine to spin, it needs steam. That steam might be created by burning gas to heat water; the water becomes steam and the steam drives the turbine blades, which in turn drive the generator.
Assume there is a PLC controlling a valve associated with the fuel supply. In one case an attacker commands that valve closed, which stops the fuel supply, reduces steam production and eventually stops the turbine because there is no longer enough steam to drive it. In another case the attacker tries to double the commanded fuel flow, but that does not mean the plant suddenly receives twice the fuel, produces twice the steam or generates twice the power because the physical limits the plant was designed around still apply.
The pipe supplying the burner is only so large, there is only so much fuel available and that fuel can only be supplied at a certain maximum rate. A digital attack does not change any of those things. An attacker might conceivably cause a relief valve to lift, disrupt the process, trip equipment or, in a sufficiently bad case, create a localised safety incident, but there are still multiple layers between changing a digital value and catastrophically destroying the plant.
The boiler and fuel system will have protections of their own, while the turbine will have dedicated protections such as overspeed trips. Operators may retain the ability to isolate fuel, close valves, trip equipment locally or otherwise place the process into a safer state, so at several points there are engineered controls intended specifically to stop an abnormal process condition turning into something like a boiler rupture or a turbine overspeeding until it destroys itself.
A Safety Instrumented System, or SIS, adds another protective layer and is normally designed to operate with an appropriate degree of independence from the basic process control system. Its job is not to run the plant efficiently; its job is to recognise defined unsafe conditions and move the process towards a safe state when those conditions occur. Where that independence is genuine, compromising the normal control system does not automatically compromise the thing that exists to protect against its failure, while environments where the safety layer has been quietly flattened back into the same control environment for convenience have lost much of that assurance.
On some of my engagements we have reached this level of access: control over the SIS itself. That is a far more serious position for an attacker to reach because if you can manipulate both the normal control system and the systems designed to protect against its failure, you have removed important layers of defence. A sufficiently capable attacker might also suppress alarms, manipulate operator displays or otherwise try to hide what is happening from the people operating the process.
Even at that point the attacker has not escaped the physical system, because mechanical protections may still exist, local controls may still exist, fuel can potentially be isolated physically, pressure can be relieved, equipment can trip independently and operators can respond to things that are physically happening in front of them. The attacker has moved much closer to consequence, but they still have to defeat the remaining engineering, physical and human layers before digital control becomes catastrophic physical effect.
What Stuxnet tells us
From what has been publicly reported, Stuxnet found a digitally controllable path into a physical variable that mattered. The centrifuge motors were controlled through frequency converters, and manipulating those converters appears to have given the attacker a mechanism for changing rotational speed in ways that could physically damage the equipment while the malware also interfered with information and protective functions available to operators.
The useful lesson is not that digital systems cannot damage physical equipment, because clearly they can. What the publicly available evidence around Stuxnet suggests is that an attacker needs a path from software, through something they can control, into a physical condition capable of producing the effect they want. Where genuinely independent protection exists between the control system and the physical consequence, the attacker has another problem to solve; where protection is absent, ineffective or relies upon the same compromised systems, the path becomes easier.
Triton provides another useful example. The malware was reported to be capable of interacting directly with and modifying Triconex safety controllers. During the incident, altered application code reportedly failed an internal validation check and caused the controllers to enter a fail-safe state, which shut down the process and ultimately helped expose the intrusion. Had the attack worked differently, the adversary may have been able to manipulate or inhibit safety functions without causing the shutdown that exposed them, although the precise intended physical outcome is not something we can know with certainty from outside the operation.
There is also a fairly mundane engineering detail from the public analysis of the incident that I particularly like: the affected controller had its physical keyswitch left in Program mode and, according to subsequent research, placing it into Run mode would have made loading arbitrary logic substantially more difficult. It is a simple physical switch, but it says quite a lot about the wider problem because a sophisticated state-level adversary can spend enormous effort getting from the outside world, through the organisation, into the OT environment, onto an engineering workstation and eventually into the safety system, while the physical operating mode of the controller still changes what is possible next.
It would be wrong to overstate the protection offered by that key because later work has shown ways in which some of these protections can potentially be bypassed under the right conditions. The point is that the key raises the cost, as do independent protection, physical isolation, different vendors, different architectures and different process designs. Cost matters when the proposed scenario is not one carefully engineered attack against one carefully studied target, but an AI attempting to produce serious physical consequences across large parts of critical infrastructure.
Attacks like Stuxnet and Triton involved nation-state resources, purpose-built tooling, specialist industrial knowledge and detailed understanding of narrow target environments.
Layer 3: Humans, operations and physical controls
There is usually a human operator or engineering team somewhere in this story, and some of the more interesting engagements I have worked on have involved understanding whether a threat actor could trick those people into taking, or not taking, the correct next action.
Imagine a worst-case scenario where we have access to essentially every digital system that matters in a power station, oil and gas facility, railway or similar critical infrastructure environment. We manipulate the underlying process while simultaneously making everything appear normal to the operator, suppressing alarms, modifying SIS logic and interfering with whatever information the engineering team would normally use to understand what is happening.
That is a genuinely serious scenario and it is also one where AI could materially improve an attacker's capability, because coordinating exploitation, process manipulation, operator deception and suppression of diagnostic information at the same time is difficult and a sufficiently capable autonomous system might eventually become very good at it.
At some point though, the process still exists in the real world. An emergency shutdown may activate, safety relays may trip, relief valves may lift, equipment may stop operating or somebody on-site may see, hear or smell something that very clearly disagrees with the indicators on the screen in front of them. This would unquestionably be a bad day and could create real risk to the people working there, but one area of industrial operations is considerably more mature than cybersecurity: safety.
There is a reason you cannot casually wander onto most industrial sites. You sit through inductions, learn where the muster points are, understand the alarms and evacuation procedures and, depending on the environment, use intrinsically safe equipment, carry gas detectors or learn how emergency shutdown arrangements work.
These organisations practise for failures because catastrophic industrial events existed long before cybersecurity did, and fires, explosions, loss of containment, mechanical failures, power failures, floods, earthquakes and human mistakes are not new phenomena. The process does not really care why something went wrong either; if a pressure relief valve lifts or an emergency shutdown activates, the immediate safety response does not fundamentally change because the initiating event happened to be malicious software instead of a failed bearing. The objective remains protecting people, protecting the environment, protecting the plant and getting the process back into a safe operating state.
AI changes the attack, but not the physics
None of this means AI is irrelevant to cybersecurity. Quite the opposite: AI will accelerate reconnaissance, vulnerability research, tooling development and the analysis of unfamiliar environments, and an AI-enabled attacker may eventually be able to identify the engineer with access to the particular application it needs, compromise that person's account, move through the environment using their privileges, analyse technical documentation and PLC logic, understand unfamiliar protocols and decide what the next useful step in the attack should be without waiting for a human operator to choose each action. Doing that quickly and at scale would be a substantial change to offensive security.
What AI cannot do is retrospectively change how the plant was constructed. It cannot make a pipe larger, create an actuator that was never installed, increase the mechanical capacity of a valve, remove a passive barrier or create energy that does not exist within the process, although it can become much better at finding ways to abuse the mechanisms which do exist.
Every cyber-physical attack ultimately requires some path from software, through a controllable component, into a physical process capable of producing the desired effect. Stuxnet appears to have found such a path and Triton appears to have targeted an even deeper layer, while future AI-enabled attacks may become extremely good at identifying those paths and understanding how to use them. The laws of physics do not prevent cyber attacks from causing physical damage; they define the boundaries within which that damage is possible.
The risk we should actually be preparing for
AI raises cyber risk substantially. It will make capable attackers faster and may reduce the level of skill required for less capable attackers to achieve things that currently need specialist knowledge. Reconnaissance will become cheaper, unfamiliar systems easier to understand and some operations that once required a team of experienced people may eventually be coordinated by a much smaller number of humans, or by software acting substantially on its own.
Critical infrastructure and operators of operational technology should be particularly concerned about this because everything I have described above assumes the protection layers actually exist and are genuinely independent, which in practice is often not the case. I have seen safety systems sitting on control networks, interlocks bypassed for convenience and never restored, remote write-access to sensitive systems that nobody remembered enabling, and manual fallback procedures that existed very convincingly on paper but had not been exercised in years.
That erosion is where I think a large part of the real risk sits, and an AI-accelerated adversary will find these weaknesses faster because they are exactly the same weaknesses a good human attacker looks for today.
I have had to use the words "mass-casualty event" in a report only once in my entire career. The possibility was real, otherwise we would not have written it down, but getting there required a very unusual sequence of things to happen: cyber compromise, specific knowledge of the physical process, manipulation of the control systems, defeat of safety systems, failure of physical protections and unsuccessful human intervention. The probability became a very small percentage of a very small percentage of a very small percentage.
AI may make some of those individual steps considerably easier, faster or cheaper, and may eventually make some of them possible at a scale we have not previously had to contend with. That is precisely why AI-enabled attacks against critical infrastructure should be taken seriously, but the practical response is to understand which assumptions stand between compromise and consequence, and whether those assumptions survive contact with a capable attacker.
So if you own or operate this kind of environment, the useful questions are not especially futuristic:
- Is the safety instrumented system genuinely independent of the control environment, rather than merely drawn that way on a network diagram?
- Can any digital path influence the physical protections intended to remain independent?
- Does the manual fallback actually work, and has anybody tested it recently?
- Could a trusted insider with full access and knowledge create the consequence you are worried about?
- Most importantly, have these assumptions actually been tested rather than simply asserted?
Those questions are also testable. This is the kind of work we do at OscarTango: start from the consequence you actually care about, work backwards through the engineering and cyber layers, and test whether the assumptions keeping that consequence unlikely are really true. The useful output is not another maturity score or a diagram showing that controls are supposed to exist; it is knowing which paths an attacker can actually take, which protection layers genuinely stop them, and where the gaps are before somebody else finds them.
If you operate critical infrastructure and cannot answer those questions with confidence, that is probably a more useful place to start than trying to predict what the next generation of AI might be capable of.
By Aaron Fenwick, Founder of OscarTango. He has spent more than 20 years on the offensive side of OT and critical infrastructure security — including engagements that reached the safety layer itself.