system
Given the following post, respond as if you're commenting on Reddit:
human
What does the development/testing process look like when working with dangerous systems?

Think things like power plants, refineries, heavy machinery, etc. When bugs can potentially be fatal, how does the industry approach problems like this? When people rely on fail-safes for these systems and the fail-safe is carried out through a computer program, how can we be certain that the fail-safe will work every time?
gpt
I worked at the UK National Grid for seven years. They update stuff as seldom as possible.

A major refresh happens about every 5 to 8 years. They run the old and new systems in parallel for about six months before it gets formally certified.

They have an upgrade cycle of six months, and every change is tested by the vendor, then by the client on a development system, and then on one of the live sites for two weeks to allow for instant roll-back.

They run an eight-way redundant system. A server has two of everything -- apps processor, database cluster, telemetry/control managers, raids, UPS, diesel generator: and three supervisory processors (so they get a majority vote) which switch the rest of the units in and out as needed.

A site has two complete server complexes, far enough apart that a terrorist attack cannot get them both.

There is a complete duplicate of that site 80 miles away, behind a range of hills, so one nuclear hit will not get both. The sites are connected by six private circuits.

And yet, the main vendor managed to cause the US north-east blackout in 2003, which killed around 100 people, and took down NYC subway, Amtrak, water supplies, cell-phones, airports, and the power supply to 50 million people. They lost stability of the power network, they lost 265 generators that turned themselves off for their own protection, they lost all the system alarms because the telemetry was overloaded, and then --- they managed to lose all power to their own control room.

When a fail-safe system fails ...