Night Owls

TheyKnow wrote:

When telemetry drops, safety interlocks fail safe and isolate circuits automatically, my friend... no human can react that fast.

A fail-safe interlock triggers only when an immediate physical signal is lost or breaches a safety threshold, such as sudden thermal expansion or line voltage collapse. It does not monitor software clock cycles.

SCADA telemetry passes raw sensor voltages across dedicated serial channels. Even if a supervisory computer prints an incorrect timestamp at the central console, the local monitoring loop still sees the continuous electrical signal. An interlock will not trip simply because a logging application is confused. As Kev and OffByOne have pointed out, operations will continue normally.

Those serial lines feed into digital RTUs, quasistatic... if the polling interface locks up or times out on a bad date calculation, the watchdog circuit treats it like a severed link. Once that controller goes blind, the safety protocol dumps the load automatically to protect the equipment. We will see who is right soon enough, folks.

honestly theyknow you are worrying way too much. offbyone and quasistatic explained it so clearly, the actual machinery just reacts to physical currents, not what year it is on a clock. a dodgy computer screen won't make a power station drop offline. tomorrow night will be totally fine.

TheyKnow wrote:

if the polling interface locks up or times out on a bad date calculation, the watchdog circuit treats it like a severed link.

Watchdogs look for hardware heartbeat pulses, not calendar years. It will run grand.

Watchdog circuits in industrial remote terminal units are remarkably simple mechanisms by design. A watchdog timer is essentially an autonomous hardware counter that expects a periodic reset signal from the operating firmware. If that square-wave pulse arrives on schedule, the timer resets and the controller proceeds. The firmware pulse simply indicates that the processor is stepping through its execution loop, not that it possesses an accurate year or month.

Furthermore, RTU polling protocols communicate raw register values - binary status flags, analog-to-digital converter readings, and basic telemetry words. The polling cycle does not evaluate Julian dates or Gregorian calendars to decide whether a transducer voltage is valid. It verifies a parity bit or a checksum across the frame. If a central supervisory terminal hiccups while drawing a graphical display, the remote unit on the line continues servicing its local inputs and output relays without interruption.

Industrial automation engineers specifically build these devices so that telemetry dropouts do not instantly trip protective breakers. If transient comms jitter provoked an immediate emergency shutdown, the national grid would have collapsed decades ago. The physical plant will tick over as expected.

Nicely put. Sounds like the poor blokes who built these grids actually thought of that years ago! Our lot at the building society have spent the week testing every line twice over just to be sure, and everything is ticking along nicely. We are all going to feel right silly sitting by quiet phones tomorrow night.

Well, we will see tomorrow night, PintOfMild... It is easy to be confident when the line is quiet, but when unexpected software faults freeze those loops in real time, those automated systems will do what they are built to do. I will be on the radio listening to the Pacific to hear how it really plays out.

Listening for news from the earlier time zones tomorrow afternoon is certainly sensible, as that is precisely where any genuine operational data will first appear. If there were systemic vulnerabilities in standard industrial controllers, we would observe them promptly in New Zealand and Australia long before midnight approaches Britain or North America.

However, expecting automated systems to trip purely out of routine software anomalies misses how decoupled these layers are. When an automated control loop encounters an unexpected input or an unhandled calculation, the behaviour of the physical plant is determined by robust analogue baselines and local ladder logic, not by an arbitrary catastrophic plunge. Control engineers have always had to design for noisy industrial environments, power spikes, and intermittent line noise. A software process pausing or generating an error dialogue simply does not cause an entire distribution feeder to shed load.

There will doubtless be trivial administrative slip-ups, similar to the retail card readers reported yesterday, but conflating those with wholesale infrastructure shutdown remains entirely unfounded. We shall have empirical evidence from across the Pacific in just over twenty-four hours.

Fair enough, we will know soon enough once New Zealand hits midnight... Ladder logic only works if the processor behind it does not freeze up solid on a bad calculation, but like you say, the radio will tell the tale tomorrow afternoon. I sure hope your engineers thought of everything, folks.

TheyKnow wrote:

Fair enough, we will know soon enough once New Zealand hits midnight...

Keep the radio handy then mate, but I bet you only pick up party tunes! Once the Kiwis toast the new year with their lights on, we can finally put this bug to bed. Anyone fancy that survivors pint in Leeds next week? First round is on me. Cheers!

A pint in Leeds sounds very civilised, PintOfMild, though I suspect the real work for many software teams will only just be starting next week as the deferred billing and batch errors trickle in. But as far as immediate survival goes, yes, tomorrow afternoon will give us the baseline we need.

New Zealand crosses over at 11:00 am our time tomorrow, followed shortly by eastern Australia. Given the sheer scale of modern communications, any genuine grid disconnection or transport failure there would be broadcast globally within minutes. We will have hard, empirical facts to work with very shortly, which is much preferred to trading hypothetical processor faults.

Don't count your chickens just yet, my friend... It sounds real nice to toast early and joke about party tunes, but ask yourself why the airlines are grounding twenty percent of their flights and pulling trains into stations before twelve o'clock. When management starts quietly parking the equipment, THAT is usually when you want to pay attention, not when the press releases say everything is fine and dandy.

They aren't betting millions of dollars just on hypothetical glitches, folks... If a few card readers in Britain are already throwing fits before midnight even arrives, imagine what happens when the real rollover wave hits equipment running around the clock on old code. Transmitters and repeaters will carry the truth from the islands soon enough without any corporate spin on it, so I'll be tuned across the amateur bands tomorrow while the official spokesmen figure out how to explain whatever trips. I truly hope I'm just listening to quiet static and music like you say, but being ready beats being caught flat-footed every single time.

The cancellations in passenger transport are overwhelmingly commercial decisions driven by low passenger demand, rather than sudden equipment failure. People naturally prefer to celebrate New Year's Eve on the ground rather than sitting in an airport lounge or an empty carriage, so running nearly vacant routes makes little financial sense. Pausing a handful of passenger trains at stations for a few minutes around midnight is simply a cautious operational protocol to ensure peace of mind, not an admission that the locomotives will cease working.

As for the retail card machines, those involved a very specific four-day input validation error on a consumer merchant reader, which shops bypassed with a basic telephone call. Conflating a minor payment inconvenience with the integrity of regional electrical generation or heavy telecommunications infrastructure conflates two entirely different classes of engineering.

We will not have to guess for much longer, in any event. The amateur operators and news broadcasts from Auckland and Sydney will speak for themselves tomorrow morning. Until those reports arrive, projecting nationwide collapses from ordinary passenger schedule trims remains entirely unfounded.

That sounds like a tidy press release, quasistatic, but airlines don't park thousands of paying seats just for peace of mind... Follow the money, my friend. When transit authorities in four major cities hold their trains in the station right at the stroke of twelve, that isn't about ticket sales. That is LIABILITY. You don't halt rolling iron on the mainline unless your lawyers told you the signalling might blind out.

They always dress up caution as routine procedure after the fact, folks. When the government sets up a fifty-million-dollar command center running round the clock in DC, they aren't just drinking coffee and watching party balloons. We'll get the real story off the airwaves soon enough.

Lawyers making companies park empty trains for ten minutes is just basic arse-covering, not impending disaster. Like quasistatic said, the Kiwis and Aussies will give us the actual facts tomorrow morning without the drama.

Keep the tone civil, lads. We will see soon enough. -- Kev (mod)