Twenty years of machines that lie
Field lessons from devices that answer ping while frozen, UPS cards that keep reporting, and screens stuck on one frame.
Machines don’t lie on purpose. But after twenty years of building devices, networks and the software around them, I’ve learned that the most dangerous state a device can be in isn’t “down”. It’s “down, while reporting that everything is fine”.
Three examples from the field.
The device that answers ping
A device shows as online in the dashboard. It even answers ping. And it’s frozen.
Ping is answered low in the operating system, by the network stack. The application doing the actual work can be hung while the system keeps replying. “Reachable” and “working” are different questions, and a lot of monitoring only asks the first one.
Lesson: a heartbeat should come from the work itself — the application showing that it’s making progress — not from a layer that stays alive after the work has stopped.
The UPS that was only a card
A UPS is usually monitored through an SNMP card. The card has its own processor and its own network connection. It can keep answering queries and reporting the UPS as healthy while the UPS itself is no longer working.
What you’re monitoring is the card’s view of the UPS, not the UPS. When the two disagree, the dashboard shows the card.
Lesson: know which component is actually making the claim, and where its knowledge ends.
The screen stuck on one frame
On an advertising screen, the picture can freeze. The screen still shows something, the device is still connected, and in the control center it appears online. Nobody notices until someone walks past it.
For an advertiser, that screen is reporting hours of playback that never happened.
Lesson: “online” is a statement about the connection, not about what’s on the screen.
The pattern
All three are the same failure. The system reports on a layer that is easy to observe, and we read it as a statement about the layer that matters: the network instead of the application, the card instead of the UPS, the connection instead of the picture.
The fix always has the same shape. Move the evidence closer to the thing you actually care about, and make it hard to produce unless the real work has happened.
That question — the device says something happened; how do you know it did? — runs through my work from vehicle telemetry and M2M networks to EV charging. It’s also the question behind ibibik, where a screen has to sign what it actually played.