For a stretch of a few hours, one of my notification channels was completely dead. No alerts came through. Nothing got flagged. And the entire time, the health check I'd built to watch over that exact channel kept reporting green.
That's the part that actually bothered me. Not that something broke, things break, that's a given when you're running automated systems across a portfolio of ventures. What bothered me was that I had a monitor specifically designed to catch this failure, and it didn't, because it was answering a different question than the one I actually needed answered.
The check was asking "can I reach it," not "is it doing its job"
The health check I'd built pinged the notification service and confirmed the connection was live. It could reach the endpoint. The endpoint responded. By every measure that check was designed to evaluate, everything was fine. What it never asked was whether a message sent through that channel actually arrived somewhere a human would see it.
Those are two different questions, and I'd conflated them without realizing it. A connectivity check tells you the pipe exists. It says nothing about whether anything is actually moving through it, or whether what's moving through it is landing where it's supposed to. I had built a system that could confidently tell me everything was fine while the thing it was supposed to protect against was actively happening.
Why this kind of gap is so easy to build accidentally
I didn't set out to build a shallow check. When I first wired up monitoring for that channel, "can I successfully connect and get a response" felt like a reasonable proxy for "is this working." It's the natural first thing to check, it's straightforward to implement, and it catches a real category of failure, the channel being completely unreachable.
The problem is that it's a proxy, and proxies degrade silently. A connection can succeed while the actual delivery downstream fails for reasons that have nothing to do with reachability, a permissions change, a rate limit, a silent API deprecation on the receiving end. None of those show up as a failed connection. They show up as messages that go nowhere while the sender reports success the entire time.
What actually fixed it
The fix was replacing the reachability check with something closer to a real activity check, confirming that a message sent through the channel actually produced a visible result, not just that the send call didn't error out. That's a meaningfully higher bar to build, because it requires closing the loop on the other end instead of trusting that a successful API response means the job got done.
It's more work to set up. It's also the only version of the check that actually answers the question I care about, which was never "can this system technically be reached," it was "if something important happens, will I actually find out."
The distinction I now apply everywhere
I've started asking a specific question of every automated check I build or inherit: does this confirm the outcome, or does it confirm a precondition for the outcome. Most monitoring defaults to the second, because preconditions are easier to check than outcomes. A server responding to a ping is a precondition for the app working. It is not the same as the app working. A database accepting a connection is a precondition for data being saved correctly. It is not the same as data being saved correctly.
The gap between those two things is exactly where silent failures live, and it's the gap I now specifically look for whenever I add a new automated check across any venture.
What I'd tell someone setting up their own monitoring
Don't stop at "is it reachable." Ask what the check would need to confirm for you to actually trust the green status without checking manually yourself. If the honest answer is "I'd still want to verify this by hand sometimes," the check isn't finished yet, no matter how green it looks. A status that lies confidently is worse than no status at all, because it replaces the instinct to double-check with false confidence, and false confidence is exactly what let this run for hours before anyone noticed.
