Alert Fatigue Masking Genuine Escalation

The volume of low-value alerts exceeded processing capacity, and the team developed — individually, then collectively — heuristics for fast closure: by familiarity (“that one again”), by source, by batch. The rare, real signal arrives through the same channel, dressed up as noise, and is subjected to the same heuristic.

This is a stable equilibrium, not an accident: every new detection rule adds volume; triage adapts by lowering the attention paid to each item; the system converges toward a state where detection nominally exists but no longer produces escalation. Nothing breaks, nothing raises an alarm: alerts are “handled,” dashboards are green. The failure is invisible in the tooling precisely because it resides in what the tools don’t measure — the quality of attention given to each event.

The cost is paid in dwell time: intrusions discovered late very often generated alerts that were seen and closed.

The precise mechanism is worth breaking down:

• the relevant alert is closed in seconds through recognition of its appearance (source, title, familiar shape), not through reading its content (machine, account, context);
• subsequent alerts from the same intrusion inherit the verdict: “already seen, already closed” — repetition, which should raise concern, instead reassures;
• escalating late requires contradicting one’s own prior closures, a cost that nothing in the process rewards.

At the post-mortem, the full chain of the intrusion is found in the tool, alert by alert, each closed within seconds. This finding — the bitterest one in the field — is the signature of the pattern.

The pattern is nearly impossible to see from the inside: the measuring instrument (the triage team’s judgment) is precisely what is degraded, and self-assessment structurally concludes “we’ve got this under control.” Identification requires measures external to that judgment:

• the escalation rate (alerts turned into investigations) and its trend: a rate trending toward zero while volume rises is the most reliable sign;
• median handling time per alert: a few seconds means closure by appearance;
• the share of batch closures and closures with no comment;
• the end-to-end test, the most conclusive one: periodically inject benign but realistic scenarios (an agreed-upon exercise, a test signal) and measure the rate of effective detection — not the alert-generation rate, but the actual human escalation rate.

If the injected signal is never escalated, the volume question is settled.

Treat volume as debt, with debt service:

• any source where nearly all alerts are closed without action gets tuned, enriched with context, or cut off. An alert one has decided never to act on is more harmful than no alert at all: it consumes attention and sustains the illusion of coverage;
• define an alert budget per analyst based on actual processing capacity, and refuse overruns the way a budget overrun would be refused;
• separate the paths: high-confidence signals (detection on a directory server, blocking on a server, a triggered decoy) do not go through the same queue as routine alerts;
• continuously measure the escalation rate and median time per alert, and review them in operational reviews on the same footing as availability.

The queue of closed alerts becomes an investigation source:

• retroactively search, across the estimated compromise window, for all alerts touching the machines and accounts in scope — including, and especially, those closed within seconds;
• treat them as timeline data points, not as faults to be prosecuted: the reckoning over triage can wait for the post-mortem, and a triage team put on trial mid-incident stops cooperating;
• ring-fence a channel during the incident: one person dedicated to new alerts within the scope, outside the normal queue rotation — the attacker keeps generating signal while everyone else is looking at old traces;
• temporarily raise sensitivity on the suspected scope, accepting the increase in noise: during an incident, the volume/attention trade-off reverses.

This pattern is not fixed by asking for more vigilance: it is a property of the triage system, not of the people running it. A fix based on individual attention reverts to the starting state within a few months. Practical corollary: self-assessment carries no weight here. Only measures external to the triage team’s judgment allow a conclusion to be drawn.