Alert thresholds look like the simplest part of monitoring: pick a number, and if a metric crosses it, send a message. Anyone who has joined an operations rota knows it is rarely that simple. A threshold set slightly too low on a busy service can produce dozens of notifications a day, each technically correct and practically useless. A few weeks of that and the people receiving them begin to treat every message as background. This primer explains, for newcomers to DevOps and site reliability work, how thresholds and the settings around them decide whether alerts get read, and finishes by defining ten words that turn up constantly in alert rule files.

Why alert thresholds decide whether anyone listens

Every alert rule makes an implicit promise: when this fires, a person should look. If that promise is broken often enough, people stop believing it. The effect is gradual and well documented in operations practice. Engineers start acknowledging without reading, then muting channels, then relying on customers to report problems. Our explainer on why responders tune out after too many warnings covers the human side of this in more depth.

Thresholds are where the promise is first made or broken. Three decisions matter most: the value that triggers the rule, how long the condition must last before it counts, and what happens to related messages once one has fired.

Choosing the trigger value

Beginners often set thresholds on resources, such as CPU above 80 percent or memory above 70 percent. These are easy to measure but loosely connected to whether users are affected. A server can sit at 90 percent CPU all day while serving requests quickly. A better starting point is the experience of whoever depends on the service: response time, error rate, failed transactions, a queue that is not draining. When those move, someone is likely to be affected. When only a resource number moves, often nobody is.

Where resource alerts are genuinely useful, such as a disk that will fill completely, consider alerting on the projected time until exhaustion rather than a fixed percentage. “Disk full in four hours” deserves attention; “disk at 81 percent” usually does not.

A practical way to test a candidate value is to replay it against the last few weeks of data before switching it on. Count how many times it would have fired, then check those moments against your incident log. If the rule would have triggered thirty times and only two of those lined up with a genuine problem, the value is too sensitive. If it would have missed an incident the team remembers well, it is too loose. This evidence-first approach replaces guesswork with a small, repeatable experiment, and it gives you something concrete to show colleagues when you propose a change.

Requiring the condition to persist

Many metrics spike briefly and recover on their own. A rule that fires the instant a value crosses its line will catch every one of those spikes. Adding a duration, so the condition must hold for several minutes before the alert is raised, removes a large share of transient noise with little delay for real problems. It is one of the cheapest improvements available.

Handling the messages that follow

Once a real issue starts, many rules may fire together, because one fault shows up in several places. Grouping those notifications into a single incident, and suppressing lower-level ones while a higher-level one is active, keeps the responder focused on the cause rather than the symptoms. Most alert routing tools support this; our entry describing the component that routes and groups alert notifications explains how such a service fits between monitoring and the people on call.

For a wider view of how operations teams approach correlation and filtering at scale, TheAIOps.com publishes articles on AIOps and IT operations, including pieces on reducing the volume of alerts that reach people. It is a useful next stop once the basics here feel familiar.

Retiring rules that never lead to action

Thresholds are not set once and forgotten. A simple monthly habit helps: list every rule that fired, and for each one note whether anyone did anything as a result. Rules that fire repeatedly without action are either set at the wrong value or measuring something that does not matter. Adjust them, demote them to a report, or remove them. Keeping coverage high does not require keeping every rule ever written.

Mini-glossary of alert threshold terms

• Threshold: the value a metric must cross for a rule to consider firing.
• Evaluation interval: how often the monitoring system checks the rule against current data.
• Pending duration (or “for” clause): how long a condition must stay true before the alert is actually raised.
• Flapping: an alert that repeatedly fires and resolves as a metric hovers around its threshold.
• Hysteresis: using a different value to clear an alert than to raise it, which reduces flapping.
• Grouping: combining notifications that share labels, such as a service or cluster, into one message.
• Inhibition: automatically holding back lower-priority alerts while a related higher-priority one is active.
• Silence: a temporary, deliberate mute, usually during planned maintenance.
• Actionable alert: one where the receiver can and should do something specific in response.
• Symptom-based alert: a rule on what users experience, rather than on an internal cause.

Putting it into practice

When you next write or review a rule, ask four questions. Does it measure something a user would notice? Does the condition need to persist before firing? Will it be grouped with related alerts? And if it fires at 3am, does the person receiving it know what to do? If any answer is no, the rule probably needs work before it earns a place on the pager. Our guide to how teams respond once an incident is declared picks up from the moment an alert becomes a real problem.

Getting thresholds right is less about clever tooling and more about discipline: fewer rules, each tied to user impact, each given time to settle, each owned by someone. Those habits are what keep alerts worth reading.