Alert rules that do not wake you for nothing

An alert has one job: get a human to act when acting is needed. Every alert that fires and leads to no action teaches the team to ignore alerts, and the next real one is ignored too. Noisy alerting is not a tooling problem; it is a design problem, and the design rules are the same in Prometheus, Grafana, Zabbix or a cloud provider’s console.

Rule 1: alert on symptoms, not causes

Users do not care that CPU is at 95 percent; they care that pages are slow or failing. Alert on what the user experiences (error rate, latency, availability, a queue that is not draining) and treat resource metrics as dashboards you look at after a symptom alert fires. A high CPU alert that fires during every nightly batch job is the canonical example of a cause alert that should have been a graph.

The exception is a cause that will become a symptom with certainty and lead time: a disk that will be full in 24 hours, a certificate that expires in 7 days, a backup that did not run. Those are worth alerting on, as tickets, because they can be fixed before anyone notices.

Rule 2: every alert needs a duration

A single scrape where a host is unreachable or latency spikes is noise. The for clause requires the condition to hold continuously before the alert fires. Five minutes for availability, ten for error rates and thirty for slow-moving resource trends are reasonable starting points. If a condition clears before the duration passes, nobody is bothered, and nothing was lost because the system recovered on its own.

Rule 3: two severities, two channels

• page: somebody must act now, at any hour. Users are affected or will be within minutes. Goes to the on-call phone.
• ticket: somebody must act this week. Goes to a chat channel or issue tracker, during office hours only.
• Anything that is neither is not an alert. Record it as a metric, show it on a dashboard, delete the rule.

The test for page: would you be comfortable waking a colleague for this at 3 a.m.? If the honest answer is “it can wait until morning”, it is a ticket.

Rule 4: let the alert manager do the arithmetic

When a switch dies, forty hosts go down; you want one notification, not forty. When a host is down, its disk alert is irrelevant. Alertmanager handles both: grouping folds related alerts into one notification, inhibition silences dependent alerts while their parent fires, and time intervals hold ticket-level alerts until office hours. Here is a Prometheus rule file that applies rules 1 to 3:

groups:
  - name: node
    rules:
      - alert: HostDown
        expr: up{job="node"} == 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.instance }} is not reachable by Prometheus"
          runbook: "https://wiki.example.lan/runbooks/host-down"

      - alert: DiskWillFillIn24h
        expr: |
          predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24 * 3600) < 0
          and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes < 0.2
        for: 30m
        labels:
          severity: ticket
        annotations:
          summary: "{{ $labels.instance }} {{ $labels.mountpoint }} fills up within a day at the current rate"

      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{code=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "{{ $labels.service }}: more than 5% of requests fail"

And the matching Alertmanager routing:

route:
  receiver: tickets
  group_by: [alertname, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: [severity="page"]
      receiver: oncall
      repeat_interval: 4h
    - matchers: [severity="ticket"]
      receiver: tickets
      active_time_intervals: [office-hours]

time_intervals:
  - name: office-hours
    time_intervals:
      - weekdays: ["monday:friday"]
        times:
          - start_time: "08:00"
            end_time: "18:00"

inhibit_rules:
  - source_matchers: [alertname="HostDown"]
    target_matchers: [severity=~"page|ticket"]
    equal: [instance]

• group_by collects alerts with the same name and service into one message; group_wait gives late arrivals 30 seconds to join it.
• repeat_interval controls how often an unresolved alert is re-sent; four hours for pages is enough to remind without flooding.
• active_time_intervals delivers ticket alerts only on weekdays between 08:00 and 18:00; they still fire and are visible, but the notification waits.
• The inhibit rule suppresses every other alert for an instance while HostDown is firing for that same instance.

Rule 5: every page links to a runbook

The person who wrote the alert knows what to do about it; the person who receives it at 3 a.m. might not. A runbook URL in the annotations, pointing at a page with what this means, how to confirm it, what to try first and who to call if that fails, turns a stressful alert into a checklist. If you cannot write a runbook for an alert, that is a strong hint that nobody knows what action it requires, and it should not page.

Rule 6: review what fired

Once a week or after every on-call shift, list the alerts that fired and ask for each: did it require action? Was the action obvious from the alert? Did it fire too early, too late, or for too long? Alerts that fired and needed no action are tuned (longer for, higher threshold), downgraded to ticket, or deleted. Incidents that happened without an alert get a new symptom alert. The alert set is never finished; it tracks how the system behaves this month.

Signs your alerting is in trouble

• People acknowledge alerts without opening them.
• A chat channel receives more than a handful of alert messages per day and nobody reads it.
• Thresholds were set once by guessing and never revisited.
• The same alert fires and resolves repeatedly (flapping) and everyone knows to wait it out.
• Nobody can say what action an alert expects.

Each one is fixable with the rules above, and fixing them is cheaper than the outage that gets ignored because it looked like all the others.

Official documentation

• Prometheus: Alerting best practices
• Alertmanager configuration
• Google SRE book: Monitoring distributed systems