Alert channels and rules

Getting alerts working takes two objects, not one, and knowing that up front saves the most common hour of confusion:

• A channel is where an alert goes — a Slack workspace, a phone number, a PagerDuty service.
• A rule is when it goes there — which monitors, which events, how often to repeat.

Creating a channel alone sends you nothing. It is a destination with no instructions. You need at least one rule pointing at it.

The channels

There are 13 types:

• Email
• Slack
• Discord
• Microsoft Teams
• Telegram
• Webhook
• PagerDuty
• Opsgenie
• Pushover
• SMS
• Voice call
• Mattermost
• Google Chat

Each has its own setup — a Slack webhook URL, a PagerDuty routing key, a phone number — entered once when you create the channel. Anything secret is encrypted before it is stored and is never shown again; the form displays *** and you replace the value rather than editing it.

Test it before you rely on it

Every channel has a Test button that sends a real message through the real adapter — not a simulation. If your Slack webhook URL is wrong, you find out immediately rather than during an outage.

The rules

A rule connects monitors to a channel and says which events should fire.

Which monitors. A rule can target one specific monitor, or every monitor with a given tag, or all of them. Tag-based rules are usually what you want: tag a monitor production when you create it and it inherits the routing you already set up, instead of needing a new rule each time.

Which events. Independently switchable:

• Opened — an incident started.
• Recovered — it ended.
• Flapping — the monitor is oscillating rather than settling. Worth routing somewhere quieter; a flap is usually a symptom of a threshold set too tight.

How often to repeat. A renotify interval re-sends while an incident stays open, so something that breaks at 2am is still visible at 8am. Off by default.

Rules stack. A monitor can match several, and each fires independently — which is how “everything to Slack, production also to PagerDuty” works, with two rules and no special case.

Severity

Incidents carry a severity of minor, major or critical, and rules can route on it. The common shape is everything to a chat channel and only critical to a pager, so the phone stays quiet for things that can wait.

Keeping the noise down

Three mechanisms exist for this and they are described in full on reducing alert noise:

• Maintenance windows suppress alerts during planned work.
• Events group incidents that open together, so one bad deploy is one event and not forty pages.
• Dependencies let you say a database backs an API, so a database outage pages you about the database rather than about everything downstream of it.

Before reaching for any of them, check the monitor’s own regions required to alert setting — see HTTP monitors. A large share of avoidable pages are a single region with a bad path to your host, and a quorum fixes that at the source rather than filtering it afterwards.

What it costs

Most channels are free to send. SMS and voice calls are billed per message because the carrier bills us; email, Slack, webhooks and the rest are not.

Free plans are limited to email channels. Every other type needs a paid plan.