baseboard
Blog

Alerts you won’t mute in a week

2 min read

Most alert channels die the same way. The first day posts forty old problems at once. A monitor flaps for a minute and posts twice. Nobody hears when things recover, so every red message stays open in someone's head. By the second week the channel is muted, and the one alert that matters goes unread.

When we built alerts into Baseboard, we wrote down those failures first and designed against each of them. Here's what came out.

Start quiet

A new rule doesn't post about anything that's already wrong. The first check after a rule is made, changed or turned back on only notes the current state. From then on you hear about new problems. If you want the existing ones, they're already on the Overview page.

Ignore blips

Some alerts are about something that happened: a deploy failed, a new error appeared. Those post once, and Baseboard remembers them for three days so the same event isn't posted again.

Others are about an ongoing state: a monitor is down, CI is red on the default branch, revenue is below a level you set. Those post only after the problem shows on two checks in a row. Checks run every five minutes, so a thirty-second blip never reaches Slack.

Say when it's fixed

When an ongoing problem clears, Slack edits the original message to say it's resolved. Teams and email get a follow-up, and a webhook gets an alert.resolved event with the same id as the alert it closes. The channel ends up showing what's still broken, not everything that ever was.

Fold bursts

A bad deploy can set off a dozen error alerts at once. If one rule has more than five posts in a check, Baseboard sends the first four and folds the rest into one summary.

Watch what you'd otherwise check by hand

The kinds cover the things people tend to open dashboards for in the morning: failed deploys and builds, failing jobs and syncs, monitors, new errors above a level, failed payments, subscriptions set to cancel, monthly recurring revenue crossing a line, a day's AI spend going over a limit, and bounced email.

Rules post to Slack, Teams, a signed webhook or members' email. Every rule has a Send test button and keeps a record of what each destination answered, so a broken webhook shows up before you need it. The details are in Alerts.

More from the blog

See every customer in one place.

Connect a sign-in provider and a billing provider on the free plan. It reads every page and changes nothing.