Yarrix
GitHub
All docsAlerts

DocsAlerts

Alert routing

View source on GitHubdocs/alert-routing.md

On this page

Alerts are set up in alerts.yaml. Three ideas: destinations (where a message can go), rules (which alerts go to which destinations), and behavior (when, and how often). Every field is in the config reference. A hub without the file offers to create it: an admin adds the first destination under Alerts → Routing, and the hub writes it as the default with the fallback rule (PUT /api/v1/config/files/alerts.yaml with If-None-Match: *). What leaves the hub (retries, formats, templates, the signing headers) is in alert-delivery.md.

A small working file#

destinations:
  chat:
    type: webhook
    default: true                       # exactly one destination is the default
    url: ${env:CHAT_WEBHOOK_URL}        # a URL can carry a token, so it is a secret reference
    format: slack                       # yarrix_json | slack | discord | ntfy | template
  mail:
    type: email
    to: [ops@example.net]
    subject_prefix: "[yarrix] "

smtp:
  host: smtp.example.net
  port: 587
  security: starttls
  username: ${env:SMTP_USER}
  password: ${env:SMTP_PASSWORD}
  from: yarrix@example.net

rules:
  - when: { severity: critical }
    send: [chat, mail]
  - fallback: [chat]                    # the last rule, always: the default destination

Check it with yarrix-hub config check, then send each destination a test: POST /api/v1/destinations/<id>/test (admin token). The reply tells you whether the receiver accepted it.

Destinations#

  • webhook: an HTTP request to a URL. format picks the body: yarrix_json (the API's own shape), slack, discord, ntfy, or template for your own. Add headers: (each value is a secret, for an Authorization header), sign: (below), events: to limit what it receives, and enabled: false to switch one off without deleting it.
  • email: to, optional cc, from, subject_prefix. The server settings come from the shared smtp: block, and a destination may override them field by field with its own smtp:.
  • telegram: messages from your own bot to a chat. bot_token (a secret reference), chat_id (the user, group or channel id; get it from the bot's getUpdates after you message it), optional thread_id for a forum topic, and api_url only for a self-hosted Bot API server. Updates and the recovery are replies to the opening message.
  • pushover: a push to your phone through your own application. app_token and user_key (secret references), optional device. priority maps each severity; by default critical is 1, warn 0 and info -1. A priority of 2 is an emergency: Pushover repeats it every emergency.retry until someone acknowledges it in the app, and stops after emergency.expire (at most 3h). Emergencies need both fields, and retry is at least 30s. A recovery never sends an emergency.
destinations:
  phone:
    type: pushover
    app_token: ${env:PUSHOVER_APP_TOKEN}
    user_key: ${env:PUSHOVER_USER_KEY}
    priority: { critical: 1, warn: 0, info: -1 }
  tg:
    type: telegram
    bot_token: ${env:TELEGRAM_BOT_TOKEN}
    chat_id: "-1001234567890"

Yarrix ships no bot and no Pushover application: each user creates their own, and the tokens stay in the environment or a file. The Telegram token sits in the request path, so Yarrix never writes that URL to a log, an error or the API.

SMTP setup#

smtp:
  host: smtp.example.net
  port: 587                 # 587 with starttls, 465 with tls, 25 or 2525 with none
  security: starttls        # starttls (default) | tls | none
  username: ${env:SMTP_USER}
  password: ${env:SMTP_PASSWORD}
  from: yarrix@example.net
  timeout: 30s
  • starttls upgrades a plain connection; tls connects encrypted from the start; none sends everything, including the password, in clear and is accepted only for a loopback or private-network server (an internal relay).
  • For a mail server with a private certificate authority, tls_ca_file names the PEM file to trust.
  • helo sets the name announced to the server.
  • Delivery follows the same retry rules as webhooks: a 5xx reply fails for good, anything else is retried.
  • Send a test (POST /api/v1/destinations/<id>/test) after changing it; a wrong password or port shows up in the reply's error.

Rules: which alerts go where#

Rules are read top to bottom and every matching rule adds its destinations: they are additive, not first-match-wins. An alert goes to the union of what its matching rules send to. A rule is only used if it has a when. If no destination is left (nothing matched, or the ones that matched are disabled or do not take that event), the default destination receives the alert even when it is disabled, so an alert is never silently dropped. The last rule must be - fallback: [<the default>] to make that explicit.

rules:
  - when: { severity: critical }
    send: [chat, mail]
    ignore_quiet_hours: true
  - when: { domain: backups }
    send: [mail]
  - when: { host: lab-1 }
    hold: true                  # lab-1 never alerts
  - fallback: [chat]

when can match severity (info, warn, critical), domain (hosts, apps, backups, heartbeats, system), host, check, tag (the tags of an app, a custom check or a heartbeat) and host_state (asleep, down, up). A field you leave out matches anything; all the fields you give must match. A destination's own events: list is checked too: it receives only the events it lists.

hold: true swallows the matching alerts, and wins over every other matching rule, which is how you mute a noisy host or the alerts of a machine that is asleep (when: { host_state: asleep }) without editing the other rules. also: is another spelling of send: for readability.

GET /api/v1/alerts/routing (operator) shows alerts.yaml as the hub loaded it, with secrets masked, and each destination's last delivery.

Behavior#

behavior:
  after_consecutive_failures: 2         # an app or custom check must fail this many runs in a row first
  send_recovery: true                   # also tell me when it resolves
  renotify_unacked_critical: 4h         # repeat an unacknowledged critical alert
  quiet_hours: { from: "22:00", to: "07:00", tz: Europe/London, hold: [warn, info] }
  digest: { at: "07:00", tz: Europe/London, send: [mail] }
  reports:
    - { every: weekly, day: mon, at: "08:00", tz: Europe/London, send: [mail] }

During quiet_hours, alerts of the listed severities wait and a single incident.summary goes out when the window ends; a rule with ignore_quiet_hours: true sends at once. maintenance: lists windows in which one host, check or backup does not alert (target: host:atlas). People can also snooze or silence from the web app (POST /api/v1/silences).

One message for several problems on a host#

When a host is up but several things on it fail together (a dead Docker daemon, say), you get one message per destination, not one per service: "atlas: 6 problems (3 critical)", worst first. Apps, backups and heartbeats of one host that finish their behavior.group_wait together (default 90s) are grouped when two or more are bound for the same destination; a lone problem is sent as itself. Each incident keeps its own row, timeline and ack.

  • A destination gets only the incidents your rules routed to it. Grouping never widens who hears what.
  • A destination whose events: list does not include incident.grouped gets the individual messages.
  • Problems that start within behavior.group_interval (default 5m) after the message join it as one follow-up ("atlas: 2 more problems"). Later ones are sent alone.
  • Recoveries inside that window are held and sent as one "atlas: all 6 resolved". If the window ends first, each recovery is sent on its own.
  • Turn it off everywhere with behavior.group_by_host: false, or for one host with alert_grouping: false in hosts/<id>.yaml (true turns it back on there). group_wait: 0s sends at once and groups nothing.
  • The wait is anchored on the first problem on the host: when it ends, every other problem on that host that is only waiting for group_wait leaves with it, however late it was detected.
  • "Ack all on <host>" in the Alerts tab, or POST /api/v1/alerts/hosts/{id}/ack, acknowledges every open problem on the host. Each incident is acked and audited on its own. Operators and read-write keys may; read-only may not.
  • A host that is down is covered by its own alert (see dependencies); there is nothing to group.

Pause and silence#

A silence and a pause both keep a notification from going out, but they differ in what the hub does with the object.

  • A silence keeps checking the object and keeps its incident open. It only stops the notification.
  • A pause stops judging the object. Its checks, SSH polls and backup probes stop, its open incidents resolve with event paused (no recovery notification), and its state reads paused rather than red. Agent pushes are still stored, so charts have no gap.

Pauses are runtime state, set from the host or object drawer or POST /api/v1/pauses, not from alerts.yaml. A host pause covers everything defined under that host. Global pause ("pause all monitoring") needs an admin and an end time of seven days or less. An operator's pause needs an end, and the paused time on that scope in the last 30 days (active, resumed and expired pauses all count) plus the new pause must come to at most 30 days; a pause with no end, or one past that window, needs an admin. Every pause with no end is counted in the banner on every page, so it cannot be forgotten. Any pause ends by itself at its until. A heartbeat whose ping was due during the pause does not alert on resume: it gets a new baseline when the pause ends.

Verifying the signature#

Give a webhook a signing secret and the receiver can prove a request came from your hub and was not altered or replayed:

destinations:
  chat:
    type: webhook
    url: ${env:CHAT_WEBHOOK_URL}
    sign: { type: hmac_sha256, secret: "${env:YARRIX_SIGNING_SECRET}" }

Every request then carries:

X-Yarrix-Timestamp: 1794182294
X-Yarrix-Signature: sha256=565859d73aebe2b03e3a9fd28f160f40d6ceff6c743234dc1c27804019cc5ca6

The signature is the lowercase hex HMAC-SHA256, keyed with your secret, of the timestamp, a dot, and the exact raw request body: HMAC(secret, timestamp + "." + body). Verify against the raw bytes, before any JSON parsing, compare in constant time, and reject a timestamp that is too old (five minutes below). These snippets were each run against a request signed by a real hub.

Node

const { createHmac, timingSafeEqual } = require("node:crypto");

// headers: lowercase header names. Returns true when the body is genuine and recent.
function verify(secret, headers, rawBody, now = Date.now() / 1000, toleranceSeconds = 300) {
  const ts = headers["x-yarrix-timestamp"];
  if (!/^\d+$/.test(String(ts)) || Math.abs(now - Number(ts)) > toleranceSeconds) return false;
  const want = "sha256=" + createHmac("sha256", secret).update(`${ts}.${rawBody}`).digest("hex");
  const got = String(headers["x-yarrix-signature"] ?? "");
  return got.length === want.length && timingSafeEqual(Buffer.from(got), Buffer.from(want));
}

Python

import hashlib, hmac, time

def verify(secret: str, headers: dict, raw_body: bytes, now=None, tolerance=300) -> bool:
    """headers: lowercase names. raw_body: the bytes exactly as received."""
    ts = headers.get("x-yarrix-timestamp", "")
    now = time.time() if now is None else now
    if not ts.isdigit() or abs(now - int(ts)) > tolerance:
        return False
    want = "sha256=" + hmac.new(secret.encode(), ts.encode() + b"." + raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(headers.get("x-yarrix-signature", ""), want)

Shell (openssl), for a script that has saved the body to a file:

#!/bin/sh
# usage: verify.sh SECRET TIMESTAMP SIGNATURE BODYFILE
secret=$1 ts=$2 sig=$3 bodyfile=$4
want="sha256=$( { printf '%s.' "$ts"; cat "$bodyfile"; } | openssl dgst -sha256 -hmac "$secret" | sed 's/^.* //')"
[ "$want" = "$sig" ] && echo good || echo BAD

(The shell version does not check the timestamp's age; compare $ts with date +%s as well if replay matters to you.)

A signature proves origin and integrity, not secrecy: use an https:// URL as well. Rotate the secret by changing it in your secret store and on the receiver together.