Alert delivery

How Yarrix sends an alert once routing has chosen a destination. Routing itself (rules, quiet hours, silences) is in alerts.yaml; this page is about what leaves the hub.

The queue#

Every (incident event x destination) is a row in the deliveries table, written in the same transaction as the incident change, so a crash cannot lose an alert. A worker claims due rows, sends them, and records the outcome:

• A 2xx response (webhooks) or a 250 after DATA (email) marks the row ok.
• A 4xx other than 408 and 429 fails permanently: retrying will not fix a bad URL or a rejected token. So does a 3xx other than a followed 307/308 redirect, and an SMTP 5xx reply.
• Everything else (5xx, 408, 429, SMTP 4xx, connection and TLS errors, timeouts) retries after min(start x 2^n, max) seconds, plus or minus 20% jitter. A Telegram 429 names its own wait (retry_after), and the next try never comes sooner than that. A retry_after over one hour is treated as a broken or hostile receiver: the row fails at once and raises the destination's delivery incident, rather than parking the queue. The defaults are start: 10s, max: 15m, retries: 6: six sends in all (at 0, 10, 30, 70, 150 and 310 seconds), then failed.
• After the last retry the row is failed and Yarrix opens an internal incident router:<destination>:delivery, sent to every other enabled destination. It resolves itself when a later delivery to that destination succeeds.
• A hub that stops mid-send leaves the row in sending; on start those go back to queued. Delivery is at least once: after a crash in the middle of a send, the receiver can see that one alert twice. thread_key (the incident id) lets it de-duplicate.

GET /api/v1/deliveries?destination=&status=&limit= lists rows. POST /api/v1/destinations/{id}/test sends a test event to one destination right now, outside the queue, and returns {ok, code, latency_ms, error}.

Formats#

Events: incident.opened, incident.updated (with renotify: true on a renotify), incident.resolved, incident.summary (one per destination when quiet hours end, listing the incidents still open), incident.grouped (one per destination for several problems on one host: {event, host, count, title, incidents[]}, plus followup: true for late joiners and resolved: true for "all resolved"; a destination whose events list leaves it out gets the individual events instead), test.

Templates#

format: template renders template with {{name}}, {{#name}}...{{/name}} (shown when the value is set; a list repeats) and {{^name}}...{{/name}} (shown when it is not). With a JSON content_type (the default) values are escaped for a JSON string. Available names: event, severity, title, reason, host, check, domain, since, url, thread_key, owner and runbook (heartbeats that set them), headline, mention (the destination's mention_on_critical, only for critical), and the flags critical, warn, info, opened, updated, resolved, summary, test, renotify.

destinations:
  chat:
    type: webhook
    url: ${env:CHAT_HOOK}
    format: template
    mention_on_critical: "<!here>"
    template: '{"text": "{{headline}} {{#critical}}{{mention}}{{/critical}}"}'

Signing#

A destination with sign: {type: hmac_sha256, secret: ${env:...}} adds two headers:

X-Yarrix-Timestamp: <unix seconds>
X-Yarrix-Signature: sha256=<hex hmac_sha256(secret, timestamp + "." + body)>

The timestamp is part of what is signed, so a receiver can reject replays. Verify against the raw request body, before parsing it:

const { createHmac, timingSafeEqual } = require("node:crypto");

// headers: lowercase header names. Returns true when the body is genuine and recent.
function verify(secret, headers, rawBody, now = Date.now() / 1000, toleranceSeconds = 300) {
  const ts = headers["x-yarrix-timestamp"];
  if (!/^\d+$/.test(String(ts)) || Math.abs(now - Number(ts)) > toleranceSeconds) return false;
  const want = "sha256=" + createHmac("sha256", secret).update(`${ts}.${rawBody}`).digest("hex");
  const got = String(headers["x-yarrix-signature"] ?? "");
  return got.length === want.length && timingSafeEqual(Buffer.from(got), Buffer.from(want));
}

Where a destination may point#

Destination URLs and SMTP hosts are admin-only. Private ranges are allowed, since self-hosted receivers usually sit on the LAN. Cloud metadata addresses (169.254.169.254, 169.254.170.2, fd00:ec2::254, and the other well-known ones, in every spelling including IPv4-mapped IPv6 and the NAT64 form 64:ff9b::a9fe:a9fe; the local-use NAT64 range 64:ff9b:1::/48 is refused outright) are refused unless the webhook sets allow_metadata_ips: true.

• A URL that names such an address is an error at config load.
• A name is resolved once per request, every answer is checked, and the connection goes to that exact address, so a DNS answer that changes between the check and the connect cannot redirect the request. The Host header and the TLS server name stay the original name.
• Redirects are followed only for 307 and 308 (so the method and body survive), only to the same origin (same scheme, host and port, so configured auth headers never reach another origin), at most three, and each hop goes through the same check. A cross-origin redirect fails permanently; set the destination to the final address.

A Telegram api_url is the one place a bot token travels in a URL path, so it must be https. Plain http is allowed only to a loopback or private IP literal (for a local Bot API server), the same rule as SMTP security: none. A URL that is neither is a load error at destinations.<id>.api_url, and the same check runs again before each send, so a value that skipped the load check still sends nothing; that delivery fails. Unlike SMTP, a hostname over http is refused even when it resolves to a private address, because the HTTP guard has no connect-time private-only check; use the IP literal or https. Telegram's default and Pushover's endpoint are public hosts, reached through the same guard.

SMTP security: none (no encryption, credentials in clear) is allowed only to loopback or private addresses. A literal public address is an error at load. A hostname is checked when connecting: every address it resolves to must be loopback or private, and the connection goes to the address that was checked.

Daily digest#

behavior.digest: { at: "07:00", tz: Europe/London, send: [chat, mail] } queues one digest.daily message per listed destination each local day. It goes through the same queue, retries, signing and formats as any alert. Disabled destinations, and ones whose events list leaves out digest.daily, are skipped. Every name in send must be a destination (checked at load).

• Layout. Fixed, the same every day: counts per domain, every item that is not green (critical, then warn, then unobserved or asleep), a backups summary (ok, late, failed, unobserved, running, and each job's last success), and delivery failures in the last 24 hours. The text is in the payload as text; slack and discord show it in a code block, ntfy sends it as the body, a template can use {{text}} and {{#digest}}, and an email destination gets it as a plain-text message with no HTML part.
• Once a day. The day is claimed in the database before anything is queued, so a restart, a second tick, or the repeated hour when clocks go back sends nothing twice. If at falls in the hour clocks skip, the digest goes out when that hour ends. A digest missed by more than six hours of wall-clock time (the hub was down) is skipped for that day.
• Quiet hours. If quiet hours hold info and the digest comes due inside them, it is delivered when they end.
• Preview. GET /api/v1/alerts/digest/preview (operator role) returns the payload as it would be sent right now, text included.

Weekly and monthly reports#

behavior:
  reports:
    - { every: weekly, day: mon, at: "08:00", tz: Europe/London, send: [mail] }
    - { every: monthly, at: "08:00", tz: Europe/London, send: [mail] }   # sent on the 1st, covers the previous month

Each entry queues one report.weekly or report.monthly message per listed destination, through the same queue, formats and signing as the digest. A weekly report covers the seven days up to its send time; a monthly one covers the previous calendar month in its zone. Disabled destinations, and ones whose events list leaves out the report event, are skipped. Every name in send must be a destination, a weekly report needs a day, and a monthly one takes none (all checked at load).

• Layout. Fixed: availability per domain (time-weighted; time nobody observed is left out and reported beside the figure, and a figure with less than half the period observed reads "not enough data"), the ten lowest objects, every object below 99.9 %, backup run success, incidents and the longest critical outage, and paused time. The text is in the payload as text, and each format renders it as the digest does.
• Once per period. Each destination's claim is written in the database with the delivery, so a restart or a later tick in the send window queues nothing twice.
• History starts when this ships. Figures come from the availability history (state_log). A period that begins before it is counted from its first row, and the report says "History starts <date>" when that is inside the period.
• Preview. GET /api/v1/alerts/reports/preview?every=weekly|monthly (operator role) returns the most recent report of that kind, with the same text the delivery carries.