Custom checks

A custom check is a command you choose, run on a host on a schedule, whose result becomes a status in Yarrix. Use one when none of the built-in readings covers what you care about: a ZFS pool, a vendor tool, a local API, a Windows service.

# hosts/nas-1.yaml
custom_checks:
  - id: zfs-health                        # lowercase, digits, - and _
    name: ZFS pool health
    run: { command: "zpool status -x" }
    interval: 5m                          # or cron: "*/10 * * * *" with tz
    parse: { mode: regex, ok_pattern: "all pools are healthy" }
    severity: critical

Every field is in the config reference (custom_checks). The shape is always the same: what to run, how often, how to read the answer.

Where it runs#

Over SSH from the hub, using the host's own ssh: block (target, key, host key). The hub opens the connection, feeds the command to the remote shell on standard input, and reads the output. Nothing is copied to the host and nothing is left behind. So the host needs an ssh: block even if it is collected by an agent:

ssh:
  target: yarrix@nas-1.example.internal
  identity_file: /etc/yarrix/keys/nas-1

(The hub image carries an ssh client; see security.md.) A host collected with an agent can instead run named commands defined in its own agent.yaml with run: { agent_command: verify-photos }. The hub names the command; it cannot send one, and the agent runs the argv you wrote locally with no shell.

run takes exactly one of:

shell is sh by default (powershell on Windows); bash, powershell and cmd are also accepted. {host} and {os} in the text are replaced with the host's id and configured OS, and nothing else is substituted. timeout defaults to 30 seconds (up to 10 minutes).

Parse modes#

How the output becomes ok, warn, failed (or critical) or unknown.

Only exit_code looks at the exit code; the other modes judge the text. If the output cannot be read (not JSON, a field missing, no number), the result is unknown with "output not parseable", never ok.

Example: a Nagios-style plugin#

Nagios plugins exit 0 (OK), 1 (WARNING), 2 (CRITICAL), 3 (UNKNOWN). Map them directly:

  - id: root-disk
    name: Root filesystem (nagios plugin)
    run: { command: "/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /" }
    interval: 5m
    parse: { mode: exit_code, ok_codes: [0], warn_codes: [1] }   # 2 and 3 fail
    severity: critical

Example: a JSON API on the host#

  - id: app-health
    name: App health endpoint
    run: { command: "curl -fsS --max-time 10 http://127.0.0.1:8080/health" }
    interval: 1m
    parse:
      mode: json
      fields:
        - { path: "$.status", equals: "ok" }
        - { path: "$.disk.free_gb", below: { warn: 50, critical: 10 } }   # alert when it falls under

Paths start with $. below alerts when the value is under the threshold, so it suits "free space". For a value that should stay low, use number mode with direction: above.

Example: ZFS or Btrfs health#

  - id: zfs-health
    name: ZFS pools
    run: { command: "zpool status -x" }
    interval: 5m
    parse: { mode: regex, ok_pattern: "all pools are healthy" }
    severity: critical

  - id: btrfs-errors
    name: Btrfs device errors on /mnt/data
    run: { command: "btrfs device stats /mnt/data" }          # may need sudo, see below
    interval: 15m
    parse: { mode: regex, bad_pattern: '_errs\s+[1-9]' }      # any non-zero error counter
    severity: warn

A script is a good home for anything longer than a line, for example run: { script: scripts/pool-state.sh }, with parse: { mode: json, ... } over what it prints (examples/config/scripts/pool-state.sh).

Example: a PowerShell check#

# hosts/win-1.yaml
  - id: backup-service
    name: Backup service running
    run: { command: "(Get-Service -Name ExampleBackup).Status" }
    shell: powershell
    interval: 5m
    parse: { mode: regex, ok_pattern: "^Running" }
    severity: warn

  - id: c-free-gb
    name: "Free space on C:"
    run: { command: "[math]::Round((Get-PSDrive C).Free / 1GB)" }
    shell: powershell
    parse: { mode: number, warn: 20, critical: 5, direction: below, unit: GB }

Windows checks need the Windows OpenSSH server on the host (platforms/windows.md), which has not been tested on real Windows yet.

When it alerts#

severity (info, warn, critical; default warn) is how serious a failure is. after_consecutive_failures (default from alerts.yaml behavior) is how many failed runs in a row it takes. domain: hosts (default) makes the check count toward the host's state; domain: apps lists it with the app checks instead. Alerts then route like any other (alert-routing.md). To run one now: POST /api/v1/checks/<id>/run.

Security model#

A custom check is arbitrary code execution on a monitored host, by design. The controls are about who may define one and what it can reach.

• Only an admin can define or change one. run, shell and env are exec fields: through the API they need an admin signed in to the web app, never an API token (security.md). Editing the files on disk is always possible for whoever owns the config folder.
• It runs with the host's ssh: account, and with nothing more. If that account is root, the check is root. Use a dedicated low-privilege user, and give it only the passwordless sudo rules the checks need (for example one for btrfs device stats), then write sudo -n in the command where it is needed (ssh.sudo governs the built-in probe, not your command).
• Use a separate low-privilege key for the collector and checks, not the terminal's. The terminal opens interactive shells for people and should use another account and key (security.md). A key pinned to the built-in probe cannot run custom checks; the check reports "key is limited to the probe".
• Secrets go in env, never in the command. env: { API_TOKEN: ${env:CHECK_TOKEN} } exports the value to the check through standard input, so it never appears in a process list or command line. Output is scrubbed of the values before it is stored or shown. The names PATH, IFS, BASH_ENV, NODE_OPTIONS, anything starting LD_, DYLD_, PYTHON, PERL5 or GIT_, and a few more that change how programs load code are refused.
• Scripts stay under scripts/. The path must be written plainly (no . or ..), every symlink on the way must resolve inside scripts/, and the check is repeated right before each run.
• Limits. 64 KB of output per stream (the rest is cut), a timeout on every run, and the connection is closed if a run times out.
• Placeholders are strict. {host} and {os} must be plain names, so they cannot carry shell syntax.