Security hardening

Yarrix can reach every machine it monitors, and with the terminal on it can open a shell on them. Read this page before you point it at anything you care about. The product never calls home; the only outbound connections are the ones you configure (alert destinations, checks, SSH, the agents' own connection to your hub).

The short list#

1. Serve the hub over HTTPS. Plain HTTP works on your own network but is not encrypted (auth.md).
2. Give the collector a low-privilege account, and the terminal a different one (below).
3. One SSH key per host, never one key for the fleet (below).
4. Pin SSH host keys for anything that matters (below).
5. Leave the Docker socket out of the agent unless you need container inventory (below).
6. Keep the terminal off unless you need it, and treat its recordings as sensitive.
7. Use ${env:NAME} or ${file:/path} for every secret, and treat ping URLs and API tokens as passwords.

SSH host keys#

Agentless collection, SSH custom checks, SSH backup probes and the web terminal all use the system ssh client. The hub gives it its own settings: a config file it writes itself (your ~/.ssh/config and the system known-hosts files are not read), batch mode (no password prompts, no password or keyboard-interactive authentication), and no agent or X11 forwarding. The one exception is the add-a-host wizard's one-time "install the key with the host's password" (below); that single login is made with a password, and nothing else the hub does is.

What happens with a host's key depends on whether you set host_key in the host's ssh: block:

• host_key not set: trust on first use. The first time the hub connects it accepts the key it is shown and records it in $YARRIX_DATA_DIR/known_hosts (a plain, unhashed file). From then on a different key is a hard failure: the host shows "host key changed" and nothing is collected. There is no button to accept a new key. If you rebuilt the machine on purpose, delete its line from that file and the next connection trusts the new key. Trust on first use means an attacker who is on the network path during the very first connection is trusted. For a home LAN that is usually acceptable; for anything else, pin.
• host_key set: pinned. The hub trusts only that key and refuses the connection if the host presents another, including the first time. Nothing is written to known_hosts.

To pin, read the key from a place you trust (the machine's console, or ssh-keyscan run from somewhere you trust):

ssh-keyscan -t ed25519 atlas.example.internal 2>/dev/null
# atlas.example.internal ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...

and put the last two fields, on one line, in the host file:

ssh:
  target: yarrix@atlas.example.internal
  host_key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."

It must be a key type and its base64, on one line. A SHA256: fingerprint is not accepted. If the target uses a port other than 22, the hub handles the [host]:port form itself.

The terminal uses the host's pinned key only when terminal.target is the same user@host[:port] as ssh.target. A terminal to a different target (another account on the same machine counts, since the target string differs) falls back to trust on first use, sharing the same known_hosts file. If you give the terminal its own account, the pin does not apply to it, so for a pinned host check the terminal's first connection by hand.

host_key is an exec field: changing it needs an admin browser session.

Status pages and badges#

These are the only pages the hub can serve with no sign-in, and they are off until an admin turns them on. A page shows only the items it lists, by the label you wrote and a state word; no host name, address or reason unless the page sets show_reasons. A badge URL holds a key the hub made, not a guessable name. The page and badge fields can only be changed by an admin signed in to the web app. See status-pages.md.

A separate account for the terminal#

The collector needs to read metrics and, with ssh.sudo: true, run a short list of commands as root. Whoever opens a terminal gets a shell. Those are different jobs, so use different accounts and different keys:

If the terminal logged in as the collector, anyone who can open a terminal would get the collector's privileges, and anyone who steals the collector key would get a shell. With two accounts, a stolen collector key can run only what the collector account may run. The same goes for custom checks: they run with the ssh: block's account, so the commands you write there should be things that account may do (custom-checks.md).

Per-key restrictions help further. On the host, a collector key can be pinned to the probe so it can run nothing else (AGENT.md). That form fits the built-in probe only: custom checks send their own command and report "key is limited to the probe" on such a key, so a host that runs custom checks over SSH needs an ordinary low-privilege key for them.

One key per host#

The add-a-host wizard does this for you. When an agentless host reaches the SSH step, the hub makes a key for that host alone (ssh-keygen -t ed25519, no passphrase) at $YARRIX_DATA_DIR/ssh/keys/<host id> (mode 0600, the public half beside it as <host id>.pub), shows its public half to authorise, and writes the host's ssh.identity_file as that path. The private key never leaves the hub's disk: it is not sent to the browser, put in the config file, returned by the API, logged or stored in the database. Back up the ssh/keys folder with the rest of the data directory.

If you delete a host's config file, the hub leaves its key file where it is and logs the path once. It may be the only copy of a key the machine still trusts, and the hub cannot tell whether you meant to keep it. Remove $YARRIX_DATA_DIR/ssh/keys/<host id> (and .pub) yourself, and take the key out of the host's authorized_keys, when you no longer want that access. Adding a host with the same name later reuses the same key.

An older hub used one key for every agentless host, $YARRIX_DATA_DIR/ssh/id_ed25519. If that file is still there, hosts with no identity_file keep signing in with it and nothing else changes; the hub no longer makes it. To move a host over, add the new host's key to it from the wizard (or by hand) and set its identity_file. Once no host uses the shared key, delete it and remove it from the hosts.

By hand, give every host its own key pair (ssh-keygen -t ed25519 -f /etc/yarrix/keys/atlas -N "") and point that host's ssh.identity_file and terminal.identity_file at it. A key shared across the fleet turns one stolen file into access to every machine. identity_file is a path read by ssh on the hub; the key never goes in the config file itself. Keep the key directory readable only by the account the hub runs as. Agents work the same way: every agent has its own token, minted once per host with yarrix-hub enroll create --host <id>; an enrollment token expires (24 hours unless you pass --ttl), and the agent swaps it for a per-host token it keeps in its state directory (mode 0600).

Which keys the hub offers a host#

A host with an identity_file is offered that key and no other. A host without one is offered nothing from the hub user's own account: not the keys in ~/.ssh and not a running ssh-agent (on a desktop install those can be your personal keys). It is offered only the old shared hub key, if an older version made one. To let such hosts try the hub user's ~/.ssh default keys and ssh-agent again, set YARRIX_SSH_USER_KEYS=1 on the hub; the hub logs a warning at start while it is on. Pointing a host's identity_file at a key under ~/.ssh is the other explicit way and needs no setting.

Passwords are never stored#

The hub runs ssh in batch mode with keys only, and has no field anywhere that keeps a password. The wizard offers one exception, for adding a key to a host that still accepts a password:

• You type the SSH password into the wizard once. It is sent in one request body to POST /api/v1/ssh/install-key, used for one ssh login that appends that host's public key to ~/.ssh/authorized_keys (what ssh-copy-id does), and dropped. Then the hub logs in again with the key alone, on a fresh connection, and reports whether it works.
• The password reaches ssh only through the environment of that one child process, where ssh's askpass helper reads it (SSH_ASKPASS with SSH_ASKPASS_REQUIRE=force, OpenSSH 8.4 and later; the Docker image has 9.2). It is not an argument, a file, the config, the database, an audit row, a log line or any response, and ssh's own output is scrubbed of it.
• The host's key is verified first. The wizard shows the fingerprints the host presents and asks you to confirm one; the hub records that key and ssh then checks it strictly, so a machine in the middle fails before any password is sent. A host whose key changed since the hub last saw it is refused. (With a pinned host_key the pinned key is used instead.)
• It needs a signed-in admin cookie session (an API token gets 403). A plaintext password is refused over plain http except on localhost; over plain http the page seals it first (see Passwords on a plain-http hub). Wrong passwords are limited: five for one account from one address locks that account out for 15 minutes.
• It makes one password attempt, so it cannot lock the account by itself.

What it cannot do: a host with password login switched off (the wizard says "does not accept password login"); Windows (the account's key may belong in administrators_authorized_keys, which this does not write, so use the PowerShell commands); an account with a restricted shell; a connection through a jump host (use the manual commands). The manual path, pasting the public key yourself, always works and stays the default.

Passwords on a plain-http hub#

A home hub on plain http:// is the default, and a browser offers no TLS there. So that the wizard (and Reinstall over SSH) still work without a manual step, the page encrypts the SSH and sudo passwords before they leave it, to a key only the hub holds:

• The hub keeps a P-256 key pair in memory only, replaced every hour; it is never written to disk. POST /api/v1/ssh/seal-key (admin cookie session) gives the page the public key and a single-use nonce good for ten minutes.
• The page makes a throwaway key pair, derives an AES-256-GCM key (ECDH, then HKDF-SHA256 salted with the nonce) and encrypts the passwords with the nonce, host and target as authenticated data. It sends sealed in place of password / sudo_password. Plain http gives a page no crypto.subtle, so the page uses the audited @noble libraries; the hub uses Node's crypto.
• The hub spends the nonce on the first try, right or wrong, so a captured request cannot be replayed, and the ciphertext cannot be redirected to another host or target. A plaintext password on plain http is still refused.
• Over https, and on localhost, nothing changes: the password is sent as before.

What this protects: someone who only watches the network (a passive sniffer on your LAN) sees ciphertext. It is the same promise sign-in makes with OPAQUE. What it does not: someone who can change traffic in transit. On plain http they can change the page itself, and the page does the encrypting, so they can have it send the password to them. The wizard says so next to the password box. If that matters on your network, serve the hub over HTTPS (auth.md).

Install the agent for me#

POST /api/v1/ssh/install-agent (the wizard's Install it for me) installs the agent over SSH. The same rules as above apply to the passwords: they are sent in that one request body, reach ssh only through the child process's environment, and are not kept, logged, audited or returned.

• Login by the account's password or by the host's own key. The host's key is confirmed first, as in the key step; nothing is sent to an unconfirmed host.
• Root: a root login runs the install directly. Another account runs it through sudo, with the sudo password typed in the wizard (the login password is reused when you log in by password). The password goes to sudo on stdin, ahead of the script, so it is never in an argument. Without a usable sudo the install stops with "needs root" and nothing is changed; the manual commands are shown.
• The remote command is a constant. The install script arrives on stdin, is written to a mode-0600 temporary file and removed afterwards. The one-time enrollment token (valid for an hour, revoked if the install fails) is inside that script and so never in a process list or argument.
• Needs a signed-in admin cookie session (API tokens get 403). A plaintext password is refused over plain http except on localhost; over plain http the page seals it first, as in the next section. One install per target at a time; password attempts are rate limited like the key step.
• Each attempt writes an agent_install audit row with the host, target, platform and outcome, and no secret.
• The download is checked against hashes the hub sends inside the script, over the authenticated SSH channel. The hub checks its signed release manifest against the built-in release key the agent trusts (or against YARRIX_RELEASE_PUBKEYS, when the operator sets it to replace that key). When the signature verifies, the hashes are that manifest's, and the script also checks that the binary reports the manifest's version; the host needs no minisign for this. When it does not verify, the script checks the manifest's signature with minisign on the host, which refuses a bad one, so an unverified manifest never leads to a checksum-only install. Either way the script does not fetch a .sha256 from the same place as the binary, so someone on the network path of a plain-http hub cannot swap both. The by-hand commands in the wizard carry the same hashes. When the hub does not serve the files, the build comes from the GitHub release over https and is checked against the .sha256 published beside it: that catches a corrupt download, and relies on https and GitHub for authenticity.
• The download, its check and the install happen in a private directory made with mktemp -d and removed afterwards, never under a fixed name in /tmp, so another user on the host cannot plant a file or symlink there for root to write through or to swap in after the check.
• Linux with systemd and macOS with launchd are automated. Unraid, Synology, UGOS and Windows are not (the hub answers not_automated, without connecting, and the wizard shows their commands). launchd has been exercised only against a stubbed launchctl in tests, not a real macOS.

A reinstall of an enrolled agent (mode: "upgrade" on the same route) follows every rule above: admin cookie session only, https or localhost when a password is sent, a confirmed host key, the constant remote command, the hub's own hash for the download, and no password stored or audited. It mints no enrollment token, and nothing is revoked if it fails, because the host keeps the enrollment it has. Automatic mode never uses it: the hub holds no SSH credentials, so an unattended reinstall is not possible.

The Docker socket#

Container inventory reads the Docker API through /var/run/docker.sock. Access to that socket is root on the host: anything that can talk to it can start a privileged container that mounts the whole disk. Mounting it :ro does not help, because the read-only flag covers the socket file, not what the API lets you do.

For that reason deploy/docker/compose.agent.yaml leaves the socket out by default, and the agent container reports no container inventory until you add it. If you want inventory:

• Prefer the native binary on the host, running as a user that is in the docker group. Be aware that this is the same exposure, just without a container around it.
• Or put a read-only socket proxy in front (one that allows only GET /containers/json) and point the agent at that instead of the raw socket.
• Mounting the raw socket into the agent container is the last resort. The agent itself only reads, but a bug in it would then be root on the host.

The hub container never needs the Docker socket.

Editing commands needs an admin session#

Some config fields decide what runs, or where the hub connects, or widen what the hub itself will do. These are the exec fields and guarded fields; the generated config reference marks each with exec or guarded:

• custom_checks[].run, .shell and .env, ssh.target, .jump, .sudo, .identity_file, .host_key, terminal.target, .jump and .startup_command, apps[].source.command, and the global terminal.startup_command.
• Guarded: alerts.yaml destinations.*.allow_metadata_ips, and self_monitor.db_backup.path and .ping_url, and agent_updates.mode (global and per host), auth.token_limits.max_mute (what a read-write key may do), config.git.commit (turning it off hides later edits from history), retention.raw, retention.rollup and availability.keep (shrinking them deletes history at the next purge), and a host's collect.token. Changing any secret reference (${env:...}, ${file:...}, ${secret:...}) is guarded too.

Changing any of them through the config API (PUT /api/v1/config/files/{path}, and the editor that uses it) needs an admin signed in to the web app. An API token, whatever its scope, is refused with 403 exec_field_requires_admin_session, and the refusal is audited. So a leaked admin API token cannot be turned into command execution on your hosts. Editing the files on disk is always allowed, since whoever can write there already controls the hub.

Read-write API keys (AI agents)#

An admin can make an API key with the operate scope. It acts as an operator and never as an admin: it cannot read or write config, manage tokens, users or enrollments, open the terminal or send a command line.

• What it can do: acknowledge, snooze, add a note, silence, pause a host or one object, run a defined check now (a custom check runs the command its admin wrote), re-verify a backup, and run a command by name that an admin declared in the host's agent.yaml. The agent is sent only the name and runs the argv it already has, with no shell. A name that is not declared is 404 and nothing is sent. Pausing everything is refused.
• Limits: a snooze, silence or pause made with a key lasts at most auth.token_limits.max_mute (default 24h) and cannot be open-ended (400). 30 actions a minute per key and one run-now per check per minute (429). A mute request body that is too large or not a JSON object is refused rather than defaulted.
• Records: every action is written to the audit log under token:<name>, including refused ones. Every mute, and every command run, is also sent as a system.action message to the default destination, even when that destination is disabled or filters events, so a key cannot hide an outage or change a host quietly. A human's own mutes are not announced.
• Revoking a key takes effect on its next request.

Agent updates#

D-04 still holds: the hub cannot make an agent run a command. An update is the one thing a hub can ask that changes what an agent runs, so it is built so that the hub only chooses the moment.

• The update message is a version and an id (unknown fields are rejected). The agent downloads from its own hub and trusts only a minisign signature made by a key compiled into the agent or listed in the machine's own agent.yaml. The hub's copy of the signature check is informational.
• Refused, whatever the hub says: an unsigned manifest, a signature from another key, a version that is not newer (no downgrades, so an old signed build cannot be replayed), a binary whose size or SHA-256 differs from the signed manifest, and any update when update.allow is false in agent.yaml. Redirects are not followed.
• The trust boundary is the machine's own agent.yaml. update.trusted_keys there adds signing keys for that host: anything signed by a listed key will be installed on it. Writing that file already needs root on the host, and whoever has root there can replace the agent anyway, so this adds no new power. The hub cannot read, write or override it; neither can anyone who only has hub access. update.allow: false in the same file turns self-update off for that host, and nothing the hub sends changes that.
• A new build that crashes or cannot reach the hub is rolled back automatically. On Windows that is a file rename (yarrix.exe.prev back to yarrix.exe) done by the new build itself; this path has not run on a real Windows machine.
• Containers and unsupervised agents refuse to update themselves.
• Choosing the mode (agent_updates.mode) is a guarded field: admin session only. Starting an update needs the admin role.
• Automatic mode is the hub starting that same version-only request on a timer. It adds no new power over the agent: the signature, the version rule and the machine's own update.allow decide what runs. It goes one canary host first, at most max_concurrent at a time, and pauses by itself when two hosts fail on a release (audited as agent_update.paused and .resumed).

What a stolen signing key allows: moving agents to a build signed with it. The project's private key is kept off the hub and out of CI: maintainers sign each release locally with scripts/release/sign-release.sh, and a release CI built stays a draft, so nothing is offered, until it is signed. Your signature is the trust gate, so sign-release.sh downloads every asset of the draft first and refuses (non-zero exit, nothing uploaded, the draft kept) if an asset in SHA256SUMS is missing or differs, if an asset exists that SHA256SUMS does not list (the SBOM is listed; only MANIFEST.json, SHA256SUMS and .sha256 companions are exempt), if MANIFEST.json disagrees with SHA256SUMS on a hash or with the asset on a size, if the draft's tag is not the commit your local git rev-parse <tag>^{commit} gives (or the tag is not in your clone), or if minisign.pub at the tag differs from your trusted key (MINISIGN_PUBKEY_FILE, default the working-tree minisign.pub). The last check matters because CI derives the public key baked into builds from minisign.pub in the tagged checkout: a tag carrying a swapped file would ship builds that trust another key. Residual risk: whoever can move a tag or push to the default branch before you sign can still change what the release contains, and you sign what you checked, not what you reviewed. Protect tags and the default branch on the hosting side (tag protection or a ruleset, required review, no force-push), and review the tagged commit before signing. If you sign your own builds, do the same. Official builds trust only the key committed as minisign.pub; to rotate, ship a release signed by the old key that carries the new public key.

The web terminal#

Off by default. When on: only a browser session opens one (never an API token), a viewer can never open one, an operator needs the host in their auth.users rule, the session must be under 12 hours old, and the request's Origin must match the hub's address. Sessions are limited in number and idle time, and each one is written to the audit log. With terminal.record: true the screen output (not your keystrokes) is saved as an asciicast under $YARRIX_DATA_DIR/recordings/ for 30 days, and it can contain anything the remote printed, passwords included; only admins can play it back. To stop every terminal at once without a restart, create the file $YARRIX_DATA_DIR/terminal.disabled. Details: terminal.md.

Secrets, tokens and URLs#

• Write secrets as ${env:NAME}, ${file:/path} or ${secret:NAME} (the hub's own encrypted store, below). A secret saved from the web app or the config API must be one of those references: the hub refuses a plain value with 422 secret_must_be_reference, except a webhook or Telegram API URL whose plain text carries no secret. A plain value already in a file keeps working, with a warning, and is left alone when you save something else. Secret values never appear in API responses or logs.
• Changing which variable, file or stored secret a field points at needs an admin signed in to the web app, not an API token: without that rule a token could repoint a webhook header at any secret the hub holds and read it back.

The hub's encrypted secret store#

Type a secret value in the web app and the hub keeps it as ciphertext in secrets.json in its data folder (AES-256-GCM, a fresh random nonce per secret, the name authenticated). The YAML holds only ${secret:NAME}. The value is never shown again, never logged and never in config git; overwriting a name rotates it, and the new value is live at once. Only an admin signed in to the web app can store one.

Three keys protect it, and you hold two of them:

• The hub key opens the store in normal running. It is YARRIX_SECRET_KEY (64 hex characters or base64 of 32 bytes) if you set that, otherwise a file secret.key (mode 0600) that the hub makes in the data folder the first time you set the store up. Back secret.key up separately from the data folder and the database backups. A copy of the data folder next to its own key file protects nothing; a copy without the key cannot be read, which is the point.
• The recovery key is made when the store is set up and shown once, as 32 characters in groups of four (XXXX-XXXX-...). Save it somewhere that is not this machine (a password manager, a printout). The hub keeps only a wrapped copy of its data key that this key can open, never the key itself, and will not store a secret until you have typed the key back to confirm you saved it. Lowercase, spaces and dashes do not matter when you type it. A signed-in admin can regenerate it; the old one stops working at once.
• Who can see what. GET /api/v1/config/secrets shows the store's state and the stored secret names with their last-changed times, never a value. An admin API token can read that list; setting, restoring and re-keying the store need an admin signed in to the web app.
• Losing the hub key (secret.key deleted, a new machine, a changed YARRIX_SECRET_KEY) does not crash the hub. Every config load reports secret NAME can't be decrypted for each reference, and Settings says why. Enter the recovery key (Settings, or as a sign-in at /recover) and the hub re-keys the store with a new secret.key; every secret reads again. Without the hub key and the recovery key the stored values cannot be recovered, by design.

The recovery key is also accepted at /recover in place of the PIN from yarrix-hub auth setup-pin: it gives an admin session (audited as via: secret_recovery_key) and is not used up. Wrong guesses are limited like the PIN's. Treat it like a master password, since it opens the store and signs you in as an admin.

• A heartbeat ping URL is a credential: anyone with it can mark a job as healthy. Do not paste it in chat or tickets.
• API tokens are shown once and stored hashed. Give a script the narrowest scope: read for dashboards, report for backup reports.
• Alert webhooks can be signed so the receiver can check them (alert-routing.md). Destination URLs that point at cloud metadata addresses are refused.
• Failed guesses against local passwords (every sign-in start counts), setup and installer PINs, login links and API tokens lock out that client for 15 minutes. 100 failed password attempts in a row switch the password off until recovery (yarrix-hub auth setup-pin). The six-digit PINs are also limited hub-wide (30 wrong guesses burn the live PINs; setup stops at 100 wrong guesses until yarrix-hub auth setup-pin), and those counts survive a restart.

SSH from the hub image#

The Docker image carries OpenSSH's client (ssh, ssh-keygen, ssh-keyscan), so agentless hosts, the web terminal, SSH custom checks and SSH backup probes work from it. It still runs as a non-root user on a read-only root file system with no shell and no package manager. The hub gives ssh a throwaway home and its own config, so nothing in the image or the container's environment is read as SSH settings.

Two things follow from running in a container:

• Keys are files you mount. Put each host's key in a folder, mount it read-only (for example ./keys:/keys:ro) and point identity_file at it. ssh refuses a private key that other users can read, so the file must be mode 0600 and owned by the user the container runs as (YARRIX_UID, or 65532 by default).
• Any user ID works. YARRIX_UID can be a uid that the image has never heard of. For ssh only, the hub adds that user through nss_wrapper, so the "No user exists for uid" error does not appear.

scripts/hub-image-ssh-proof.ts builds the image from deploy/docker/compose.yaml, starts a throwaway sshd container next to it, and checks that the hub collects from it and that the terminal opens a shell there. It removes everything it creates.

Windows#

The Windows agent and the Windows agentless probe have not been tested on a real Windows machine yet. They are cross-compiled and unit-tested against hand-written samples of Windows output. Read platforms/windows.md, and run them first on a machine you can afford to lose. That includes agent self-update: the swap of yarrix.exe and the rollback have not run on real Windows hardware (see Agent updates).

Running helper tools as root#

When the agent runs as root it looks for smartctl, systemctl and nvidia-smi only in standard system directories and refuses one that a non-root user can write to, so an unprivileged user cannot plant a program that root then runs.