This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Host security maintenance

Repeatable security scans and package updates without automatic reboot

Security scan execution and reportsSecurity scan execution and reports

Run and copy label Ansible actions; report data returns to the controller.

The security playbook composes the existing owner inventories, runs the pinned Lynis audit, updates all installed packages, and enables native automatic security updates. OSTree hosts stage complete deployments for a later boot. Immediate and automatic reboots are disabled. The playbook verifies unchanged boot identities; it does not claim that an audit proves absence of compromise.

The default host pattern excludes the legacy Vault VM group and the XCP-ng/XOA appliance groups. Access failures are reported and excluded from execution; failed access is not evidence that a machine is inactive. The inventory mapping contains source labels only. Correct host membership in the owning inventory. Owner deployment group variables are intentionally not imported: this workflow needs only inventory identities and explicit connection overrides.

Create a private controller report directory before running:

umask 077
mkdir -p out/security/reports out/security/tmp
TMPDIR="$PWD/out/security/tmp" bazel run //infra/security/ansible:ansible -- \
  --extra-vars "lynis_audit_report_directory=$PWD/out/security/reports"

Use --limit for a subset. The security_audit tag performs the audit only; update_all_packages updates packages and reconciles automatic updates after separately reviewed audit evidence. Full execution performs both, sequentially per host. Audit-only runs may set security_serial=3. An explicit additional inventory may supply connection overrides, including a local connection for Host Bot when running on that exact machine. Never apply a fleet-wide local connection override.

Authentication uses the existing operator AppRole, temporary signed SSH keys, and the public SSH host CA. It does not inject unrelated deployment secrets. The SSH target uses the same boundary:

bazel run //infra/security:ssh -- ansible@host1.ingress.alwaldend.com uname -r

Lynis reports and logs are private operational artifacts. Review warnings before maintenance; the role never applies scanner hardening suggestions. Distribution updates can restart services despite preserving the host boot. Installed kernel fixes can remain inactive until a separately authorized reboot.

//infra/security/ansible:clamav runs the shared native streaming scan role against the existing fleet inventories. It is independent of the maintenance playbook. //infra/security/appliances:clamav uses the appliance authentication boundary and defaults to XOA only. XCP-ng remains unscanned because the official native payload requires a newer glibc; its Lynis and update workflows are unchanged.

Both callers import one shared three-play lifecycle: prepare the controller, scan selected hosts, and clean up before reporting accumulated failures. The thin callers retain their separate host defaults and connection variables; packaging retains each inventory and authentication boundary.

The controller uses the pinned official ClamAV engine in a temporary private directory and a bounded transient systemd user unit. Endpoints stream files to its private Unix socket through the shared role’s SSH transport; they do not load the signature database. Scans install no persistent service and perform no reboot, deletion, or quarantine.

Startup verifies native signature metadata and daily freshness, then uses one bounded daemon VERSION polling task to require the verified daily version and a server response shape. The client’s local-version fallback is rejected. signatures.json owns signature metadata; lifecycle evidence records the actual server version without duplicating it in a separate engine.json.

Remote scans require reverse Unix-socket forwarding permission for the SSH account. The workflow preserves server SSH policy and stops before scanning when forwarding is denied. Review any required exception through the owning SSH configuration; the scanner does not relax login or forwarding restrictions.

Create private report and temporary directories, and retain one signature cache across runs. For example:

umask 077
mkdir -p out/security/clamav/reports out/security/clamav/database out/security/clamav/tmp
TMPDIR="$PWD/out/security/clamav/tmp" \
ANSIBLE_LOCAL_TEMP="$PWD/out/security/clamav/tmp" \
  bazel run //infra/security/ansible:clamav -- \
  --extra-vars "clamav_scan_report_directory=$PWD/out/security/clamav/reports" \
  --extra-vars "clamav_database_directory=$PWD/out/security/clamav/database"

Run one controller workflow at a time. Preserve the cache, including cooldown state; do not create a new cache to retry a download failure. Every --limit must include localhost, for example --limit 'localhost,host1.ingress.alwaldend.com' or --limit 'localhost,xoa'. For a local scan of Host Bot, explicitly exclude the entire task scratch directory through clamav_scan_exclude_paths, including controller files, signatures, and reports. Only Host Bot should receive a local connection override.

The controller must have a working systemd user manager, at least 4 GiB available RAM, and at least 5 GiB free disk. Its unit runs at Nice 19 with idle I/O priority and a 6 GiB memory ceiling. clamav_controller_timeout_seconds bounds its total lifetime (default 172800); clamav_scan_timeout_seconds bounds each endpoint (default 86400). If Ansible is interrupted, the unit can remain until that maximum runtime; the recorded unique unit can also be stopped explicitly.

Host failures are retained while later selected batches continue. The final play attempts controller cleanup before reporting accumulated failures; host-failure reporting still runs if cleanup fails. Inspect native scan logs and each role receipt: completion, detections, read errors, exclusions, and size/parser limits are distinct outcomes. Successful execution is not proof of complete coverage or absence of compromise. The controller documentation describes signature verification, engine limits, lifetime, and private evidence.

The shared SSH role supplies the forwarding default for ansible. Apply SSH configuration through the owning host deployment with --tags ssh and --limit; this is normal SSH setup, including certificate handling and the upstream restart handler. The completed one-time maintenance playbook and helper are retired. Their historical rollout evidence remains in the change’s validation record; normal owner deployment does not expose that retired reload-only or rollback workflow. The scanner does not apply server configuration.

1 - Appliance security maintenance

Private XCP-ng and XOA audits with separately selected no-reboot updates

The default target audits the XCP-ng and XOA groups from //infra/xcp_ng/ansible:inventory. It reuses the owner’s xcpng_host and xcp_ng_ansible credential selectors. SSH uses the controller’s existing ~/.ssh/known_hosts, requires an ED25519 match, and never adds keys. The Host Bot configuration owns approved key provisioning and provenance.

XOA’s existing firewall limits new SSH connections, so its transport reuses one authenticated connection. The controller control socket uses a per-run mode 0700 /tmp/appliance-ssh.* directory: the workspace’s absolute path exceeds Linux’s 108-byte Unix-socket limit. The playbook closes the connection and removes this temporary directory. An unreachable host can prevent Ansible’s cleanup; ssh-control-directory.txt records the exact controller path for operator cleanup. Audit evidence remains in the requested workspace directory.

Both appliances use alwaldend.main.lynis_audit with an already installed Ansible-compatible interpreter. XCP-ng’s host-level interpreter declaration and earlier manual bootstrap exception belong to the owner inventory. Ansible reports runtime failures directly; the audit installs no interpreter or scanner package.

Create an existing controller-owned mode 0700 report directory. The default audit uses the pinned Lynis archive:

umask 077
mkdir -p out/security/appliances/tmp
TMPDIR="$PWD/out/security/appliances/tmp" \
ANSIBLE_LOCAL_TEMP="$PWD/out/security/appliances/tmp" \
  bazel run //infra/security/appliances:ansible -- \
  --extra-vars "appliance_report_directory=$PWD/out/security/appliances"

Use --limit xcpng or --limit xoa for one appliance group. Use separate report directories to compare runs. Each host gets a unique private Lynis result directory containing scanner reports and the caller’s boot identities. Files are mode 0600. Scanner exit codes 0 and 78 mean the scan completed; 78 includes warnings. They are not a finding that a host is uncompromised.

:ansible.status performs only a boot-ID and uptime probe with one SSH connection per appliance and retains connection failures.

The shared Lynis role owns archive extraction, the audit profile, private evidence collection, failure handling, and remote scratch cleanup for both appliances. Its unique result directory is directly under appliance_report_directory; there is no second appliance audit lifecycle. Set lynis_audit_timeout_seconds to change the shared role’s default bound. The caller records before/after boot identities and the XOA connection recovery path there after the role returns, including scanner failures. An interruption can prevent these final observations; it does not establish an unchanged boot or a completed audit. Raw status and boot checks remain available without the audit’s Python runtime.

The local-only :syntax target checks the playbooks without AL authentication or a host connection. Building :ansible_bin checks packaging; neither is a substitute for inspecting a successful live audit’s evidence.

Auditing never selects an update playbook. The separate :ansible.update_xcpng and :ansible.update_xoa targets require appliance_report_directory and explicit operator authorization for the selected mutation. Both preserve and compare boot IDs.

The XCP-ng path permits only a single-host XCP-ng 8.3 pool coordinator with HA off, no active XAPI tasks, no XOSTOR, no attached CD media, and no unreviewed yum-cron. It records the host’s enabled state, temporarily disables new VM placement, and runs yum update using only the stable xcp-ng-base and xcp-ng-updates repositories with signature/TLS verification intact. After successful package installation it invokes the vendor xe-toolstack-restart, waits for XAPI readiness, and restores the original enabled state. Existing guests are not evacuated, stopped, or rebooted. Restoration is also attempted after failure when XAPI responds; failure to recover XAPI is reported rather than claiming restoration. Kernel and hypervisor fixes can remain inactive until a separately authorized host reboot.

This vendor yum workflow remains on raw transport. A newer Ansible interpreter does not provide the vendor Python’s yum module bindings or replace its maintenance procedure.

If the selected download mirror fails, set appliance_xcpng_use_origin_mirror=true to select the vendor’s documented https://updates.xcp-ng.org origin for this transaction. The override applies only to the same stable 8.3 repositories and preserves signature and TLS verification; it does not change installed repository configuration. See the vendor mirror documentation.

The XCP-ng vendor procedure documents the disable, yum update, control-plane restart, and enable sequence without a host reboot. It requires pool coordinator ordering and HA precautions. Its Rolling Pool Update procedure reboots hosts and therefore cannot satisfy a no-reboot run. The CLI reference directs package maintenance to yum instead of xe update-*. Multi-host and XOSTOR maintenance require a separately reviewed coordinated workflow.

The XOA path verifies Debian and invokes the shared package-update role using the existing interpreter and APT bindings. Missing bindings fail rather than being installed implicitly. The role installs the authorized standard unattended-upgrades component if needed and configures native OS automatic updates with immediate and automatic reboots disabled.

The XOA application updater is separate from Debian OS maintenance. Its installed version and unattended reboot semantics must be verified before application automation is changed. This workflow does not claim or enable application automatic updates. Native unattended XCP-ng updates meeting the pool and no-reboot constraints have not been established; appliance_automatic_updates=true fails before mutation.

The independent :clamav target reuses the appliance inventory and authentication but defaults to XOA only. XCP-ng remains unscanned because the official native ClamAV payload requires a newer glibc. This does not change either appliance’s Lynis or package-update workflow.

Follow the ClamAV invocation instructions, selecting //infra/security/appliances:clamav and including localhost in any limit, such as --limit 'localhost,xoa'. The shared native role streams endpoint data to the private controller; its signature cache and bounded transient unit are owned by the controller package. :clamav_syntax checks this playbook locally without authentication or live scanning.

2 - ClamAV controller lifecycle

The fleet and XOA callers import one shared Ansible lifecycle playbook, passing their host patterns and connection variables while retaining separate inventory and authentication packaging. Native tasks and configuration templates prepare an unprivileged controller. The official pinned payload is extracted into a mode 0700 invocation directory; it is not installed as a host package. No custom controller helper or database parser is used.

The playbooks require clamav_scan_report_directory and clamav_database_directory as existing operator-owned mode 0700 absolute paths under ignored task scratch. Paths use ASCII letters, numbers, underscores, periods, slashes, and hyphens. Preparation requires a working systemd user manager, native tar/flock/timeout tools, at least 4 GiB available RAM, 5 GiB free disk on both paths, and writable directories.

One native freshclam invocation preserves the existing database cache and its cooldown state. An exclusive nonblocking flock prevents overlapping update commands. Use one controller scan workflow at a time against the cache. The relocated tools receive LD_LIBRARY_PATH=prefix/usr/local/lib and CVD_CERTS_DIR=prefix/usr/local/etc/certs; freshclam/clamd configuration and sigtool’s command also specify the certificate directory. The environment is needed by freshclam’s database-test subprocess as well. TLS, code-signature verification, and TestDatabases yes remain enabled.

Native sigtool --info verifies daily/main/bytecode containers. Ansible filters retain their versions, signature counts, build times, and ages. Daily signatures may be at most 72 hours old by default (clamav_daily_max_age_seconds=259200). Main and bytecode ages are recorded without a daily-age limit because their release cadence differs; future timestamps beyond five minutes are rejected. A successful freshclam return code alone is not evidence of freshness.

systemd-run --user starts foreground clamd as a uniquely named transient unit with RuntimeMaxSec (default 172800 seconds), TimeoutStopSec=15, KillMode=control-group, UMask=0077, and default Nice=19, IOSchedulingClass=idle, and MemoryMax=6G. clamav_controller_memory_gib selects the integer memory ceiling in GiB (default 6, range 1–64), mapped to systemd’s MemoryMax. The shared caller boolean clamav_scan_background defaults to true; false selects Nice=0 and IOSchedulingClass=best-effort. Supply booleans through JSON/YAML extra variables, for example --extra-vars '{"clamav_scan_background": false}'. Controller preparation supplies the default before the endpoint role is loaded. Select overrides only after checking controller headroom; the existing RAM/disk minimums remain. The unit owns the process group, its private working directory, and native log output. One native clamdscan --version polling task waits for a daemon response containing the engine version, verified daily version, and database timestamp; the client’s local-version fallback is rejected. One outer timeout bounds the whole loop, including a stalled client, with a 15-second forced-stop grace. The archive pin owns the engine version; the actual server version is retained in lifecycle evidence. Real clean-file and EICAR validation remain separate acceptance steps.

The daemon binds a mode 0600 Unix socket inside a mode 0700 native RuntimeDirectory, under the operator’s XDG_RUNTIME_DIR. Systemd owns its creation and removal with the transient unit. This gives clients a short absolute path without process/cwd aliases. Reports, logs, extraction and the signature cache stay under ignored workspace scratch. The native runtime root must be operator-owned mode 0700. SSH controls use separate private runtime subdirectories, retained until authenticated recovery is finished.

clamav_controller_threads selects native MaxThreads (default 2, integer range 1–32); MaxQueue is four times that value (default 8). Thread and memory inputs accept integers or decimal digit strings; invalid values fail before engine execution. Lifecycle receipts retain the selected resources and explicit scheduling settings in their controller object.

The fixed engine policy uses no concurrent reload, a 100 MiB file limit, matching client/server stream limits of 100 MiB plus one byte, 400 MiB aggregate scan size, 10,000 archive members, recursion 17, and 120,000 ms scan time. Heuristic alerts, encrypted-content alerts, and limit alerts are enabled; heuristic precedence is disabled. ClamAV 1.5.4 has no ScanEncrypted option. Encrypted content and limit alerts are limitations or heuristics, not confirmed malware. Neither deletion nor quarantine is configured. The extra stream byte makes oversized inputs produce an explicit file-limit alert instead of accepting a truncated prefix or aborting directory traversal at the protocol limit.

The first play prepares the controller; the selected-host play invokes the shared role and retains failures while continuing later serial batches; the final play attempts controller cleanup before reporting accumulated failures. Host-failure reporting still runs when cleanup fails. Every --limit must include localhost. Startup failure stops the named unit before failing the first play. If Ansible is killed or cleanup is otherwise skipped, RuntimeMaxSec is the fallback: shutdown is bounded, not immediate on parent-process exit. The private lifecycle receipt records the exact unit for systemctl --user stop UNIT recovery. No process-name search or unverified PID signal is used. The unit is transient; no permanent user service or host configuration is installed.

Mode 0600 configuration, signature, freshclam, clamd, and lifecycle artifacts remain in the invocation directory. Signature metadata is owned by signatures.json; lifecycle evidence retains the actual server version without duplicating signature metadata. The initial receipt is incomplete before engine execution. Cleanup confirmation is not an endpoint scan verdict. Native role reports must be checked for detections, read errors, exclusions, and coverage limits. For a local Host Bot scan, the caller explicitly excludes the whole task scratch tree, including the database cache and controller extraction.

Build //infra/security/ansible:clamav_bin and //infra/security/appliances:clamav_bin to check packaging. Their separate :clamav_syntax targets parse playbooks locally without AL authentication or engine/host execution. The parent owns real-engine, clean/EICAR, and fleet validation. The appliance scan defaults to XOA; XCP-ng is unscanned because of the native payload’s glibc requirement.