This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

VictoriaMetrics

VictoriaMetrics VM inside DC1 infrastructure, with dashboard access and Traefik scrape pathsVictoriaMetrics VM inside DC1 infrastructure, with dashboard access and Traefik scrape paths

Bazel renders the Mermaid source in light and dark themes.

One Fedora VM on XCP-ng runs VictoriaMetrics single-node under systemd. The guest starts with 2 vCPUs, 4 GiB RAM, and a persistent 40 GiB boot and data disk. The database lives at /var/lib/victoria-metrics. Retention is 30 days, with an ingestion cutoff when free disk space falls below 5 GiB. Bazel verifies the release archive and packages the executable for Ansible.

VictoriaMetrics scrapes its loopback endpoint every 15 seconds and the Traefik proxies every 30 seconds. The traefik_ingress job uses active ingress hosts from the WireGuard inventory. The traefik_dc1 job covers Forgejo, dev1, three OpenHands hosts, the 3x-ui control plane, and the local download host. Each proxy firewall allows metrics traffic from the VictoriaMetrics guest address. Cloud-only proxies need a private route before they can join these scrape jobs.

Add scrape jobs to the scrape YAML template. Bazel packages the ingress inventory as an extra-vars file. Ansible renders the active WireGuard targets from it.

  1. Provision the dedicated Vault AppRole, including its DNS policy, SSH host certificate signer, and the Vault ACME role. Run the web credential setup to generate and store the dashboard login and verifier. Reconcile Vault memberships and synchronize the OIDC identity and resource set through the Xen Orchestra workflow.
  2. Verify that the address in the DNS declaration is available on the DC1 VM network. Review the component’s DNS and VM plans separately before applying them.
  3. Once SSH is available and cloud-init has finished, verify the guest’s host key and run bazel run //infra/victoria_metrics/ansible. The configured SSH destination on DC1 is victoria-metrics.alwaldend.com. The deployment also allows an explicit ansible_host override before DNS is available.
  4. Verify Vault resolves the service hostname through DC1 DNS and can reach guest Traefik on port 80 for ACME validation. Check that Traefik serves the Vault certificate and shared ingress can reach the backend. Guest Traefik renews the certificate automatically. Shared ingress uses Let’s Encrypt.

Open https://victoria-metrics.alwaldend.com/vmui/ with a trusted ingress client certificate and the shared Basic Auth login. Members of Vault’s global_admins group can read the credentials at secrets/alwaldend.com/vault1/approles/src_infra_victoria_metrics/web_auth. Ingress verifies the guest’s Vault-issued certificate. On the DC1 LAN, use the same dashboard URL and Basic Auth login. DC1 DNS points the service name directly at guest Traefik, so the browser must trust the Vault server CA. HTTP redirects to HTTPS. Shared ingress alone uses the dc1.victoria-metrics.alwaldend.com upstream name. Select Dashboards → Monitoring overview for scrape health, Traefik traffic and certificate expiry, and VictoriaMetrics disk and ingestion metrics. The panels are defined in monitoring-overview.json. For a table of targets that are down, run up == 0 in Query and select Table.

The HTTP listener on guest loopback also remains available through SSH local port forwarding to 127.0.0.1:8428 using the Ansible account. This deployment has one failure domain. Guest or disk loss can make the stored metrics unavailable. Off-VM backup storage is outside this component.

1 - Vault AppRole

This Terraform root owns the AppRole for the VictoriaMetrics deployment. Provision the shared Vault prerequisites, apply this root, then reconcile the shared AppRole and Ansible memberships before synchronizing its Xen Orchestra identity.

Use bazel run //infra/victoria_metrics/approle:tf.plan to review the AppRole and tf.apply to deploy it. The root uses the Vault infrastructure credentials for its policies and HTTP state.

The AppRole can enroll Traefik through the component’s Vault ACME directory. The PKI role allows the VictoriaMetrics service hostname.

The global_admins group inherits read access to the login through the component’s readers group. The component AppRole manages the web_auth entry in its own subtree. The separate tf_setup package generates and stores the credentials using these permissions.

2 - VictoriaMetrics DNS

The DNS declaration gives the service hostname two records: a public CNAME to shared ingress and a DC1 A record for the guest. A third record, the public dc1.victoria-metrics.alwaldend.com A record, lets shared ingress resolve the guest’s private address through public DNS. Terraform provisioning uses the same guest address. Verify the address is available on the DC1 network before deployment.

Review DNS changes with bazel run //infra/victoria_metrics/dns:tf.plan.

3 - VictoriaMetrics web credential setup

Run bazel run //infra/victoria_metrics/tf_setup/web_auth:tf_setup.apply after configuring the component AppRole and before running Ansible. The component AppRole writes the dashboard login and bcrypt verifier to web_auth in its own Vault subtree. Ansible injects the verifier file for Traefik.

The credential state uses an admin-owned backend because it contains the generated password. Reapply the guest playbook after rotating the credentials to load the updated verifier.