This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Vault

Setup for vault.dc1.alwaldend.com

This diagram describes the maintained architecture source, not live inventory or service health. The Mermaid source is authoritative; Bazel renders its light and dark SVGs.

Vault architectureVault architecture

bazel run //infra/vault/tf_setup:tf.apply # Create VMs (requires an active Vault host)
bazel run //infra/vault/ansible # Set up hosts (BM and VMs)
bazel run //infra/vault/tf:tf.apply # Configure vault

bazel run //infra/vault:backup

With a working Vault:

bazel run //infra/vault:unseal

Without a working Vault:

bazel run //infra/vault:unseal_standalone

bazel run //infra/vault/ansible:fix_quorum

bazel run //infra/vault/ansible:ansible.vm # Set up only VMs

bazel run //infra/vault/ansible:ansible.bm # Set up only bare metal

Plan:

bazel run //infra/vault/tf:tf.plan

Apply:

bazel run //infra/vault/tf:tf.apply

Run terraform directly:

bazel run //infra/vault/tf:tf.direct -- -chdir="${PWD}" plan

bazel run //infra/vault/tf:tf.apply -- -replace 'module.vm_ha["host2"].proxmox_vm_qemu.vm' -replace 'module.vm_ha["host3"].proxmox_vm_qemu.vm

username="username"
bazel run //infra/vault:gen_client_cert -- --user "${username}" --output_dir "${PWD}"
bazel run //tools/ykman -- piv certificates import 9A "${PWD}/${username}.pfx"
bazel run //tools/ykman -- piv keys import 9A "${PWD}/${username}.pfx"

bazel run //infra/vault:gen_client_cert -- --host some-host --output_dir "${HOME}/.al/client_cert"

bazel run //infra/vault:gen_client_cert -- --host some-host --user username --output_dir "${HOME}/.al/client_cert"
  • Prepare encrypted unseal token
  • Run and input the encrypted token: 
    bazel run //infra/vault:unseal
  • Prepare encrypted unseal token
  • Run and input the encrypted token: 
    bazel run //infra/vault:gen_root_token -- --pgp_key path_to_public_gpg_key_in_base64

bazel run //infra/vault -- write -f pki/ica_servers/roles/ica_servers_dc1_pve1/acme/new-eab

bazel run //:vault -- write ssh/clients/sign/admins ttl=30000000  public_key=@"${HOME}/.ssh/key"

bazel run //infra/vault -- token revoke -mode=path auth

Renew the Vault server certificate through its Ansible role. The vault_tls tag issues a certificate, installs its chain and private key, and reloads Vault with SIGHUP. It stops on certificate errors and does not run the host setup, package installation, or service restart tasks.

Run from the repository root. Keep SSH host verification enabled by deriving a temporary known_hosts entry from the repository-owned public SSH CA:

vault_recovery_dir="${PWD}/out/vault-certificate-recovery"
mkdir -p "${vault_recovery_dir}"
printf '@cert-authority host1.vault.alwaldend.com,host1.vault.dc1.alwaldend.com %s\n' \
  "$(cat infra/vault/tf/output/ssh_ca_servers.crt)" \
  >"${vault_recovery_dir}/ssh_known_hosts"

For an expired server certificate, temporarily disable Vault certificate verification for this recovery command only. The SSH options trust that CA only for this invocation and retain the existing SSH certificate’s host alias:

env VAULT_SKIP_VERIFY=true \
  ANSIBLE_SSH_COMMON_ARGS="-o StrictHostKeyChecking=yes \
-o HostKeyAlias=host1.vault.alwaldend.com \
-o UserKnownHostsFile=\"${vault_recovery_dir}/ssh_known_hosts\"" \
  bazel_agent bazel run //infra/vault/ansible:ansible -- \
  --limit host1.vault.alwaldend.com --tags vault_tls

Vault must be reachable and unsealed, and the host token must allow the component’s AppRole login. If that token has expired, connect the login YubiKey and refresh it first:

bazel_agent bazel run //tools/vault/login -- simeonwarren --insecure

This login replaces the host token. Stop if it fails; a missing YubiKey cannot be fixed by disabling TLS verification. Neither command changes the persistent TLS verification configuration. Do not add the bypass to shell profiles, al.lua, or Ansible defaults, and do not use verbose or diff output for the certificate tasks. The temporary public known_hosts projection does not modify the host’s SSH configuration; do not replace it with an unverified ssh-keyscan result or disable SSH host verification.

After renewal, verify Vault with certificate validation enabled:

env -u VAULT_SKIP_VERIFY bazel_agent bazel run //:vault.status

For renewal before expiry, omit VAULT_SKIP_VERIFY=true from the Ansible command. The current inventory serves Vault from host1; review the inventory before changing the host limit. Its inventory name remains host1.vault.alwaldend.com to preserve the Vault node identity; ansible_host connects through the DNS-owned name host1.vault.dc1.alwaldend.com. Certificate alternate names include this connection hostname for the existing unseal endpoint, as well as the shared Vault service names.

bazel run //infra/vault -- read identity/oidc/client/src_infra_dc1_forgejo1_provider

 bazel run //infra/vault -- list -format json identity/entity-alias/id | jq ".[]" | xargs "-I{}" bazel run //infra/vault -- read "identity/entity-alias/id/{}"

1 - Ansible

Ansible setup

2 - OpenSpec

2.1 - Specifications

2.1.1 - Download identity

2.1.1.1 - download-identity Specification

Provide a dedicated Vault identity for download infrastructure while keeping state, credential references, and provider permissions scoped to its needs.

Vault configuration SHALL declare src_infra_download using the existing AppRole backend and reusable identity modules. Its state permissions SHALL cover the component’s own environment and DNS state keys, not another component’s state namespace.

  • WHEN the download identity’s effective state policy is evaluated
  • THEN it grants access to download-owned state and locks
  • AND it does not grant access to unrelated component state

The identity SHALL expose the membership and identity outputs needed for Yandex folder provisioning and XCP-ng resource-set assignment. It SHALL receive the required component SSH permissions and DNS provider-secret reads for global and local records without administrative provider credentials.

  • WHEN provider owners consume the declared download identity
  • THEN they can assign its Yandex folder and XCP-ng resource set
  • AND no unrelated component’s identity is reused for deployment

Checked-in declarations SHALL contain secret references only. Download certificate automation SHALL receive role-scoped EAB access for internal Vault HTTP-01 certificates. The role SHALL permit only alwaldend.com, download.alwaldend.com, and www.alwaldend.com, without subdomain or client certificate permissions. Public Let’s Encrypt HTTP-01 SHALL require no DNS challenge credential. Adding this identity SHALL NOT broaden unrelated identities’ policies.

  • WHEN the role and its policy declarations are inspected
  • THEN they name Vault paths and permissions without embedding credential values
  • AND unrelated identity permissions remain unchanged

The download identity SHALL be deployed from its own Terraform root and backend under infra/vault/approles/src_infra_download. Shared identity IDs SHALL be resolved by name through data sources rather than parent-module inputs or another root’s state. Shared group membership lists SHALL retain one owner in the core Vault stage.

  • WHEN shared Vault mounts and the administrative/operator identities already exist
  • THEN the standalone root can create the download identity using administrative authentication
  • AND a subsequent core Vault apply resolves the new identity/group by name and adds their memberships

2.1.2 - Infra vault

2.1.2.1 - Infrastructure Vault

Describe the Vault host configuration, authentication and certificate services, and packaged recovery entry points. This baseline concerns checked-in desired state and supported wrapper structure; it does not establish live unsealed state, successful backups, quorum, or certificate freshness.

Baseline source revision: 550d7e79b1f5fdbc2b6017b75178471d6914082f. Observation date: 2026-09-08. Sources are linked in full; no excerpts are used.

Sources: operator documentation, operational wrappers, Ansible entry points, Vault host template, authentication backends, storage engines, server PKI, client PKI, and public CA outputs.

The documented workflow SHALL separate tf_setup VM provisioning, Ansible host setup, and the tf Vault configuration stage. Ansible SHALL expose combined, VM-only, and bare-metal-only setup entry points using the shared host and Vault roles.

  • WHEN an operator selects ansible.vm or ansible.bm
  • THEN the wrapper selects the matching checked-in playbook and inventory
  • AND Vault’s API configuration remains owned by the separate Terraform stage

The host template SHALL configure TLS API and cluster endpoints on ports 8200 and 8201 and Raft storage under /opt/vault/raft. It SHALL derive each Raft node identifier from the inventory hostname and emit retry-join addresses for hosts in the Vault inventory group.

  • WHEN Ansible renders vault.hcl for a member of the Vault group
  • THEN its API and cluster URLs use its inventory hostname
  • AND Raft receives that hostname as node ID and the group’s HTTPS join URLs

Terraform SHALL declare userpass, approle, and cert authentication backends, a version-two KV engine at secrets, and a transit engine at transit/default. Client and server certificate authorities SHALL remain distinct, and server PKI ACME SHALL require external account binding with the default directory policy set to forbid.

  • WHEN the main Terraform configuration is evaluated
  • THEN authentication, KV, transit, client PKI, and server PKI are distinct declared resources
  • AND server ACME is enabled with eab_policy = "always-required"
  • AND published CA output files contain certificate or public-key material

The package SHALL expose separate wrappers for backup, ordinary unseal, standalone unseal, client certificate generation, and root-token generation. Standalone unseal SHALL use the default_no_auth environment without selecting the ordinary unseal plugin; these wrappers SHALL NOT imply that recovery has been executed successfully.

  • WHEN an authorized operator invokes //infra/vault:unseal_standalone
  • THEN the wrapper selects the packaged unseal utility and declared Vault endpoint with its no-auth environment
  • AND success remains an operational result to be observed separately

Every owner that manages DNS records through Terraform SHALL use its existing component AppRole or a dedicated owner AppRole when one is missing. A DNS-only identity SHALL retain the shared AppRole module’s own-state and named shared secret access without cloud provisioning, SSH, or PKI permissions. Existing component identities and unrelated authentication flows SHALL remain stable.

  • WHEN a project has DNS records but no existing component AppRole
  • THEN Vault configuration declares an AppRole named for that owner
  • AND the identity can manage its own Terraform state without access to another owner’s state
  • WHEN host_bot uses its dedicated DNS AppRole
  • THEN its existing Ansible authentication remains unchanged

DNS identities SHALL receive read access to the existing Cloudflare credential only when they own global records and to the existing RouterOS DNS credential only when they own dc1 records. DNS access SHALL NOT grant writes to either provider secret or duplicate credential values.

  • WHEN an owner declares only global records
  • THEN its DNS policy grants Cloudflare credential reads
  • AND its DNS policy grants no RouterOS credential access
  • WHEN an owner declares global and dc1 records
  • THEN its DNS policy grants reads for both existing credential references
  • AND neither read policy grants mutation of the provider credential

The packaged Vault Ansible workflow SHALL expose a vault_tls tag that issues and installs the configured server certificate and private key, then reloads Vault. Selecting only this tag SHALL NOT update packages, the service binary, host configuration, or Raft peers. Certificate material SHALL be suppressed from task output. Explicit certificate renewal SHALL stop on issuance or installation failure before reloading the service.

  • WHEN the operator selects vault_tls and limits the play to one Vault host
  • THEN only certificate issuance, certificate and key installation, and service reload are selected, in addition to read-only fact gathering
  • AND a failed certificate task prevents the reload

The recovery procedure SHALL use an invocation-scoped verification override only when an expired server certificate prevents the authorized renewal. It SHALL retain the existing Vault authentication and injection flow, require a valid bootstrap login, and verify the recovered service with certificate validation restored. The procedure SHALL NOT persist insecure TLS defaults.

  • WHEN an authorized operator performs the documented recovery
  • THEN the temporary override applies to credential injection and certificate issuance within that command
  • AND a failed bootstrap login prevents proceeding to renewal
  • AND success is established separately with normal TLS verification

2.1.3 - Owned dns

2.1.3.1 - owned-dns Specification

Define Vault service DNS management through the owner’s tf_setup root, including canonical declarations, scoped credentials, and offline source checks before adopting live records.

The tf_setup root SHALL consume this owner’s canonical dnsconfig.json through the shared DNS Terraform module, preserving declared record identities and views. DNS resources SHALL default to disabled until the authorized adoption revision and SHALL retain enabled ownership after existing records are imported into the owner’s state. Reconciliation against unchanged declarations and adopted state SHALL propose no record additions, changes, replacements, or deletions.

  • WHEN the checked-in root is evaluated with default inputs before its authorized adoption revision
  • THEN it reads this owner’s declaration and disables managed DNS records
  • AND the package contains the module and declaration inputs
  • WHEN the authorized adoption completes the shared writer audit and applicable recovery prerequisites and enables the owner’s record management
  • THEN the owning root imports existing records by exact provider ID into its state while preserving their declared identities and views
  • WHEN the adopted root uses its checked-in source defaults
  • THEN DNS ownership remains enabled and the package retains the module and canonical declaration inputs
  • WHEN the owning root plans against its adopted state and unchanged declarations
  • THEN it proposes no record additions, changes, replacements, or deletions

The dns.plan, dns.show, and dns.apply entrypoints SHALL select src_infra_dc1_vault through the repository AL flow using dns=1 and operate on module.dns in the existing tf_setup root and Vault HTTP backend. The apply entrypoint SHALL require a saved plan. Ordinary setup and service wrappers SHALL retain their stage labels, authentication, and backend paths. Secret values SHALL remain in injected variables. Real DNS credentials and Vault policy grants SHALL be prerequisites for operational DNS calls. RouterOS DNS credentials SHALL remain isolated from unrelated RouterOS resources. The package SHALL expose a format test that does not authenticate to Vault or contact DNS providers.

  • WHEN the package format test executes
  • THEN it checks the packaged Terraform configuration without live credentials
  • AND existing non-DNS authentication and backend paths remain unchanged
  • WHEN a scoped DNS entrypoint selects dns=1
  • THEN it selects DNS credential injection through the owner AppRole and existing setup backend, with plan and show scoped to module.dns and apply requiring a saved plan
  • AND disabled DNS resources do not imply offline provider configuration
  • AND ordinary setup and service authentication retains its original labels

3 - Tf

Terraform config

Missing DNS-owner AppRoles compose the existing reusable AppRole module with an owner-specific DNS policy in approles/<name>/. DNS access grants read-only access to the existing provider credentials for the owner’s views. Existing component identities receive these policies through their existing module declarations.

DNS-only group membership adds no policies. These identities retain their own state and named shared-secret access without joining the general infrastructure AppRole group or receiving cloud provisioning, SSH, or PKI permissions.

The operator bootstrap membership follows the existing component pattern. An authorized Vault apply must create the roles and policies before their DNS stages can authenticate; source validation does not establish deployed access. Follow the DNS migration for the coordinated cutover and the AppRole bootstrap procedure when distinguishing missing deployment from insufficient operator access.

3.1 - Output

Output files

4 - Tf setup

Terraform setup

This root owns Vault’s canonical DNS declaration and retains its existing src_infra_dc1_vault AppRole and setup state backend. Use //infra/vault/tf_setup:dns.plan, dns.show, and dns.apply for the scoped DNS workflow. They select dns=1 and module.dns; apply requires a reviewed saved plan. Ordinary setup and service wrappers retain their existing authentication flow.

dns_enabled defaults to true after verified adoption. Keep it enabled to retain existing records; disabling it would propose deletion. The adoption record contains the import and verification evidence. Targeted DNS checks do not establish the health of other resources in the root.

Offline formatting is available through //infra/vault/tf_setup:tf_tests.fmt_test.