This tree contains infrastructure as code. Tracked source follows the repository’s public-source policy. Infrastructure facts are not confidential merely because they are operational, generated, or live. Reports may include them unless they contain credentials, other secrets, or personal information. Inspect raw state, plans, inventories, and decrypted configuration because those artifacts can contain prohibited content; do not track the artifacts themselves.

Requirements

Bazel targets MUST use repository-internal visibility.
Infrastructure definitions MUST NOT be published as production artifacts.
Infrastructure targets MUST NOT be dependencies of production build targets.
Public checked-in documentation MAY be included in the repository documentation site, including non-secret, non-personal operational facts.

New project

The commands below are state-changing operator examples. An agent must use bazel_agent, apply the repository Terraform and secret-handling procedures, and receive explicit authority for the exact operation and environment before running an equivalent command.

Create an approle: example
Add it to approles: example

Run apply:

bazel_agent bazel run //infra/vault/tf:tf.apply

Update Yandex Cloud folders:

bazel_agent bazel run //infra/yandex_cloud/org1/tf:tf.apply

Update Proxmox resource pools:

bazel_agent bazel run //infra/pve/tf:tf.apply

Set up al config: example
Set up a bucket for Terraform state: example

1 - Architecture

Rendered infrastructure architecture diagrams

These diagrams render every page of the checked-in Drawio source. They describe the source document, not a live inventory or health check. Pages marked Archive retain the source’s historical classification. Open an image for its full-size SVG.

DC1

Vault

Flux

Forgejo

www

Threexui

DNS

Ingress

T3code

Truenas

Archive/Harvester

Archive/Opencode

Archive/Hermes

Archive/Proxmox

Archive/Harbor

Edit and regenerate

Open the canonical source with the pinned Drawio desktop tool:

bazel_agent bazel run //infra/arch

Regenerate the maintained SVGs after editing arch.drawio:

bazel_agent bazel run //infra/arch:update
bazel_agent bazel test //infra/arch:update_tests

The update target renders all 15 named pages in Bazel sandboxes with Drawio 30.2.6 web assets from the pinned desktop archive and the repository’s pinned headless Chrome. A pinned Liberation font set and isolated Fontconfig configuration keep text measurement independent of host fonts. The renderer loads only local assets, fails on missing pages and export errors, and records the source SHA-256 in each SVG. No display server, live infrastructure access, or runtime download is required. The freshness test compares rendered output with the maintained SVGs; documentation consumes those checked-in images.

2 - Ceph

Ceph

Links

Proxmox docs: https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster
Single node crushmap: https://imanudin.net/2025/05/12/proxmox-ceph-single-node-installation-for-test-lab-or-home-setup/

See current crush map

Go to Nodes -> Host -> Ceph -> Configuration

Crush map update

Ssh to the PVE host and update the map:

sudo ceph osd getcrushmap -o crushmap.cm
sudo crushtool --decompile crushmap.cm -o crushmap.txt
sudo vim crushmap.txt
sudo crushtool --compile crushmap.txt -o new_crushmap.cm
sudo ceph osd setcrushmap -i new_crushmap.cm
sudo ceph -s

3 - Dns

Dns setup for alwaldend.com

Links

Docs: https://docs.dnscontrol.org
Rules: ../../projects/rules_dnscontrol

Deployment

//infra/dns:config_test validates the generated configuration with the pinned DNSControl check command. It requires no credentials and does not access DNS providers.

Operational wrappers pass providers.json explicitly. Its environment references are populated by the AL Vault injector; DNSControl reads the values directly without a rendered credentials file.

Project landing records live next to their projects in projects/<project>/dnsconfig.json. Project files own their subdomain CNAMEs, which point directly to alwaldend.github.io. as required by GitHub Pages custom-domain setup.

Interactive:

bazel run //infra/dns

Just preview:

bazel run //infra/dns:dns.preview

Just deploy:

bazel run //infra/dns:dns.deploy

Deploy modifies the bind file, which will cause //infra/alwaldend.com/dns:preview_test to fail

dc1 BIND

{{/< readfile file=“zones/dc1/alwaldend.com.zone” code=“true” lang=“zone” >/}}

global BIND

{{/< readfile file=“zones/global/alwaldend.com.zone” code=“true” lang=“zone” >/}}

3.1 - Tf

Terraform config

4 - flux

Fluxcd deployment

Links

Site: https://fluxcd.io/

Set up port forwarding

ssh -L 6443:127.0.0.1:6443 -N flux.alwaldend.com

Run flux CLI

bazel run //infra/flux/cl:flux

Show flux status

bazel run infra/flux/cl:flux -- get all -A

Run bootstrap

bazel run //infra/flux/cl:flux.bootstrap

Run flux operator

bazel run //infra/flux/cl:op

Show oidc info

bazel run //infra/flux/cl:oidc

Renew certificate

bazel run //infra/flux/cl:cmctl -- renew traefik-gateway-websecure-tls -n traefik

Update secrets

Generate secret id and the token:

SECRET_ID=$(bazel run //infra/flux/cl:vault.secret_id | jq -r .data.secret_id)
TOKEN=$(echo "${SECRET_ID}" | bazel run //infra/flux/cl:vault.ops_token | jq -r .auth.client_token)
echo "Secret id: ${SECRET_ID}, Token: ${TOKEN}"

Patch secret-id in ./cl/cert-manager/issuer-approle.yaml
Patch token in projects/kustomization/flux-repo/flux-sops-secret.yaml

Encrypt:

bazel run //infra/flux/cl:sops.encrypt infra/flux/cl/cert-manager/issuer-approle.yaml
bazel run //infra/flux/cl:sops.encrypt projects/kustomization/flux-repo/flux-sops-secret.yaml
bazel run //infra/flux/cl:sops.encrypt projects/kustomization/flux-repo/flux-git-src-secret.yaml

4.1 - Ansible

Ansible setup

4.2 - cl

Cluster manifests

Links

Cert manager charts: https://artifacthub.io/packages/helm/cert-manager/cert-manager

4.3 - Tf setup

Terraform setup

5 - forgejo

forgejo.alwaldend.com

Links

Docs: https://forgejo.org/docs/latest/admin/installation/binary/
Config reference: https://forgejo.org/docs/latest/admin/config-cheat-sheet/

Deployment

bazel run //infra/forgejo/tf_setup # Create hosts
bazel run //infra/forgejo/ansible # Configure hosts
bazel run //infra/forgejo/tf # Configure forgejo

5.1 - Ansible

Ansible setup

5.2 - Tf

Terraform config

5.3 - Tf setup

Terraform setup

6 - Forgejo runner

Forgejo Actions runner deployment

Deploy VMs

bazel run //infra/forgejo_runner/tf_setup # Create VMs
bazel run //infra/forgejo_runner/ansible # Configure VMs

6.1 - Ansible

Forgejo runner configuration

6.2 - Terraform setup

Forgejo runner VM

7 - GitHub

GitHub Pages repositories for project landing sites

7.1 - GitHub Terraform

GitHub Pages repositories for project landing sites

This package creates the public repositories that host the project landing pages. Each repository receives its built Hugo site from the reusable deploy target //projects:deploy_landings in the root workspace. Terraform also manages the custom domain and Pages source (pages, /).

For a new site, first plan and apply with bootstrap_projects containing only the new projects whose pages branches do not yet exist. For example, pass -var='bootstrap_projects=["new_project"]' to //infra/github/tf:tf.plan and the reviewed tf.apply. This creates the repository without trying to enable Pages on a nonexistent branch. Do not include an existing live site: doing so would remove its Pages configuration.

Publish the built site with //projects:deploy_landings, then plan and apply with the default empty bootstrap_projects to enable Pages. Keep all applies limited to reviewed, authorized repository and Pages changes. DNS is managed separately by //infra/dns; validate the public custom domain after rollout.

8 - harbor

Harbor deployment

Links

Site: https://goharbor.io/
Releases: https://github.com/goharbor/harbor/releases

Go to User Profile, copy CLI secret
Run:
```
podman login harbor.alwaldend.com
```

Update secrets

Generate approle secret id:

SECRET_ID=$(bazel run //infra/harbor/cl:vault.secret_id | jq -r .data.secret_id)
echo "Secret id: ${SECRET_ID}"

Patch secret-id in ./cl/cert-manager/issuer-approle.yaml

Encrypt:

bazel run //infra/flux/cl:sops.encrypt infra/harbor/cl/cert-manager/issuer-approle.yaml

8.1 - Ansible

Ansible setup

8.2 - cl

Cluster manifests

8.3 - Tf

Terraform config

8.4 - Tf setup

Terraform setup

9 - ingress

ingress

Run ansible

bazel run //infra/ingress/ansible

Services in traefik_services must use a backend target that resolves from the ingress hosts without resolving back to ingress. Define a dedicated site-local address, conventionally prefixed with dc1., in the owning service’s DNS configuration and use that address as the target.

Apply terraform

bazel run //infra/ingress/tf

Update signed image url

Create a signed url: https://yandex.cloud/ru/docs/storage/operations/objects/link-for-download
Update the secret: alwaldend.com/vault1/approles/src_infra_ingress/image

Recreate VMs

bazel run infra/ingress/tf -- -replace 'yandex_compute_disk.vpc["*"]'

Regenerate wireguard keys

Regenerate private and public keys:

wg genkey | tee host1.privatekey.txt | wg pubkey >host1.publickey.txt
wg genkey | tee host2.privatekey.txt | wg pubkey >host2.publickey.txt
wg genkey | tee router.privatekey.txt | wg pubkey >router.publickey.txt
cat - >data.json <<EOF
{
  "wg_public_keys": {
    "host1": "$(cat host1.publickey.txt)",
    "host2": "$(cat host2.publickey.txt)",
    "router": "$(cat router.publickey.txt)"
  },
  "wg_private_keys": {
    "host1": "$(cat host1.privatekey.txt)",
    "host2": "$(cat host2.privatekey.txt)",
    "router": "$(cat router.privatekey.txt)"
  },
  "wg_preshared_keys": {
    "host1": "$(openssl rand 32 | base64)",
    "host2": "$(openssl rand 32 | base64)"
  }
}
EOF
bazel run infra/ingress:vault.kv_put -- -format json alwaldend.com/vault1/approles/src_infra_ingress/wireguard "@${PWD}/data.json"
rm data.json *.privatekey.txt *.publickey.txt

9.1 - Ansible

Ansible setup

9.2 - Tf

Terraform config

10 - Mikrotik

Mikrotik setup for dc1.alwaldend.com

Deployment

Open Winbox
Open new terminal
Run /export
Copy output

Links

Exports

Router1

/interface bridge
add admin-mac=78:9A:18:38:6C:CA auto-mac=no comment="bridge1 (wired)" name=bridge1
add comment="bridge2 (wireless)" name=bridge2
/interface wifi
set [ find default-name=wifi1 ] channel.frequency=5000-5400 .skip-dfs-channels=10min-cac comment="wifi1 (5GHz)" configuration.country=Russia .mode=ap .ssid=divinity-5GHz datapath.client-isolation=yes disabled=no security.authentication-types=wpa2-psk,wpa3-psk .connect-priority=0 .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.skip-dfs-channels=10min-cac comment="wifi2 (2GHz)" configuration.country=Russia .mode=ap .ssid=divinity-2GHz datapath.client-isolation=yes disabled=no security.authentication-types=wpa2-psk,wpa3-psk .connect-priority=0 .ft=yes .ft-over-ds=yes
/interface ethernet
set [ find default-name=ether1 ] comment=ether1 l2mtu=1500 mac-address=F4:28:53:7F:A4:59
set [ find default-name=ether2 ] comment=ether2
set [ find default-name=ether3 ] comment=ether3
set [ find default-name=ether4 ] comment=ether4
set [ find default-name=ether5 ] comment=ether5
/interface wireguard
add comment="tf[users/simeonwarren/hermes/tf_setup]" disabled=yes listen-port=13232 mtu=1420 name=hermes-vpc
add comment="tf[infra/ingress/tf]" listen-port=13231 mtu=1420 name=ingress-vpc
/interface ethernet switch
set switch1 cpu-flow-control=yes
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=accept-forward-WAN
add name=accept-input-DNS
add name=accept-input-DHCP-server
add name=accept-input-ICMP
add name=accept-input-winbox
add name=accept-input-web-ui
add name=accept-input-mikrotik-neighbor-discovery
add name=accept-forward-LAN
add name=accept-output-LAN
add name=accept-input-NTP
add name=accept-input-API
/ip pool
add comment=bridge1 name=bridge1 ranges=192.168.1.10-192.168.1.254
add comment=bridge2 name=bridge2 ranges=192.168.2.10-192.168.2.254
/ip dhcp-server
add address-pool=bridge1 comment=bridge1 interface=bridge1 lease-time=10m name=bridge1
add address-pool=bridge2 interface=bridge2 name=bridge2
/ipv6 pool
add name=dc01 prefix=fd2e:546d:5738::/48 prefix-length=64
/user group
add comment=src_infra_dns name=src_infra_dns policy=read,write,api,rest-api,!local,!telnet,!ssh,!ftp,!reboot,!policy,!test,!winbox,!password,!web,!sniff,!sensitive,!romon
add comment=src_infra_ingress name=src_infra_ingress policy=read,write,api,rest-api,!local,!telnet,!ssh,!ftp,!reboot,!policy,!test,!winbox,!password,!web,!sniff,!sensitive,!romon
add comment=users_simeonwarren name=users_simeonwarren policy=read,write,api,rest-api,!local,!telnet,!ssh,!ftp,!reboot,!policy,!test,!winbox,!password,!web,!sniff,!sensitive,!romon
/interface bridge port
add bridge=bridge1 comment=bridge1-ether2 interface=ether2
add bridge=bridge1 comment=bridge1-ether3 interface=ether3
add bridge=bridge1 comment=bridge1-ether4 interface=ether4
add bridge=bridge1 comment=bridge1-ether5 interface=ether5
add bridge=bridge2 comment=bridge2-wifi1 interface=wifi1
add bridge=bridge2 comment=bridge2-wifi2 interface=wifi2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=WAN
/interface list member
add interface=bridge1 list=LAN
add interface=ether1 list=WAN
add interface=bridge2 list=LAN
add interface=bridge2 list=accept-forward-WAN
add interface=bridge1 list=accept-forward-WAN
add interface=bridge1 list=accept-input-DNS
add interface=bridge2 list=accept-input-DNS
add interface=bridge1 list=accept-input-DHCP-server
add interface=bridge2 list=accept-input-DHCP-server
add interface=bridge1 list=accept-input-ICMP
add interface=bridge2 list=accept-input-ICMP
add interface=bridge1 list=accept-input-winbox
add interface=bridge1 list=accept-input-web-ui
add interface=bridge1 list=accept-input-mikrotik-neighbor-discovery
add interface=bridge1 list=accept-forward-LAN
add interface=bridge1 list=accept-output-LAN
add interface=bridge1 list=accept-input-NTP
add interface=bridge1 list=accept-input-API
add comment="tf[infra/ingress/tf]" interface=ingress-vpc list=accept-input-ICMP
add comment="tf[infra/ingress/tf]" interface=ingress-vpc list=LAN
add comment="tf[infra/ingress/tf]" interface=ingress-vpc list=accept-forward-LAN
add comment="tf[users/simeonwarren/hermes/tf_setup]" interface=hermes-vpc list=accept-forward-LAN
add comment="tf[users/simeonwarren/hermes/tf_setup]" interface=hermes-vpc list=accept-input-ICMP
add comment="tf[users/simeonwarren/hermes/tf_setup]" interface=hermes-vpc list=LAN
/interface ovpn-server server
add mac-address=FE:B3:B4:C4:A4:48 name=ovpn-server1
/interface wireguard peers
add allowed-address=10.10.0.2/24 comment=host2 endpoint-address=103.76.53.6 endpoint-port=51820 interface=ingress-vpc name=ingress-vpc-host2 persistent-keepalive=5s public-key="Z2JamOjZYOGaf4tPZzchyHjLw/XlOtUtQObyROEQ9DM="
add allowed-address=10.10.0.1/24 comment=host1 endpoint-address=158.160.196.128 endpoint-port=51820 interface=ingress-vpc name=ingress-vpc-host1 persistent-keepalive=5s public-key="xmyl+frvngmzRB9z5yEURxQj4vTw47tKQV7EZrTAREw="
add allowed-address=10.20.0.1/24 comment=host1 endpoint-address=158.160.220.223 endpoint-port=51820 interface=hermes-vpc name=hermes-vpc-host1 persistent-keepalive=5s public-key="oA4ZpsmrclIOIWh3ECsb4ZFKH1hQMDtuW3xNXat3IyQ="
/ip address
add address=192.168.1.1/24 comment="bridge1 (LAN)" interface=bridge1 network=192.168.1.0
add address=192.168.2.1/24 comment="bridge2 (Wireless)" interface=bridge2 network=192.168.2.0
add address=192.168.10.1/24 comment=host1.pve1.dc1.alwaldend.com interface=bridge1 network=192.168.10.0
add address=10.10.0.0/24 comment="tf[infra/ingress/tf]" interface=ingress-vpc network=10.10.0.0
add address=10.20.0.0/24 comment="tf[users/simeonwarren/hermes/tf_setup]" interface=hermes-vpc network=10.20.0.0
/ip dhcp-client
add comment=defconf interface=ether1 name=ether1 use-peer-dns=no
/ip dhcp-server lease
add address=192.168.1.250 client-id=1:2c:cf:67:67:b5:13 mac-address=2C:CF:67:67:B5:13 server=bridge1
add address=192.168.1.218 client-id=1:e0:be:3:2b:9a:1a mac-address=E0:BE:03:2B:9A:1A server=bridge1
add address=192.168.1.216 client-id=ff:60:8:6d:aa:0:1:0:1:31:93:31:1a:34:5a:60:8:6d:aa mac-address=34:5A:60:08:6D:AA server=bridge1
/ip dhcp-server network
add address=192.168.1.0/24 comment=defconf dns-server=192.168.1.1 gateway=192.168.1.1
add address=192.168.2.0/24 dns-server=192.168.2.1 gateway=192.168.2.1
/ip dns
set allow-remote-requests=yes servers=1.1.1.2,1.0.0.2 use-doh-server=https://odoh.cloudflare-dns.com/dns-query verify-doh-cert=yes
/ip dns static
add address=185.199.108.153 name=alwaldend.com ttl=5m type=A
add address=185.199.109.153 name=alwaldend.com ttl=5m type=A
add address=185.199.110.153 name=alwaldend.com ttl=5m type=A
add address=185.199.111.153 name=alwaldend.com ttl=5m type=A
add address=2606:50c0:8000::153 name=alwaldend.com ttl=5m type=AAAA
add address=2606:50c0:8001::153 name=alwaldend.com ttl=5m type=AAAA
add address=2606:50c0:8002::153 name=alwaldend.com ttl=5m type=AAAA
add address=2606:50c0:8003::153 name=alwaldend.com ttl=5m type=AAAA
add mx-exchange=mail.protonmail.ch mx-preference=10 name=alwaldend.com ttl=5m type=MX
add mx-exchange=mailsec.protonmail.ch mx-preference=20 name=alwaldend.com ttl=5m type=MX
add name=alwaldend.com text="_globalsign-domain-verification=0QBJgVV_uwcFLTi1Rot3bb1LyJ5uW1WD0ygvIS4OM5" ttl=5m type=TXT
add name=alwaldend.com text="protonmail-verification=bdcd133d3f472fa17f66328950d02fbeae1bef75" ttl=5m type=TXT
add name=alwaldend.com text="v=spf1 include:_spf.protonmail.ch ~all" ttl=5m type=TXT
add name=_dmarc.alwaldend.com text="v=DMARC1; p=quarantine; adkim=s" ttl=5m type=TXT
add cname=protonmail.domainkey.djgwfzcu5fgjtpoijqqomgifmqj6zeiuwdd4mzim4hrxab3zsgwkq.domains.proton.ch name=protonmail._domainkey.alwaldend.com ttl=5m type=CNAME
add cname=protonmail2.domainkey.djgwfzcu5fgjtpoijqqomgifmqj6zeiuwdd4mzim4hrxab3zsgwkq.domains.proton.ch name=protonmail2._domainkey.alwaldend.com ttl=5m type=CNAME
add cname=protonmail3.domainkey.djgwfzcu5fgjtpoijqqomgifmqj6zeiuwdd4mzim4hrxab3zsgwkq.domains.proton.ch name=protonmail3._domainkey.alwaldend.com ttl=5m type=CNAME
add address=192.168.1.222 name=bm1.dc1.alwaldend.com ttl=5m type=A
add address=192.168.1.216 name=bm2.dc1.alwaldend.com ttl=5m type=A
add address=fd2e:546d:5738:0:365a:60ff:fe08:6daa name=bm2.dc1.alwaldend.com ttl=10m type=AAAA
add address=192.168.1.218 name=bm3.dc1.alwaldend.com ttl=5m type=A
add address=fd2e:546d:5738:0:e2be:3ff:fe2b:9a1a name=bm3.dc1.alwaldend.com ttl=10m type=AAAA
add cname=bm2.dc1.alwaldend.com name=host1.pve1.dc1.alwaldend.com ttl=10m type=CNAME
add address=192.168.10.10 name=cloudinit-test.vm.pve1.dc1.alwaldend.com ttl=5m type=A
add address=192.168.1.1 name=router1.dc1.alwaldend.com ttl=5m type=A
add address=fd2e:546d:5738::1 name=router1.dc1.alwaldend.com ttl=10m type=AAAA
add address=192.168.1.254 name=switch1.dc1.alwaldend.com ttl=5m type=A
add address=192.168.1.218 name=vault.dc1.alwaldend.com ttl=5m type=A
add mx-exchange=mx1.simplelogin.co mx-preference=10 name=simplelogin.alwaldend.com ttl=3h type=MX
add mx-exchange=mx2.simplelogin.co mx-preference=20 name=simplelogin.alwaldend.com ttl=3h type=MX
add name=simplelogin.alwaldend.com text="sl-verification=bxfzzfjiggzsxyzxhhmkmjqkaskjgy" ttl=3h type=TXT
add name=simplelogin.alwaldend.com text="v=spf1 include:simplelogin.co ~all" ttl=3h type=TXT
add name=_dmarc.simplelogin.alwaldend.com text="v=DMARC1; p=quarantine; pct=100; adkim=s; aspf=s" ttl=3h type=TXT
add cname=dkim._domainkey.simplelogin.co name=dkim._domainkey.simplelogin.alwaldend.com ttl=3h type=CNAME
add cname=dkim02._domainkey.simplelogin.co name=dkim02._domainkey.simplelogin.alwaldend.com ttl=3h type=CNAME
add cname=dkim03._domainkey.simplelogin.co name=dkim03._domainkey.simplelogin.alwaldend.com ttl=3h type=CNAME
add cname=alwaldend.com name=www.alwaldend.com ttl=5m type=CNAME
add mx-exchange=mx.yandex.net mx-preference=10 name=yandex.alwaldend.com ttl=6h type=MX
add name=yandex.alwaldend.com text="v=spf1 redirect=_spf.yandex.net" ttl=5m type=TXT
add name=yandex.alwaldend.com text="yandex-verification: b83672f59b3dbe16" ttl=5m type=TXT
add name=mail._domainkey.yandex.alwaldend.com text="v=DKIM1; k=rsa; t=s; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCcYzFVgkeDOhaIIkWM8gNQjxVsv0/aXfU+ax5urB5y6hA6lSjRnjRo6tm0bXbkOJf41GmiwMNgdXpwRtzgzAlX1i2aJbtEr4b9jzibEGLQ7Cvqs44bOYES9f/K3ueQpnvdTOJmFqlRReFL7ZrUyDFCoQ7f4+7h4i8s01cCcRrt5wIDAQAB" ttl=5m type=TXT
add address=192.168.1.218 name=host1.vault.dc1.alwaldend.com ttl=5m type=A
add address=192.168.10.60 name=flux.alwaldend.com ttl=5m type=A
add address=192.168.10.60 name=host1.flux.alwaldend.com ttl=5m type=A
add cname=flux.alwaldend.com name=openid.flux.alwaldend.com ttl=10m type=CNAME
add cname=flux.alwaldend.com name=operator.flux.alwaldend.com ttl=10m type=CNAME
add address=192.168.10.40 name=forgejo.alwaldend.com ttl=5m type=A
add address=192.168.10.40 name=host1.forgejo.alwaldend.com ttl=5m type=A
add address=192.168.10.50 name=harbor.alwaldend.com ttl=5m type=A
add address=192.168.10.50 name=host1.harbor.alwaldend.com ttl=5m type=A
add address=192.168.1.216 name=pve.alwaldend.com ttl=5m type=A
add address=192.168.10.80 name=threexui.alwaldend.com ttl=5m type=A
add address=192.168.10.80 name=host1.threexui.alwaldend.com ttl=5m type=A
add address=45.142.141.133 name=njalla1.nodes.threexui.alwaldend.com ttl=5m type=A
add address=2a0a:3840:8078:141:0:2d8e:8d85:1337 name=njalla1.nodes.threexui.alwaldend.com ttl=10m type=AAAA
add address=192.168.1.218 name=vault.alwaldend.com ttl=5m type=A
add address=103.76.53.6 name=ingress.alwaldend.com ttl=5m type=A
add address=158.160.196.128 name=ingress.alwaldend.com ttl=5m type=A
add address=158.160.196.128 name=host1.ingress.alwaldend.com ttl=5m type=A
add address=103.76.53.6 name=host2.ingress.alwaldend.com ttl=5m type=A
add name=yc.threexui.alwaldend.com ns=ns1.yandexcloud.net ttl=5m type=NS
add name=yc.threexui.alwaldend.com ns=ns2.yandexcloud.net ttl=5m type=NS
add cname=host1.nodes.yc.threexui.alwaldend.com name=yc1.nodes.threexui.alwaldend.com ttl=10m type=CNAME
add address=192.168.10.100 name=runner1.forgejo-runner.alwaldend.com ttl=5m type=A
add cname=host1.yc.hermes.simeonwarren.users.alwaldend.com name=hermes.simeonwarren.users.alwaldend.com ttl=10m type=CNAME
add cname=host1.yc.hermes.simeonwarren.users.alwaldend.com name=host1.hermes.simeonwarren.users.alwaldend.com ttl=10m type=CNAME
add name=yc.hermes.simeonwarren.users.alwaldend.com ns=ns1.yandexcloud.net ttl=5m type=NS
add name=yc.hermes.simeonwarren.users.alwaldend.com ns=ns2.yandexcloud.net ttl=5m type=NS
/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid log-prefix=drop-invalid
add action=accept chain=input comment="defconf: accept ICMP" in-interface-list=accept-input-ICMP protocol=icmp
add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN log-prefix=drop-not-coming-from-lan
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid log-prefix=drop-invalid
add action=drop chain=forward comment="defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN log-prefix=drop-from-wan-not-dstnated
add action=accept chain=input in-interface-list=WAN protocol=gre
add action=accept chain=forward comment="accept forward WAN" in-interface-list=accept-forward-WAN out-interface-list=WAN
add action=accept chain=forward comment="accept forward LAN" in-interface-list=accept-forward-LAN out-interface-list=LAN
add action=accept chain=input comment="accept input DNS (udp)" dst-port=53 in-interface-list=accept-input-DNS protocol=udp
add action=accept chain=input comment="accept input DNS (tcp)" dst-port=53 in-interface-list=accept-input-DNS protocol=tcp
add action=accept chain=input comment=accept-input-NTP dst-port=123 in-interface-list=accept-input-NTP protocol=udp
add action=accept chain=input comment="accept input DHCP-server" dst-port=67 in-interface-list=accept-input-DHCP-server log-prefix=accept-DHCP protocol=udp
add action=accept chain=input comment="accept input winbox (tcp)" dst-port=8291 in-interface-list=accept-input-winbox protocol=tcp
add action=accept chain=input comment="accept input winbox (udp)" dst-port=20561 in-interface-list=accept-input-winbox protocol=udp
add action=accept chain=input comment="accept input web ui" dst-port=80,443 in-interface-list=accept-input-web-ui protocol=tcp
add action=accept chain=input comment="accept input mikrotik neighbor discovery" dst-port=5678 in-interface-list=accept-input-mikrotik-neighbor-discovery protocol=udp
add action=drop chain=forward comment="drop forward" log=yes log-prefix=drop-forward
add action=drop chain=input comment="drop input" log=yes log-prefix=drop-input
add action=accept chain=output comment=accept-output-LAN out-interface-list=LAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
/ip ipsec profile
set [ find default=yes ] dpd-interval=2m dpd-maximum-failures=5
/ip service
set www-ssl certificate=alwaldend.com_acme disabled=no
set reverse-proxy certificate=alwaldend.com_acme
set api-ssl certificate=alwaldend.com_acme
/ipv6 address
add address=::1 from-pool=dc01 interface=bridge1
add address=::1:0:0:0:1 from-pool=dc01 interface=bridge2
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" in-interface-list=accept-input-ICMP protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" dst-port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" in-interface-list=accept-input-ICMP protocol=icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="accept forward WAN" in-interface-list=accept-forward-WAN out-interface-list=WAN
add action=accept chain=forward comment="accept forward LAN" in-interface-list=accept-forward-LAN out-interface-list=LAN
add action=accept chain=input comment="accept input DNS (udp)" dst-port=53 in-interface-list=accept-input-DNS protocol=udp
add action=accept chain=input comment="accept input DNS (tcp)" dst-port=53 in-interface-list=accept-input-DNS protocol=tcp
add action=accept chain=input comment=accept-input-NTP dst-port=123 in-interface-list=accept-input-NTP protocol=udp
add action=accept chain=input comment="accept input winbox (tcp)" dst-port=8291 in-interface-list=accept-input-winbox protocol=tcp
add action=accept chain=input comment="accept input winbox (udp)" dst-port=20561 in-interface-list=accept-input-winbox protocol=udp
add action=accept chain=input comment="accept input web ui" dst-port=80,443 in-interface-list=accept-input-web-ui protocol=tcp
add action=accept chain=input comment="accept input mikrotik neighbor discovery" dst-port=5678 in-interface-list=accept-input-mikrotik-neighbor-discovery protocol=udp
add action=drop chain=forward comment="drop forward" log=yes log-prefix=drop-forward-ipv6
add action=drop chain=input comment="drop input" log=yes log-prefix=drop-input-ipv6
add action=accept chain=output comment=accept-output-LAN out-interface-list=LAN
/ipv6 nd
set [ find default=yes ] advertise-dns=yes interface=bridge1
add advertise-dns=yes interface=bridge2
/system clock
set time-zone-name=Europe/Moscow
/system identity
set name=router1.dc1.alwaldend.com
/system ntp server
set enabled=yes
/system routerboard settings
set auto-upgrade=yes
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Router2

# 2025-09-07 11:04:01 by RouterOS 7.19.4
# model = L009UiGS-2HaxD
/interface bridge
add name=bridge01
/port
set 0 name=serial0
/interface bridge port
add bridge=bridge01 interface=ether2
add bridge=bridge01 interface=ether3
add bridge=bridge01 interface=ether4
add bridge=bridge01 interface=ether5
add bridge=bridge01 interface=ether6
add bridge=bridge01 interface=ether7
add bridge=bridge01 interface=ether8
add bridge=bridge01 interface=ether1
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ipv6 settings
set accept-router-advertisements=yes
/ip dhcp-client
add interface=bridge01
/system clock
set time-zone-name=Europe/Moscow
/system identity
set name=router02.dc01.alwaldend.com
/system routerboard settings
set enter-setup-on=delete-key

11 - Nas

TrueNAS deploy

Links

Docs: https://www.truenas.com/

12 - Pve

Proxmox cluster pve.alwaldend.com

Links

Docs: https://www.proxmox.com/en/products/proxmox-virtual-environment/overview

Deployment

bazel run //infra/pve/ansible # host setup
bazel run //infra/pve/tf/tf.apply # tf setup

Recreate the test VM

bazel run //infra/pve/tf:tf.apply -- --replace module.vm_cloudinit_test.proxmox_vm_qemu.vm

Cloud-init snippet update

bazel run //infra/pve/ansible -- --tags pve_snippets

Update ACME account

Create an EAB:

bazel run //infra/dc1/vault -- write -f pki/ica_servers/roles/src_infra_dc1_pve1_pki_server/acme/new-eab

Login as root@pam
Go to Datacenter -> ACME

Create a new account with the EAB and a directory:

https://vault.dc1.alwaldend.com:8200/v1/pki/ica_servers/roles/src_infra_dc1_pve1_pki_server/acme/directory

Pve token

Create a token with Privilege Separation
Grant it required roles

Create json:

{
  "token_id": "",
  "token_secret": ""
}

Write the data:

bazel run //infra/pve:vault.kv_put alwaldend.com/vault1/approles/src_infra_dc1_pve1/pve_token @"${PWD}/data.json"

12.1 - Ansible

Ansible setup

12.2 - Tf

Terraform config

13 - threexui

3x-ui

Links

Site: https://docs.sanaei.dev/
Releases: https://github.com/MHSanaei/3x-ui/releases

Fetch and fix subs for a particular subscription id

bazel run //infra/threexui:fix_subs -- --hosts njalla1.nodes.threexui.alwaldend.com,yc1.nodes.threexui.alwaldend.com --sub_id subid

Fix subs from a local file

bazel run //infra/threexui:fix_subs -- --sub_file path_to_file

13.1 - Ansible

Ansible setup

13.2 - sub

Fetch subscriptions and fix them

13.3 - Tf

Terraform config

13.4 - Tf setup

Terraform setup

14 - Vault

Setup for vault.dc1.alwaldend.com

Links

Intermediate CA: https://developer.hashicorp.com/vault/tutorials/pki/pki-engine-external-ca
ACME: https://developer.hashicorp.com/vault/docs/secrets/pki/acme

Deployment

bazel run //infra/vault/tf_setup:tf.apply # Create VMs (requires an active Vault host)
bazel run //infra/vault/ansible # Set up hosts (BM and VMs)
bazel run //infra/vault/tf:tf.apply # Configure vault

Backup

bazel run //infra/vault:backup

Unseal

With a working Vault:

bazel run //infra/vault:unseal

Without a working Vault:

bazel run //infra/vault:unseal_standalone

Fix quorum

bazel run //infra/vault/ansible:fix_quorum

Set up only VMs

bazel run //infra/vault/ansible:ansible.vm # Set up only VMs

Set up only bare metal

bazel run //infra/vault/ansible:ansible.bm # Set up only bare metal

Tf

Plan:

bazel run //infra/vault/tf:tf.plan

Apply:

bazel run //infra/vault/tf:tf.apply

Run terraform directly:

bazel run //infra/vault/tf:tf.direct -- -chdir="${PWD}" plan

Replace VMs

bazel run //infra/vault/tf:tf.apply -- -replace 'module.vm_ha["host2"].proxmox_vm_qemu.vm' -replace 'module.vm_ha["host3"].proxmox_vm_qemu.vm

Generate and import a user cilent certificate

username="username"
bazel run //infra/vault:gen_client_cert -- --user "${username}" --output_dir "${PWD}"
bazel run //tools/ykman -- piv certificates import 9A "${PWD}/${username}.pfx"
bazel run //tools/ykman -- piv keys import 9A "${PWD}/${username}.pfx"

Generate a host cilent certificate

bazel run //infra/vault:gen_client_cert -- --host some-host --output_dir "${HOME}/.al/client_cert"

Generate a user device cilent certificate

bazel run //infra/vault:gen_client_cert -- --host some-host --user username --output_dir "${HOME}/.al/client_cert"

Unseal

Prepare encrypted unseal token
Run and input the encrypted token:
```
bazel run //infra/vault:unseal
```

Root token

Prepare encrypted unseal token

Run and input the encrypted token:

bazel run //infra/vault:gen_root_token -- --pgp_key path_to_public_gpg_key_in_base64

Generate an EAB for ACME

bazel run //infra/vault -- write -f pki/ica_servers/roles/ica_servers_dc1_pve1/acme/new-eab

Sign a client ssh key

bazel run //:vault -- write ssh/clients/sign/admins ttl=30000000  public_key=@"${HOME}/.ssh/key"

Revoke all tokens

bazel run //infra/vault -- token revoke -mode=path auth

Vault certificates

Vault certificates (tls_cert_file, tls_key_file) should be updated manually

Read OIDC client info

bazel run //infra/vault -- read identity/oidc/client/src_infra_dc1_forgejo1_provider

Read all entity aliases

 bazel run //infra/vault -- list -format json identity/entity-alias/id | jq ".[]" | xargs "-I{}" bazel run //infra/vault -- read "identity/entity-alias/id/{}"

14.1 - Ansible

Ansible setup

14.2 - Tf

Terraform config

14.2.1 - Output

Output files

14.3 - Tf setup

Terraform setup

15 - XCP-ng

XCP-ng infrastructure

The dc1 DNS configuration maps xcp-ng.alwaldend.com and host1.xcp-ng.alwaldend.com to 192.168.1.213.

Terraform lives in internal/tf and declares the pinned vatesfr/xenorchestra provider. It includes empty provider and HTTP backend blocks, with no resources, endpoints, or credentials. The provider requires Xen Orchestra connected to the XCP-ng pool when resources are added.

al.lua authenticates with the src_infra_xcp_ng Vault AppRole and configures the HTTP backend using secrets/alwaldend.com/vault1/approles/src_infra_xcp_ng/tf_backend. The AppRole is declared in infra/vault/tf and must be provisioned before running the Terraform commands. Provider connection settings are left to the operator.

Check Terraform formatting with:

bazel_agent bazel test //infra/xcp_ng/internal/tf:tf_tests.fmt_test

The formatting test does not authenticate to Vault.

16 - Yandex Cloud

Yandex Cloud (yandex.cloud)

Links

Site: https://yandex.cloud

16.1 - Org1

Organization 1

Links

Site: https://yandex.cloud

16.1.1 - Tf

Terraform config

Infra

Requirements

New project

1 - Architecture

DC1

Vault

Flux

Forgejo

www

Threexui

DNS

Ingress

T3code

Truenas

Archive/Harvester

Archive/Opencode

Archive/Hermes

Archive/Proxmox

Archive/Harbor

Edit and regenerate

2 - Ceph

Links

See current crush map

Crush map update

3 - Dns

Links

Deployment

dc1 BIND

global BIND

3.1 - Tf

4 - flux

Links

Set up port forwarding

Run flux CLI

Show flux status

Run bootstrap

Run flux operator

Show oidc info

Renew certificate

Update secrets

4.1 - Ansible

4.2 - cl

Links

4.3 - Tf setup

5 - forgejo

Links

Deployment

5.1 - Ansible

5.2 - Tf

5.3 - Tf setup

6 - Forgejo runner

Deploy VMs

6.1 - Ansible

6.2 - Terraform setup

7 - GitHub

7.1 - GitHub Terraform

8 - harbor

Links

CE Login

Update secrets

8.1 - Ansible

8.2 - cl

8.3 - Tf

8.4 - Tf setup

9 - ingress

Run ansible

Apply terraform

Update signed image url

Recreate VMs

Regenerate wireguard keys

9.1 - Ansible

9.2 - Tf

10 - Mikrotik

Deployment

Links

Exports

Router1

Router2

11 - Nas

Links