kalexlab - GitOps Homelab

Overview

This repository keeps all the Infrastructure as Code (IaC) for my homelab, composed of:

Host	Hardware	IP	Role
zeus	HP EliteDesk 800 G3M, i7-6700T, 16G DDR4	192.168.20.4	Control plane (schedulable)
hermes	Dell Optiplex 3050M, i5-6600T, 16G DDR4	192.168.20.3	Worker
athena	Raspberry Pi 4B, 4G	192.168.20.2	Worker, VPN egress gateway
-	TP-Link Archer AX23 (OpenWRT)	192.168.10.1	Router, WireGuard remote access

Nodes live on a dedicated VLAN (192.168.20.0/24, router 192.168.20.1). The Kubernetes API is exposed on the control plane VIP 192.168.20.254. LoadBalancer services get IPs from 10.20.0.16/28, advertised via BGP (Cilium, ASN 64512) to the router (bird2, ASN 64513).

Cluster nodes run Talos, whose configuration is managed with topf. Secrets are encrypted with SOPS using an AGE key.

Once the cluster is up and running, the app-of-apps manifest installs all the ArgoCD applications, finalizing the cluster setup:

kubectl apply -f ./bootstrap/app-of-apps.yaml

Note

TODO: document the cluster bootstrap procedure. After the migration from talhelper to topf, the cluster bootstrap playbook (which installs Cilium, CoreDNS and ArgoCD) still references the old talos/ and helm/ folders and needs to be reworked.

Folder structure

• ansible: everything related to Ansible
  • playbooks/openwrt: sets up the router with the WireGuard VPN (remote access to the LAN, including cluster applications) and the egress VPN (see Cluster VPN)
  • playbooks/cluster: cluster bootstrap playbook (outdated, see the note above)
  • roles/wireguard-mesh: role to set up a WireGuard mesh
• apps: ArgoCD Application manifests, one per application, grouped by category (apps, management, misc, monitoring). Each one points to the matching kubernetes/<category>/<app> folder
• bootstrap: a single resource that bootstraps all the cluster applications with the app-of-apps ArgoCD pattern, recursing into apps
• kubernetes: the Kustomize bases for all the applications deployed on the cluster
• topf: the Talos configuration
  • topf.yaml: cluster definition (Talos/Kubernetes versions, nodes, per-node data such as disks, labels and taints)
  • all, control-plane, worker, node: config patches applied to all nodes, by role, or to specific nodes
  • schematic.yaml.tpl: the Talos Image Factory schematic

Application layout

Every application under kubernetes follows the same structure:

kubernetes/<category>/<app>/
├── kustomization.yaml   # entrypoint, optionally including Helm charts
├── manifests/           # plain manifests (ingresses, secrets, extra resources, ...)
├── patches/             # patches to kustomization-rendered manifests, if any
└── values/              # Helm values, if any Helm chart is used

Helm charts are rendered by Kustomize through the helmCharts field (ArgoCD runs with kustomize.buildOptions: --enable-helm), and are cached locally in .charts. For example, longhorn:

helmGlobals:
  chartHome: ../../../.charts

helmCharts:
  - name: longhorn
    repo: https://charts.longhorn.io
    version: 1.13.0
    releaseName: longhorn
    namespace: longhorn-system
    valuesFile: values/longhorn.yaml

resources:
  - manifests

Secrets are stored next to the other manifests as *.sops.yaml files, encrypted with SOPS (see .sops.yaml).

Tooling

Common tasks are available as just recipes in the justfile:

Recipe	Description
just init	Install the pre-commit hooks
just topf <args>	Run topf from the topf folder
just topf-talosconfig	Generate the talosconfig file
just tal <args>	Run talosctl with the generated talosconfig
just sops <file>	Toggle encryption of a secret file
just encrypt-all	Ensure all *.sops.yaml files are encrypted
just kust <path> [-a]	Render a Kustomize folder (with Helm), and optionally apply it
just debug-pod <ns>	Start a throwaway pod with DNS/network tools
just pvc-pod <ns> <pvc>	Start a pod mounting the given PVC on /mnt

pre-commit hooks run yamllint, enforce Conventional Commits, and prevent committing unencrypted secrets. Dependencies (container images, Helm charts, Talos/Kubernetes versions) are kept up to date by Renovate.

Cluster VPN

For increased privacy, it is possible to ensure selected pods route their egress traffic through a VPN.

In order for a pod to be routed through the VPN, it needs to have the following label: egress.kalexlab.xyz/policy: vpn.

The VPN is implemented as following:

• Cilium's EgressGateway, configured in egress-gateway.yaml, redirects the egress traffic from pods with the above label through the node labeled egress.kalexlab.xyz/gateway: vpn (athena, see topf.yaml). The node is also tainted with kalexlab.xyz/taint=vpn:NoSchedule, so that regular workloads are not scheduled on it
• On the router, an WireGuard interface is configured
• Finally, still on the router, Policy-based Routing is applied to the athena node's IP. The rule routes all the non-local traffic through the VPN interface