Three nodes, one apply
OpenTofu builds the guests, cloud-init builds the Kubernetes cluster, and Argo CD keeps it honest. Failure modes that only show up when you own the hypervisor.
- homelab
- kubernetes
- opentofu
- platform
The homelab has been through a few shapes. A hypervisor with hand-built VMs. A
Docker host with compose files in a directory called docker. A Kubernetes
cluster that existed mostly so I could learn how it failed.
This is the current one: three nodes, one OpenTofu configuration, and a rule that anything not in the repository is either a credential or a mistake.
The goal was not to build a data center. It was to make the homelab reproducible enough that I could tear it down on a Saturday and have it back by dinner. It was also to run the pattern I would use for a small production platform: infrastructure in OpenTofu, workloads in git, one way in.
Addresses and hostnames in this post are placeholders. The shape is real; the numbers are not.
The shape
One Proxmox host. Three VMs, four vCPU and eight gigabytes each, every one a combined k3s control-plane and worker with embedded etcd. A floating virtual IP (VIP) in front of the API server. Longhorn on a dedicated disk per node. Traefik, bundled with k3s. A tunnel connector. Argo CD.
Two things are deliberately missing: an external load balancer and a certificate manager. Cloudflare terminates public TLS, and the tunnel connects outbound. Nothing in the cluster listens on a public address, and every hop after Cloudflare — tunnel, Traefik, pods — is plain HTTP on a private network. That removes a category of things to get wrong.
The VIP matters more than it sounds. Three etcd members means one node can be
down and the cluster still has quorum. It also means the API endpoint should not
move when that happens. kube-vip holds one address on the control-plane nodes, so
kubectl has one endpoint that survives a node going away.
OpenTofu is the hypervisor's API
The root module creates the Proxmox guests, the Cloudflare DNS records and the
tunnel's ingress rules. Three providers: bpg/proxmox, cloudflare, and
random for the k3s token.
State is local and gitignored. For a single operator on one hypervisor, a local
state file and a backup is honest. It is also a credential: the state knows the
k3s join token and every resource attribute, and terraform.tfvars holds the
Proxmox and Cloudflare API tokens. Only the example file is committed.
The state file being local is a choice. The lock file being committed should not be.
# .gitignore
*.tfstate
*.tfvars
!*.tfvars.example
.terraform.lock.hcl stays. It records the exact provider versions and their
checksums, which is what makes a plan reproducible. It used to be gitignored. The
constraint in providers.tf was ~> 0.66, which permits anything below 1.0, so
tofu init re-resolved it on every machine. One day a configuration written for
bpg/proxmox 0.66 was running against 0.113.1 — roughly forty-seven minor
versions later — with no record of when that changed.
Nothing broke that day. That is the problem with drift: it does not announce
itself. Now the constraint is still a range, the lock file is committed, and
tofu init without -upgrade is the rule.
Guests are config files
Every Kubernetes node is a clone of one Debian cloud-init template. The clone gets two disks — OS and Longhorn data — one network interface on the guest VLAN, an address, and a user-data snippet.
The snippet is uploaded to Proxmox as a file, over SSH rather than the API. That
means two things fail in the middle of an apply if they are not ready: the
Proxmox host needs the key, and the datastore needs Snippets enabled as a
content type. Both are in the preflight script, because both are much cheaper to
catch before tofu apply starts creating guests.
The VM resource is ordinary except for two lines that are load-bearing.
resource "proxmox_virtual_environment_vm" "k8s" {
for_each = var.k8s_nodes
name = each.key
node_name = var.proxmox_node
# Never let an apply power-cycle the cluster.
reboot_after_update = false
clone {
vm_id = var.vm_template_id
full = true
}
initialization {
user_data_file_id = proxmox_virtual_environment_file.k8s_cloud_init[each.key].id
}
}
reboot_after_update = false is there because a change to cores or memory does
not take effect until QEMU restarts, and the provider's default is to do that
itself. The nodes are a single for_each, so a resize would restart all three at
once. Three-node embedded etcd means losing quorum; Longhorn with two replicas
means losing both copies of every volume. The new values still land in the VM
config — the provider just warns that a reboot is outstanding. The reboot is then
a separate, deliberate, one-node-at-a-time operation.
The ignore_changes on user_data_file_id and clone is the other half.
Cloud-init runs once. The snippet ID is a ForceNew input. Changing the
snippet — even because the SSH public key on a rebuilt workstation changed — made
the plan want to destroy and recreate all three nodes. It would have delivered a
key that would never be installed on a guest that had already booted. The snippet
still updates for future nodes; the running ones are left alone.
The clone block needs the same treatment for a different reason. A VM imported
from an existing Proxmox guest has no clone block in state; without
ignore_changes, every plan after an import proposes replacing the whole cluster.
Adopting an existing guest is never a no-op. I learned that one the week it
rebooted seven VMs, including the database behind the forge.
The cloud-init user-data is short, because most of the work is in scripts it drops on disk and runs once.
#cloud-config
packages:
- open-iscsi
- nfs-common
- qemu-guest-agent
runcmd:
- /usr/local/sbin/k8s-longhorn-disk.sh
- /usr/local/sbin/k8s-install.sh
qemu-guest-agent is not optional. With agent { enabled = true } and no agent
in the guest, the Proxmox provider asks for the VM's network interfaces, gets a
500 QEMU guest agent is not running, and retries every four seconds forever.
The plan does not fail. It hangs, which is a worse user experience and a much
worse error message. The Debian cloud images do not ship the agent, so
cloud-init installs it before anything asks.
The Longhorn disk script is deliberately idempotent. It looks for a disk with the right label, falls back to the first whole disk with no partitions and no filesystem, formats it, and mounts it at Longhorn's default data path. cloud-init is not the only thing that will ever run it.
The cluster builds itself
k3s is installed by the same cloud-init run. The first node in sorted order
starts the etcd cluster with cluster-init; the other two join it. The API
certificate includes the VIP, every node address and the internal names, so the
VIP is valid from the first connection.
The joiners wait. A single tofu apply creates all three VMs at once, and without
a wait the second and third nodes race the first node's API server and the
installer fails. The install script polls the bootstrap node's /livez endpoint
for up to fifteen minutes before it joins.
# -k because the bootstrap API's certificate is not trusted by this node yet.
for _ in $(seq 1 180); do
curl -sk --max-time 3 "https://${bootstrap}:6443/livez" >/dev/null 2>&1 && break
sleep 5
done
kube-vip is a DaemonSet on the control-plane nodes. The kubeconfig is pulled off
the first node and rewritten to point at the VIP — otherwise it points at
127.0.0.1 and only works on the node it came from.
One detail that cost an hour: kube-vip concatenates the address and the subnet
verbatim to build a CIDR. The subnet has to include the slash. /32, not 32.
The failure mode is invalid CIDR address, and the API VIP never comes up.
Storage that can lose a node
Each node has a second disk, formatted and mounted at Longhorn's data path by cloud-init. The Helm values do not need per-node disk configuration; they mostly need restraint.
defaultSettings:
defaultReplicaCount: 2
replicaReplenishmentWaitInterval: 600
persistence:
defaultClass: false
Two replicas on three nodes means one node can be down and every volume still has a copy. It also means a rebooted node's replicas take up to ten minutes to rebuild. Roll a second node before the first is back and you can run out of copies. Longhorn will salvage a volume whose replicas all failed — it is a recovery path, not a normal one. The roll script refuses to touch more than one node at a time for exactly this reason.
The default StorageClass is disabled. k3s already ships local-path, and two
defaults is a coin flip you do not want in the middle of an incident. Volumes ask
for storageClassName: longhorn explicitly.
The only way in
There is no port forward and no public address on the cluster. cloudflared
makes an outbound connection to Cloudflare; Cloudflare terminates TLS and sends plain
HTTP back through the tunnel to Traefik. Two tunnel replicas, spread across nodes,
with a PodDisruptionBudget, so losing a node does not take ingress with it.
The tunnel's ingress list is managed by OpenTofu, and that is a sharp edge. The resource owns the entire ordered list. It does not merge. Removing a hostname takes it offline and deletes its DNS record with it. Before editing the list, I re-run the discovery script and diff it against the live configuration — because the alternative is finding out from a 404.
Cloudflare Access sits in front of the admin surfaces: the Argo CD UI, the secrets manager, anything with a login. It is free, it is a dashboard setting, and it is the difference between a password being the only thing in front of cluster administration and a password plus an identity provider.
The cluster follows git
Exactly one object is applied by hand: the root Argo CD Application. It points at
kubernetes/argocd/apps/, and every Application in that directory — plus anything
added later — is reconciled from git. Adding a service is a directory and an
Application file. Deleting it prunes the namespace, which is why stateful things
get a Retain reclaim policy before anyone touches the Application.
The repository is private, and Argo CD discovers credentials by label, not by
name. type=git is a data key. The label is
argocd.argoproj.io/secret-type=repository. Without it the Secret is inert, the
repository is treated as anonymous, and the error says authentication required
even when the key inside is perfectly valid. That one cost an evening.
Images are built by CI on the forge and pushed to a private registry. Argo CD Image Updater
watches for immutable sha-<short> tags and commits the tag bump
back to the repository. The read-only deploy key Argo uses cannot push, so the
updater gets its own write credential — and it infers the credential type from the
repository URL, so the write-back URL has to be the SSH form. A read key and a
write key for the same repository is not redundant; it is the difference between
a bot that can commit and one that can only read.
The result is the loop I wanted: a merge reaches the cluster without anyone bumping a tag by hand, and git remains the record of what is running. The site you are reading is one of those applications; how it gets from a merge to a pod is A working app is not a working deployment.
What stays out of git
terraform.tfvars, the state file, the kubeconfig, and every credential that
bootstraps the credentials. The kubeconfig is cluster-admin. The tunnel token is
in a Kubernetes Secret, created out of band. The Cloudflare API token that manages
DNS and tunnel routes is not the tunnel token: one can edit the tunnel
configuration, the other can run the tunnel, and neither can do the other's job.
Application secrets live in a self-hosted secrets manager, with its database on a separate VM rather than a Longhorn volume. That is deliberate. The source of truth for secrets should not depend on the storage system whose credentials it might be holding. Its encryption key is the one thing in the stack that is unrecoverable if lost; a backup of the database is useless without it. It is backed up somewhere that is neither the cluster nor the workstation.
The manual steps are: the Proxmox API token, the tunnel token, the Argo CD repository key, the secrets manager bootstrap, and the image updater's write key. Five things, each written down.
The goal was never zero manual steps. It was that every manual step is written down once, and has a failure mode you can recognize.
If you are building one
- Commit the lock file and pin the provider. A plan that resolves differently on a different day is not a plan.
- Turn off automatic reboots for anything in a cluster. Do the reboot yourself, one node at a time.
- Treat cloud-init as one-shot.
ignore_changeson the user-data ID and the clone block, or the next SSH key rotation is a cluster rebuild. - Install the guest agent before the provider asks for it. A hung plan is harder to read than an error.
- Give the tunnel list one owner. Whole-list resources are not merge-friendly.
- Put Access in front of the admin UIs. It is the cheapest control in the stack.
- Keep state and secrets out of git, and write down where they live. The cluster is reproducible; the credentials are the part you cannot re-derive.
The cluster is not the interesting part. The interesting part is that the cluster is a repository, and the repository is the thing I can read, review and revert. Everything else — the VMs, the VIP, the volumes, the tunnel, the applications — is a consequence. When it breaks, I read the plan.