A working app is not a working deployment

The tests passed. The pages rendered. The CSS compiled. The site still was not live.

I rebuilt blakeridgway.com as a small Rails application. The content is markdown in git, parsed at boot and served from memory. There is no database in the read path. That part was done before I started on the deployment, and it worked on the first try.

The deployment took longer.

What I built

The application is deliberately small. It runs Rails 8.1 with Puma and Thruster, serves precompiled assets, and reads posts and pages from files baked into the image. Publishing is a commit. Rolling back a bad post is a git revert.

The runtime is less small. I run a three-node k3s cluster at home, with Argo CD for the application layer, Traefik for ingress, and a Cloudflare Tunnel for public traffic. Images are built by Forgejo Actions and published to a private registry on the same forge. The deployment lives in homelab-infra, one directory and one Argo CD Application, with an Image Updater watching immutable sha-<short> tags.

I chose two replicas and no persistent volume. The content is in the image. The only SQLite databases are Solid Cache, Solid Queue and Solid Cable, and there are no jobs and no user data in them. The entrypoint recreates them on every boot. That makes a normal rolling update safe and removes a Longhorn volume from the picture.

That was the design. The first deploy still found three things the design did not cover.

The first red build

The Forgejo workflow runs bin/ci before it builds an image. Setup, RuboCop, Brakeman, the test suite, the seed check and bundler-audit. I like that gate. It means a broken dependency is a failed pipeline rather than a surprise in production.

It also means a CVE published overnight can stop a deploy that has nothing to do with the change. That is what happened. The advisory database had new entries for Puma 8.0.2 and msgpack 1.8.5. I bumped them to 8.1.1 and 1.8.7, ran the suite again, and pushed the lockfile.

The interesting part was not the bump. It was the reminder that the build stage only runs after the test stage, so a red audit means there is no image at all. I had been thinking about the application. The pipeline was thinking about the artifact.

The image that did not exist

The Argo CD Application synced. The namespace, Deployment, Service and Ingress appeared. The pods did not start.

waypoint-...   0/1   ImagePullBackOff
failed to resolve reference ".../waypoint:sha-2384a24": not found

Argo was doing exactly what I asked. It was asking the registry for a tag that had never been published. The registry had no waypoint package at all.

The pod status is the wrong place to start debugging that. Kubernetes can only tell you what it tried. The registry tells you whether there is anything to run. Once the image existed, a rollout restart pulled it and both pods were healthy. No code changed. The deployment had been pointed at an artifact that did not exist yet.

The order matters. Build and publish the image, confirm the tag, then let the Application sync. An Image Updater will fix a stale tag, but it cannot invent a package.

DNS is state, not a text box

The domain had already been moved to Cloudflare, but the records were still pointing at the old host. The apex and www answered with a 301 to blakerides.cc. A nameserver change is not a cutover.

The old A records were real records with real IDs. Creating a tunnel CNAME beside an A record of the same name does not work. I imported both records into the Terraform state, changed them to CNAMEs pointing at the tunnel, and added the two hostnames to the tunnel's ingress list. The MX and SPF records for Spacemail were left alone. The domain still needs to receive mail after the website moves.

There are two redirects left on purpose. www goes to the apex, and plain HTTP goes to HTTPS. Both preserve the path and query string. The app sees HTTPS because the request reaches it through Cloudflare, Traefik and finally an HTTP hop to Puma, so assume_ssl is on. force_ssl is on too, with /up excluded from the redirect so the kubelet probe stays a plain 200.

Without assume_ssl, Rails generates http:// canonical and Open Graph URLs even though the browser is on HTTPS. That is the kind of bug that looks like an SEO problem until you notice the request never left the proxy.

The cache remembers

After the cutover, the homepage was a 200. The favicon was a 301 to the old domain.

The response had cf-cache-status: HIT and an Age of 904 seconds. Cloudflare had cached the old origin's redirect. The new server was serving a valid icon; the edge never asked it. A purge fixed it, and the same URL returned 200 with image/vnd.microsoft.icon.

This is the part that is easy to miss because DNS looks finished. A record can point at the new tunnel while the edge still serves yesterday's answer. The cutover checklist needs a cache check, not just a dig.

The boring choice

The stateless deployment is the part I would defend.

A personal site does not need a database in the read path. It does not need a persistent volume, a migration window or a Recreate strategy. The content is a set of files. The cache can be per pod. If a pod restarts, it rebuilds a cache and continues.

The tradeoff is explicit. Solid Cache is not shared between the two replicas, so a cache hit on one pod is a miss on the other. There is no shared queue and no shared cable database. If I add comments, search or real background jobs, that changes. At that point the site needs a real database and a different deployment shape, and I should make that change on purpose rather than discover it during an incident.

For now the application is a process, a health check and some files. That is the right amount of infrastructure for a site that mostly publishes text.

What I would do differently

I would check the registry tag before merging the Argo CD Application. One curl to the registry's tag list is cheaper than fifteen minutes of ImagePullBackOff.

I would import the DNS records before writing the tunnel config. The import is not optional bookkeeping. It is the difference between a clean apply and a collision.

I would purge the cache as part of the cutover, not after someone notices a favicon pointing at the old domain.

And I would test canonical URLs through the public hostname, not on localhost. The application was correct locally. The public edge is where the scheme, the host and the cache actually meet.

The last mile

The application was the easy part. The deployment is the product.

That is true for a personal site, and it is more true for a regulated SaaS platform. The same last mile shows up as an image that never reached the registry, a DNS record that outlived its host, a cache that served a redirect from the previous owner, or a health check that was green because nobody checked what the probe actually did.

I run Arcline, where this is the work: reliability, cost and audit readiness for teams that answer to an auditor. The scale is different. The failure modes are not. A working application is a good start. It is not the same thing as a working deployment.

The site is live now. The canonical is HTTPS, the favicon is a 200, and the only 301s left are the ones I meant to keep.