Fri 17 July 2026

I still run nomad at home. This surprises some people, who assume that anyone with opinions about production has long since capitulated to kubernetes, but the whole point of the lab is that it's mine and it's small and I get to pick. Nomad is still the right amount of machinery (probably even too much at that) for the number of jobs I actually run, and I've written about various bits of it (secrets, DNS, the works-on-my-cluster tag generally) before.

The itch I've been meaning to scratch for ages is the deploy loop. For a long time my "gitops" was me, on the couch, running a shell script that ran nomad job run against a pile of hcl files. That works right up until you forget which files you changed, or you run the wrong one, or you're away from the box and the state of the cluster is whatever you last remembered typing. The whole appeal of gitops is that git is the thing that's true, and the cluster's job is to catch up to git. I wanted that, without dragging in a great big platform to get it.

So there's now nomad-gitops. It watches a repo, and when the repo changes, the cluster changes. That's more or less the entire pitch.

A few things I decided up front and stuck to:

• Git as the source of truth. If a job exists in the cluster but not in the repo, that's drift, and drift is a bug. I didn't want a tool that only ever adds things and leaves a graveyard of jobs nobody remembers deploying.
• Boring and legible over clever. I want to be able to read the logs and know exactly why it did a thing. No magic, no reconciliation loop I can't explain to myself six months from now when it's misbehaving at an inconvenient hour.
• It should be fine to turn off. If nomad-gitops falls over, the cluster keeps running whatever it was running. Nothing depends on it being up to stay up. This is the same rule I apply to everything in the lab, because the lab is not allowed to become a second job.

None of this is novel. Every gitops tool worth anything does versions of this. But writing it down for my particular small case, and deciding what I was not going to do, was most of the work.

The code was mostly written by Claude, and I want to be honest about the shape of that, because it's not "I typed a sentence and got a repo". I did a previous thing with claude on more of a 'yolo mode', but this time around I started with the philosophy, almost.

Before any code existed, I wrote the documents. A design doc for what reconciliation actually meant here, a conventions doc for how the repo should be laid out, and a fairly opinionated spec for behaviour in the annoying cases: what happens on a partial push, what counts as drift, what it should refuse to do. Those documents were the first-order artifacts, they're in the docs directory of the repo. I founf it useful to ask Claude to refine them by asking me questions. My brin works a little better with pointed questions sometimes, rather than wide-open updates and thinking.

I've found when you hand a model a vague idea it will cheerfully produce something plausible that isn't quite what you wanted, and you spend the afternoon arguing with it. When you hand it a doc that already says "drift is a bug and here's how we treat it", or "new persistent state because that is faff and we don't need it", it mostly builds that thing, and the arguments happen at the level of the doc, where they're cheap. The documents are the part I actually care about.

Aside: I found that Claude is not amazing at structuring docs in the way I like to read them, so the usage docs aren't very understandable. It's all there, though.

As with CringeSweeper, I read all of it, it's unremarkable Go, and it's been quietly doing its job in the lab for a while now. Have a poke at the repo if any of this sounds useful. The docs are probably more interesting than the code.

Category: Tech Tagged: log #worksonmycluster tech