Back in January, in First thoughts, I wrote that I was setting up a proper server in the basement, and that the endeavour “shall partly be documented here”. Then I built it, and said nothing for seven months. Not because it went badly — because the interesting part only started once the case was closed.
Let me get the shopping out of the way first, since it is the part people ask about and the part that mattered least.
I did not build it from parts. I found a used machine whose specifications fit my plan so precisely that it felt slightly like cheating: fourteen cores, 64 GB of memory, room for the disks I wanted. It ended up with five data drives, 80 TB total with one of them as parity, and two NVMe drives mirrored as a cache. A small UPS sits above, idling at about seven percent load, which is enough to let the whole thing shut down politely instead of falling over.
There was a graphics card in it. I sold it, for two reasons that both seemed excellent at the time: it would have drawn power around the clock for a workload I did not have, and selling it made the already excellent used price even better. Both of those are still true. Both of those are also why I do not have a graphics card now, at a moment when the used price has climbed considerably and when running a language model on my own hardware has stopped being a novelty and started being something I would actually use. I was right, and yet I regret it. Oh well.
The rest of the money went into rack accessories. Shelves, compartments, a bag of very specific screws for a mounting standard I have used precisely never. The case is comically oversized for what is inside it. I like it anyway: there is room for networking equipment I do not own yet, and the emptiness looks less like waste than like an invitation. At least that’s what I’m telling myself.
Why the basement, then. Five spinning disks and a parity check are not a living-room sound, the heat has to go somewhere, and there was simply space down there. The third reason is the one I am quietly, immensely pleased with: I had an ethernet line pulled from the fibre socket on the first floor straight down to the machine, so the server sits at the front of the house network rather than dangling off the end of it. This is a rented flat, so I asked the management first and paid for it myself. No regrets. It hums away in a little storage room, out of everyone’s way, and it makes me happy every time I walk past. Except when I think about what would happen to my little machine in case of a flood. Which is not likely, I’m told, but in this climate? Who knows.
So much for the hardware. Here is the part that took the months.
The container count went from 28 at the first inventory in June to 38 by early August. The growth was fine; each one turned up for a reason. What was not fine was the answer to a question I only thought to ask much later: if this machine lost its Docker setup tomorrow, how much of it comes back on its own?
Twenty-seven of them come back, because they were installed from templates and the system remembers those. Eleven do not. Nine are compose projects that exist only as files in a folder — nothing in the interface knows they are there. And one is a bridge network I created by hand (read: i pasted some code generated by AI), once, and never thought about again. Without it the database layer refuses to start at all, and the error message tells you nothing whatsoever about why. So I prompted AI, and begged for redundancy and fail-safes and backup-scripts. AI delivered (I think), and while I can’t code for the life of me I’m knowledgeable enough to ask the (hopefully) right questions: “What is a state of the art backup protocol?” “How can I make sure a restore work?” “How to setup a system so I’m alerted in case a backup fails silently?”
I did not learn this from an outage. I learned it on many ordinary evenings, because I sat down to find out how recovery works and discovered there’s a lot I didn’t know. An uncomfortable bit: gaps have been there for months. They probably still are, as are matters of OpSec (stuff for another article!). Live and learn. Script and test. Copy and paste.
The backups told the same kind of story. I had set retention to seven days, which sounds like a week of safety net, and the job runs weekly. Put those two numbers next to each other and every backup falls out of the window before the next one is written, so what is actually kept is a minimum of two snapshots. Not seven days — two copies. I had chosen that number myself and simply never followed it through to the end of its own sentence. Logic never was my strong suit, thank god for smarter people than me and AI.
I’ve messed up configs, pasted scripts that weren’t tested, deleted stuff I didn’t want to delete. So far, restores were painless. But there’s always a silent voice in my head: Have I forgotten anything? I tend to create convoluted setups that backup to multiple locations, sometimes even vice versa. Syncthing, Restic, UrBackup, Snapshots and Unraid-Scripts. The backups must keep growing! Fingers crossed.