Catching Up: A Server Outage, a Home Assistant Overhaul, and Six Weeks of Homelab Work

It’s been a while since the last one of these, and there’s a good reason: the homelab spent about a week completely dark. A cooling fan failed on the main server, thermals climbed, and the box took itself offline to avoid frying anything. What followed was equal parts firefighting and a hard lesson in how resilient — and how fragile — a self-hosted setup really is. So instead of a normal weekly recap, this is a catch-up covering the outage, the recovery, and the pile of infrastructure work that came out of it.

The Outage

The failure itself was mundane: one fan, dead. But a single fan on the wrong part of the chassis is enough to make an entire rack of services vanish. Everything that lived on that machine went with it — the media stack, the dashboards, the automation brain, the databases.

The saving grace was that not all of my work is anchored to one box. A second machine picked up the day-to-day while the server was down, so momentum didn’t stop even though the primary was cold. When the hardware came back online, the real job began: reconciling a week of work that had happened somewhere else back onto the machine that’s supposed to be the source of truth. Git repositories, config files, and notes all had to be merged carefully, with a bias toward “never lose anything” over “move fast.” It’s tedious, and it’s exactly the kind of work you’re grateful you set up the tooling for beforehand.

The bigger takeaway came when I looked at where my backups were landing: all of them, on the same storage array that had just been at risk. Backups that only exist on the machine you’re backing up are not backups — they’re a copy that dies with the original. That’s now near the top of the fix list.

A 160-Finding Home Assistant Audit

Once the server was stable, I turned a hard eye on the smart home, because a rushed recovery tends to leave debris. I ran a full audit of the Home Assistant setup, and it surfaced 160 issues — everything from cosmetic dashboard rot to a couple of genuinely alarming ones.

Three root causes stood out. First, the post-outage restart never fully finished: hundreds of entities were frozen in a stale state, all stuck in the same three-minute window when the reboot stalled. Second, an incomplete migration months earlier had left two automation engines running shadow copies of the same logic, so some things fired twice — including a notification handler with enough reach to be worth taking seriously. Third, the backup problem again: the automation configs were only being saved to the same disk that had just failed.

The remediation pass fixed the things that were safe to fix without physical access. The alarm system got the most attention: I added an occupancy guard so it can’t arm while family is home, wired up auto-disarm on arrival, and re-registered the sensors so it can actually trigger the way it’s supposed to. On the housekeeping side, sixteen dashboard cards pointing at long-deleted devices got cleaned up, and I cut the database’s write churn by more than half by excluding the noisiest, least-useful entities from history. A handful of items are deliberately left for a hands-on session — anything involving a physical door test or a full restart of a live security system is not something to do blind and remote.

Busting a Plex Transcoding Myth

For months I’d been told — and half-believed — that the GPUs in this server couldn’t do video transcoding. Remote streaming had been stuttering, and I’d chalked it up to that limitation. It turned out to be wrong on two counts.

The stutter wasn’t bandwidth. The home connection is fine, the port forwarding was open, and the stream wasn’t being relayed. The actual problem was that Plex was transcoding on the CPU at slower than real time — about 0.8× — because the container had never been given access to the GPU. It was asking for hardware acceleration, silently failing, and falling back to software.

The fix was giving the container proper GPU passthrough. Once I did, hardware decode and encode both lit up, transcode speed jumped to 1.7×, and the CPU load on the transcoder dropped from roughly 1,680% (yes, across a lot of cores) to about 50%. The buffering was gone. The lesson: “this hardware can’t do X” is worth testing directly before you accept it, because the real answer was a one-line configuration gap, not a hardware limit.

The Database That Had Been Broken for Over a Year

While chasing why new media wasn’t showing up, I found an indexer service whose database had been corrupt since April of last year. It had been running from an in-memory copy for months, quietly working, until the outage forced a restart and it reloaded the broken file from disk. A recovery pass rebuilt the database cleanly, and grabs started flowing again. It’s a good reminder that “it’s been working fine” and “it’s healthy” are not the same statement.

A Monitor That Watched Nothing

One more war story, because it’s a good one. I have a watchdog process that’s supposed to keep the scheduler alive. Its logs showed it faithfully “relaunching” over and over — which looks like health, until you notice the thing it’s supposed to be advancing never moved. The bug was subtle: the watchdog guards itself with a lock, and when it started the scheduler, the scheduler inherited that lock file handle and held it open forever. So every relaunch immediately failed to acquire its own lock and exited, doing nothing. A repeating “restarted!” message that never makes progress isn’t a flap — it’s a permanent wedge wearing a healthy costume.

Self-Hosted AI, and the Cost of a Cold Start

A lot of my tooling leans on a locally-hosted large language model instead of a paid API — it’s free, it’s private, and it’s plenty good for summaries, drafting, and routing. But a big model is a big file, and loading a 60-gigabyte-plus model off a nearly-full disk takes almost six minutes the first time. Several of my automations had timeouts shorter than that cold-load, so they’d ask for a response, give up mid-load, and get an empty answer back — which then crashed the script that wasn’t expecting it. The fix was patient pre-warming, retries, and never trusting that a response is non-empty. Boring plumbing, but it’s the difference between an automation that runs for months and one that silently dies and nobody notices for weeks.

Alongside that, I spent time making my own AI usage honest — instrumenting the tooling so I can actually see which model did which work and what it cost, rather than guessing. And I’ve leaned harder into running many small AI agents in parallel for big review-and-audit jobs, which is how that 160-item Home Assistant sweep got done thoroughly instead of superficially.

Apps

The app side kept moving through all of this, mostly as a steady stream of updates driven by tester feedback — a pro-AV field-calculator app that picked up a stage-lighting layout planner, a kids’ handwriting-practice app, and continued work on a set of audio and sound-level measurement tools. QR WiFi Share got a production update on the Play Store. A game side project also continues in the background; still early, still fun, and still under wraps.

What’s Next

The priority list writes itself after a week like this one. Off-machine backups come first — a copy of the important data that lives somewhere the next hardware failure can’t reach. Then finishing the Home Assistant recovery properly, which means a careful, deliberate restart rather than another rushed one. And more of the same unglamorous reliability work: the automations that quietly hold everything together only earn their keep if they actually survive the bad days. This was a bad week. The setup mostly held. Now to make “mostly” into “completely.”