Misadventures of the week
I woke up on Saturday to find that my media server’s ReFS (opens in a new window) volume had gone offline. Somehow it could not read “Container Table”… a vital data structure of ReFS… Resilient File System (opens in a new window). Which is mirrored AND striped with parity.
I tested the UPS (opens in a new window), and it was fine. One of the disk drives was reporting a bad sector or two, but its SMART (opens in a new window) report was still reporting “OK”. I still ordered a replacement, and it is on the way, but it wasn’t likely my culprit.
Breaking out my google-foo, I searched for the errors in the Event Log. I found the errors in Microsoft’s docs, but the big clue was "Container Table". I found a few posts referring to a revoked Windows Update that updated the ReFS (opens in a new window) version… Could it have been this? I was one feature update behind, so I updated as far as I could.
Nope, still dead. 💩
I’ve been subscribed to Backblaze’s Personal Backup (opens in a new window) plan for years, so I just ordered restore disks, and got to rebuilding the server. I could have tried upgrading to Windows 11, but I decided to take this as an opportunity to ditch Windows and switch to a hypervisor and run all my services as VMs & containers.
I had to plug a monitor and keyboard in to perform this maintenance, so while I was here I decided to update the bios, and ensure all the virtualization features were turned on. Especially the settings to allow PCIe forwarding.
Once Proxmox (opens in a new window) was installed, I moved on to picking a NAS (opens in a new window) OS… I deliberated between OpenMediaVault (opens in a new window) and TrueNAS (opens in a new window). The major deciding factor was ZFS (opens in a new window)… and the ability to grow the volume by replacing drives with larger ones over time. I know Proxmox (opens in a new window) can manage ZFS volumes too, but I want the storage to be bare metal with the NAS OS directly controlling the drives. Instead of mapping virtual drives into the VM. Less overhead and feels safer.
✅ VMs created. Now…
…Now, the VMs needs their hardware assigned. Easier said than done. 😅 Learning the tricks to get the SAS (opens in a new window) controller and iGPU (opens in a new window) forwarding into the hosted VMs was a pain. — And, frankly, I gave up on the GPU. The instructions I found online were all good but failed to cover some edge cases… that I happen to hit. For example: If you exactly allocate memory, and do not leave any for the hardware mapping, the entire machine will crash 😛. The GPU is initialized and claimed very early in the boot process and since it’s an onboard GPU vs a dGPU (opens in a new window), I had a bugger of a time getting the system to use it, or not crash! Anyways, the SAS (opens in a new window) controller was able to be used by a VM running TrueNAS (opens in a new window) which is the most important part. With that, I could now set up the NAS and crate the ZFS (opens in a new window) pool to hold all the data.
Without Hardware Transcoding, watching content from [Plex] on devices that need video content re-encoded would be pretty taxing… many 4K videos would simply stutter. Simply put, [Plex] needs access to the hardware (my Sandy Bridge iGPU (opens in a new window) with Quick Sync (opens in a new window)) to enable fast transcodes. Sadly, every time I started a VM with the iGPU (opens in a new window) passed to it, the entire machine crashed. So, I changed course and looked at how others achieved this. And it looks like they used a Container! I should have thought of this… With a tweak to cgroups (opens in a new window), and device mapping, it just works. 😅 (oversimplified explainer: Containers do not virtualize the hardware, they just namespace processes)
- NAS ✔️
- Plex ✔️
Container and VMs created and working! Time to restore all my files! ⏳
This is where I’ve hit a brick wall.
Backblaze (opens in a new window) backup client only runs on Windows and macOS and ignores network-mapped drives. — I know this, because I created a Windows VM, mapped a drive letter, and installed the Backblaze client… no go. The mapped drive is not available to select in the backup.
Tried the B2 (opens in a new window) integration on TrueNAS… only to realize this would change my bill from $7/mo to ~$100/mo …and B2 doesn’t have as good of restore options… it appears to be optimized for applications and live data, not archival off-site backup.
So, I think I may just be out of luck with the off-site backup. :/
Backblaze, if you’re reading this, I would pay more than $7/mo for a “NAS Edition” of personal backup… some TrueNAS integration?? 😃 Please? 😬