Misadventures of the week

I woke up on Saturday to find that my media server’s ReFS volume had gone offline. Somehow it could not read “Container Table”… a vital data structure of ReFS… Resilient File System. Which is mirrored AND striped with parity.

💀 How did this happen?

I tested the UPS, and it was fine. One of the disk drives was reporting a bad sector or two, but its SMART report was still reporting “OK”. I still ordered a replacement, and it is on the way, but it wasn't likely my culprit.

🤨 Well, What the heck?

Breaking out my google-foo, I searched for the errors in the Event Log. I found the errors in Microsoft's docs, but the big clue was "Container Table". I found a few posts referring to a revoked Windows Update that updated the ReFS version… Could it have been this? I was one feature update behind, so I updated as far as I could.

Nope, still dead. 💩

I’ve been subscribed to Backblaze’s Personal Backup plan for years, so I just ordered restore disks, and got to rebuilding the server. I could have tried upgrading to Windows 11, but I decided to take this as an opportunity to ditch Windows and switch to a hypervisor and run all my services as VMs & containers.

Step 0: UEFI Changes

I had to plug a monitor and keyboard in to perform this maintenance, so while I was here I decided to update the bios, and ensure all the virtualization features were turned on. Especially the settings to allow PCIe forwarding.

Step 1: Install Proxmox, pick a NAS, build Plex

Once Proxmox was installed, I moved on to picking a NAS OS… I deliberated between OpenMediaVault and TrueNAS. The major deciding factor was ZFS… and the ability to grow the volume by replacing drives with larger ones over time. I know Proxmox can manage ZFS volumes too, but I want the storage to be bare metal with the NAS OS directly controlling the drives. Instead of mapping virtual drives into the VM. Less overhead and feels safer.

✅ VMs created. Now…

Step 2: $#!%{#‼️

…Now, the VMs needs their hardware assigned. Easier said than done. 😅 Learning the tricks to get the SAS controller and iGPU forwarding into the hosted VMs was a pain. — And, frankly, I gave up on the GPU. The instructions I found online were all good but failed to cover some edge cases… that I happen to hit. For example: If you exactly allocate memory, and do not leave any for the hardware mapping, the entire machine will crash 😛. The GPU is initialized and claimed very early in the boot process and since it's an onboard GPU vs a dGPU, I had a bugger of a time getting the system to use it, or not crash! Anyways, the SAS controller was able to be used by a VM running TrueNAS which is the most important part. With that, I could now set up the NAS and crate the ZFS pool to hold all the data.

Step 2. Plan B: Solving Hardware Transcoding in Plex

Without Hardware Transcoding, watching content from [Plex] on devices that need video content re-encoded would be pretty taxing… many 4K videos would simply stutter. Simply put, [Plex] needs access to the hardware (my Sandy Bridge iGPU with Quick Sync) to enable fast transcodes. Sadly, every time I started a VM with the iGPU passed to it, the entire machine crashed. So, I changed course and looked at how others achieved this. And it looks like they used a Container! I should have thought of this… With a tweak to cgroups, and device mapping, it just works. 😅 (oversimplified explainer: Containers do not virtualize the hardware, they just namespace processes)

Done

• NAS ✔️
• Plex ✔️

Container and VMs created and working! Time to restore all my files! ⏳

Bonus: Re-establish backups

This is where I’ve hit a brick wall.

Backblaze backup client only runs on Windows and macOS and ignores network-mapped drives. — I know this, because I created a Windows VM, mapped a drive letter, and installed the Backblaze client… no go. The mapped drive is not available to select in the backup.

Tried the B2 integration on TrueNAS… only to realize this would change my bill from $7/mo to ~$100/mo …and B2 doesn’t have as good of restore options… it appears to be optimized for applications and live data, not archival off-site backup.

So, I think I may just be out of luck with the off-site backup. :/

Backblaze, if you're reading this, I would pay more than $7/mo for a "NAS Edition" of personal backup... some TrueNAS integration?? 😃 Please? 😬