Someone once told me a bit of advice:
“When you’re building your homelab, you’re going to break things. When that happens, don’t reinstall from scratch; take the time to fix what you broke. It’ll be frustrating but you’ll learn more that way.”
I recently acquired 5 Asus Mini PCs at a great price. I decided to make them into a Proxmox cluster. I wanted this cluster to handle network management: firewall, certificate authority, etc.
So far, it’s been great at handling whatever tasks I throw at it, but I made a crucial mistake on the primary node: pve1.
pve1 has 16GB of RAM: enough for a firewall VM and a few basic services. At least according to the threads I read in preparation for this build.
But I made a critical error: I did not allocate enough RAM for my OPNsense VM. I didn’t realize that changing VLAN settings could cause RAM spikes. I also wasn’t aware this could happen even if the device isn’t handling traffic yet. While configuring my VLANs, the VM hit 99%+ RAM utilization and stopped responding. My host device also started misbehaving so I did what I usually do at home in this situation and I rebooted pxe1.
Upon reboot, the VM was no longer visible and the Proxmox Web GUI for pve1 wasn’t reachable at all. I was able to SSH into the system and found that most the Proxmox services were offline or boot-looping. I read through dozens of forum threads to figure out what happened. I realized that this RAM spike happened at the worst possible time. The system crashed in the middle of writing to pmxcfs. This left the filesystem corrupted.
pmxcfs is central to the functions of Proxmox and it’s read-only. This makes it very hard to fix the problem. Proxmox can’t get into a stable state because of pmxcfs and I can’t touch pmxcfs without Proxmox being stable.
“Don’t reinstall; fix what you broke.”
I took that homelab advice to heart, but in this specific case, it seems my hands are tied. I will be re-imaging this system after I finish writing this. But I did try my best to rebuild the system. In doing so, I learned more about how Proxmox works. I also learned about the critical importance of careful memory management/allocation.
Moving forward, I’m dedicating an entire mini PC to this single OPNsense VM. I’ll move my other services to the other nodes.