I’ve had various bits of hardware running as servers in my house for many a year. Latterly I have an Asus Terminator T3-P5G965 running Linux as my main dev and pissing about box, an HP Microserver N36L running FreeNAS as my storage (it runs a Plex server too) and sat alongside a pfSense box I built about a year ago with Asus N3150I-C mobo (overkill, I know). So three boxes, networked, from a TP-Link switch. The rest of the house networking runs into there too, for the WiFi, desktops and such. I guess its pretty typical of a home setup.
A few things about the setup began to bug me though. For one, the Linux box was starting to feel a little slow and the FreeNAS disks were over 5 years old. I’d had one fail and replaced it, but thats still 3 disks of a RAID-Z pool that are older than spinning rust should be. Also, a couple of the boxes had developed a habit of hanging on reboot, which given they sit in a slightly hard to get to place made life annoying at times. So, I figured it was time to sort the mess out and make things better.
Since everything I do with work these days is virtualised I figured I might as well eat my own shit, so to speak, and virtualise my home kit too. So I needed a fairly beefy box. I needed a storage controller I could pass though to FreeNAS. I needed ECC memory. I needed remote KVM. I needed either 3 on-board NICs or enough PCI slots to add a couple of cards. And I needed it not cost an arm and a leg.
I started looking at Xeon E3 processors and motherboards, 4 cores weren’t going to be enough. E5s then, but boy there are pricey and boy do they start sucking the wall juice. I looked on eBay for some old server kit but that was large and noisy and also juice sucking (not to mention not cheap for anything new enough to run VT-d with remote KVM). So…
Xeon-D. Its pretty much designed for what I’m looking to do. There are a bunch of CPUs from 4 cores to 12, all with hyper-threading. It can use ECC too. Its a SoC design so you get the CPU soldered onto the motherboard with limits future upgradability, but thats OK with me, I tend to build stuff and leave it for years. There are a few manufacturers who make motherboards, in the UK though the only ones you can really get are Supermicro X10SDV series, but thats fine ‘cos they fit the bill nicely. It’d probably have been sensible to get one with an on-board LSI controller, and I would have got an X10SDV-7TP4F, but post Brexit vote those went from £750 to £920 and thats kinds rich. So I picked up an X10SDV-TLN4F for rather a lot less.
Now the thing about the TLN4F is it doesn’t on the face of it have a storage controller I can pass through to FreeNAS. But, I figured that I’d try FreeNAS 10, which has FreeBSD’s Bhyve virtualisation baked right in (I know 9.10 does also, but I wanted to try 10). FreeNAS on the bare metal, a Linux box or two as virtuals and pfSense as a VM too, but with both the i350 onboard NICs passed through in hardware. That leaves FreeNAS with the two 10GBase-T onboard NICs. Sounds nice, should work, I like it.
Or it should have been nice. I know FreeNAS 10 is Beta, but the VM side is rough. There is no hardware pass-through from within the GUI or the CLI. Sure, you can do it from shell, like you can in 9.10, but meh. I had a few other issues with 10 too, but mostly I guess due to being beta. Anyway, if I wasn’t going to virtualise with FreeNAS what was I going to use. I could do KVM, but for once in my life I wanted easy, so I figured if its good enough for work I’d use ESXI here too.
Now ESXI is great, but how was this going to work. I wanted to pass through the on-board SATA (C610) through to FreeNAS. FreeNAS can serve an NFS datastore back to ESXI, but I’d still need somewhere to stick the datastore for ESXI. USB? Well, yes. You can partition unused blocks of the USB boot media as VMFS and use it. So I did. It worked, not very fast but fast enough to get FreeNAS up with the on-board SATA passed through. Then it crashed, hard, repeatedly. If I was spinning up a new VM in ESXI, even with that VM on the NFS datastore, it would loose the USB datastore and crash out ESXI. Sucky.
What to do? Well, I had two options as I saw it. One safe and one fun. The safe way is the tried and tested way. Buy a cheap LSI controller that can be flashed to IT mode and passed through to FreeNAS, using the on-board SATA for an ESXI datastore disk. Simple, but boring and it would use up the one PCI slot on the mobo that I might have a plan for down the line. The fun way then, use the M.2 slot to drop in an NVMe SSD which ESXI will use, whilst leaving the on-board SATA still free to pass through to FreeNAS.
So thats what I now have. ESXI with three primary VMs.
1 – FreeNAS with the onboard SATA passed through (Replaces the HP N36L)
2 – My main Linux box (replaces the Asus Terminator)
3 – pfSense with the on-board i350 passed through (Replaces the N3150I-C)
I’ve also got a Windows VM and a couple of test Linux instances. Its really useful. And all in the one box (‘cos I love a single point of failure, so you shouldn’t do this if you can’t not have things working).
In the next blog post I’ll run though how I made ESXI do what I needed, ‘cos it wasn’t quite out-of-the-box easy…
Leave a Reply