This repository will give an overview of my homelab journey.
I wanted a small, silent, efficient but powerful home server and was considering the below options:
Hardware:
- Something like STH’s TinyMiniMicro series and buy half a dozen 1L mini pcs.
- Dell/HP server tower server from Bargain Hardware.
- Raspberry Pi 'cluster'.
- Custom server type miniITX system using something like this asrock rack board.
- Nuc type Microcluster like Canonicals cloud in a box.
- Custom built tower gambling with an AliExpress AMD EPYC.
- AsRock DeskMeet X300.
Software
- OpenStack / DevStack
- I could see this being kinda fun but even though I do have some basic OSP experience I do risk spending too much time on the OSP side of things.
- OpenShift with KubeVirt and Hypershift
- Future looking, pretty interesting, but will require some outside supporting services (DNS etc).
- Proxmox
- No experience with this, but seems fairly simple.
In the end I opted to go for the fun DIY route and go with a small custom build tower with some dubious AliExpress parts and go for bare metal SNO OCP. However, I really was tempted by the nuc type cluster and go full DIY after coming accross this! In terms of software though, I think for my purpose I'd get better bang for my buck and remove any virtulization layer and go for running containers on bare metal.
I bought the parts from a mix of AliExpress, eBay, Scan and Amazon. I was able to source everything with my budget of £1k through timing and bargaining.
| Item | Component |
|---|---|
| CPU | AMD EPYC 7551P CPU (32 cores, 64 threads) |
| Motherboard | Supermicro H11SSL-i motherboard |
| Memory | Samsung 32gb 2133MHz DDR4 ECC RAM - x8 (256gb) |
| PSU | Super Flower LEADEX III HG 850w Gold PSU |
| SSD1 | Crucial P5 Plus 2TB M.2 PCIe Gen4 NVMe |
| SSD2 | Samsung SSD PM1735 HHHL PCIe card 1.6TB NVME SSD |
| GPU | NVIDIA Tesla P4 8GB |
| Cooler | Noctua NH-U14S TR4-SP3 for AMD sTRX4/TR4/SP3 (140mm, Brown) Noctua NF-A15 PWM |
| Fan | Noctua NF-A15 PWM, Premium Quiet Fan, 4-Pin (140mm, Brown) |
| Case | Raijintek Thetis Aluminium ATX case |
I was quite happy with the build. It's fairly small, powerful, quite and the power consumption isn't bad.
I did come accross a few issues when putting this sytem together:
- Networking - IPMI was coming up on a static IP address on another subnet.
- Resolved by removing the existing static IP config for the IPMI in the BIOS.
- BMC - Admin password had been changed to what was on the motherboard.
- Resolved by loading supermicro’s ipmicfg utility onto a USB and booting into UEFI shell tool to add/reset users/passwords.
- SSD - M.2 SSD was not being recognised.
- Resolved by a BIOS setting to change pci/m.2 device compatibility from being 'Vendor Defined' to 'AMI Native Support.
- Memory - DIMMH1 was not recognising the installed memory module.
- Resolved by removing a case standoff that was touching the underside of the board; only an issue due to using a server motherboard in a consumer case.
- Fans - Supermicro fans were ramping up and down though using consumer grade fans.
- Resolved by setting the fan speed to max; it’s still pretty much dead silent thanks to the Noctuas and their anti-vibration mounts. See Supermicro fan speed ramps up and down.
- BMC Licence - "Not licensed to perform this request. The following licenses SUM DCMS OOB were needed".
- Resolved by Reverse Engineering Supermicro IPMI.
- Disks - SNO requires two disks if wanting to use kubevirt.
- Resolved by adding a second disk.