Jeff Minard

Home Cluster (Part II): Machine Setup, PiXiEs!

With my new cluster hardware in place, on thing I really did not want to do was setup all three machines by hand with install usb sticks, etc. Especially since I had the foresight to realize that I’d probably had to re-image these machines a few times in the future.

Operating System Selection

I am most familiar with Centos 67 (any RHEL based system, really). Normally I would have gone with that OS, but I planned on relying very heavily on Docker for these servers. Now, while Docker is supported in RHEL distributions, it’s a bit of a second-class citizen. Ubuntu has all the best features, is most stable, and least PITA to work with when you’re dealing with docker. So I went with that.

Bootstrapping Bare Metal Machines

Also all network cards for the last forever support PXE boot mode. Good google that for way more details, but it boils down to the idea that the network card can grab an IP address from your DHCP server and also get instructions on where to fetch a boot image from. It will grab that boot image and start your system. Typically you’ll use this to show a menu of options.

RT-N66U Router and PXE Boot

I have the very nice RT-N66U router. Since it’s a mini-linux like box, it’s using dnsmasq interally. dnsmasq is a powerful dns server which can serve PXE info and act as the TFTP server for remote computers. You will need a USB thumb drive to store large ISO’s for the router – just pop one into the back and format it.

Turn on JFFS and SSHD

First, you must have the JFFS drive writable and the SSH server turned on. The jffs filesystem allows you to make chnages that the router will persist during restarts. You turn on SSH so you can ssh to the server to make direct changes.


Once done, you can ssh to the router and login using the admin credentials.

Adding DNSMasq Configs

We need to edit the router to load extra params into dnsmasq. SSH in, create the right directory, and setup your extra configs:

mkdir -p /jffs/configs/

If /jffs/configs/dnsmasq.conf.add file exists, add to it, otherwise put in:


What that does is setup the router to tell machines that they can PXE boot from it, and turns on the TFTP server (within dnsmasq) to faciliate as much. The tftp-root will be different based on where the system decides to mount your USB drive.

Now just restart dnsmasq to get it up and working:

# Weird syntax, but it works
service restart_dnsmasq

Setting Up The TFTP Root

First you’re going to need to define what your Boot Menu will look like, this is mine:

# /tmp/mnt/sdc1/tftp/pxelinux.cfg/default
# The Boot Menu!

MENU TITLE Pick Your Path

menu color title 1;34;49 #eea0a0ff #cc333355 std
menu color sel 7;37;40 #ff000000 #bb9999aa all
menu color border 30;44 #ffffffff #00000000 std
menu color pwdheader 31;47 #eeff1010 #20ffffff std
menu color hotkey 35;40 #90ffff00 #00000000 std
menu color hotsel 35;40 #90000000 #bb9999aa all
menu color timeout_msg 35;40 #90ffffff #00000000 none
menu color timeout 31;47 #eeff1010 #00000000 none

default vesamenu.c32

prompt 0
timeout 300

label Ubuntu-16.04
    menu label ^0 Ubuntu 16.04
    kernel ubuntu/16.04/install/netboot/ubuntu-installer/amd64/linux
    append auto=true interface=eth0 hostname=cluster domain=home url=tftp:// initrd=ubuntu/16.04/install/netboot/ubuntu-installer/amd64/initrd.gz --

label Memtest
    menu label ^1 Memtest
    kernel memtest/memtest

I only use two boot options at this point: install ubunut or run memtest. Being able to boot a machine over the network straight to a utility like memtest is pure amazing.

To setup the ubuntu source, you’ll need to download their ISO and extract it to /tmp/mnt/sdc1/tftp/ubuntu/16.04/. Once done, you need to steal a few files from the ISO and place them in the tftp root:

cp /tmp/mnt/sdc1/tftp/ubuntu/16.04/install/netboot/ubuntu-installer/amd64/pxelinux.0 /tmp/mnt/sdc1/tftp
cp /tmp/mnt/sdc1/tftp/ubuntu/16.04/isolinux/chain.c32 /tmp/mnt/sdc1/tftp
cp /tmp/mnt/sdc1/tftp/ubuntu/16.04/isolinux/ldlinux.c32 /tmp/mnt/sdc1/tftp
cp /tmp/mnt/sdc1/tftp/ubuntu/16.04/isolinux/libcom32.c32 /tmp/mnt/sdc1/tftp
cp /tmp/mnt/sdc1/tftp/ubuntu/16.04/isolinux/libutil.c32 /tmp/mnt/sdc1/tftp
cp /tmp/mnt/sdc1/tftp/ubuntu/16.04/isolinux/vesamenu.c32 /tmp/mnt/sdc1/tftp

And if you want memtest, grab this:

gunzip memtest86\+-5.01.bin.gz
mkdir /tmp/mnt/sdc1/tftp/memtest/
mv memtest86\+-5.01.bin.gz /tmp/mnt/sdc1/tftp/memtest/memtest

Compressed from these instructions

Preseed – Ubuntu Automatic Install

At this point, a machine configured for PXE boot (see your motherboard bios options) should boot, ask for DHCP, boot the menu and present you with options. If you were to select Ubuntu at this point it’d either break (not sure) or go into an interactive install. As we started at the beginning of the post, I want this to be fully automatic, so we need to tell the installer all the options to get it installed without manual intervention. In ubuntu (via debian) this is called the preseed.

If you take a look at the boot menu options, you see

label Ubuntu-16.04
    menu label ^0 Ubuntu 16.04
    kernel ubuntu/16.04/install/netboot/ubuntu-installer/amd64/linux
    append auto=true interface=eth0 hostname=cluster domain=home url=tftp:// initrd=ubuntu/16.04/install/netboot/ubuntu-installer/amd64/initrd.gz --

The preseed.cfg is what the installer will attempt to fetch and use to auto install the system. My preseed looks like this – lots of comments explaining what I picked and why are embedded.

# preseed.cfg

# Language setting -- I'm in the us
d-i debian-installer/locale string en_US

# Keyboard setting
d-i console-setup/ask_detect boolean false
d-i console-setup/layoutcode string us
d-i console-setup/charmap select UTF-8
d-i keyboard-configuration/xkb-keymap select us
d-i keyboard-configuration/layoutcode string us

# Network setting
# While I start with the hostname cluster, it will get changed later by
# editing the machine once its up and running
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string cluster
d-i netcfg/get_domain string home

# I have turned on the FTP server on my router and pointed it at the extracted
# ISO root so that I can boot and install packages from there. This way I can
# radically speed up my install since I don't need to fetch items from
# the web. This does, however, come with a downside -- these settings stick
# around AFTER the install is complete, so since this is not a REAL mirror, but
# merely a static copy, you'll need to go and fix your apt sources later on.
d-i mirror/protocol string ftp
d-i mirror/ftp/hostname string
d-i mirror/ftp/directory string /sdc1/tftp/ubuntu/16.04
d-i mirror/ftp/proxy string

# Clock setting
d-i clock-setup/utc boolean true
d-i time/zone string America/Los_Angeles
d-i clock-setup/ntp boolean true

# Partitioning: use lvm, the entire disk, wipe anything there, one big file system (/home, etc)
d-i partman-auto/method string regular
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-auto-lvm/guided_size string max
d-i partman-auto/choose_recipe select atomic
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true

# Skip creating the root user and, instead, create a user (who will have sudo) as myself.
# I do not create a sane password since I will disable password login and add my SSH key later.
d-i passwd/root-login boolean false
d-i passwd/make-user boolean true
d-i passwd/user-fullname string Jeff Minard
d-i passwd/username string jeff
# I have literally no idea what this password is. Just did:
#  $ mkpasswd -m sha-512 $(pwgen -yns 64 1)
# to make one up. PKI is how we'll really get in here (see late_command)
d-i passwd/user-password-crypted password $6$PqEsuaIV4$OfUKm5dRaFRPX1TP40OvwHg3PEEQ1S453fMziL.H0JDGamHCO.N5OIXjLH3DUjV0vhfmb.0tSBVsGx1kO5Wmb/
d-i user-setup/encrypt-home boolean false

# Base system installation
d-i base-installer/kernel/image string linux-image-amd64

# package selection
tasksel tasksel/first multiselect standard
d-i pkgsel/update-policy select unattended-upgrades
d-i pkgsel/include string openssh-server
d-i pkgsel/upgrade select none

# This is a critical step -- the preseed can only do steps from what the
# installer offers with this one exception, the "late_command" option can run
# any arbitrary command. However, you can only specify a single instance of
# the late_command option, but you can make it do multiple things. To keep this
# file sane, I have it fetch a bash script, chmod it, and execute it in the
# machine which is being built.
d-i preseed/late_command string \
 in-target wget -O /tmp/ ;\
 in-target chmod +x /tmp/ ;\
 in-target /tmp/

# Grub
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
d-i finish-install/reboot_in_progress note

That file all put together will get the system installer to completely finish an ubuntu install. You would, however, need the script. A slightly redacted version of mine looks like this:


# Update to proper sources -- as noted in preseed, the sources setup for
# install are local and static. This gets us back to normal sources AND adds docker
cat <<'EOF' > /etc/apt/sources.list
# Ubuntu Main Repos
deb xenial main restricted universe
deb-src xenial main restricted universe

# Ubuntu Update Repos
deb xenial-security main restricted universe
deb xenial-updates main restricted universe
deb-src xenial-security main restricted universe
deb-src xenial-updates main restricted universe

# Docker
deb ubuntu-xenial main

# Add Docker Key
apt-key adv --keyserver hkp:// --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

# Update apt, upgrade everything
apt-get update
apt-get upgrade -y

# Install docker
apt-get install -y docker-engine

# Setup SSH Keys for `jeff`
mkdir /home/jeff/.ssh
echo 'ssh-rsa insert-your-ssh-public-key-here' >> /home/jeff/.ssh/authorized_keys
chmod -r 600 /home/jeff/.ssh
chown -r jeff:jeff /home/jeff/.ssh

# passwordless sudo
echo "%sudo   ALL=NOPASSWD: ALL" >> /etc/sudoers

# Prevent root login
sed -e 's/PermitRootLogin.\+/PermitRootLogin no/g' -i /etc/ssh/sshd_config

# Disallow password auth (not that anyone has a clue what the fucking password is)
sed -e 's/.\+PasswordAuthentication.\+/PasswordAuthentication no/g' -i /etc/ssh/sshd_config

# speed up ssh
echo "UseDNS no" >> /etc/ssh/sshd_config

Once the machine is booted, PXE loaded, Ubuntu selected and installed I can ssh directly to the new machine and start playing with it. You can find the machine’s IP address in your router, but I have them pegged to specific IPs by using the MAC address binding in the router.


Lot’s of stuff to play with and explore. Give yourself a while to get through this step – it’s complicated tech, pretty deep in the stack and took me several hours over the course of a few weeks to really get locked in how I liked. I had to spend a lot of time learning about PXE, dnsmasq and just generally searching for solutions. And I probably rebooted those machines 100 times. I eventually turned my brain on and realized that it was smarter to test the PXE boot with a virtual machine (Virtualbox works great for this). Give that a try, was very successful.

And the end of the day, I can completely erase any of my cluster machines and have them fully rebuilt in about 30 minutes, ready to rejoin the cluster.

Home Cluster series: