Justin Garrison
January 3, 2017

On-Prem Development Kubernetes Cluster Options

Posted on January 3, 2017  •  6 minutes  • 1134 words

Kubernetes has a lot of options for running in cloud environments. If you’ve never experienced the simplicity of starting a cluster in GKE I highly recommend it. But how can you test cluster setups in an on premises environment?

Here are just a few reasons to have your own development cluster

There are plenty of other reasons but these are some that have driven me to find options. Below are some of the best options I found and what I went with.

Virtual Machines

Virtual machines might be fine in a work environment with OpenStack or VMware but will probably be harder to find resources if you are doing development at home. They also might make testing certain types of development harder depending on what hypervisor you’re using.

Minikube  is an obvious choice. It is by far the easiest way to develop against the Kubernetes API but is limited to a single VM and can’t do some things a full multi-VM cluster could do like testing network segmentation, load balancing, distributed storage, or network overlays.

VMware  is supported and can easily be used for development but it might be too much to manage for a home lab. There are also some options for starting multi-node clusters with Vagrant  which can be used but you will need to make sure your laptop/desktop has enough resources.

Development Hardware

Here’s where things get tricky. Hardware requires more commitment because you need to order it, maintain it, and pay for running it (power). But this is also where things get fun. Hardware costs more up front and what you order is dependent on your budget and development goals. Let’s look at some options.

Raspberry Pi

This is an obvious choice. In fact it even had a whole blog series on the Kubernetes blog  to explain how to do it.

It is one of the cheapest clusters you can buy. For a 5 node cluster it will cost you $300-$400. However, if you’ve ever built a Raspberry Pi cluster you’ll quickly notice some limitations.

The benefits of Raspberry Pis include ubiquitous platform for development and small footprint.

You can easily search and find dozens of examples  for setting up a Raspberry Pi cluster so I wont go into any more details here. You can even get a full kit or pre-assembled cluster from some vendors like https://www.picocluster.com/ .

PINE A64+

There are other ARM boards  that are worth looking at but I’m not going to go into them because they all typically have the same limitations as the Pi.

Beyond the Raspberry Pi

Of course the Raspberry Pi isn’t the only single board computer worthy of a home cluster. Lots of alternatives have shown up in the past year and some of them are very capable. Some benefits of these alternatives are.

The obvious downsides being the boards are more expensive. Instead of $35 for a Raspberry Pi you’re probably going to spend $100 per board for 1–2 GB of RAM and $170–$200 per board for 4GB of RAM.

If you have the money here are some great options for you to look at. (click the pictures for links)

Sub-atomic cluster

https://hackernoon.com/diy-kubernetes-cluster-with-x86-stick-pcs-b0b6b879f8a7#.6eivoptm1

A few more honorable mentions include LattePanda , Kangaroo Desktop Pro , and the Udoo . Each board would provide compute and RAM for a cluster but vary in price, options, and accessories.

The DIY route for x86 boards is only getting better. Here’s a great example of a nice looking, 5 node cluster https://hackaday.io/project/27869-kuberdoo .

Desktop Class Hardware

This last category is going to give you lots of options. You’ll have a relatively powerful cluster (possibly more powerful than many cloud instances) but will cost you. Most of these options will cost you $1500-$2500 for a five node cluster but have a few benefits the others don’t.

Of course downsides include

One thing to note, just because something is a “desktop” doesn’t mean it has to take up a lot of space.

DIY nono ITX build

Typically when people look for a desktop cluster they automatically think of the Intel NUC computers (example NUC cluster here ). These are great but are going to be one of the more expensive options. I’d suggest you also look at the DIY route. With a little more time investment you can build something more specialized and cheaper.

The nano ITX build (pictured left) is a great example. In the links there are parts and other options for hardware.

I’d also suggest you look at onboard CPU combos and PC Part Picker  to find cheap options. Here are some onboard CPU options from Gigabyte for AMD  and Intel  CPUs. I found multiple options from $60-$90 per node which is a great price if you’re on a budget.

One of the cheapest desktop solutions I found is retired company systems.

You can often find old corporate systems on eBay, local electronics recycling, colleges, or even thrift stores. The benefit to these systems is they are usually enterprise grade hardware, decently spec’d (Intel Core i3/i5 with 4+ GB RAM), and cheaper than other options. You can regularly find desktops in the $40–$80 range. And although they’re bigger and take more power than other options they cost 1/4 of the price and are more easily upgraded. I bought hardware for my 5 node cluster for only $20 per system. That’s 1/2 the price of a Raspberry Pi + case + storage and my systems came with 4GB RAM, 500 GB hard drives, and Dual Core Intel i5 processors.

No matter what you’re building, there are plenty of options available. If you have more examples and resources please leave a comment so others can benefit.

Follow me

Here's where I hang out in social media