Two Years Blogging - A Review of Designing Homelab Projects

The Art of The Homelab

I’ve been homelabbing/blogging about side projects for about two years now and thought I’d compile a list of things that have been useful to me in my homelabbing journey. These are just my opinions and everyone’s goals for their homelab are different. I view my homelab as a blank canvas upon which to experiment, fail, learn and share said failures (or successes) with others. It’s a labor of love, but nothing makes my day like a Youtube/Reddit comment that says “hey this is awesome, thank you for sharing”.

Having this small blog, writing down side projects I’ve done over the past two years has greatly helped my professional career. Even if you think a project is small, things you’re working on may help someone else in the future. I highly recommend having a place to showcase some of your work. With that that being said, here are some thoughts on things like buying hardware, designing side projects to deploying infrastructure.

Design, Analyze and Execute - Making Strategic use of Your Freetime

There are only 24 hours in a day, and the reality is those hours are not the same for all of us. Between work and the responsibilities you have in your own life, your actual free time is likely very short. Making sure you’re maximizing the use of it when experimenting with new technology or learning something new is important. Here are some questions I roll around in my head when I’m thinking of tinkering with something over the weekend.

What am I trying to learn?

Is this something I’m really interested in and will take multiple days? Or is this re-creating a neat exploit I saw on Twitter and I could finish it in a night? Will this distract from a longer-term goal like studying for a cert? Or is this task complementary to longer-term goals? Am I task hopping like crazy right now and should just put it on the TODO list and focus on what I started last week? One thing I struggle with is frequent task hopping. It largely creates a feeling of leaving several things undone. To combat this, plan on dedicating longer periods (weeks/months) to a given project from the get-go. Sometimes you’ll just be researching things to gain a deeper perspective and not have a lot of code written/infrastructure deployed so it feels like you haven’t made progress. Don’t be discouraged, this is just part of the journey.

What resources are available for this project?

Books, videos, blogs and are they up to date? Also, do I have enough previous knowledge to be successful in the task or do I need to plan on this being a longer project to build prerequisite knowledge? Seek out those knowledgeable in a subject or recommend books/blogs/videos as a part of your “syllabus” for what you’re diving into. Additionally, organizing this knowledge and making it readily searchable is important too. I use Zotero and tag resources appropriately. I find if I use bookmarks, I end up with just a folder full of links that I never revisit. Experiment and find what works for you.

What will this project allow me to demonstrate to current/future employers?

Do I care if this is not related to career goals? I like experimenting with Red Team/C2 frameworks, but I’m not a pentester/red teamer. I do it because I’m interested in experimenting with something new. If you’re job hunting and trying to break into a different field (say DevOps to Red Team or Blue Team to DevOps/etc…) try to find a way to combine your current knowledge set with what you’re looking to learn. For example, I have a DevOps background but enjoy reverse engineering (CTFs/malware/etc…). My malware pipeline blogs while not overly complex, highlight various skillsets I was trying to highlight when interviewing for some positions recently.

How long will this take?

Will I be disappointed when Sunday at 6 pm rolls around and this side project is still not done? Setting reasonable time expectations for yourself is important to not lose motivation but also not end up with a hundred half-finished projects.

What is the long-term return on time-investment?

Should I run this in the cloud?
Is this more appropriate for a hardware purchase?

Often when asked about starting up a homelab people assume their needs can be met with a cloud provider. While this is likely true, it can also get expensive quickly. Consider the cost of having a cloud lab and the associated VMs you want when all you need is a few VMs and would be better off with say VMWare workstation on your desktop. That’s not to say a cloud provider won’t have a place in your experimentation (honeypots or even just learning the services provided), but for raw compute you may be better off with just a VM.

What can I automate?

From Ansible, Chef, Salt, Puppet, Terraform, Docker, Kubernetes, neat Bash onliners, etc… there are numerous platforms and tools designed to make certain processes repeatable, scalable and overall automate away the boring tasks so you can focus on the primary goal of your project. While there’s great value in performing some tasks manually the first time so you know what’s happening under the hood, be sure to leverage the technology available to you to carve back as much of your valuable free time as you can.

Defining Goals and Avoiding Spending Too Much on Hardware

As a subscriber to /r/homelab I admire those who have full server racks illuminated in RGB with dedicated NAS storage, battery backup, separate dedicated internet lines, etc… However, most of this is impractical for a typical end-user and can really hurt the budget with regard to energy costs. Looking back to the blog post I made two years ago

I shouldn’t have bought three NUCs. Was it nice to have? Absolutely! Could the majority of my “workload” and side projects exist on, say two NUCs? It sure could, with room to spare even. Having one less NUC would have saved a decent chunk of money to be invested in other ways that I’ll discuss later in this blog. So if you’re reading this considering about buying out that decommissioned data center sale you found on eBay, consider the following questions:

What are your goals for your homelab?
What types of projects do you plan on deploying? How resource intensive will those be?

Sure running the latest and greatest Thread Ripper from AMD is cool, but does it really matter for the dozens of containers you’re going to deploy? Let me give an example that in hindsight I would do differently having bought more compute than I needed.

I’m a big fan of Elasticsearch, and therefore use it as a SIEM among other things. However, for my red team range blog, I didn’t need to set up 3 nodes across separate NUCs for high availability with 100GB of storage each when I was barely pulling in a 1GB of data a week. Was it “cool” to have this? Sure, but was it necessary? Not really, as I’m not running a production-grade workflow. The extra time spent setting this up was fine for a learning exercise, but in a world of ever-shrinking free time, I would likely have a single node cluster for development purposes in the future.

What additional equipment will you need to achieve your goal? (product keys/OS licenses/books/courses/ etc…)

Beyond hardware, perhaps there are video courses or books you’re looking to buy. Always consider the entire scope of what your goals for the year currently look like and what they might turn into before spending your entire budget on hardware.

While one could argue buying more hardware when your budget allows for it ensures room for growth and they would be right. However, at the same time, there will always be a “ceiling” to what one can run with regards to VM capacity. Understanding what’s reasonable for your goals will also help to ensure you’re not going to overspend.

Picking a Hypervisor - Opinions, Budgets and Hardware

Two years ago I was running a 3 node ESXi cluster licensed via my VMUG membership with failover configured and the whole 9 yards. I think VMUG provides great value for the enthusiast community and if you’re leveraging ESXi at work it’s a great way to have a cluster at home to tinker with and learn more about the product.

Outside of the official compatabiltiy hardware list for ESXi, VSphere takes up, in my opinion, a significant amount of memory for a homelab deployment. If you’re in a situation where you have a rack server with greater than 100GB of RAM then this doesn’t affect you. However, for my NUC deployment, it did. If you don’t need centralized management at all, and say you have an old laptop kicking around using KVM and virt-manager to remotely manage a single host works as well. Not everything needs to be a rack server with crazy hardware specifications.

If you’re just looking for a low-hassle solution to deploy VMs and containers I think information about Proxmox should be on your reading list. Or if you’re looking for something crazy to sink your teeth into here’s a variety of thoughts on the subject:

MicroSnap: Canonical’s snap packages are making it easier than ever to deploy complex pieces of software. If you’re looking for something with a strong API to provision and create resources, take a look here. Also, be aware that at the time of this writing it’s in Beta.
Kubevirt: Kubernetes is the dominating platform for container orchestration and deployment. Kubevirt leverages KVM for the ability to deploy VMs via the kubectl utility.
Harvester: Harvester is an interesting project that looks to solve those that need VMs and containers deployed in their bare-metal environments. Similar to Kubevirt, but maintained by Suse. It’s an early project, but looks to be very interesting.

One thing I see a lot of that I don’t personally think is worth the investment is the Raspberry Pi Kubernetes cluster. is it neat? Sure, but is it expensive? Yes. Unless you already have the Raspberry Pis kicking around at home, I personally don’t think it’s worth the investment. If you’re looking to focus specifically on an ARM cluster, that’s another reason to go the Kubernetes on Pi route, but if you’re looking to learn Kubernetes there are several other solutions (1 2 3) that in my opinion make more sense.

Beyond The Blog - Two Years and Beyond

As I come to my two year anniversary I still am as excited to write up these random projects and share with the world. Whether you’re just starting out with some Raspberry Pis or have been homelabbing for quite some time I hope you found this helpful.

Reach out to me on Twitter!