Tower of Kubes

Home Assistant on Kubernetes

2025-11-13T00:00:00Z

Today I learned Home Assistant can run on K8s using this Helm Chart: pajikos/home-assistant-helm-chart: Helm Chart for Home Assistant

Quote

This Helm chart bootstraps a Home Assistant instance on Kubernetes, supports configurable persistence, controller types, add-ons (e.g. code-server), and is auto-updated with new Home Assistant releases.

Telegram: View @KubeBuilders

My Opinion

For over two years, I have been running Home Assistant on Home Assistant Green, which comes pre-installed with Home Assistant OS.

The device has been working perfectly well for all of my smart home needs. Even though it is not the most cost-effective way to run Home Assistant, it is a well-designed device, fast enough for my needs and power efficient.

If I were buying a new dedicated device for Home Assistant today, I may have preferred to get a mini PC instead, since some mini PCs are similar in price to the HA Green but significantly more powerful (though maybe not as power efficient). However, I would still strive to run Home Assistant with Home Assistant OS.

Why standalone device for Home Assistant

On recent podcast episodes of Linux Unplugged (including LINUX Unplugged 637: Chris’ Smart Home Disaster), Chris talked about considering a move away from the Home Assistant Yellow (which is more powerful than the HA Green), perhaps towards a mini PC running multiple services (rather than just a mini PC). Chris also debated the benefits of running Home Assistant on NixOS vs Home Assistant OS. Nevertheless, I tend to agree with Chris’s long-standing stance that it’s best to give Home Assistant its own device, because of how essential it can be to a home.

Why Home Assistant OS

I run all my other self-hosted services in containers. Why not Home Assistant as well? The reason is that Home Assistant OS makes everything easy. Notably, Home Assistant Container installations don’t have access to add-ons.

Quote

Add-ons are additional standalone third-party software packages that can be installed on Home Assistant OS. \[Learn more\]

Installation - Home Assistant

Although Add-ons are really just containers, and many Home Assistant users manage to install them as separate containers, this requires elaborate configurations to make the different containers work together with Home Assistant. Even though I’ve been doing Docker Compose stacks (for example, applications that have multiple containers including a database), the moment I found out that HAOS allows one-click installation of Add-ons, I immediately gravitated towards that simplicity. Some examples of Add-ons that I use and rely on are Matter Server, Zigbee2MQTT and Music Assistant.

Backups are also fairly simple on HAOS.

Benefits of the Home Assistant Helm Chart

Nevertheless, I do find the idea of this Home Assistant Helm Chart compelling. Features such as replicas and partial add-ons support make this an interesting alternative to HAOS. I may run a test deployment in my parent’s home, since that’s where my homelab cluster is.

Featured image by Jakub Żerdzicki on Unsplash.

Istio Gateway

2025-10-19T00:00:00Z

Istio as a Gateway and Ingress Controller

Istio is known for its service mesh capabilities, however it can also serve as a Gateway and Ingress Controller, with support for both Ingress resources and Gateway API resources. Some view this use-case as overkill. However, my own testing of using Istio exclusively as a gateway (without a service mesh) proves that it can in fact work quite well for this purpose. This is strengthened by the benchmarks done by Howard John.

My Rationale for using Istio

Here’s one potential reason I found to use Istio for ingress/gateway instead of Envoy Gateway: Coraza WAF.

I was previously using ingress-nginx which has easy-to-enable support for ModSecurity and OWASP CRS (Core Rule Set). Since ingress-nginx is planned to eventually be replaced with InGate, I decided to look at the currently available Gateway API implementations and what WAF (Web Application Firewall) support they have. I found out that the more modern alternative to ModSecurity is OWASP Coraza WAF. From my research it seems to be able to use Coraza with Envoy Gateway you have to use Tetrate Enterprise Gateway.

However, upon further research I found this OpenShift guide: Creating a Web Application Firewall in Red Hat OpenShift. This guide uses Coraza Proxy WASM with Istio. Istio seems to be required in order to be able to use the WasmPlugin custom resource. I believe that following this guide it should work with Istio on non-OpenShift K8s just the same.

My Installation

I can confirm this works even without OpenShift! I tested this on my Talos staging cluster. Once Istio is installed and configured with the WASM Plugin for OWASP Coraza WAF, test malicious requests get blocked as expected.

The difficult part for me was getting Istio installed and figuring it out how to configure it as a gateway for Ingress and HTTPRoute resources. I wanted to avoid using the more advanced features of Istio (service mesh, ambient mesh etc.), at least for now. I have not used Istio before so there was a learning curve, certainly more complex than ingress-nginx. However, once I got Istio working as a gateway like I wanted, applying the WASM Plugin was relatively straightforward.

This is the solution that I am now using for my “homelab-as-code” Talos cluster.

Istio Gateway Installation

The main resources which I followed are:

I followed these resources, then adapted them for my own Argo CD GitOps structure and made them work with my existing adyanth/cloudflare-operator and cert-manager deployments. I used some Istio custom resources to make the same Istio Gateway work with both Ingress resources and Gateway API resources. This essentially made Istio a drop-in replacement for my previous ingress-nginx deployment (any existing Ingress resources now use Istio as the default ingress class), with the added ability to now use Gateway API.

Benchmarks

Howard John works on Istio, so is not entirely without bias (which he admits). Nevertheless, he has created Gateway API Benchmarks, a common set of tests to evaluate a Gateway API implementation. Istio comes out quite favorably in the benchmark ("✅ No issues were found"): howardjohn/gateway-api-bench: Gateway API Benchmarks provides a common set of tests to evaluate a Gateway API implementation.

UPDATE: John Howard has released Gateway API Benchmarks - Part 2. According to the new benchmarks, Istio is still among the leading Gateway API implementations; however, the new Agentgateway has better performance in the Route Scale and ListenerSet Scale benchmarks.

Featured image by Sam Moghadam on Unsplash.

KYAML

2025-08-11T00:00:00Z

Today I learned, in Kubernetes v1.34, kubectl will also support a new strict subset of YAML called KYAML.

Resources

Shell Script

I coded a simple script to convert all Kubernetes manifests in a directory from YAML to KYAML.

Initially, I wanted to code my own converter, but then found out that the upstream Kubernetes project already has a new yamlfmt tool (different from google/yamlfmt).

#!/bin/sh
# kyamlify.sh — Rename *.yaml -> *.kyaml then format to KYAML (POSIX)
# Usage: ./kyamlify.sh [ROOT_DIR]
# Env: YAMLFMT_VERSION (default: master)

set -eu

ROOT_DIR="${1:-kubernetes}"
YAMLFMT_VERSION="${YAMLFMT_VERSION:-master}"

# Require Go
command -v go >/dev/null 2>&1 || { echo "error: Go not found in PATH" >&2; exit 1; }

echo "→ Installing yamlfmt @ $YAMLFMT_VERSION"
go install "sigs.k8s.io/yaml/yamlfmt@${YAMLFMT_VERSION}"


echo "→ Formatting all YAML files under ${ROOT_DIR} as KYAML"
find "${ROOT_DIR}" -type f -name '*.yaml' -print0 \
  | xargs -0 -n1 "$(go env GOPATH)/bin/yamlfmt" -o kyaml -w

KYAML Rules

Quote

KEP-5295 introduces KYAML, which tries to address the most significant problems by:

Always double-quoting value strings
Leaving keys unquoted unless they are potentially ambiguous
Always using {} for mappings (associative arrays)
Always using [] for lists

Support for KYAML: a Kubernetes dialect of YAML | Kubernetes v1.34 Sneak Peek | Kubernetes

These rules are similar in practice to JSON5. However, while JSON5 is a superset of JSON (as well as a subset of ES5), KYAML is a subset of YAML.

In fact, I suspect that by adding --- to the first line of a JSON5 file, it would be valid KYAML.

By the way, starting the file with --- is required for KYAML (while it’s optional in YAML).

Experimentation and Additional Observations

I was initially excited about converting all my Kubernetes manifests to the “safer” KYAML format.
I ran my script then followed it by running yamllint, which introduced a few warnings post-conversion.
After fixing all yamllint warnings, I had well-formatted KYAML files.
I considered whether to rename all converted manifest files to use a *.kyaml suffix. I decided against this since I couldn’t find any evidence of this file extension.
KYAML files are 100% valid YAML files, and work with existing tooling. This includes existing Kubernetes versions and tooling.
The main thing introduced with Kubernetes v1.34 is a kubectl get -o kyaml option.
Keeping the *.yaml file extension makes sense since KYAML is still valid YAML and existing tools expect *.yaml or *.yml file extensions, not *.kyaml
After running the script, fixing formatting, and deciding to keep the filenames the same, I could add all modified files in homelab-as-code to a new kyaml branch and make a commit.
I considered opening a Pull Request, however, am still undecided.
My main consideration is whether the KYAML format would impact usability, making it harder for me to write and edit manifests.
I am not sure whether KYAML solves any real problems for me.
I understand YAML limitations but know how to avoid them by quoting values when needed, using linting and formatting tools (manually, with pre-commitand in CI).

My Opinion on the Format

In a way, KYAML is itself “yet another markup language” (despite using existing YAML rules). It is far from the first solution to problems with existing markup languages.

One notable limitation of standarad JSON is no comments. Both JSON5 and Microsoft’s JSONC (JSON with comments, primarily used in VS Code’s setttings.json file) previously addressed this. KYAML has the benefit of being a subset of YAML and designed to work with all existing YAML tooling.

In theory, KYAML could be a “safer” way to write production-grade manifests. However, this was already possible to do with JSON files. Kubernetes manifests can all be written in JSON, but there is a reason that this is rarely done in practice.

JSON files are arguably less readable and harder to work with (for humans, not machines) than YAML. At the same time, JSON files are very much machine-parsable with a lot of existing tooling like jq (though YAML tooling exists as well).

In imitating JSON but staying YAML, KYAML can feel like the worst of both, rather than the best of both world. Not as clean as JSON, and not as “human-readable” as YAML.

Featured image by Marvin Meyer on Unsplash.

The Ultimate Guide to Kubernetes Books: From Beginner to Certified Expert

2024-06-26T00:00:00Z

While studying for the Certified Kubernetes Administrator certification last year, I wanted to take a deep dive into Kubernetes topics, both those covered by the certification and beyond.

As a develeap employee, I had access to all of develeap’s courses which I used to learn more about Kubernetes. Alongside that, I wanted to supplement my learning with books and practice labs. After I earned the CKA certification, I decided to review the field of Kubernetes books, labs, and courses. In part one of this article series I reviewed several of the CKA courses I took in Review of CKA Courses. In part two, I reviewed Kubernetes practice labs and mock exams in Review of Kubernetes Practice Labs. In this part (part three), I will review several of the books and written materials that I have read while learning about Kubernetes.

The Book of Kubernetes

Author: Alan Hohn

Publisher: No Starch Press

Publication Date: August 2022

Link: No Starch Press | Amazon | Penguin Random House | O’Reilly

The Book of Kubernetes (not to be confused with The Kubernetes Book by Nigel Poulton, which I have not read) is written by Alan Hohn and published by No Starch Press. I was already a fan of No Starch Press, having read many of their previous books, and this book was brand new when I first read it, so it was naturally my first choice when wanting to read more about Kubernetes.

This book serves as an introduction to Kubernetes. Although it assumes some sysadmin knowledge (and gets quite technical fairly fast), you don’t need to know anything about containers, as everything is explained—especially in the first part.

The book is split into 20 chapters across three parts. The first part, “Making and Using Containers,” serves as a fairly comprehensive overview of what containers actually are, as well as their benefits. In fact, I would go as far as to say this book might be the best explanation I’ve read about containers; too often I see simplified explanations about containers boiled down to “containers are like virtual machines but not really”. This book actually goes through the effort of explaining the technical underpinnings of containers, and how they use the fundamental capabilities of the Linux kernel, like namespaces and cgroups, to provide containers process isolation, resource limiting and network namespaces. This is first explained using Docker, the most used container tool, but later on the chapter shows how to achieve the same results using containerd as well as CRI-O and crictl. Showing how the same container image can be run through different container runtimes and tools, helped me understand the universal nature of OCI containers and the way Kubernetes can interface with containers through the CRI protocol.

The next two parts, “Containers in Kubernetes” and “Performant Kubernetes,” focus on many important Kubernetes topics. Along the book are examples which you can deploy yourself from the book’s GitHub repo. These are automated exercises that can be deployed using Ansible and run on either Vagrant (virtual machines) or AWS. These examples are very useful for hands-on understanding of the topics, and having them all automated while offering different ways to deploy – makes them very useful.

I will note that this book isn’t specifically designed to prepare readers for the Certified Kubernetes Administrator (CKA) exam. While it covers many relevant topics, there are some key CKA subjects not included, and conversely, some topics explored here might not directly align with the CKA curriculum (for example, Custom Resource Definitions). This, however, isn’t a drawback – the book excels in its own right, just keep this in mind if your primary goal is CKA certification.

Certified Kubernetes Administrator (CKA) Study Guide

Author: Benjamin Muschko

Publisher: O’Reilly Media, Inc.

Publication Date: June 2022

Link: O’Reilly | Amazon | eBooks.com

My one “complaint” about the No Starch book was that it was not a CKA study guide (it was not really a complaint, just an observation). Well, this book is! After No Starch Press, my other favorite publisher for technical books is O’Reilly, which has maintained a high standard for decades.

This book has everything you’d expect from a study guide. There are detailed explanations of everything you need to know in the CKA curriculum. There are useful tips for the exam and practice questions which are at a good level. For some of the practice questions you get automated Vagrant environments from the book’s GitHub repo, however for many questions you will have to set up a practice cluster yourself (or use Killercoda).

In some places the book mentions optionally using Kubernetes environments in Katacoda. This was a service offered by O’Reilly, unfortunately public use of Katacoda was recently shut down. This decision must have happened around the same time this book was published, but due to the unfortunate timing, Katacoda is still mentioned in this book. Fortunately this book does not really rely on using Katacoda to understand its material, it is merely offered as another option for practice. Now, Killercoda is available as a form of replacement for Katacoda (reviewed below).

Overall this book serves its purpose well as a written CKA study guide.

Learning CoreDNS

Author: John Belamaric, Cricket Liu

Publisher: O’Reilly Media, Inc.

Publication Date: September 2019

Link: O’Reilly | Amazon | eBooks.com

This book is, of course, about CoreDNS, the modern DNS server used for Kubernetes. Despite being about CoreDNS, it also focuses on Kubernetes networking and service discovery. It’s a good read for anyone who wants to understand Kubernetes networking on a deeper level. It’s more detailed than what is required to know for the CKA exam, but more knowledge is always good.

The book starts by explaining what CoreDNS is and why a new DNS for the container and micro-service was needed. It compares CoreDNS with BIND 9 (a more traditional DNS server), and explains the pros and cons of each. Spoiler: CoreDNS is not necessarily a replacement for BIND 9, they have different usages.

The book then dives deeper into how CoreDNS works with Kubernetes and etcd for service discovery, and delves into modern workflows and best practices (including Prometheus monitoring).

If you want to understand CoreDNS well, then there is probably no better book. For learning Kubernetes networking specifically for the CKA exam, this book is not the best option as it focuses on concepts that are not that important for the exam itself, while at the same time not delving too deeply into concepts which are important (such as Kubernetes Ingresses).

Kubernetes Documentation

Link: https://kubernetes.io/docs/

Of course, this is not a book but a website, one that everyone working with Kubernetes must be familiar with—and especially anyone taking one of the certification exams where this resource is allowed.

The Kubernetes Documentation has tons of relevant information; almost everything you would need to know for the CKA and CKAD exams is included in these docs. Of course, the documentation goes beyond what you need to know for the exams. If you want to become an expert on everything you can do with “stock” Kubernetes, you could read the entire documentation. Or you can focus on reading the sections that are most relevant to you.

Since I used other resources to study for the CKA certification, I did not read the documentation cover-to-cover.¹ I did, however, use the documentation frequently to reiterate the concepts that I learned, as well as during practice labs and mock exams. Whenever I needed to find out how to do something in Kubernetes, rather than searching Google, I searched through the Kubernetes Documentation first. For example, say I wanted to find a YAML template for deploying a PersistentVolumeClaim; I would go into the Kubernetes Docs and type “PVC” into the search bar, then click “ Configure a Pod to Use a PersistentVolume for Storage”, which explains PVCs and other relevant concepts.

I found this learning method to be very effective. When preparing for the CKA/CKAD exams, you should be adept at finding relevant information from the documents quickly and efficiently.² This is why I recommend using the documents as much as possible while studying. You don’t have to read everything in the documents, but you should be familiar with the content they include and how to find it using the built-in search. Configure a Pod to Use a PersistentVolume for Storage, I highly recommend exploring the Kubernetes Docs yourself and seeing what they offer. There are sections explaining common tasks, a One-page API Reference for Kubernetes v1.30, and even Tutorials.

Featured image by 🇸🇮 Janko Ferlič on Unsplash.

In fact the Kubernetes Docs can’t be read cover-to-cover because they are not a book and therefore lack a cover. ↩︎
The CKA exam has its own way of scoring things and is not public. You get a score sometime after completing the exam, but you won’t know which questions you answered correctly. ↩︎

Review of Kubernetes Practice Labs

2024-06-26T00:00:00Z

While studying for the Certified Kubernetes Administrator certification last year, I wanted to take a deep dive into Kubernetes topics, both those covered by the certification and beyond.

As a develeap employee, I had access to all of develeap’s courses which I used to learn more about Kubernetes. Alongside that, I wanted to supplement my learning with books and practice labs. After I earned the CKA certification, I decided to review the field of Kubernetes books and practice labs. In part one of this article series I reviewed several of the CKA courses I took in Review of CKA Courses. In this part (part two), I reviewed Kubernetes practice labs and mock exams. In part three, I share my take on written materials in The Ultimate Guide to Kubernetes Books: From Beginner to Certified Expert.

Killercoda

Link: Killercoda CKA Area

CKAD Equivalent: Killercoda CKAD Area

Killercoda offers free labs! Although Killercoda does offer subscriptions, the paid features which are currently included are not major. I’d say the main reason to consider a subscription is if you are using Killercoda frequently and want to support what they are doing. Honestly, what they are doing is excellent and worth supporting.

With a free account you can access various different environments and scenarios. The ones offered are very good and worth doing. I especially like the CKA Playground environment, designed to be similar to the current environment used by the CKA exam. You can use the Killecoda playgrounds to freely play around with Kubernetes, without being constrained to a specific scenario. This is really useful, for example if you just want to try out some commands to see what they do.

In addition, the environments load fast. This is impressive considering this is all available on the free tier.

Killer Shell

Link: Killer Shell CKA Simulator

CKAD Equivalent: Killer Shell CKAD Simulator

Killer Shell is created and maintained by the same team as Killercoda (led by Kim Wüstkamp), however they have notable differences. You can read their own comparison here; to summarize, while Killercoda allows you to practice exam topics, it is not an exam simulator.

Killer Shell is an exam simulator. In fact, it is the official exam simulator for the CKA, CKAD and CKS certifications (and soon the LFCT certification as well). Although using Killer Shell on its own is not free, you get two sessions included when signing up for any of these certifications.

Each session lasts 36 hours and includes an exam simulation that is designed to closely mirror the real exam environment. This means you get not only a terminal, but also a full XFCE desktop to work in. You get to work on 25 questions (plus a few extra questions) across several different Kubernetes clusters. You have a two-hour timer. However, after the two hours pass, you can continue to answer the questions at your own pace (up to the 36-hour time limit of the session).

To be honest, it’s very hard to answer all 25 questions within the two-hour limit… The questions in the Killer Shell exam are designed to be slightly harder than in the real exam. In addition, in the real exam, you are likely to get less than 25 questions to answer within the two-hour limit. My advice is not to focus too heavily on the time limit. However, do make sure you can answer all questions, even if it takes you longer than two hours. Nevertheless, you should be trying to find ways in which you can work fast and most efficiently; for example setting aliases and environment variables, and learning Vim shortcuts. These tricks will help you in the real exam, and the exam simulation is the best place to practice.

After the two-hour timer passes (which you can also end early), you get to see an automated scoring system that checks which questions you answered correctly. The exam, however, still continues in the background, and you can go back to questions and fix your answers. I noticed the automated scoring system sometimes takes a few minutes to update, which led to some confusing situations where I tried to fix an answer but had to wait a few minutes to confirm I answered correctly. Another nitpick is the automated scoring system is a bit overly strict in the way it likes its answers. For example, in one of the early questions, you have to write a shell script to do a simple task; this can be done in several ways, but in practice, the script has to be written in the exact way that the automated scoring system expects (you can’t even use a shebang, as a friend of mine found out while trying out Killer Shell). I don’t know if the real exam is as strict with its scoring as Killer Shell.

In the real CKA exam you don’t get to see which questions you answered correctly, that’s why you should practice checking your own answers. After answering questions, you should verify that you did things correctly using different commands. Again this is something which of course can be practiced within the Killer Shell environment.

As I mentioned, you get two Killer Shell sessions after signing up for a certification. It’s worth noting that both sessions will have the same questions. In my case I found that one session was sufficient for preparing for the real exam. However if you didn’t do well in the first session, then that’s a good sign you should go back to practice and learn the concepts that you struggled with. Once you are able to get a good score at Killer Shell, you will know you are ready for the exam.

As for when to take the Killer Shell exam, it is often recommended to take it relatively close to the real exam (say within one week), after you have thoroughly learned using other resources. If you are already experienced in using Kubernetes and wondering if you need to spend time on courses, you could test yourself by doing a Killer Shell simulation session first and see how well you do.