<![CDATA[Alex Ellis' Blog]]>https://blog.alexellis.io/https://blog.alexellis.io/favicon.pngAlex Ellis' Bloghttps://blog.alexellis.io/Ghost 1.21Tue, 10 Aug 2021 08:35:18 GMT60<![CDATA[Two year update: Building an Open Source Marketplace for Kubernetes]]>https://blog.alexellis.io/kubernetes-marketplace-two-year-update/610c0063d34c2900015a8b3aThu, 05 Aug 2021 16:34:50 GMTTwo year update: Building an Open Source Marketplace for Kubernetes

It's 22 months since I found myself frustrated with writing boilerplate instructions to install simple, but necessary software in every tutorial I wrote for clients and for my own open source work.

In this article post I'll walk you through the journey of the past two years from the initial creation, through to growing the community, getting the first sponsored app and what's next. There will be code snippets, and technical details, but there should be something for everyone as we celebrate the two year anniversary of the project.

Two year update: Building an Open Source Marketplace for Kubernetes

A timeline of the various open source projects that I've started. They all have one thing in common: a focus on developer experience and community.

The prototype

In 2019, Helm 2 was still widely used and I had set out on my own path to build independent software and help companies reach developers on a full-time basis.

But for one of my earliest clients, I wasn't supporting them with OpenFaaS, I was building deeply technical content for their blog and finding that the Kubernetes ecosystem made this incredibly repetitive and fragmented.

Do you want a secure, TLS-enabled registry, with Ingress Nginx, cert-manager and Docker's open source registry? It ended up being 5000 words, and every other tutorial tended to need K3s or Kubernetes installing, followed by ingress-nginx or traefik and cert-manager.

I thought that a dedicated CLI would remove duplication and help with sourcing applications, rather than scrolling through huge Helm chart README files and guessing good defaults.

I can still remember it now. I didn't want to create another CLI that I'd have to maintain, release and update, so asked around and Duffie Cooley suggested extending k3sup, so I did.

k3sup is a CLI that I had written for a similar reason - to make the developer experience better for bootstrapping K3s. Installing K3s was also rather too manual and had meant I was repeating boilerplate instructions in dozens of blog posts.

commit 0b4997e408d5b2dc101f16c317f1fd946e3f91b1
Author: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>
Date:   Tue Sep 24 12:35:28 2019 +0100

    Add prototype for installing apps
    Installs OpenFaaS, but not on armhf, optional loadbalancer
    Pre-reqs are not checked i.e. helm/tiller.
    Signed-off-by: Alex Ellis (OpenFaaS Ltd) <alexellis2@gmail.com>

 main.go         |   3 +++

The first commit to k3sup to add a prototype app for openfaas. Creating an app for OpenFaaS in k3sup meant that we didn't have to create our own installer and that was a bonus.

k3sup was created to install K3s over an SSH connection to build Kubernetes clusters, and I wanted it to have a user-experience like Docker Swarm:

$ docker swarm init

Your join token is $SECRET

$ docker swarm join --join-token $SECRET

So it ended up looking like this:

k3sup install --ip --user pi
k3sup join --server-ip --ip --user pi

Then, the app functionality just became a sub-command:

k3sup app install openfaas
k3sup app install kubernetes-dashboard
k3sup app install mongo

I added an info command printed out curated instructions on how to get started:

k3sup app info openfaas

If you'd installed Minio, it would tell you how to access the service, and how to obtain the generated username and password. If you'd installed Postgresql, it would explain where to find the connection string.

After a few months, a few people (including my friend Stefan Prodan started to say: "why would I use your marketplace to install apps? I don't use a Raspberry Pi! I don't care about K3s!"

Well that was unexpected.

k3sup app install worked against any kind of Kubernetes cluster, even managed engines like EKS and GKE. I'd added support for Raspberry Pi clusters because that was something that other tooling neglected and I knew the community needed this.

The creation of arkade

So arkade was born:

  • k3sup app install -> arkade install
  • k3sup app info -> arkade info

I wrote up why we needed arkade, and since most of the contributors were from the OpenFaaS community, published it on the blog:

Why did the OpenFaaS community build arkade and what's in it for you?

arkade lives at github.com/alexellis/arkade. Why isn't it in the openfaas organisation? For the same reason that the app is now called arkade instead of k3sup, I know people will assume it can only be used with openfaas.

Why isn't it in its own organisation? Because I don't want people to detach the work required to make this possible from human beings, from maintainers. Funding is required, it's not trivial to build an ecosytem and to motivate and mentor a community. You'll hear a little more on this towards the end of the article where I mention Sponsored Apps.

The main points from the blog post were:

  • It was more secure than Helm 2 because it used templating instead of relying on Tiller
  • It helped you discover the options you actually cared about for charts
  • There was no more searching through docs and GitHub repos to find out how to install software
  • It made your next action obvious through the info command
  • It just worked for multi-arch
  • It put everything under one roof - Apps, plus the CLIs required to drive them

The thing I was most excited about was composite apps, the ability to get a compound benefit. A blog post I wrote for Civo, a cloud provider in the UK needed 5000 words to show how to deploy a TLS-enabled registry to Kubernetes using Cert-Manager.

Arkade meant that could be condensed down to 4 commands:

arkade install ingress-nginx
arkade install cert-manager
arkade install docker-registry
arkade install docker-registry-ingress \
  --email $EMAIL \
  --domain $DOMAIN

docker login $DOMAIN

But what about GitOps?

Hardcore GitOps fans couldn't quite understand why arkade was needed, they criticised it as an imperative tool. They asked why they would use it in production instead of FluxCD or Argo

Here's the sweet spot for arkade:

You need to install some apps, quickly. Probably in development, maybe in production. At the end of the day, it will use the official CLI of a project like Istio or Linkerd, or its official helm chart, the end result being that it installs the app in the recommended way.

GitOps tools are wonderful things, keep on using them, but it's unlikely that they will fit your needs for local development in the way that arkade can.

Don't take my word for it. I've included a few notes from SREs and DevOps engineers later on the article.

The community - more than the sum of its parts

And with arkade, came the community, and new contributors who also wanted to make Kubernetes less repetitive.

Ustav Anand designed a colourful logo for us, which is something I don't think I could have done on my own.

Two year update: Building an Open Source Marketplace for Kubernetes

Alistair Hey started contributing in the early days of arkade and added an SDK to make writing apps even easier and less repetitive:

	postgresql.RunE = func(command *cobra.Command, args []string) error {
		kubeConfigPath, _ := command.Flags().GetString("kubeconfig")
		updateRepo, _ := postgresql.Flags().GetBool("update-repo")

		arch := k8s.GetNodeArchitecture()
		fmt.Printf("Node architecture: %q\n", arch)

		if arch != IntelArch {
			return fmt.Errorf(OnlyIntelArch)

		ns, _ := postgresql.Flags().GetString("namespace")

		if ns != "default" {
			return fmt.Errorf("please use the helm chart if you'd like to change the namespace to %s", ns)

		persistence, _ := postgresql.Flags().GetBool("persistence")

		overrides := map[string]string{}

		overrides["persistence.enabled"] = strings.ToLower(strconv.FormatBool(persistence))

		customFlags, err := command.Flags().GetStringArray("set")
		if err != nil {
			return fmt.Errorf("error with --set usage: %s", err)

		if err := mergeFlags(overrides, customFlags); err != nil {
			return err

		postgresqlAppOptions := types.DefaultInstallOptions().

		_, err = apps.MakeInstallChart(postgresqlAppOptions)
		if err != nil {
			return err

		return nil

You'll notice that flags with strong types are used to populate values in helm charts, and we personally curate and bubble up what we think is going to be important, from dozens of options in the values.yaml files.

func MakeInstallPostgresql() *cobra.Command {
	var postgresql = &cobra.Command{
		Use:          "postgresql",
		Short:        "Install postgresql",
		Long:         `Install postgresql`,
		Example:      `  arkade install postgresql`,
		SilenceUsage: true,

	postgresql.Flags().Bool("update-repo", true, "Update the helm repo")
	postgresql.Flags().String("namespace", "default", "Kubernetes namespace for the application")

	postgresql.Flags().Bool("persistence", false, "Enable persistence")

	postgresql.Flags().StringArray("set", []string{},
		"Use custom flags or override existing flags \n(example --set persistence.enabled=true)")

Now it turned out that most people installing apps to Kubernetes clusters also needed dozens of CLIs - kubectl, helm, linkerd, kubectx, faas-cli to name a few.

There are die-hard fans who will refuse to use anything other than brew, but brew is anything but instant. Have you got 20 minutes whilst we update that catalog from GitHub? Oh you got rate-limited, better issue a personal access token and let brew use that so it can scrape the GitHub API. If you're unlucky, then brew will also download reams of dependencies and build them all from scratch, when all you wanted was a simple Go binary, like now.

That's when I came up with arkade get, it simply downloads CLIs from GitHub releases, but it wasn't as easy as I thought. Everyone names their CLIs a different way, and there are no standards.

I solved this by introducing Go templates, which can also be unit tested to make sure they produce the correct URLs.

arkade get also has to contend with archives like zip files and non-standard naming of binaries vs release names, when you get /releases/v0.1.0/name_0.1.0.zip and suddenly have to remove that extra v in the filename, despite it being the canonical release name.

Then there's multi-arch and multi-OS, arkade automates all of that for you.

Here's the code to define a multi-arch, multi-OS download for faas-cli:

	tools = append(tools,
			Owner:       "openfaas",
			Repo:        "faas-cli",
			Name:        "faas-cli",
			Description: "Official CLI for OpenFaaS.",
			BinaryTemplate: `{{ if HasPrefix .OS "ming" -}}
{{- else if eq .OS "darwin" -}}
{{- else if eq .Arch "armv6l" -}}
{{- else if eq .Arch "armv7l" -}}
{{- else if eq .Arch "aarch64" -}}
{{- else -}}
{{- end -}}`,

And this is the Go template for Helm, which uses an archive for its download:

	tools = append(tools,
			Owner:       "helm",
			Repo:        "helm",
			Name:        "helm",
			Version:     "v3.5.2",
			Description: "The Kubernetes Package Manager: Think of it like apt/yum/homebrew for Kubernetes.",
			URLTemplate: `{{$arch := "amd64"}}

{{- if eq .Arch "armv7l" -}}
{{$arch = "arm"}}
{{- end -}}

{{- if eq .OS "linux" -}}
	{{- if eq .Arch "aarch64" -}}
	{{$arch = "arm64"}}
	{{- end -}}
{{- end -}}

{{$os := .OS}}
{{$ext := "tar.gz"}}

{{ if HasPrefix .OS "ming" -}}
{{$os = "windows"}}
{{$ext = "zip"}}
{{- end -}}


What about the unit tests? These verify that the URLs are being produced correctly. Here's how we handle kubectl itself, which uses a custom download URL and needs a special URL template since the binaries are not on GitHub:

func Test_DownloadKubectl(t *testing.T) {
	tools := MakeTools()
	name := "kubectl"

	var tool *Tool
	for _, target := range tools {
		if name == target.Name {
			tool = &target

	tests := []test{
		{os: "darwin",
			arch:    arch64bit,
			version: "v1.20.0",
			url:     "https://storage.googleapis.com/kubernetes-release/release/v1.20.0/bin/darwin/amd64/kubectl"},
		{os: "linux",
			arch:    arch64bit,
			version: "v1.20.0",
			url:     "https://storage.googleapis.com/kubernetes-release/release/v1.20.0/bin/linux/amd64/kubectl"},
		{os: "linux",
			arch:    archARM64,
			version: "v1.20.0",
			url:     "https://storage.googleapis.com/kubernetes-release/release/v1.20.0/bin/linux/arm64/kubectl"},

	for _, tc := range tests {
		got, err := tool.GetURL(tc.os, tc.arch, tc.version)
		if err != nil {
		if got != tc.url {
			t.Fatalf("want: %s, got: %s", tc.url, got)

It's hard to say how many people contributed apps and tools to arkade, because the code used to live in K3sup, but I'll have a go:

  • k3sup has 42 contributors
  • arkade has 39 contributors

That's a lot of individual PRs and commits that I've merged. I like to encourage contributors to join OpenFaaS Slack, to learn and if they are open to it, I'll coach them and get their input into how the project should work.

Ramiro Berrelleza of Okteto added a progress bar which helps with larger binaries like kubectl:

$ arkade get kubectl
Downloading kubectl
38.37 MiB / 38.37 MiB [-----------------------------] 100.00%
Tool written to: /home/alex/.arkade/bin/kubectl

# Add (kubectl) to your PATH variable
export PATH=$PATH:$HOME/.arkade/bin/

# Test the binary:

# Or install with:
sudo mv /home/alex/.arkade/bin/kubectl /usr/local/bin/

Yankee came up with the idea of adding a table for downloads, and I think it looks great to see the data represented that way:

|       TOOL       |                         DESCRIPTION                          |
| argocd           | Declarative, GitOps continuous delivery tool for Kubernetes. |
| argocd-autopilot | An opinionated way of installing Argo-CD and managing GitOps |
|                  | repositories.                                                |
| arkade           | Portable marketplace for downloading your favourite devops   |
|                  | CLIs and installing helm charts, with a single command.      |
| buildx           | Docker CLI plugin for extended build capabilities with       |
|                  | BuildKit.                                                    |
| civo             | CLI for interacting with your Civo resources.                |
| cosign           | Container Signing, Verification and Storage in an OCI        |
|                  | registry.                                                    |
| docker-compose   | Define and run multi-container applications with Docker.     |
There are 63 apps, use 'arkade get NAME' to download one.

There are too many contributors to mention individually, but each of them has had an impact in their own way. Thanks also to Engin Diri, Carlos Panato and Batuhan Apaydın.

Sponsored Apps

My intent with arkade was to create something that solved pain in the Kubernetes ecosystem, and made life better for developers.

If you use arkade as an individual, you can sponsor me via GitHub so I can continue this work.

I also wanted to be cautious of my prior mistakes with Open Source, as Alexis Richardson first told me at KubeCon Austin in 2017: "if your software is too easy to use, never crashes and is feature complete, nobody will pay for support"

With arkade, I wanted to offer value to companies where I would develop a dedicated app for them, to improve the onboarding experience.

Why is that important? If you can lower the barrier to entry, it can encourage experimentation, and I believe that "mean time to dopamine" is an important measure for developer tools.

So the first Sponsored App was for Venafi, you may have heard of them when they acquired JetStack, the authors of cert-manager. Venafi care about machine identity - that is to say, certificates for everything and have their own add-ons for cert-manager too.

Here's the issue and linked PR and the press release which walks you through how the app works and what pain it solved in the onboarding process.

I'm also excited to confirm that Kasten have purchased a Sponsored App bundle which includes their CLI apps being added to arkade, and their Kubernetes backup solution as an app. With each Sponsored App, I write up a blog post and in some circumstances will put on a webinar.

The idea is to make things easier, and then get feedback, sign-ups, leads and to be a net win for arkade and the sponsor.

What are people saying about arkade?

Here's what Ivan Velichko, SRE @ Booking.com has to say about arkade:

I was setting up a new dev environment yesterday. Kind, helm, kustomize, kubectl, all this stuff. My take is - arkade is highly underappreciated.
I'd spend an hour in the past to install such tools. With arkade it was under ten minutes.

Greg runs Fullstack JS and is a JavaScript developer, he says:

This is real magic get #kubernetes up and going in a second; then launch #openfaas a free better than lambda solution that uses docker images.

@arghzero says:

for getting the basics installed, nothing beats arkade
it can install commonly used cli tools like kubectl locally for you, as well as common k8s pkgs like ingress-nginx or portainer

@Yankexe says:

It's hard to use K8s without Arkade these days.
My team at @lftechnology absolutely loves it.

From Michael Cade @ Kasten

I finally got around to installing Arkade, super simple!
quicker to install this than the argocli standalone commands, but there are lots of handy little tools in there.
also, the neat little part about arkade, not only does it make it easy to install a ton of different apps and CLIs you can also get the info on them as well pretty quickly.

Alistair and I recorded a live-stream over the summer and talked about how arkade works, and how to add new apps:

Two year update: Building an Open Source Marketplace for Kubernetes

Watch now: Hands-On with Arkade: K8s Marketplace

Wrapping up

It's now been two years since I set out to simplify finding, configuring and installing apps for Kubernetes, I couldn't have got the project to where it is today without everyone's help.

There are 47 apps, but several open pull requests, so I expect to see that bump up to 50 by the end of the month. Are there more than 50 apps for Kubernetes? Of course, but we are building a focused marketplace, with high quality apps that are likely to be used and are driven by demand from users.

What does that demand look like? Well there are now 47,000 downloads of arkade and 108 releases and 2k GitHub Stars. Between 50 and 100 contributors, if we factor in those who hacked on the code whilst it was stil called k3sup.

The list of CLIs we can install is growing every week with 63 apps, all with unit tests and a Go template that can be extended easily when projects release an ARM binary for their software or start supporting Windows.

If you want to try out arkade, it's just one command away:

# You move the binary to $PATH
curl -sLS https://get.arkade.dev | sh
mv arkade /usr/local/bin/

# Or have the script move it for you
curl -sLS https://get.arkade.dev | sudo sh

Or if you're one of those people who don't like to run bash, feel free to read what the script does, or head over to the releases page

What can you learn?

I'd also encourage you not to rush out there and force your users to deal with YAML or JSON manifests. Arkade has shown that Go, with a suitable SDK is just as good as manifests and dynamic files, in fact it may even be better suited to the job than YAML, which everyone in DevOps loves to hate.

Onward journey

Do you want product feedback on your developer experience? Do you want to learn how to grow a community or open source project? Book a call with me

<![CDATA[The Internet is my computer]]>https://blog.alexellis.io/the-internet-is-my-computer/60f53911d34c2900015a8b22Tue, 20 Jul 2021 08:23:10 GMT

In 1984 John Gage of Sun Microsystems was credited as saying "The Network is the computer." Almost four decades ago, John had a vision of distributed systems working together to be greater than the sum of their parts.

In this article, I'll introduce I want to dig a little deeper into John's words and what that means for us today. We'll also look at a very practical use-case for the Internet is my computer through hosted IDEs which have a number of different benefits and users. As a CNCF Ambassador, I'll also be asking what role Kubernetes has to play here and giving my personal take on hosted IDEs.

More than the sum of its parts

The core idea was that, if you had your own computer or terminal, then you could tap into the resources such as CPU, memory and storage of others on the network. This would allow for individuals to process computationally hard problems without owning the necessary hardware.

Today, the Internet has opened possibilities that may have seemed like science-fiction in the 1980s. Public cloud has all but removed issues around capacity planning and server management, whilst retaining many of the benefits, and going as far as to offer per millisecond billing for resources that would have previously involved serious capital investment.

Moving Closer to Users, and to Hardware

I’ve known the Equinix Metal (formerly Packet) folks for a while. Our worlds crossed back when my interest in Docker and Raspberry Pi overlapped with their early entrance into bare-metal and Arm CPUs. Since then, Packet has joined Equinix, which is where much of the underlying infrastructure of the Internet — think fiber optic cables, hyperscaler networks, and wireless carriers — connects. I caught up with Jacob Smith (he co-founded Packet and now runs bare metal engagement and marketing for Equinix) on this concept of the internet as a my computer. Here’s what he shared:

"As workloads scale and interact with users distributed all over the world, we see them reach down and out. Down to get closer to the hardware, and out to get closer to users. This is made possible by better tooling and the rising tide of open source, which have abstracted complexity while still allowing developers to do increasingly powerful things."

"For instance, building and running your own IDE," added Jacob.

Hacking from my shed

Hacking on the k3sup project from my shed with an 11" iPad Pro, but there is more promise to hosted IDEs than this.

While we’re talking about bare-metal servers, this is not the "rack and stack it" of the 1990s and early 2000s. Packet developed an API-driven system to automate server provisioning and management for bare-metal. The culimation of this work is the Tinkerbell project, which Equinix has contributed to the Cloud Native Computing Foundation (CNCF)'s Sandbox, building upon other support they’ve quietly given to open source, like the CNCF Community Infrastructure Lab.


As we explore use-cases for a hosted IDE, bare-metal servers present an attractive opportunity: fully dedicated infrastructure that is hosted at the edge of the network paired with the automation and "cheap to get started" benefits of on-demand and API-driven cloud.

As a CNCF Ambassador and maintainer of various OSS projects for Kubernetes, I was also curious what role Kubernetes had to play here.

The most popular IDE

According to PYPL's index of IDE popularity, VSCode, which first appeared in 2015, was ranked as the most popular editor.

Features include support for debugging, syntax highlighting, intelligent code completion, snippets, code refactoring, and embedded Git. Users can change the theme, keyboard shortcuts, preferences, and install extensions that add additional functionality.

via Wikipedia

I started my career using Visual Studio and continued on for 10 years, before discovering alternative light-weight IDEs like Atom and VSCode. Visual Studio was the way to build and develop .NET applications with C# and there was no real alternative.

The lightweight IDEs were more similar to a text editor, and initial versions lacked many features of the more traditional products. I was attracted to the simplicity and responsiveness of working on a single file or folder at a time vs. an entire multi-project solution.

VSCode is built with the Electron framework. With electron, you can build cross-platform desktop apps with JavaScript, HTML, and CSS.

Can you see where this is going? Where else does JavaScript, HTML and CSS run particularly well? In a web browser. The Internet is my IDE.

The Internet is my IDE

I asked my Twitter followers what value they saw in a hosted IDE and fully expected to hear responses similar to John's initial vision in 1984 - gaining access to fast hardware and the near unlimited resources of the Internet.

It seems like there are a number of different personas for hosted IDEs ranging from the individual hacker, to the engineering manage, to large open source projects.

A hosted IDE immediately opens up possibitilies like working on an iPad, Chromebook or other thin terminal like a 35 USD Raspberry Pi. You can find out more in Joe Previte's article: What I learned from coding on an iPad for two weeks. Joe works at Coder, a company that specialises in hosted VSCode solutions.

A CTO of a small European business DM'd me and told me that using a hosted IDE for his remote team would reduce the Total Cost of Ownership (TCO) vs. sending out 2500 USD laptops to each new joiner.

Chris Aniszczyk, CTO of the CNCF told me that the Prometheus monitoring project has been using hosted IDEs as a way of launching a code-review environment. Here, we start to see convenience of a pre-built environment augmented with the vast resources of the Internet.

I've had my own prior experience of combining hosted IDEs with open source. In 2018, before the pandemic, a number of us from the OpenFaaS community were putting on free in-person events at conferences to teach our community workshop on serverless.

I will never forget the one event where it took the whole class 3 hours to deploy a Kubernetes cluster and install a couple of CLIs.

The setup was repetitive, and whilst it was a necessary skill, it detracted from the purpose of the event - to teach the capabilities and use-cases of OpenFaaS. So I used Coder's VSCode Docker image and a number of scripts so that we could give preinstalled VMs to students: workshop-vscode. The repo shows how to pre-install Docker and Kubernetes, and can be customised to your own projects too.

As a GitHub Star, I was given early access to GitHub's Codespaces program. For any given repository, I can launch a VM with Docker and other common developer tools preinstalled and start hacking immediately.

My first use of Codespaces on arkade Kubernetes marketplace

There are a couple of my friends that work for GitPod so I reached out to the founders to ask them more about their customers. Johannes told me that a number of their clients use costly Virtual Desktop Interfaces (VDIs) to make a locked-down environment available to their developers, to work comply with various regulations. GitPod runs in containers and means that it scales well for this use-case.

Augmented solutions

Marc Campbell of Replicated messaged me and told me how much of a fan his team were of hosted IDEs, but that they did things differently.

For Marc, a local IDE provides the best experience when it comes to responsiveness and keyboard bindings. Instead of hosting the IDE in the browser, Marc provisions a beefy VM or bare-metal host on public cloud and then plumbs the terminal to that machine. So he gets the best of both worlds - that initial promise of The Network is my Computer - bandwidth, CPU and memory, coupled with a responsive and native application.

Pictured: I love my bash keyboard shortcuts, but the browser wouldn't allow me to type in Control + W

David Flanagan (né McKay) of Equinix Metal invited me to record a live-stream and to hack on a Kubernetes Operator together. If you've ever used software like Zoom, then you know how it goes - you spend most of the time calling out line numbers and saying go up a bit, go down a bit, no there!

Watch now

Jump to 00:21:00 in the video to see our plan of action, then the shared editor experience.

This time was different - David enabled VSCode's live sharing setting and we both worked on an IDE running on our local computers, but we could both edit and scroll at once and keep in sync.

Your computer is the Internet

We are seeing increasing commoditisation of hardware and with that comes both lower costs and a more accessible user experience. This new wave of use-cases and value proposition for hosted IDEs is a refelection of that progress and from the responses I've seen, it looks like we are only just getting started.

If you'd like to try out a hosted IDE (like GitPod / Coder), or running a powerful on the cloud and a local browser (like Marc), then I've compiled some up to date links for you:

Kubernetes, the Operating System

VScode can be run as a stand alone process, or within a container using Docker, but things really start to get interesting when we scale and move to multi-user environments. As I mentioned earlier, GitPod, Coder and others are using Kubernetes to orchestrate containers and VMs to share resources efficiently.

I was told that Kubernetes not only helped slice up hardware more efficiently, but cutting edge projects like stargz can improve load times by starting a container before it has been fully downloaded. Projects like Falco integrate with eBPF and can be used to detect abuse or misuse within multi-tenant systems.

Try it out with bare-metal

Until the end of the year, new customers to Equinix Metal can get 200 USD of free credits to spin up their own cloud native IDE. Just use the code johngage, which Jacob created for this article. You'll be able to run the IDE 24/7 for a week (7 days x 24 hours x 1.00 USD/hr = $168) and see if it's for you. Not only can you find servers with hundreds of GB of RAM, but their bonded 2 x 10 Gbps uplink is faster than most of us could ever hope to afford for a home connection.

I'd recommend trying out Coder's solution and have included a micro-tutorial for you to try out today. I used their utility script and had everything working for Go development within 10 minutes. And if you can't get used to the keyboard shortcuts in the browser, you try the remote SSH approach linked above.

If you have plenty of time on your hands, then you could install K3s and self-hosted GitPod which would enable multiple people in your team to share a single dedicated host, making the cost / efficiency ratio work out better.

Just bear in mind that you'll need to install any development software you plan to use via the integrated terminal such as Go, Docker, K8s or Node.js.

Disclosure: Equinix Metal is a client of OpenFaaS Ltd.

My take for 2021

As the maintainer of several large open source projects, you may be wondering what my take is? You may remember my blog post about building a local Linux computer for work and development. My prediction is that we'll continue to see IDEs running in a browser, but for me, this innovation is not about that. Developers are quick to point out that a fast Internet connection is not always available or guaranteed.

So for me, it's not about replacing my workstation or laptop, which provides a great development experience today. What excites me is the opportunity for collaboration, live coding, sharing and making open source contributions more accessible. A prime example was the OpenFaaS in-person workshop, where we were able to remove that initial 3-hour setup time for new users of Kubernetes and Docker by having everything primed ahead of time. We are only just getting started here.

You may also like

<![CDATA[I wrote a book about Everyday Go]]>https://blog.alexellis.io/i-wrote-a-book-about-golang/60d20c4754bd100001662071Tue, 22 Jun 2021 16:47:24 GMT

Today I've published my first eBook on Go called "Everyday Golang". I wanted to share the news with you, what to expect and a bit about the story behind it.

How did I learn Go?

If you've ever searched for "Golang unit-testing", then you may have found one of my tutorials or videos about Go. I started my journey with the language in around 2015, when I had a strong interest in contributing to and learn about Docker.

As I learned, I noticed that my Go started off looking like the last language I'd used, but over time it improved. Sometimes it was not comfortable being told "Your code is not idiomatic", but it only spurred me on.

Most of what I learned was through trial and error, or reading other codebases, but I also list books that I found helpful at the time.

There are various blog posts I've published that get a lot of hits, but they were never about teaching the fundamentals like structs and variable types. They were practical and served a purpose. Oftentimes the purpose was to help onboard new open source contributors, or to publish something I had learned.

Everyday Go is a fast-track for learning concepts and techniques that you can apply to your everday day work and projects.

Why write a book?

Isolating dependencies

A sample from the ePub version, displayed on my MacBook.

You may ask who I am to write a book on Go. I certainly haven't worked at Google, and I haven't spoken at Gophercon, but I do use the language on a daily basis and maintain several popular open source projects used in production by companies small and large.

Some of the more popular examples include:OpenFaaS - Serverless Framework, inlets - network tunnel, k3sup - Kubernetes installer and arkade - Kubernetes marketplace and CLI downloader.

This is now my third eBook having sold 450 copies of Serverless For Everyone Else and a course on Kubernetes on Raspberry Pi. These both taught me lessons about tooling, formats, preferences, pricing and most importantly, that I could write about what I knew, and people would pay money for it.

"..most importantly, that I could write about what I knew, and people would pay money for it."

I started an independent software company to look after the open source projects I mentioned earlier, and my plan is for eBooks to provide some passive revenue between client projects and developing open-core features.

What about pricing?

The base price for this eBook is slightly higher than my previous eBooks. As developers, we often baulk at the idea of paying for something we could learn on our own. It's so easy to confuse price with cost.

Zig Ziglar has an explantion that opened my eyes to the difference between price and cost.

If you're unhappy with the price, then please feel free to take 6 years to learn, test out, write-up and maintain everything contained in the 100+ pages of my new eBook.

I would say that over the course of a year or two, the cost would be more than justified on its own, if it only made unit-testing slightly easier, more fun or interesting at work.

I've added a Team edition that works out cheaper, with this tier, a manager, director or team lead can order a copy for five people in their team to use. You can probably put it on your OKRs or MBOs (yearly training goals).

Team edition purchase

Engin Diri bought 15 copies of my new eBook for the developer community at Schwarz IT. That was a lovely surprise and has got total sales up to 375 copies!

Everyday Go

The title of the book is Everyday Go, but for the sake of SEO and discoverability, you'll also hear it called "Everyday Golang".

It includes my top ranking tutorial on unit-testing, updated and extended, along with many other topics. You'll see examples of Goroutines, HTTP servers, Prometheus metrics, connecting to databases, releasing binaries via GitHub Actions and I'm adding more as I get feedback from readers.

It was very encouraging to see such a strong response when I tweeted about the book for the first time.

Does Alex ever sleep? 😴 He just wrote a Golang book! I had a chance to read an early copy and its really useful, real life content! I liked the testing part the most! 👍

Engin Diri, Developer Advocate @ Schwarz IT

Book launch!

I want to thank everyone who's bought their copy already. I hope you'll enjoy learning, applying and adapting the examples and recipes in your own work.

"Alex is a seasoned pro building software and products with Go. He’s a great communicator. You’ll be in a better place for having learned from his experience."

Jeff Nickoloff, Director Venmo

So who is it for?

These examples and concepts will help you most as you start out on your Go journey, but even if you've been writing Go for a while, you may find that there's a new perspective or technique you didn't know. Even I go back to my material on unit-testing, Goroutines, GitHub Actions and Prometheus metrics.

The way I see it, what's better than a series of well-maintained, up to date and idiomatic examples and patterns that you can copy and paste into new and existing projects to go faster?

The material is available as PDF & ePub by standard, with a Serverless Supplement, Code samples (ZIP file) and a Team edition available separately.

If you buy a copy and aren't fully satisfied for some reason, then I'm open to feedback so that I can improve the material for everyone else. All the topics are listed precisely, so that you know exactly what to expect from the book.

Check out the topics covered and what people are saying on Gumroad. Pick up a copy today, or subscribe with your email list for updates from me in the future.

Did you know? GitHub Sponsors with access to the Treasure Trove can get 20% off all of my eBooks.

You may also like to..

<![CDATA[Join The Grow Lab Challenge]]>https://blog.alexellis.io/the-grow-lab-challenge/607305654cf36d00019e77f1Sun, 11 Apr 2021 15:58:40 GMTJoin The Grow Lab Challenge

In 2017 I put together a small project with a Raspberry Pi Zero and its camera module to monitor the growth of seeds. I started off with cress, moved onto pea shoots and finished up with some basil. A few people around the world took part and built their own labs.

News: we have a new website for growlab!

This year, I wanted to expand on the original idea and instead of focus on Tweeting, and taking a photo from the side, I wanted the image to be taken from above. This would give a better view, and let me use a seed tray, where I had more than one seed germinating.

Join The Grow Lab Challenge

Taken from my Grow Lab, the view from above can capture up to 40 different seeds germinating at once.

I want to invite you to join me and a few others in building your own Grow Lab.

To make it more interesting, it'll be a contest with milestones that you can reach by completing different tasks and sharing the results with the community. There's no such thing as perfect here, and putting your own spin on the project is encouraged.

The goals of the project are to bring the excitement of watching nature at work, directly in your own kitchen window sill and to give you a practical project, to get started with IoT and embedded electronics.

A global community challenge

There will be three awards you can win by participating in the project - bronze, gold and platinum. For each award you'll get a mark 1/10, so you can get up to 30 points in total.

Bronze - assemble your Grow Lab using the recommend parts, customise the design. Tweet a photo of your build.

Silver - install the software and Tweet a photo from the camera in its position.

Gold - Wait until at least one of your seeds has germinated, and Tweet a second photo.

Platinum - use the software to create images over two weeks. Compile the images into a timelapse and upload it to YouTube. Tweet a link to the video.

Tweet with the hashtag #growlab to participate.

To encourage you, I'll be sharing and re-tweeting your images and messages as the project goes on, and including the best in a final write-up at the end of the summer. To help me keep track, I'll be adding tweets that I see into the alexellis/growlab repository on GitHub. You can also send your own Pull Requests here if you like.

I'll also speak to my friends at Pimoroni, Hackster and Raspberry Pi to see if they can help with a set of small prizes or shout-outs for those of you who get to the Platinum level.

Most of all, I hope you learn, have fun and enjoy building your own Grow Lab.

First participants

Richard Gee who took part in my original experiment in 2017 has already put his build together and put a spin on the design:

Richard told me that he used a pipe bender to save costs on pipe elbows. The elbows in my project cost around 3 GBP or 5 USD each.

Build your own Grow Lab

There are two parts to the build and you're welcome to customise either or both.


If you're not an adult, then you'll need adult supervision for this project. You take this project on at your own risk. It goes without saying that we need to be careful when water, electricity, and powertools are all in close proximity.

The hardware

For Raspberry Pi components:

  • Raspberry Pi Zero W
  • Raspberry Pi camera module with wide-angle 160 degrees lens
  • RPi zero camera cable (yellow)
  • Power supply with a long cable ~ 2-2.5M
  • 32-64GB SD Card

You can use a Raspberry Pi 4 if you wish to host some additional software, but if you just want to record photos and/or take sensor readings, then the Raspberry Pi Zero is small, light and very much up to the task.

Next, you can either mirror my build and customise it as you need, or use the build that I used for my 2017 project. The 2017 project takes photos from the side and uses small holes in a plastic box to mount the Raspberry Pi Zero and its camera.

The Raspberry Pi components are the same for either build.

For my build:

  • A length of 15mm copper pipe
  • 2x 15mm copper elbows (friction fit)
  • Pipe cutter or hacksaw
  • An adjustable spanner
  • Gorilla glue or A/B epoxy
  • Small sheet of MDF
  • Small sheet of wood from a pallet, shelf, or an old piece of wood for the base board
  • 15mm flat drill bit
  • Drill

The method:

  • Mark a hole in the center of the base board, about 1-2cm from the edge and drill at 15mm hole into it
  • Cut a section of your 15mm pipe to 11 inches in height, this will be the main upright
  • Attach the olive to the pipe and slide on the nut. Tighten the eblow and you have your first part of the vertical piece with its right angle
  • Next, cut another piece of pipe for the horizontal section. This should be 50% of the length of your seed tray. If you're using a single seed pot, then make this 50% of the base board
  • Now attach the piece of pipe you just cut into the elbow, so that you have right angle when you're finished.
  • Attach the second elbow to the horizontal piece of pipe
  • Cut a very small piece of pipe now, around 1.5 inches that will be the final piece, that we'll attach the MDF to with the Raspberry Pi and camera on the other side
  • Mark a hole in your piece of MDF in the center. Drill a 15mm hole into it, but try not to go all the way through to the other side.
  • Attach your Raspberry Pi Zero and its camera to the MDF using small self-tapping wood screws. You may be able to use small tack nails, but be careful not to hit the components.
  • Now glue the smallest piece of pipe into the piece of MDF on the opposite side to the Raspberry Pi.

This is how everything should look at this point:

Join The Grow Lab Challenge

I mounted the camera and Raspberry Pi to the underside of a piece of MDF and glued it to the pipework.

Next, you'll need to glue the large piece of pipework into the base board. After an hour, or when the glue is set, attach the MDF board using the smaller piece of pipe you glued into the larger piece of pipe and tighten the nut.

Your completed lab should look like this:

Join The Grow Lab Challenge

My second Grow Lab in 2021

I ran the power cable over the top of the pipe from a mains plug, but if you're confident and have something to hold the pipe still, you could cut a small hole and run the cable through the pipe and route it through the other end.

The software

For me, the beauty of using a Raspberry Pi instead of an Arduino or Pico board is that we get access to a full Linux Operating system, but can still use sensors and cameras.

  • Flash Raspberry Pi OS Lite to your SD card
  • Create a file called ssh on the boot partition to enable remote login
  • Connect the camera cable before powering on

Boot up the Raspberry Pi. I tend to use a USB network adapter to get remote access without a screen, but you can use a USB hub, keyboard, and HDMI monitor if you wish.

Make sure the camera is working with sudo raspistill -o image.jpg, then copy the file using scp pi@raspberrypi.local:~/image.jpg Desktop/.

Connect with your keyboard, or over SSH using ssh pi@raspberrypi.local. If you don't have ssh available and are using a Windows computer, install Git Bash

  • Now run raspi-config and configure your WiFi network, hostname and enable the camera under Interface Options
  • Run an update sudo apt update && sudo apt-upgrade -qy then reboot
  • Add git and tmux - sudo apt install tmux git

There are two options for software.

The first option is to use my seeds2 software from 2017 which will tweet photos of your seeds live to the Internet on a periodic basis. It will require a Twitter API key which is harder to get than it was back then. Keep an eye out, as I'm planning on creating "seeds3" - the next generation of the project that records temperature, humidity and air pressure using a Bosch BME280 sensor.

The second option is the simplest and where I recommend that you start. You can download my phototimer project to create your timelapse offline. Then you can tweet photos or share them on social media as and when you like. When you have enough images, you can even create your own timelapse using command-line tools like ffmpeg or iMovie.

Here's a very quick example of what you can expect. I took this from 6am to 3pm the morning after I finished my build.

git clone https://github.com/alexellis/phototimer
mkdir -p ~/image

Use nano to edit the config.py file:

config = {}
config["am"] = 600
config["pm"] = 1900

config["flip_horizontal"] = False
config["flip_vertical"] = False
config["metering_mode"] = "matrix"
config["exposure"] = "auto"

config["base_path"] = "/home/pi/image"
config["height"] = 1944
config["width"] = 2592
config["quality"] = 35

Photos are only taken between the hours am <> pm, so from 6am to 7pm in the evening. The image can also be flipped depending on how you've positioned your camera. I use a low quality number of 35 to fit more images onto my SD card and to make it quick for me to transfer the images over the network to see the latest images.

Then test out taking a photo with

sudo python take.py 15

This takes a photo every 15 seconds and saves it under /home/pi/image/year/month/day/hour/.

Run tmux now, so that we can disconnect from the Raspberry Pi and keep the program running. Watch for a green or yellow bar at the bottom of your screen. That's how you know tmux is running.


cd phototimer

# Every 10 minutes = 60*10 = 600
sudo python take.py 600

Now hit Control A + B to detach from tmux and log out.

[detached (from session 0)]

Whenever you log in over SSH, just type tmux attach to get back to the photo timer.

You can find out more about Tmux in my tutorial video.

When you want to copy your photos from your Raspberry Pi to your laptop you can use the scp utility.

scp -r pi@raspberrypi.local:~/image Desktop/

Then open the image folder on your Desktop and browse the images as organised in a structure of year/month/day/hour.

To copy a specific day or hour, just add it to the path:

# The 4th April
scp -r pi@raspberrypi.local:~/image/2021/4/10 Desktop/

# 6AM on the 4th April
scp -r pi@raspberrypi.local:~/image/2021/4/10/6 Desktop/

Congratulations, you have built your own Grow Lab and downloaded your first images!

The seeds for your lab

You can grow whatever you like in your lab. I would suggest you try these out if you are not sure where to start:

  • A pip from an Apple, Orange or Lemon
  • A pea
  • A dwarf bean
  • A tomato
  • A broad bean
  • A courgette
  • A sunflower

You'll find most in your local super market or garden centre.

Putting your own spin on things

If you want to grow just one or two plants, perhaps you could add a small submersible 5V pump and a capacitive soil sensor, to auto-water your plants. It's relatively simple to put this together on your own, or you can use a Grow kit from Pimoroni.

If you're filling up your SD card quicker than you expected, then why not add a larger, removable USB pen-drive and mount it under /home/pi/image? That way you can simply turn off the lab and copy the files off the USB pen drive whenever you want.

Could you aggregate the data from a BME280 sensor into a time-series database like InfluxDB, so that you can plot the data over time? See also: Create an Environmental Monitoring Dashboard

Could you add an LED like a Pimoroni Blinkt! so that the plants can receive light during the night?

Rather than using a regular camera, could you use a night-vision camera and IR LEDs, so that you can see how your plants move and change shape during the night?

Could you use the inlets project to host a small HTTP server to show the latest image from your Grow Lab?

Perhaps you'd rather use the Raspberry Pi 4 and host some OpenFaaS functions alongside your experiment? Perhaps you'll also add a database, and a dashboard like Grafana.

Wrapping up

I hope you enjoyed this blog post and that you'll take part in the community challenge this Spring and Summer. It'd be great to see your messages on Twitter, just use the hashtag #growlab or Tweet to me if you have any questions or comments.

As you make progress, don't forget to see the "A global community challenge" and the various levels you can achieve.

As I hear about your build, I'll keep track of your achievements through a GitHub repository: alexellis/growlab


A good place to get Raspberry Pi components in the UK is Pimoroni and Adafruit in the US.

<![CDATA[State of netbooting Raspberry Pi in 2021]]>https://blog.alexellis.io/state-of-netbooting-raspberry-pi-in-2021/606b35364cf36d00019e77d7Mon, 05 Apr 2021 17:25:25 GMT
State of netbooting Raspberry Pi in 2021

TLDR; When I tried netbooting 24 RPi3s in 2017, it failed miserably. Find out what changed when I built my private cloud with the newer Raspberry Pi 4s and K3s.

Just over three years ago, I wrote up a blog post exploring why netbooting was interesting for Raspberry Pi, and what kind of problems I ran into due to issues with the firmware available at the time.

My original rig had 24 early-generation Raspberry Pi 3s, which had a timing bug rendering them unreliable for netbooting.

In this post, I'll give you an update (things are much better now) and I'll also tell you about my new rig which I'm running K3s on.

State of netbooting Raspberry Pi in 2021

My private cloud with 40GB of RAM for running K3s

You'll also find a link to my new workshop and eBook which has a Bill of Materials and step-by-step instructions for netbooting your own Raspberry Pis.

Why did I want to do it in 2017?

My original hopes were that I could save on the costs and management of 24 SD cards. These are notoriously unreliable due to their limited write cycles. When used in a Raspberry Pi cluster running Kubernetes, the cards would be written to constantly meaning the cards will eventually fail.

State of netbooting Raspberry Pi in 2021

Given the bug with netbooting the variant of the RPi3 that I had, this was the only way to get it working

Even with automation, it's a time-consuming and manual process to flash, label and update 24 SD cards. I did actually go through this process and got all of the nodes functioning.

State of netbooting Raspberry Pi in 2021

My original rig refreshed with K3s

I ran Docker Swarm on the cluster originally, before Swarm fell out of fashion, and Kubernetes took over to bring clustering to containers. Swarm was always light-weight and fast on Raspberry Pi, and Kubernetes using kubeadm was always slow and difficult to use.

Fortunately Darren Shepherd came up with a novel idea of taking way the things he didn't want in Kubernetes and created a "distro" called K3s (3 is less than 8).

Where kubeadm created a headache for Raspberry Pi, K3s made things simple with an easy boostrap, low resource requirements and a single binary for distribution and updates.

Given how much I'd liked the Docker experience of "docker swarm init" and "docker swarm join", I decided to re-create a similar experience for K3s in the k3sup tool.

k3sup installs K3s over SSH and also makes creating a multi-server configuration easier through CLI flags.

State of netbooting Raspberry Pi in 2021

Conceptual workflow showing the install and join commands run with a hostname or IP address over SSH.

Why did I revisit netbooting in 2020?

I was helping the team at Equinix Metal get their Open Source provisioning software Tinkerbell ready for general use. That meant going through examples to check that they worked, building new examples and giving feedback on the developer- and user-experience.

As part of the project I wrote up a feature-length article for The New Stack to link the ideas of Cloud Native computing (a relatively new and trendy area) to bare-metal provisioning (something which has used the same basic tools for 20-25 years).

My post Bare Metal in a Cloud Native World also helped me refresh my own memory on TFTP, DHCP and on NFS.

State of netbooting Raspberry Pi in 2021

My conceptual diagram showing how PXE boot works

One of the other things I built out for the client was a GitHub repository collecting other Open Source awesome baremetal projects: awesome-baremetal

As I was kicking the tires with Tinkerbell and testing out netbooting with a few Intel NUCs, my interest in the Raspberry Pi got stirred again. With the popularity of K3s and my K3s bootstrapping tool k3sup ('k3sup'), it seemed like a good time to revisit the problems I'd had in 2017.

My build in 2021

In 2021, I wanted to take what I'd learned during the previous year and bundle it up into a workshop that anyone could follow to access netbooting. It took me about two weeks to fully understand, document, automate and test the setup.

Whilst you can spend a lot of money here, I was fortunate to have an old PC I could use as a server, and the kit that BitScope had sent me. Note: the 2GB Raspberry Pi only costs 35 USD now, and you only really need one to try this out.

For the initial configuration I used an Intel NUC as the netbooting server with a four Raspberry Pi 4s with 2GB of RAM each. That would be enough to run Kubernetes and a few workloads like OpenFaaS.

I used the official 32-bit Raspberry Pi Operating System which supports PXE booting. Ubuntu 20.04 (Focal Fossa), a 64-bit Operating System also works with some modifications and is needed if you want to run a project that is only made available for 64-bit Operating Systems. The Raspberry Pi is capable of running both, but the 32-bit OS is better supported and understood by the community.

After assembling the hardware, the process involves flashing a new firmware to each node, setting up a boot order, configuring a server with TFTP, NFS, a DHCP server and IP forwarding to act as a router.

State of netbooting Raspberry Pi in 2021

I flashed each of the Raspberry Pis using the same SD card, then I could close the case and take a note of each MAC address and serial number for later on.

This is the network topology, where everything runs in its own subnet, on a separate LAN and switch.

State of netbooting Raspberry Pi in 2021

The network topology from my workshop

Once you have everything configured, it's then a case of picking a case or enclosure to host your nodes.

BitScope had sent me a newer version of their product called the Cluster Blade - this time just for 8 nodes. It was aimed at edge computing and could be rack-mounted. Do not confuse this with the previous generation product called "The BitScope Blade", which is designed for Raspberry Pi 3.

They also kindly sent over an additional 4x nodes for me so that I could fully populate the cluster with 8 nodes.

State of netbooting Raspberry Pi in 2021

This gave me a total of 40GB of RAM and 32 CPU cores. My netbooting server also had a 500GB NVMe installed and shared for NFS, so there was plenty of space available too.

With each of the nodes running, I observed around 30-31 Watts of power being drawn through the power supply. The 24-node RPi3 cluster drew ~ 21 Watts, so considering the increase in I/O speed and RAM per node, this seems reasonable.

A private cloud for production

The use-case that BitScope see for these racks is where customers want to move off AWS EC2 and onto their own private cloud, which they can run on-premises or at the edge at a customer site.

If an application already runs in a container, then it shouldn't take too much work to port it over to K3s. One of the tricky parts is in the ecosystem, and making sure that everything that needs to run is available for an Arm CPU.

App ecosystem

Things are much better than they were in 2017, with the following key components available with Arm support: cert-manager, ingress-nginx, Postgresql, OpenFaaS, NATS, Prometheus, Docker registry, Grafana, Minio, Portainer, .NET Core, Node.js, Python 3, Go and many others. If you're a Kubernetes or K3s user, then you may also find the arkade tool useful for discovering and installing apps with a single command line.


People often ask me about storage. Many modern microservices are built to be stateless, and not to need to read or write from a disk, but you may need underlying object storage or a SQL database. That is where I tend to need persistent storage when working with OpenFaaS functions.

Fortunately, K3s comes with a storage driver that can use the local disk of the node and is called Local path provisioner. There are other options available such as OpenEBS and Longhorn, but since we already have NFS in place, we may as well use the NFS volume provisioner.

Ingress and remote access

Some people are content with having no Ingress to their clusters at all, but I'm not one of those people. If I need to expose a Kubernetes LoadBalancer, then I want it to have a public and accessible IP address. For that reason I built the inlets-operator, a Kubernetes controller that can provision a VM and then create a network tunnel to enable TCP or HTTPS traffic to enter the cluster from behind NAT, firewalls and captive portals.

State of netbooting Raspberry Pi in 2021

Credit goes to Ivan Velichko for the animation.


Loosely defined, GitOps (coined by Alexis Richardson) is storing deployment configuration in a Git repository for continuous deployment and is very much in vogue in 2021.

Johan Siebens wrote up a blog post showing how ArgoCD can be used to bring GitOps to one or more Raspberry Pi clusters, or a fleet of clusters. ArgoCD for your private Raspberry Pi k3s cluster

Rancher's Fleet project (another one of of Darren's inventions) is designed to manage fleets of up to 1 Million clusters.

Wrapping up

Whatever I said about netbooting in 2017 is still large correct apart from one thing. It is now much more reliable and easier than ever to get started with, whether you have one or 100 Raspberry Pis.

So however many Raspberry Pis you have, booting over the network means you can quickly provision and manage your nodes, even if they are just running PiHole, Plex, or a Ghost blog. Removing the SD cards makes the filesystem faster, and more reliable.

It should be noted that a local NVMe over USB3 is faster, by a large margin, however it is also much more costly to scale 8x NVMe drives plus their caddies, vs. one NVMe in an old PC or server that you may already own.

Let me leave you with some resources and further links so that you can keep on learning.

Take it further

Is there something I didn't cover? It may be included in my workshop, or you can reach out to me over Twitter if you already have a copy.

If you'd like to build your own private cloud, or just learn more about networking, you can pick up my eBook or video workshop package with 30 USD off until 9th April.

State of netbooting Raspberry Pi in 2021

Would you like to learn more of the journey over the past 5 years of Docker and Kubernetes on Raspberry Pi? Are you curious about the costs and the real-life use-cases? The Past, Present, and Future of Kubernetes on Raspberry Pi

Would you like to try booting your Raspberry Pi from a local disk? Upgrade your Raspberry Pi 4 with a NVMe boot drive

Want to try K3s? You can do so with an SD card in around 15 minutes with my K3s walk-through

<![CDATA[How I test OpenFaaS changes with Kubernetes]]>https://blog.alexellis.io/how-i-test-openfaas-with-kubernetes/605071b24cf36d00019e77a9Tue, 16 Mar 2021 09:53:04 GMT

There are several ways to test changes in OpenFaaS with Kubernetes. I wanted to outline an approach that I find useful, so that new and potential contributors can get started quickly and experiment. I'll also give you a very brief overview of what OpenFaaS is, and what gets deployed when you install it.

If you work with Kubernetes controllers or open-source projects that are written in Go, feel free to read on and send comments, questions and suggestions my way on Twitter.

First of all, what is OpenFaaS?

OpenFaaS is an open-source project which I started in 2016 so that I could run serverless functions on my own infrastructure using containers. Since then, it's become much more. It has a much larger community of users and contributors than I could have ever imagined.

Despite the misconceptions, this project is not backed by Venture Capitalists (VCs), or an external company. So the community relies upon volunteers (including myself) to do the heavy lifting.

How it started

OpenFaaS has over 26k GitHub stars across 40 repositories and ~ 350 contributors.

When you add OpenFaaS to Kubernetes, you get a number of components installed to your cluster, each performing a different task. There are components which we have built, and some that come from the CNCF. To date, every OpenFaaS component is written in Go.

Conceptual architecture

Pictured: Conceptual architecture

These core components can be found in the openfaas organisation:

Then, the following CNCF projects are installed as dependencies:

  • Prometheus is used to monitor Rate, Error and Duration (RED) metrics for each function and for the core services.
  • AlertManager is used to monitor functions for high rates of invocation, and then triggers auto-scaling. When any alert is resolved, the function will scale back to its original minimum replica count.
  • NATS provides a way to execute tasks in the background through a queue.

From there, they are configured through environment variables for any non-confidential configuration data, for things such as:

  • Timeouts
  • Concurrency levels
  • How many times to retry something
  • A feature flag
  • Log verbosity

Secrets are used to inject confidential data like API keys, the main example is the authentication token used to authenticate to the gateway.

These configuration items can be altered through the helm chart, or through the arkade Kubernetes marketplace, which abstracts away the mechanics of the chart, to make it look like a one-line installation.

The functions that OpenFaaS can deploy to Kubernetes are not magic, they are actually very similar to these services known as "core services" or the "openfaas stack"

How do I deploy OpenFaaS?

The way I deploy OpenFaaS tends to be using kind - Kubernetes in Docker. KinD is a community project led by Ben Elder, at Google.

There are dozens of ways to run Kubernetes, you can run it on your local computer, or on the cloud. Running locally tends to be the cheapest option, and using a local Kubernetes distribution that runs inside a Docker container means that the same instructions work for Linux, MacOS and Windows users.

Meet arkade. arkade is an open-source Kubernetes marketplace - it offers apps and also offers CLIs - around 60 combined.

# Note: you can also run without `sudo` and move the binary yourself
curl -sLS https://dl.get-arkade.dev | sudo sh

# Windows users with Git Bash
curl -sLS https://dl.get-arkade.dev | sh

Download kind with arkade:

arkade get kind

Make sure that you have Docker installed, then create a cluster:

kind create cluster

The context for kubectl will be set to the new kind cluster. It can support multiple, but I tend to create/delete when needed rather than managing more than one local kind cluster.

Next install OpenFaaS using its helm chart, and all the defaults:

arkade install openfaas

The arkade app has a number of flags that you can use, which translate to helm chart options.

  • --gateways 2 converts to --set gateway.replicas=2
  • --load-balancer generates --set serviceType=LoadBalancer

See what options are available in the helm chart README

From there, the app will tell you how to port-forward the gateway and UI to your local computer. If you want to see this information again at any time, then run arkade info openfaas.

The easiest way to access OpenFaaS will through the CLI, where you can discover sample functions and deploy something.

faas-cli store list | grep nodeinfo
faas-cli store describe nodeinfo
faas-cli store deploy nodeinfo

You'll get a URL that's accesible on your gateway via and you can invoke it via curl.

Making the first patch

In this example, we'll make a patch to the Kubernetes controller for OpenFaaS. If you just need to test a patch for the helm chart, you can see instructions in the chart README.

Fork and branch

Fork the repository you want to change such as faas-netes.

Clone your fork of the project, into the GOPATH

mkdir -p $GOPATH/src/github.com/openfaas/
cd $GOPATH/src/github.com/openfaas/
git clone https://github.com/openfaas/faas-netes
cd faas-netes

If you want to avoid pulling in all the history, add --depth=1 to the git clone statement.

Now create a branch to start working in. We'll make the Kubernetes operator print the text "Hello World" upon start-up.

git checkout -b alexellis/print-hello

Now, edit main.go and add to func main() {

log.Println("Hello world")

Would you like to improve your Go skills? Checkout my new book Everday Go for practical tips and hands-on examples.

Test out the new version

We will build a new container image, push it to a registry, then install openfaas again using its helm chart, and the new image reference. Some components need to be developed this way, due to the way they reference internal components, or are bundled as part of a Pod.

This workflow may not be the most optimized, but it's good enough for me.

Check the Makefile to see if it has a target like make local or make docker that you can use, if not, just go ahead and build your own image:

export USERNAME="alexellis"
docker build -t $USERNAME/faas-netes:print-hello .
docker push $USERNAME/faas-netes:print-hello

Then you can deploy the new version to the cluster using arkade install again:

export USERNAME="alexellis"

arkade install openfaas \
  --set faasnetes.image=$USERNAME/faas-netes:print-hello

If you are working on the gateway, then you should stop the port-forwarding and run the command again, since the pod will be updated.

Now check the logs for the component you updated. In this instance, the Kubernetes controller is bundled in the same Pod as the gateway, so you need to pick the specific container:

kubectl logs -n openfaas deploy/gateway \
  -c faas-netes

At the time of writing, the OpenFaaS Kubernetes controller can run in two modes: as a controller or CRD operator.

The CRD operator allows for someone to use kubectl apply to create functions, instead of using faas-cli. For instance:

# nodeinfo.yaml
apiVersion: openfaas.com/v1
kind: Function
  name: nodeinfo
  namespace: openfaas-fn
  name: nodeinfo
  image: ghcr.io/openfaas/nodeinfo:75d61698aaaa14345c7983f5a5db91f6da80e5bb

This may change in the future, but for now, there are often two codepaths that you need to update. An example would be adding custom runtime support for gVisor and kata containers. We had to edit the HTTP handlers for the Controller, then for the Operator afterwards.

To change the image for the CRD operator add:

export USERNAME="alexellis"

# Add the operator.create flag:
arkade install openfaas \
  --set operator.image=$USERNAME/faas-netes:print-hello \
  --set operator.create=true
# Check its logs:
kubectl logs -n openfaas deploy/gateway \
  -c operator

How do you rinse and repeat this?

After each change, build a new image tag such as alexellis/faas-netes:print-hello1, push it, and run the arkade install command.

When you are done, you can run kind delete cluster or leave the cluster running for further testing.

Send a Pull Request

Providing that you have raised and Issue, and someone from the team has approved your change, you should send a Pull Request (PR). It's important that we fill out the whole PR template to avoid causing unnecessary work for everyone involved.

Part of the PR template asks: "How has this been tested?". You will be surprised how many people leave this field out. Copy and paste the arkade command you used, along with the output you got.

Why? So that the person verifying your patch can do so quickly. You may also want to share the container image so that someone else could test out your code without having to do a local build of their own.

Wrapping up

We took a look at the OpenFaaS core components, and what gets deployed when you install the project to your cluster. We covered the kind tool for a local Kubernetes cluster and arkade to install OpenFaaS without having to think deeply about helm. Using docker, we build and pushed a test version of our code and verified that it worked.

This approach can be used for most of the components in the project: fork, clone, edit, docker build, docker push, arkade install with a --set flag, then send a PR.

The techique of building, pushing and deploying a container to a cluster should work all the time, but sometimes you can build and run a controller on your own computer, and make the feedback loop even faster.

I use both techniques for other projects that target Kubernetes like the inlets-operator. It can add cloud-style LoadBalancers to KinD and private clusters by creating VMs in a cloud provider and starting a tunnel server there. Whenever we add a new cloud like Azure or AWS EC2, it can be easier to test the controller on my own computer, and have it point at the cluster, instead of running inside the cluster.

Get involved

Do you want to help, but cant't contribute code or time? Check out the options for becoming a GitHub Sponsor

Here are some further links for learning and contributing:


Is there an alternative to OpenFaaS on Kubernetes?

If are left wondering whether there's an easier alternative to OpenFaaS, that doesn't require any Kubernetes knowledge, then you may be in luck. Check out the faasd project, which uses the same components, on a single VM or host.

<![CDATA[My first eBook launch - results & feedback (10k USD in 27 days)]]>https://blog.alexellis.io/my-first-ebook-results-feedback/601bbde94cf36d00019e7746Thu, 04 Feb 2021 09:30:54 GMTMy first eBook launch - results & feedback (10k USD in 27 days)

I wanted to write to you all and share that I've launched my first eBook called "Serverless For Everyone Else" - within the first three hours of launch, nobody bought a single copy and I thought that I'd got it all wrong.

The "everyone else" is a reference to how complicated and difficult cloud-based and Kubernetes-based FaaS has become. The eBook sets out to help indie developers, and individuals within corporations build and deploy functions using a new version of OpenFaaS (faasd) which is open source and can run on a single VM or VPS without much overhead.

The real power of functions comes in extending existing systems through webhooks, polling, or querying and updating state. I'll share some of these use-cases below including how I used functions to extend how the Gumroad marketplace worked.

Why write an eBook anyway?

Why did I decide to write an eBook? If you've ever written about tech or spoken at a conference, then Packt has probably approached you on LinkedIn about writing for them. From what I hear, that kind of arrangement means writing in a word doc, with tight deadlines and cranking out 300+ pages. It wasn't for me.

I'd seen Daniel Vassallo create an eBook about AWS and Twitter. Daniel had really good results using Gumroad to sell his self-published eBooks. Then there's Adam Wathan who built Tailwind CSS and trained himself up as a UI designer. He wrote his own eBook and in his yearly review mentioned that it did 600k USD in passive sales, without any promotion.

Whilst I didn't expect to get anywhere close to those numbers with a specialist topic, I did want to know if people would buy something from me.

The tiers at launch

I launched with three tiers, with names and clear differentiation:

  • Minimalist - 25 USD - just the eBook
  • DevOps PRO - 50 USD - the eBook, plus a Grafana dashboard and the YAML for faasd to enable it
  • The learner - 99 USD - the eBook, plus a Grafana dashboard, the YAML for faasd to enable it and a video workshop ready in 7 days
  • The team player - the learner, but for 5 people in your company - for a lunch & learn, brownbag, or to better understand Serverless

As long as the options add some value, it seems that people are willing to pay a little more and enjoy the choice.

There was no pre-launch and I didn't collect email addresses. I just tweeted about the eBook on Friday afternoon, and then nothing happened for three hours.

I interpreted this as a failure because I'd been reading "The Right It: Why So Many Ideas Fail and How to Make Sure Yours Succeed".

You tend to get what you focus on, like when you think of installing a satellite dish, then realise that everyone on your street has one, and can't stop seeing them.

In the same way, I saw failure to find market fit. In those first hours I convinced myself that the lack of sales was a sign that nobody wanted the material.

Then I got my first sale, and felt a little relief, maybe I was wrong?


The sales graph had one initial spike on the Friday, and kept going into the Sunday. What had happened? My Twitter audience and GitHub Sponsors came to lend their support.


Then things died down, and over the course of the week I wondered again if I'd failed to find market fit. Gumroad was sending me almost no organic traffic, and nobody was landing on the page from search engines.

Still, it was more money than I had expected to earn from the eBook within the first week, but I felt like with so much interest in OpenFaaS, that there was a broader audience to reach.

Getting a second peak

After the first week, I released the video workshop to the people who'd purchased that tier (99USD), and to get some feedback, I offered a free upgrade to the workshop for the tier just below (50USD).

The discount resulted in another large spike in sales.


Gumroad has no way to offer a discount for a single tier, or to enable upgrades. So how did I fulfil the upgrade?

I did it by writing a function and deploying it to my Raspberry Pi. Gumroad can be configured to send a webhook (also called "a ping") to a HTTPS endpoint whenever there is a sale. My code just had to validate the sender, parse the dollar amount and email address, and then send out an email to the customer. I evaluated a few email services and settled on AWS SES for its simplicity and predictable pricing. I used inlets PRO to receive webhooks.


You can see the source code here: alexellis/gumroad-responder

After the second peak, I saw a sharp drop-off in sales, again. I started to wonder why I was seeing this pattern?

At that time 99% of referrals had come from Twitter and from email. Had I exhausted my network? Had everyone who was going to buy an eBook on learning open-source serverless, already become a customer?

I went ahead and added a link on openfaas.com and on my blog at the end of each post, to try and get more eyes on the eBook. It's too early to tell if that is going to work out, but has resulted in a few conversions already.

One of the bits of feedback I got on the eBook was how users enjoyed the use-cases, and practical examples. I'd recently started video streaming, so invited a friend to my YouTube channel to talk about serverless use-cases.


Watch the recording

The recording generated a lot of interest and engagement for existing eBook customers, but seemed to have no obvious effect on sales.

You can get the show notes here which have lots of links for use-cases from hobbyists, indie devs and corporate users of OpenFaaS.

My next milestone

I am pleased with the results so far, and have learned a lot along the way. I also had a few people reach out to me directly asking for help who had taken inspiration from seeing someone actually go ahead and take a calculated risk. As they say, "nothing ventured, nothing gained".

I haven't quite got to 10k USD, which is a nice round number to aim for, but I can roughly understand what each peak means here, and whether they are repeatable.


Once the sales do cross that threshold, I'll write-up a how to guide and share the command lines and tools I used to generate the eBook.

What's next

I committed to providing free updates - because the eBook was partially experimental and I wanted to validate that it was useful. Using the OpenFaaS Slack helped here - I opened a new channel for people who'd bought the book, and enjoyed the feedback and engagement from this.

Some people charge extra for community, but it felt too hard to enforce the access for this.


I generated my own eBook cover originally and then only later, got a professional designer involved.


It turns out that the cover may not matter with Gumroad, as long as it's not awful. Since changing cover, I've only seen two new sales.

Now that I have some initial feedback and have learned that people will buy things from me, I want to let goodwill rebuild, and my audience "recover" before launching a second eBook. I've done a lot of free blog posts and writing on Kubernetes, Docker, Golang and Raspberry Pi and analytics from my blog shows me what's popular.

Just because something you give away for free is popular, doesn't mean that anyone will be willing to pay for it. OpenFaaS itself is testament to that, with many corporations using it in production, without contributing code or finances back. Part of my 2021 goals is to find a way to make OpenFaaS sustainable.

"If You Always Do What You've Always Done, You'll Always Get What You've Always Got." - Henry Ford

The eBook has already brought in more revenue in a month, than the OpenFaaS GitHub Sponsors account has in a year.

The Right It has taught me not to invest too much effort up front in any one idea, because it is likely to fail. That's just the way things go. Instead I'll be testing ideas out through small experiments that take a few hours or days, and using data to decide whether to invest more.

I'm happy to answer questions via Twitter.

Update (12 Feb 2021)

It took 27 days, but the eBook has now reached the milestone of 10k USD in sales! Most of the purchases being made now are from links on the OpenFaaS website, the faasd repo or my own blog. This is also known as long-tail, a slow and (mostly) steady stream of revenue.


Visualised: Launch event, offers and longtail

I've also released revision 6 of the eBook which has a much more detailed section on CI/CD and recipes for multi-arch builds. I'm using it to deploy endpoints directly to my Raspberry Pi from GitHub Actions.

If you're curious about how Serverless functions can be of use for your next side-project or idea, why not try the eBook and / or video workshop? You can get your money back within 7 days if you're not satisfied.

Events and webhooks can be used to build custom extensions for SaaS products or existing systems. You can see an example in my gumroad-responder which forwards messages to Slack, and sends emails via AWS SES to customers during a promotion.

🎁 To say thanks for reading, the next 20 people can get 20% off any tier - including the video workshop package using the discount link below:

Check the options on Gumroad: Serverless For Everyone Else on Gumroad

<![CDATA[Still in the game: My 2020 Year in Review]]>https://blog.alexellis.io/still-in-the-game-my-2020-year-in-review/5feb2c70759e7300016c1868Tue, 29 Dec 2020 14:57:15 GMTStill in the game: My 2020 Year in Review

I'm still in the game, so join me as I explore the highlights of a second year of independent business and creating Open Source software. I'll share my top blog posts by views, top interviews, books that have influenced me and a little about each project I maintain. At the end of the post I'll talk about what 2021 has in store.

I had a lot of help this year, so I want to thank all my company's customers, my GitHub Sponsors, and everyone who contributed or helped to share.

Most popular blog posts of 2020

Over the past year, I've focused on developing the OpenFaaS and inlets brands, and I've also experimented with Medium's Partner Program. These are the four blogs where you would have seen me:

It's been encouraging to see my posts trending on Hacker News, Reddit and Twitter, along with being included in newsletters like KubeWeekly and cron.weekly.

Here are the most popular blog posts from the year ranked by views:

From openfaas/inlets:

I now have hundreds of blog posts going back several years, so it's difficult to read too much into the numbers, as it depends on how much traffic the site gets and where it has trended. A post on Hacker News could get 15k views overnight, and another more impactful post could get a steady stream of 3k per day from organic searches.

This year I've mainly focused on use-cases, and on capturing the imagination. In 2021, you'll see more of the same along with more guest posts on the blogs.

Best interviews / features

As part of GitHub's readme program, I was invited to write a feature-length piece on independent business and Open Source development

I hope this gives you a deeper perspective on what it's like to be on the hook for a large OSS project.

Craig Box from Google reached out to me and asked me: "Why aren't you working for a FAANG company for 350K USD / year"?

Before I knew it, Craig had sketched out a brief history of my career in a Google Doc, and we were recording a podcast with his co-host

Learn how a developer at a payroll company became a Docker Captain, caused a stir with a prototype, and quit his job to create an independent software business.

Influential reads

I've read a number of books this year that have had an influence on me. I can't say that I finished them all, but I did get something from each that has helped me with my journey.

  • Don't just roll the dice - a short guide to software pricing by Neil Davidson - this book taught me about the "pricing curve" - and how once a price rises above 0USD (think open source), you lose part of your potential market. There is a sweet spot on the curve. It also taught me about "price anchoring" - where a similar product becomes the way people expect your offering to be priced.

  • Before The Exit: Thought Experiments For Entrepreneurs by Dan Andrews - this book showed me that a product or project gives a platform and outlet, that once taken away can leave a void. It also challenged the idea that selling a business is the only successful outcome. And that receiving a windfall of cash, isn't necessarily as fulfilling as we would hope.

  • The Irresistible Consultant's Guide to Winning Clients by David A. Fields - the main learning in this book is that success is found not in selling what you think you are able to, but in finding what pain a customer might have, and offering a solution that could help. Likewise, they need to sense urgency and enough pain to buy the solution. It also has to be simple enough to make sense to them.

A good example of this would be Tailwind UI - a series of HTML and CSS snippets built by a designer and marketed to developers. Most backend developers are hopeless at making things look nice on a webpage.

  • The Visual MBA by Jason Barron - having spent so much of my life giving all of the value I was creating away for free, this book helped me understand a framework for testing business ideas and iterating on them.

Ultimately, from a business and sustainability perspective, Open Source software fails its creators and only benefits its users.

  • Start with Why by Simon Sinek - the premise of this book was enough for me to stop and question things. When you're responsible for picking what you're working on, then asking Why? becomes even more important. Why did I start OpenFaaS? Why am I still working on it?

Having a why and challenging it over time could save us all a lot of wasted time.

A common theme in all of these books was that customers buy ways to reduce and remove pain. It needn't be a 10x improvement, but it should be an obvious solution, and they need to have a sense urgency about fixing the problem.

I would also say that I've become more aware of "Indie Hackers" - these are solo developers who come up with what's usually a very trivial idea, but execute on it hard. One example that caught my attention was a service that just makes banner images for blog posts. That was it. That was the product. From the outside, it looks like a simple idea that is easily copied, but by pressing in, he is now earning 8k USD per month from a platform, which I imagine costs very little to run.

When I compare this to my journey with OpenFaaS, and consider that its recurring revenue on GitHub Sponsors is only around 500 USD / mo, from one end-user company, it makes me question the ideology of Open Source Software.

Project highlights

inlets - Cloud Native tunnels

inlets was created to make tunnels something we could self-host, to get away from the limitations of SaaS products, and to integrate deeply with VMs, containers and Kubernetes.

This year I launched a landing page and blog for inlets PRO. To begin with it was just a few pieces of Tailwind UI tied together that showed pricing and features. It was a starting-point, but then over time it's evolved and now even has its own blog with guest posts.

Highlights for inlets:

  • inlets blog launched - the first post was about my inlets presentation on "Advanced cloud patterns" at cert-manager's community day. Since then I've seen several guest posts and an uptick in visits to the domain because of SEO and social sharing.

  • Business customers - inlets PRO has seen lots of customer interest over the year, and had its first business customers. I've got used to qualifying leads, following-up on free trials and more used to rejection.

  • inletsctl/inlets-operator providers - both gained new cloud providers like Hetzner, Azure and Vultr. I couldn't have done this without the support of the community

  • inlets OSS + PRO syntax - both were quite different and are now almost identical to help with adoption and to make it more intuitive for new users

  • Personal customers - some customers feel the price is fair, others expect it to be free. This is just the pricing curve doing its thing.

Most personal customers are choosing inlets PRO to enable their local development, self-hosting needs or to get Ingress into their homelabs.

It only takes a few seconds to create a tunnel server with inletsctl, but next year, if there's enough interest then I may look into building a managed inlets service.

OpenFaaS - Serverless Functions Made Simple

My top highlights:

For OpenFaaS my highlight would be faasd. faasd was initially an experiment to see if containerd would make a good back-end for functions instead of Kubernetes. By removing the support for clusters which have natural latency through their event-drive nature, things like cold-starts got so rapid, that they stopped existing. The cost to run OpenFaaS dropped from "a Kubernetes cluster" to a 3 EUR / mo VPS.

We're still only scratching the surface of what faasd will mean for the community.

You end up with something that costs less than cloud functions, and has a much better developer experience, with the ability to "scale-up" into Kubernetes in the future.

See it in action in this community blog post: Tracking Stripe Payments with Slack and faasd

arkade - portable Kubernetes marketplace

arkade now has 40 apps that you can install with one command, and dozens of CLIs that you can install much quicker than using brew or apt-get. This year I've seen around half a dozen projects copy the approach arkade uses to fetch binaries, so I think we're on to something there.

Its real value comes in compounding - when you need several CLIs or Apps for a tutorial or cluster, the speed-up in getting something deployed is huge.

Want self-hosted OpenFaaS, with a public IP and TLS certificate, but running on your laptop, with KinD?

# Get your CLIs
arkade get kind
arkade get kubectl
arkade get faas-cli

# Create that cluster
kind create cluster

# Deploy inlets-operator to get a public IP for your IngressController
arkade install inlets-operator \
  --token-file $HOME/digitalocean-token \
  --region lon1 \
  --license-from $HOME/license.txt

# Install a HTTP and TLS stack
arkade install ingress-nginx
arkade install cert-manager

# Install OpenFaaS and generate an ingress record
arkade install openfaas
arkade install openfaas-ingress \
  --email web@example.com \
  --domain reg.example.com

You'd then have a TLS cert for OpenFaaS gateway running on your laptop at https://reg.example.com

The same can be done with other combinations Apps and CLIs too.

arkade also gained its first Sponsored App for Venafi

k3sup ("ketchup") - bootstrap K3s over SSH

k3sup gained support for K3s' new HA mode using etcd and was used in several of my newest tutorials:

If you've not seen it yet, it looks a bit like this:

Still in the game: My 2020 Year in Review

Through the k3sup install command you can create a cluster with an initial server, then use the k3sup join command and the server's IP to join in your nodes.

For example, provision two hosts, then install k3s to one as a server, and join the second to it. You'll be able to merge the resulting configuration to your local Kubernetes config file too.

k3sup install --host k3sup-server1.ec2.example \
 --merge \
 --path $HOME/.kube/config \
 --context k3s-ec2

k3sup join --server-host k3sup-server1.ec2.example \
 --host k3sup-agent1.ec2.example

Then make use of your new cluster:

kubectl config set-context k3s-ec2
kubectl get nodes

I continue to see lots of tutorials appearing for the project, and launched a range of SWAG which was popular with the community.

You can still buy a t-shirt, mug or hoodie and support the project.

2021 outlook


Many of you will be aware of the challenges of developing and maintaining Open Source Software for free, without any form of compensation or value capture. Popularity, contributions and traction are all encouraging, but compound the problem.

I still don't know the solution to making OpenFaaS sustainable in the long-term, but it's clear that giving everything away for free makes for an extremely difficult path to walk.

For that reason I want to thank my GitHub Sponsors who have taken the hardest step of reaching into their pockets. Through your weekly Insiders' Updates, you've got a personal view into the peaks and troughs of my year.

Still in the game: My 2020 Year in Review

I also created The Treasure Trove for you, where you can go back in time and see over 18 month's worth of updates, tips and private downloads. Did you know that it's built as a Go function, deployed to a faasd instance on a Raspberry Pi 3, linked to the Internet through inlets?

Over the past year I've helped several companies reach and engage developers through engineering as marketing, writing, social media and my expansive network of maintainers and influencers. I want to thank Equinix Metal, Dialog exe AS, The LinuxFoundation, Sysdig, Venafi, Rancher, Civo and others who worked with my company this year. We are just getting started here.

I don't know what 2021 will hold for us all, but I am ready for the challenge and still in the game.

How you can help

Has my work helped you in some way? Have I inspired you on some level? Did you ever think: If Alex can do it, then maybe I can too? Maybe you needed to get some help at work, and when you Googled Kubernetes, Golang, or Docker, my name kept coming up. Maybe you use multi-stage Docker builds, did you know I wrote that tutorial you used?

You can support my continued Open Source and community activities in one of these ways:

  • Join my Insiders' program via GitHub sponsor - you'll get weekly emails and updates from me, but more than that - enable me to invest more of my time in Open Source
  • Buy my new eBook - Serverless For Everyone Else - a great starting point - especially if you've ever said Serverless isn't for me
  • If you use OpenFaaS, k3sup, arkade or inlets at work - whether for testing, pre-production or with customers then Sponsor the OpenFaaS account

If you're building your own thing, and know how important developer-experience and community are, then check out OpenFaaS Ltd consulting services and reach out to the company. We have helped brands big and small to drive up engagement and expand their reach.

Did you know that I write tutorials about Kubernetes and Docker and build cloud native applications for a living? I now offer my time to companies to help them understand how to fit together a Cloud Native architecture. We have a lot to talk about, you can start today with a one-off call -

Connect with me

Still in the game: My 2020 Year in Review

Here I am with my wife Erin wearing our OpenFaaS hoodies

Every professional can benefit from a coach, someone to help you find the answers, to suggest ideas and to keep you accountable. Book a 1:1 session with me via Zoom.

<![CDATA[Self-hosting Kubernetes on your Raspberry Pi]]>https://blog.alexellis.io/self-hosting-kubernetes-on-your-raspberry-pi/5fe21498759e7300016c1845Wed, 23 Dec 2020 12:00:13 GMTSelf-hosting Kubernetes on your Raspberry Pi

The aim of this post is to show you how to build a Kubernetes cluster with Raspberry Pi 4s for self-hosting APIs, websites and functions so that you can expose them on the Internet and serve traffic to users. Use it to evaluate your current setup, or to plan a new build including: hardware options, HA, Ingress, storage and Operating System choices.

We'll start by addressing the usual questions, before going on to explore a number of topics in detail. These are my preferences built-up after building clusters on Raspberry Pi for 5 years. I'd also like to learn from you, so feel free to tweet to me @alexellisuk.

Why self host?

The only valid reason here is: because you want to. And if the idea of self-hosting something turns you off, then stop reading now and post a derisive comment on Hacker News.

Self-hosting Kubernetes on your Raspberry Pi

KubeCon talk: The Past, Present, and Future of Kubernetes on Raspberry Pi

The costs of cloud

I was speaking to a friend recently that told me they were paying 7USD / mo to host 20 small projects on a well-known PaaS. That would have been costing them around 1700USD per year, moving those workloads to an RPi would save a fortune.

Running a Kubernetes cluster on one of the cheapest providers like DigitalOcean can still set you back 60-120 USD / mo for a modest cluster. You can get much better value with your own hardware, whether that's with an old PC or some Raspberry Pis. I'm not saying that you should call your boss and give him the good news, but there is a certain crowd who enjoy building and running infrastructure, they also tend to enjoy cost savings.

"But, I have no use-case"

Quite often my content on Raspberry Pi can be divisive. It brings the best out of some people, and the worst out of others.

Let me put this succinctly - the Raspberry Pi is a Linux server. If you write code, or "do DevOps" for a day-job, you have a use-case. If you use a computer on a daily basis, you have a use-case.

Self-hosting Kubernetes on your Raspberry Pi

I'll just leave this here.

Why use the Raspberry Pi?

If you have the budget to buy some used workstations, NUCs, or servers on eBay, they will perform better, but they also take up more space and draw more power. Fortunately, most of what we'll cover will apply to Raspberry Pi and whatever Intel kit you can get your hands on.

But, I thought SD cards were unreliable?

Now that the RPi 4 can boot from an SSD or NVMe over USB, the reliability of SD cards no longer needs to be a concern. The RPi 4 has much faster I/O, up to 8GB of RAM available along with a quad-core processor. Not only is it suitable for self-hosting many types of application, but it is likely to outperform smaller shared VPSes or EC2 instances since it's a dedicated host.

It's now trivial to build a cluster that can tolerate a node failure using K3s and its now embedded etcd mode which was introduced in K3s 1.19.

There are naturally some draw-backs to self-hosting: you have a dependency on your ISP's uptime, your significant other or children could unplug your infrastructure (yes it happens), but I think you will have fun and hone your skills.

So whether you're building a home-lab, or looking to host websites, APIs and automation tools, I hope you'll find the recommendations useful.

Picking tools and buying parts

Self-hosting Kubernetes on your Raspberry Pi

The parts from my latest K3s build

Bill of materials (hardware)

You will need at least one Raspberry Pi 4 with 2GB RAM or more. I would recommend that you buy three if you wish to run a HA cluster, that can tolerate a node failure.

Managing costs

Let me put this in bold: you do not need to spend a fortune to build a cluster. 4x RPi 4 including taxes in the UK runs to about 33 GBP per unit ~ about 130 GBP total. If you just want the experience of self-hosting, and to learn about Kubernetes without having to spend too much, you could buy a single 4GB RPi 4 for around 54 GBP including taxes.

A note on the RPi3

The Raspberry Pi 3 and CM3 are both having issues with K3s, particulary if you are running anything later than 1.17. If you have an existing investment (as I do), I would suggest selling them or giving them away.

Cases and cooling off

Whatever case you decide to use, make sure it has a fan. You can compare options in my post Cooling off your Raspberry Pi 4.

The best options for cases are either one of the sets of acrylic plates and stand-offs that can be bought for 20-25 USD on Amazon, or a professional-grade BitScope Cluster Blade. Whatever you do, do not buy the original first-generation BitScope Blade which doesn't support the RPi4 or have cooling.

Buy the best power adapter

Do not buy a multi-charger for the RPi4, they may appear to work, however if you look into things you will find that they are "browning out". Official adapters are not expensive, my advise is not to be stingy.


When it comes to storage, you will need at least a 32GB SD card, but I would encourage you to go one further and buy solid state storage. Read about the performance differences in my post: Upgrade your Raspberry Pi 4 with a NVMe boot drive. Some ask whether an M2 SSD is sufficient with the USB-bus of the RPi4 causing a bottleneck. They make a good point however the NVMe performs better, has a similar cost and can be used in one of your PCs at a later date.

Kubernetes - kubeadm vs. k3s?

Until late 2019, most of us were still struggling to run upstream Kubernetes on our RPis with kubeadm, and it was not a pleasant experience. Why? Because kubeadm was designed to run on high-performance Intel cloud servers, not on low-powered ARM SoCs.

Self-hosting Kubernetes on your Raspberry Pi

The headlines for K3s claim 500MB of RAM is required for a server, increasing 256MB when using embedded etcd. The claim for an agent is 50MB of RAM. Whatever the actual usage is, it's going to be much less than kubeadm and put less of a load on the system.

32-bit or 64-bit OS?

From my perspective the main limitation of a 32-bit OS is the 3GB process limit. That isn't something that I'm particularly worried about, but maybe you are? What the question really comes down to is: do you want to use Ubuntu or RaspiOS?

Self-hosting Kubernetes on your Raspberry Pi Self-hosting Kubernetes on your Raspberry Pi

The Raspberry Pi Foundation do have a beta version of RaspiOS out, however it isn't the lite edition and comes with a fully-loaded desktop.

In my experience with K3s, Ubuntu 20.10 works equally well as RaspiOS Lite on the RPi4. I tend to recommend with RaspiOS with its 32-bit OS because most Raspberry Pi users already use this and are comfortable with it.

Sourcing apps and charts

Sourcing apps and charts can be tricky with ARM, as many maintainers have not yet considering porting their software to this architecture.

For that reason I built the arkade project as an Open Source Kubernetes Marketplace. You can use it to install apps whether they are packaged as Helm charts, YAML manifests or shipped with their own CLI.

Why not use helm? Well in some instances it does, but where a package is not available for ARM, it will tell you. Where a chart has 150 configuration options, the --help text tells you the ones we think you'll need. The arkade info APP command gives you instructions on how to use the chart, and you can get that back at any time without installing the app again.

Self-hosting Kubernetes on your Raspberry Pi
There are now over 40 packages and many contributors. Notable packages that work on Raspberry Pi are: Minio, OpenFaaS, cert-manager, ingress-nginx, Traefik 2, Linkerd, SealedSecrets and the Kubernetes dashboard.

You can install of them using arkade install:

arkade install APP

arkade also offers a fast download of CLIs to prevent breaking the flow when you're following instructions. How many times have you ran "brew update" and then instantly regretted it?

arkade get kubectl [--version 0.19.5]
arkade get kubectx
arkade get helm
arkade get terraform

Each package is downloaded as a static binary and put in your PATH. It doesn't matter if you're using Intel or ARM, Linux, MacOS or Windows, the correct binary (if available) will be downloaded.

Warning: Whatever you do, resist the urge to run "third-party" or community builds of tools you recognise. This is asking for trouble, the code could be doing anything to your computers and network. See also: "Poison docker images"

Ingress and LoadBalancers

There are two main ways I would expose services on Kubernetes. The first is to use a LoadBalancer, this is what you will be most used to from using managed Kubernetes clusters. By setting your Service as a type: LoadBalancer and giving a number of ports, a cloud load-balancer is provisioned and starts routing traffic on the given IP address.

When running at home, you can use a tool like kube-vip or MetalLB, but they will only give you IP addresses on your local network, which external users cannot access. If you are thinking of using these tools, you may as well just use a NodePort and forego the additional tooling. A NodePort makes a high port number such as 31112 available on all your nodes, which then routes traffic to the exposed service.

Many homelab'ers will be aware of port-forwarding, where your home router can be set to forward traffic on a given set of ports to hosts on your internal network. This is achieved through mapping a high non-standard port number to the internal port. Unfortunately, many home users are inadvertently giving away their location, their ISP customer number (Virgin Media, I'm looking at you), and in some instance, a good estimation of their home address. Now on top of all of that, you cannot take that IP address with you, so if you close the lid of your laptop running K3s, and open the lid on a captive portal in a coffee shop, your services will no longer be available.

Likewise, your IP is likely to change. The mitigation is to run a daemon that continually checks your IP address and updates your DNS records.

That all sounds like a lot of hard work to me.

In the early days, I used to use Ngrok which tunnels out from your internal network, but unfortunately I kept running into its connection and rate-limiting. It also has no Kubernetes integration, and it's not for lack of trying - Joe Beda (a Kubernetes co-founder) once built a project to try and fix this problem.

Over the past year I've developed a tool called inlets which fixes the issues of port-forwarding and Ngrok: no manual configuration, no high ports, no exposing your location, the IP moves with you - even to a coffee shop and it has absolutely no artificial rate-limiting because you self host the tunnel server on your own VM.

inlets is a Cloud Native Tunnel built from the ground up to work in containers, VMs, Kubernetes, Linux, Windows, MacOS, ARM - you name it.

It scales better than a VPN, since you don't have to allocate IP addresses and subnets for each host. You can have thousands of inlets tunnels and set them up within a matter of minutes, where as customers have told me they often have to send an engineer or contractor on-site to configure their VPNs. Unlike an SSH tunnel, there's no need for additional tooling just to keep it running and it doesn't require extra configuration to go over a HTTP proxy.

You run the client inside your restrictive network, and the server on a public VM. When the two are connected you get to access your private service from the public VM's IP. Since the client speaks first, it tends to "just work".

Self-hosting Kubernetes on your Raspberry Pi

inlets integrates into Kubernetes using the inlets-operator and you can install it using arkade or helm:

arkade install inlets-operator \
 --provider digitalocean \
 --region lon1 \
 --token-file ~/do-api-key.json \
 --license-file ~/LICENSE

In the README you'll find out the differences between inlets OSS and PRO, for self-hosting PRO is recommended and more featureful. There are around half a dozen cloud providers available now, added by community members, but I prefer DigitalOcean for the fast boot-time and low cost (around 5 USD / mo).

After running that one command, any LoadBalancer you expose will create a VM on public cloud and deploy an inlets server. The inlets client will then be run within your cluster and you get to self-host whatever you like.

Now a well-known trick with Kubernetes is to only expose one LoadBalancer, and host all your sites on the same IP address through different DNS names. This can be done with Kubernetes Ingress.

Unless you configure K3s otherwise, it will add Traefik 1.x for you during installation. arkade ships with several IngressController apps that you can use such as Traefik 2 and Kong, but I tend to prefer ingress-nginx.

So whether you're using public cloud, or the inlets-operator and self-hosting at home, you now only have to pay for one VM or Cloud LoadBalancer instead of one per service.

Using inlets you can also get TLS certificates by running cert-manager and tunnelling port 80 and 443 from the exit-server to your Kubernetes ingressController.

Hosting services

You can build your own helm chart for a service and expose it via your IngressController.

For an example of a helm chart for Node.js, see my example of an Express.js microservice with a Dockerfile, Kubernetes YAMLs and a non-root user: alexellis/expressjs-k8s.

I am biased here, but I prefer to use OpenFaaS to host microservices and APIs. The workflow looks a bit like this:

arkade install openfaas
arkade install openfaas \
  --domain openfaas.example.com \
  --email alex@example.com

That gives me https://openfaas.example.com for my admin panel and https://openfaas.example.com/functions/ for anything I deploy.

From there I can use one of the many language templates in the store to create a function and deploy it:

faas-cli template store pull golang-http
faas-cli new 
  --lang golang-http \
  --prefix alexellis2 \

faas-cli publish --platforms linux/amd64,linux/arm/v7

faas-cli deploy -f https://openfaas.example.com

That publishes an image I can run locally or on my RPi.

I'll be able to access the API at: https://openfaas.example.com/functions/my-api

Custom domains such as my-api.example.com can also be deployed using the FunctionIngress add-on: TLS and custom domains for functions

You'll also be able to re-map your APIs and websites into a single REST style API.

OpenFaaS comes with built-in monitoring, scaling and a web UI:

Self-hosting Kubernetes on your Raspberry Pi

Pictured: Grafana dashboard showing function scaling, HTTP codes and execution duration.

Self-hosting Kubernetes on your Raspberry Pi

Pictured: The OpenFaaS UI showing a machine-learning model being used

Find out more: OpenFaaS docs


As far as CI/CD is concerned, if you're using OpenFaaS you can use the existing API to deploy functions and GitHub Actions to build multi-arch images, see this post by Utsav Anand: Build and deploy OpenFaaS functions with GitHub Actions.

If you picked a 64-bit Operating System, then Flux may be an option to look into, but requires considerably work to configure with a configuration repository detailing which manifests to apply, and a separate job to build and deploy images.

Remote access with kubectl

You can also expose your Kubernetes API server with a public IP, see my guide: Get kubectl access to your private cluster from anywhere.

Whist you won't need this to use OpenFaaS, you could use this in conjunction with a CI/CD pipeline if you are deploying with Helm or kustomize.


You are going to need a PersistentVolume for your stateful applications like Minio, or Prometheus - where files are stored on the filesystem of the container.

K3s ships with a "local path provisioner" which uses the host's storage for Persistent Volumes. It does have some limitations which you can read about here.

I won't claim to be an expert on self-hosted Kubernetes block storage, but Longhorn, NFS and Ceph all appear to be popular.

Just to show how quickly things are moving with ARM support, 5 days ago Longhorn v1.1.0 was released which adds support for 64-bit ARM OSes, like Ubuntu 20.10.

The OpenEBS project also gained beta support for 64-bit ARM in November 2020 in version 2.3.0

For the time-being, if you're using RaspiOS, the nfs-provisioner is worth trying:

arkade install nfs-provisioner

For object storage you can use Minio, which provides an S3-like API:

arkade install minio

Taking advantage of managed services

If I were to host an application that required a database, S3, or a Redis cache, then I'd probably use a managed service for those parts, particulary if I cared about the data. On DigitalOcean managed databases start at a reasonable 15USD / mo, and their managed S3 equivalent "Spaces" starts at 5USD / mo.

It's easy to rebuild a cluster, and redeploy stateless microservices, what's more difficult is managing stateful applications and recovering from data loss.


There is no doubt that hosting your own registry will be faster than using the Docker Hub's mirror on the West Coast of the USA, especially if you live in Europe like I do. This is not strictly necessary, but you should try it and see if it's for you.

With your IngressController tunneled with the inlets-operator, you can run a couple of arkade commands and get your own registry:

arkade install docker-registry
arkade install docker-registry-ingress \
  --email web@example.com \
  --domain reg.example.com \
  --ingress-class traefik

From there you'll get a valid TLS cert and can use it from anywhere, it also has auth:

docker login reg.example.com

docker tag alpine:3.11 reg.example.com/alpine:3.11
docker push reg.example.com/alpine:3.11
docker pull reg.example.com/alpine:3.11

See the full tutorial: Get a TLS-enabled Docker registry in 5 minutes


Now perhaps I was a little self-deprecating in the previous section. One thing I do know a lot about is netbooting, so much so that I took a week to build a workshop for my GitHub Sponsors so that they can save themselves that time. Putting aside a dollars per month is decent value when you consider how long it took to build and test the instructions.

Self-hosting Kubernetes on your Raspberry Pi

Pictured: a netbooting sequence that hasn't changed in 30 years.

containerd is the usual choice for running containers on Kubernetes clusters, but it doesn't support NFS, which is the main option you'll find for netbooted machines. It's what Mythic Beasts use for their own RPi cloud, and unfortunately why users cannot run K8s or Docker there.

My lab explains the workarounds and gives you what you need to netboot on RPi3 and RPi4. The neat thing is that once you host the root filesystem on another computer, you can increase the speed by backing that with NVMe.

This is what my NFS server looks like:

alex@nuc7:~$ ls /nfs/
1b9a8834  1befbd15  3c104aa8  7feb56e7	a7dc46e5  c3e76471

Whenever I run into an issue with a node, let's say the one with the serial-number c3e76471 - I just run rm -rf /nfs/c3e76471 and then my provisioning script and cycle power on the unit. When using the BitScope Cluster Blade, I can even cycle the power remotely using its microcontroller.

If you'd like to learn how netbooting works, you can read more in my article on The New Stack: Bare Metal in a Cloud Native World

For access to my advanced netbooting workshop for the Raspberry Pi, just sign up for my GitHub Sponsors on The Treasure Trove tier and click on the Insiders Update.

Putting it all together

Let's say that you wanted to put this all together so that you had a highly-available Kubernetes cluster with a public IP for its IngressController using K3s, Traefik, OpenfaaS and cert-manager. That's just like the setup I built for my KubeCon demo.

K3s can support two modes of multi-master / High-Availability using etcd or an external database:

k3sup is a tool that I built to install k3s over SSH.

Imagine you'd purchased three RPi4s with 4GB of RAM to be your servers, and two with 8GB of RAM to run your workloads. Here's how you could use k3sup to configure them:

set -e


echo Installing Server 1
k3sup install --user pi --ip \
  --k3s-channel $CH \

echo Installing Server 2
k3sup join --user pi --ip --server-ip \
  --k3s-channel $CH \

echo Installing Server 3
  --k3s-channel $CH \
  --server \

echo Installing Agent 1
  --k3s-channel $CH
echo Installing Agent 2
  --k3s-channel $CH

Then we'd use arkade to install the various helm charts and Ingress records we need:

# Install cert-manager
arkade install cert-manager

# Install OpenFaaS
arkade install openfaas

# Install inlets-operator
arkade install inlets-operator --provider digitalocean \
  --region lon1 --license-file ~/LICENSE \
  --token-file ~/Downloads/do-access-token

# Get the public IP from Traefik
INGRESS_IP=$(kubectl get svc/traefik -n kube-system -o jsonpath="{.spec.loadBalancerIP}")

# Create a DNS A record for that IP
doctl domain create selfhosted.example.com --ip-address INGRESS_IP

# Generate an Ingress record for OpenFaaS
arkade install openfaas-ingress \
  --ingress-class traefik \
  --email alex@selfhosted.example.com \
  --domain selfhosted.example.com

export OPENFAAS_URL=https://selfhosted.example.com

# Log in to openfaas
faas-cli login

I can then create a new function and deploy it:

docker login --username alexellis2

# Deploy a function (alexellis2 is my Docker Hub name)
faas-cli new --lang node12 \
  --prefix alexellis2 \

faas-cli publish --platforms linux/arm/v7,linux/arm64,linux/amd64

faas-cli deploy

# Invoke the function
curl https://selfhosted.example.com/function/slashcommand

Wrapping up

Over the course of 5 years I've written many tutorials about containers and clustering on Raspberry Pi. Things are sill evolving, and support is getting better. If things were rough last time you tried it out, give it another shot.

If you enjoyed the tutorial, then try it out over the holidays and let me know what you think of inlets. You can reach me on Twitter @alexellisuk

My KubeCon talk

You can watch my talk from KubeCon and learn the various ways to build a cluster, see a live demo of deploying to my RPis from GitHub and learn what I am using for my own cluster.

The Past, Present, and Future of Kubernetes on Raspberry Pi

"Kubernetes isn't for me"

Now if Kubernetes is "not for you", or your budget won't allow for a few Raspberry Pis at this time, then that's understandable and there is another way.

Keeping costs low

Together with the OpenFaaS community I launched a new project called faasd. faasd hosts APIs, microservices, websites and functions using a single node, instead of a whole Kubernetes cluster. That makes it much cheaper, but in the event of the node crashing, you may have some downtime. That's the main trade-off vs. cost and complexity.

Why not checkout my new faasd eBook on Gumroad?

Within a few minutes you'll be deploying your own functions with TLS enabled. Just write a few lines of code and deploy from your laptop or your favourite CI/CD solution. Best of all, you can run it on a 5USD VM or even a Raspberry Pi.
You'll learn how to deploy code in any language, lift and shift Dockerfiles, run requests in queues, write background jobs and to integrate with databases. faasd packages the same code as OpenFaaS, so you get built-in metrics for your HTTP endpoints, a user-friendly CLI, pre-packaged functions and templates from the store and a UI.

For me, I found that hosting my GitHub Sponsors portal on a Raspberry Pi 3 worked out better than maintaining a cloud instance. So whenever you log into The Treasure Trove, you're actually hitting a Raspberry Pi in my house through an inlets tunnel.

"Self-hosting isn't for me"

That's entirely understandable - modern Cloud Native infrastructure can be hard to maintain, and you may not have anything you want to host.

Just keep an eye on your cloud costs, particulary when you're doing local development and consider whether having a machine on your local network could save you some of that spend over the course of the next one or two years.

Learn how to build your own lab


Self-hosting Kubernetes on your Raspberry Pi

Check out my netbooting workshop

You may also like



<![CDATA[Bare-metal Kubernetes with K3s]]>https://blog.alexellis.io/bare-metal-kubernetes-with-k3s/5fe0c022759e7300016c1839Mon, 21 Dec 2020 16:13:44 GMT
Bare-metal Kubernetes with K3s

Learn how to configure K3s on bare-metal to run a Kubernetes cluster with just as much resilience and fault tolerance as a managed service.

This tutorial is a follow-on from my post Kubernetes on bare-metal in 10 minutes from 2017. The original post focused on getting Kubernetes working across a number of bare-metal hosts running Ubuntu, and then it went on to deploy a microservice and the dashboard. It was a primer, and written to help new users kick the tires with Kubernetes. It used production-ready kubeadm, but only used one master node, meaning it couldn't tolerate a failure.

In this post, I want to provide you with a HA, production-ready cluster on bare-metal infrastructure. We will use Equinix Metal for the hosts, automate the boring parts, but do the rest step by step, so that you can get an understanding on what's going on under the hood.

In the conclusion, we will review the setup and discuss other options for running on-premises or in a homelab with Raspberry Pis. We will also compare kube-vip to other solutions for bare-metal and self-hosted networking.

Bare-metal Kubernetes with K3s

Conceptual diagram: kubectl access provided by the stable EIP. The IngressController also has an EIP which routes traffic for services exposed with Ingress such as the OpenFaaS gateway.

This post also moves from kubeadm to using K3s (a CNCF project), which requires fewer resources for its control-plane and has a built-in HA mode using embedded etcd. K3s has been generally available (GA) and production ready since 2019.

Follow me on Twitter @alexellisuk


We will use Terraform to create the nodes on Equinix Metal (aka Packet) servers, k3s to create a HA control-plane, and kube-vip to configure a HA IP address for the API server.

The setup will be able to tolerate at least one failure of a server. So unlike the 2017 post, our servers will form a HA cluster, and will also have an Elastic IP (EIP) configured so that the Kubernetes API server will remain accessible if one or more of the servers becomes unavailable.

In a Kubernetes cluster, the Cloud Controller Manager add-on has several responsibilities including node management, routing and managing services. At the end of the tutorial, not only will your cluster have a HA control-plane, a stable IP for the API Server, but Equinix Metal's (CCM) will also allow you to expose services as type LoadBalancer. Each IP address costs around 3.25 USD / mo, you can find out more details on the IP Addresses section of the dashboard.

Skill level: intermediate to advanced. You should be familiar with Kubernetes, networking and public cloud infrastructure.

Install the pre-requisites

Download arkade, a portable Kubernetes marketplace and downloader for DevOps CLIs. It will be used to download the tools we need for the tutorial. You are of course welcome to do things the hard way, if you prefer.

curl -sLS https://dl.get-arkade.dev | sh

# Install the binary using the command given as output, such as: 
sudo cp arkade-darwin /usr/local/bin/arkade

# Or on Linux:
sudo cp arkade /usr/local/bin/arkade

Download the following CLIS:

  • kubectl - the Kubernetes CLI
  • k3sup - a k3s installer that works over SSH
  • terraform - an Infrastructure as code (IaC) tool, which we'll used to create the initial hosts
  • kubectx - a fast way to check and switch your Kubernetes context between clusters
arkade get kubectl
arkade get k3sup
arkade get terraform
arkade get kubectx

Follow the step to put these binaries into your PATH:

# Add (k3sup) to your PATH variable

Provision your nodes

We will use terraform to provision three servers and two agents. You can alter these numbers, but three is the minimum number required for K3s' built-in HA mode using etcd.

You may well ask why the entire blog post isn't just a single Terraform command. I can hear you Hacker News. The point of this post isn't to do all the work for you, but to help you understand what is required, and in what order. Creating hosts is boring, so we are going to automate that and save you some time.

Save main.tfvars with the following:

terraform {
  required_providers {
    packet = {
      source = "terraform-providers/packet"
      version = "~> 3.2.1"
  required_version = ">= 0.13"

variable "api_token" {
  description = "Equinix Metal API token"

variable "project_id" {
  description = "Equinix Metal Project ID"

variable "servers" {
  description = "Count of servers"

variable "agents" {
  description = "Count of agents"

provider "packet" {

resource "packet_ssh_key" "key1" {
  name       = "k3sup-1"
  public_key = file("/home/alex/.ssh/id_rsa.pub")

resource "packet_device" "k3s-server" {
  count		         = var.servers
  hostname         = "k3s-server-${count.index+1}"
  plan             = "c1.small.x86"
  facilities       = ["ams1"]
  operating_system = "ubuntu_20_10"
  billing_cycle    = "hourly"
  project_id       = var.project_id
  depends_on       = [packet_ssh_key.key1]

resource "packet_device" "k3s-agent" {
  count 	         = var.agents
  hostname         = "k3s-agent-${count.index+1}"
  plan             = "c1.small.x86"
  facilities       = ["ams1"]
  operating_system = "ubuntu_20_10"
  billing_cycle    = "hourly"
  project_id       = var.project_id
  depends_on       = [packet_ssh_key.key1]

output "server_ips" {
  value = packet_device.k3s-server.*.access_public_ipv4

output "agent_ips" {
  value = packet_device.k3s-agent.*.access_public_ipv4

Then create main.tfvars

api_token   =   ""
project_id  =   ""
servers     =   3
agents      =   2

Edit api_token and project_id with the values from the Equinix Metal dashboard. The API key is found on your user profile.

If you wish to change the size of the nodes, you can edit c1.small.x86 and use a different value. You can find the options in your dashboard.

Now run the terraform to create the hosts:

terraform init
terraform plan -var-file=main.tfvars
terraform apply -var-file=main.tfvars

When the hosts have been created, you'll get output with your IP addresses.

agent_ips = [
server_ips = [

Open a terminal and write environment variables for each:

export SERVER1=""
export SERVER2=""
export SERVER3=""

export AGENT1=""
export AGENT2=""

Save this file as hosts.txt as you may need it again later.

Obtain an IP address for the control-plane

K3s can run in a HA mode, where a failure of a master node can be tolerated. This isn't enough for public-facing clusters, where a stable IP address for the Kubernetes control-plane is required.

We need a stable IP for port 6443, which we could also call an Elastic IP or EIP. Fortunately BGP can help us there. One of our three master nodes will advertise its IP address as the route for the EIP, then if it goes down, another will start advertising. This means our clients can always depend on a stable address of: https://$EIP:6443 for connecting to the Kubernetes API server.

Obtain an IP address from your EM dashboard:

  • IPs & Networks

  • Request IP Address

  • Select the same region that you'll use for the nodes:

  • Pick a /32 for a single IP for the control-plane.

It may take some time for an engineer to approve the request.

Once you have the IP, set it in an environment variable:

export EIP=""

Add this line to hosts.txt so that you have it, if you need it later.

Enable BGP for each server

Click on each server in the Equinix Metal dashboard. Click on its details page, then BGP. Find the Manage button on the IPv4 row and click "Enable BGP".

This setting allows BGP to be advertised from this server.

Provision the first master

For various reasons, we can only set up kube-vip after K3s has started, so we will create the first master node:

k3sup install \
  --ip $SERVER1 \
  --tls-san $EIP \
  --cluster \
  --k3s-channel latest \
  --k3s-extra-args "--no-deploy servicelb --disable-cloud-controller" \
  --merge \
  --local-path $HOME/.kube/config \
  • --tls-san is required to advertise the EIP, so that K3s will create a valid certificate for the API server.
  • the --k3s-channel is specifying the latest version of K3s, which in this instance will be 1.19, by the time you run this tutorial, it may have changed, in which can you can give 1.19 as the channel, or a specific version with --k3s-version
  • note the --cluster flag, which tells the server to use etcd to create a cluster for the servers we will join later on
  • --local-path, --context and --merge all allow us to merge the KUBECONFIG from the K3s to our local file.
  • in --k3s-extra-args we disable the built-in K3s service load-balancer, and also disable the cloud-controller, because we will use the Equinix Metal Cloud Controller Manager instead.

Now check your kubeconfig file is pointing at the right cluster:

kubectx em-ha-k3s

Try querying the nodes in the cluster:

[alex@nuc ~]$ kubectl get nodes
NAME           STATUS     ROLES         AGE   VERSION
k3s-server-1   Ready      etcd,master   7s    v1.19.5+k3s2

Apply the RBAC so that kube-vip can access the Kubernetes API and configure services:

kubectl apply -f https://kube-vip.io/manifests/rbac.yaml

Now log into the first server using SSH. There is a one-time configuration option required for each server.

ssh root@$SERVER1

ctr image pull docker.io/plndr/kube-vip:0.3.0

alias kube-vip="ctr run --rm --net-host docker.io/plndr/kube-vip:0.3.0"
export INTERFACE=lo
export EIP=""

kube-vip vip /kube-vip manifest daemonset \
  --interface $INTERFACE \
  --vip $EIP \
  --controlplane \
  --services \
  --inCluster \
  --taint \
  --bgp \
  --packet \
  --provider-config /etc/cloud-sa/cloud-sa.json | tee /var/lib/rancher/k3s/server/manifests/vip.yaml

The manifest for kube-vip will be created at /var/lib/rancher/k3s/server/manifests/vip.yaml, any files placed here will be run by K3s.

Log out of the server

Deploy the Equinix Metal Cloud Controller Manager

Now deploy the Equinix Metal Cloud Controller Manager (CCM), which will create the /etc/cloud-sa/cloud-sa.json file referenced in the command above.

As configured, the CCM will obtain IP addresses from the Equinix Metal API for any services that are exposed.

Create a secret as follows:

export API_KEY=""
export PROJECT_ID=""

cat <<EOF > ccm-secret.yaml
apiVersion: v1
kind: Secret
  name: packet-cloud-config
  namespace: kube-system
  cloud-sa.json: |
      "apiKey": "$API_KEY",
      "projectID": "$PROJECT_ID"

Now apply the secret, then the modified CCM for use with kube-vip:

kubectl apply -f ./ccm-secret.yaml
kubectl apply -f https://gist.githubusercontent.com/thebsdbox/c86dd970549638105af8d96439175a59/raw/4abf90fb7929ded3f7a201818efbb6164b7081f0/ccm.yaml

The gist contains a forked version of the Equinix Metal CCM. This was developed by the author of kube-vip and will be upstreamed, at which point the URL will redirect to the original repo for the CCM.

Check that the CCM and kube-vip pods have started:

kubectl get pods -n kube-system -w

kube-vip-ds and packet-cloud-controller-manager should be "Running", and have no errors.

Provision the additional master servers

Now that we have the first server up and running, and advertising its EIP, we can use it to join our other servers, and later our agents using the EIP.

This is important, because if we used the IP of one of the servers, and that server went down, the agent would no longer be able to communicate with the API server.

k3sup join \
  --ip $SERVER2 \
  --server \
  --server-ip $EIP \
  --k3s-channel latest
  • --server - tells K3s to join the cluster as a server
  • --server-ip - note that we specified the EIP, and not the IP of any one server here

This command will write a kubeconfig file into your local directory, ignore it and continue to use the one that was merged into your kubeconfig file.

Add the third server:

k3sup join \
  --ip $SERVER3 \
  --server \
  --server-ip $EIP \
  --k3s-channel latest

Add your agents

To add an agent (or worker) node, use the join command from above, but remove the --server flag.

agents=("$AGENT1" "$AGENT2")

for i in ${agents[*]} 
    k3sup join \
    --ip $i \
    --server \
    --server-ip $EIP \
    --k3s-channel latest

Switch your KUBECONFIG to use the EIP

From your own computer, switch your KUBECONFIG to use the EIP instead of the SERVER1 IP which k3sup will have set for you by default.

Edit ~/.kube/config and replace the IP $SERVER1 with the $EIP value.

On MacOS:

export EIP=""
export SERVER1=""

sed -ie s/$SERVER1/$EIP/g ~/.kube/config

Then check the file is pointing at the EIP:

cat ~/.kube/config |grep $EIP

Try using kubectl with the new EIP in place:

kubectl get nodes -o wide

Deploying workloads

The arkade tool we installed earlier not only installs CLIs, but is an open-source Kubernetes marketplace. You can use it on your Raspberry Pi, your Equinix Metal cluster, at work, on your laptop, wherever you want. It provides around 40 Kubernetes applications and is open source, so you can fork it and add your own.

kube-vip works in tandem with the CCM. The CCM will obtain a new IP address for you from Equinix Metal's API, then assign it to any LoadBalancer service in the Pending state. kube-vip then starts advertising the IP address and routing traffic to the nodes.

The quickest example you can find is taken from the inlets-operator README:

kubectl apply -f https://raw.githubusercontent.com/inlets/inlets-operator/master/contrib/nginx-sample-deployment.yaml
kubectl expose deployment nginx-1 --port=80 --type=LoadBalancer

The IP will show up relatively quickly, within a few seconds:

kubectl get svc -o wide -w
kubernetes   ClusterIP     <none>        443/TCP        151m   <none>
nginx-1      LoadBalancer   <pending>     80:30822/TCP   3s     app=nginx
nginx-1      LoadBalancer   <pending>     80:30822/TCP   5s     app=nginx
nginx-1      LoadBalancer   80:30822/TCP   5s     app=nginx

NGINX_IP=$(kubectl get svc/nginx-1 -o jsonpath="{.spec.loadBalancerIP}")

curl -s http://$NGINX_IP | grep "<title>"
<title>Welcome to nginx!</title>

kubectl delete svc/nginx-1

To get the Kubernetes dashboard up and running simply run:

arkade install kubernetes-dashboard

The output will tell you how to obtain a token and how to log into the dashboard. You can get this information back at any time by typing in:

arkade info kubernetes-dashboard

By default Traefik 1.x is installed with K3s, you can disable that or delete it and then install your preferred IngressController with:

kubectl delete svc/traefik -n kube-system
kubectl delete deploy/traefik -n kube-system

arkade already ships with four options for IngressControllers. You can also run arkade get helm and then use your favourite helm chart, if it's not already available below.

arkade install ingress-nginx
# Or
arkade install traefik2
# Or
arkade install nginx-inc
# Or
arkade install kong-ingress

Add cert-manager:

arkade install cert-manager

Once you find the IP address of your IngressController, you can create a DNS record for any ingress records you may create, and then get TLS certificates for free from LetsEncrypt.

To install OpenFaaS, with an encrypted TLS endpoint, run:

export DOMAIN=gateway.example.com
arkade install openfaas
arkade install openfaas-ingress \
  --email webmaster@$DOMAIN \
  --domain $DOMAIN

In this example, you would need to create a DNS A record for gateway.example.com using the IP of your IngressController. Then you can go ahead and log into OpenFaaS or open it in a browser using its URL of: https://gateway.example.com.

To host a Docker registry simply run:

export DOMAIN=registry.example.com
arkade install docker-registry
arkade install docker-registry-ingress \
  --email webmaster@$DOMAIN \
  --domain $DOMAIN

To get the domain working, you'd just create the DNS record as per above for registry.example.com and point it at the IngressController.

By using Ingress, you can save on costs and on IP addresses, which as I mentioned in the intro cost around 3.25 USD / mo.

Linkerd and Istio are also just one command way.

Type in arkade install --help for a complete list.

Wrapping up

In the tutorial, we showed how kube-vip can integrate with BGP when used on Equinix Metal's bare-metal cloud to provide a stable IP.

The control-plane is HA and can tolerate at least one failure, because we configured K3s to use its built-in clustering mode. The built-in clustering mode uses and embedded version of etcd to synchronise the servers.

The API server on port 6443 was made HA and can also tolerate a failure. This is because the IP address of the API Server was the EIP, provided by BGP. We updated our kubeconfig file and our additional servers and agents to use the EIP for their cluster join commands.

The Cloud Controller Manager also uses BGP to provide an IP address for any services you wish to expose with a Kubernetes LoadBalancer.

Whilst this blog post took much longer to write, and is more involved than the initial post from 2017, it produces a cluster that is highly available, fast, and ready to run production workloads.

Bare-metal without BGP

If you are running Kubernetes at your workplace or within your homelab and do not have access to BGP, then there are other options.

For instance, you may be able to use ARP instead. ARP will broadcast a virtual IP (VIP) within your local network, so that your clients can still use a stable IP address. ARP can also be used with your Raspberry Pi clusters, but bear in mind that K3s is having issues with the older RPi3 and CM3.

DNS round-robin is also an option for configuring a stable endpoint, but caching and the time taken to propagate to your clients could make it less practical.

Alternatively, if you are using a managed cloud which offers native L4 load-balancers you do not need to use a tool like kube-vip. Clouds such as AWS, GCP, Azure, and DigitalOcean all offer their own managed load-balancers.

See an example of using DigitalOcean's managed LB in a post I recently wrote for Rancher: Set up Your K3s Cluster for High Availability on DigitalOcean

The other missing piece for on-premises clusters is the lack of Ingress, of reachable, routable IP addresses. This can be a particular problem if firewall ports cannot be opened, or port-forwarding rules are unavailable. Then solutions like Cloudflare's Argo Tunnel provide a way to expose services.

More recent open source projects like the inlets-operator integrate directly into Kubernetes like a Cloud Controller Manager to provide TCP tunnels for services you expose, such as your IngressController or a microservice. inlets PRO has also become popular for self-hosting on a Raspberry Pi cluster or homelab.

MetalLB is also a popular tool for on-premises Kubernetes networking, however its primary use-case is for advertising service LoadBalancers instead of advertising a stable IP for the control-plane. kube-vip handles both use-cases, and is under active development by its author, Dan.

Further resources

Follow me on Twitter @alexellisuk

Here are two additional blog posts on K3s and advanced networking, which may be of interest to you:

The tools we used are open source and available on GitHub:

  • k3sup - K3s installer using SSH
  • arkade - portable Kubernetes marketplace
  • kube-vip - Kubernetes control-plane and service load-balancer, developed by Daniel Finneran
  • inlets-operator - get Ingress and TCP LoadBalancers on private hosts behind a firewall or NAT

For full disclosure: Equinix Metal is a client of OpenFaaS Ltd. Thank you to the team for providing clients and to Dan for bringing support to kube-vip for K3s.

<![CDATA[containerd development with multipass]]>https://blog.alexellis.io/containerd-development-multipass/5fd39342759e7300016c1816Fri, 11 Dec 2020 16:22:49 GMT

About 18 months ago I started a project which developed directly against containerd. This presented a problem which I'd not really encountered before - Docker and Kubernetes on my Mac were no longer enough, I needed a Linux environment.

To begin with I just used an old 2016 model Dell XPS which gave me everyting I needed, but when others started to contribute, they were using Macs and so we had a problem. Multipass was the answer to our woes and we were pleasantly surprised by it and wondered why more people weren't using it every day.

I want to tell you a bit about our experience in the OpenFaaS community developing faasd - a portable FaaS framework, just like OpenFaaS, but without the complexity and overheads of a Kubernetes cluster.

What's containerd ever done for us?

Some time ago the original version of Docker was actually written in Python, and then morphed over time into a Go re-write. The Go version was seen as monolithic by some consumers, particularly the Kubernetes community.

Docker did many things, and at one point clustering and multi-node orchestration was even added to that list (think Docker Swarm and Docker EE)

As the codebase was refactored two projects emmerged: containerd and runc.

  • runc was a tiny Go binary that had one job: run a container based upon a spec. runc was also the driver for the OCI specification
  • containerd's job was to get things ready for runc - such as pulling images and defining specs

Docker then remained as a thiner layer on top of both of these tools to bring a user-friendly developer-experience, networking, high-level API and CLI.

Over time containerd has shifted into the lime-light and in Kubernetes 1.20, it will take over duties for running containers in Kubernetes clusters. Now not a lot changes, because containerd was always there along with runc, we just skip a few levels of indirection.

containerd doesn't provide networking out of the box, and that was one of the hardest challenges.

Not because it's technically complex, but there was a severe lack of documentation.

Container Network Initiative (CNI) filled a gap for us and enabled us to build a network between our containers.

When you squint at faasd, you see something that looks a lot like a single-node Kubernetes cluster, using the same projects you'd find on most nodes: containerd, runc and CNI.

Why would you develop with containerd?

There were two main reasons for creating "faasd" - the first was that we were hearing from users that they didn't want to run an entire Kubernetes cluster just to run a handful of functions, APIS or webpages. The second was that I fancied doing some learning and low-level coding.

Whilst containerd has a socket available, and can be mounted or forwarded, it doesn't work as you would expect. The containerd client tries to run containers on the host it's executing on and just synchronises state with the containerd socket. I found that very confusing, but was reassured that is the way it was designed to work.

You can see our code which has developed into two main services shipped in a single binary.

  • faasd - starts all the core services for OpenFaaS: the gateway, NATS, Prometheus and our queue worker. A docker-compose file is used to define versions of images, and the dependency graph for starting up services.
  • faasd-provider - a HTTP interface that performs invoke and CRUD for functions and secrets

Both are installed with systemd unit files, it was refreshing to lean on the host system for once, instead of using abstractions.

The short-version

multipass is a Canonical and its open source components and usage instructions are available on GitHub.

Multipass is a lightweight VM manager for Linux, Windows and macOS. It's designed for developers who want a fresh Ubuntu environment with a single command.

Since it supports metadata for cloud-init, you can simulate a small cloud deployment on your laptop or workstation.

On MacOS it is currently using hyperkit to run VMs, which is another thing we can thank Docker for building. On Windows and Linux it uses different virtualization technology, but has the same simple user-experience which means we get to write one tutorial and to be done with it. As a busy maintainer, I see that as a big win.

Here's how you get a VM launched with the latest version of Ubuntu Server:

multipass launch --name faasd

Shell into the VM:

multipass exec faasd /bin/bash

We then realised that multipass supported cloud-init, so morphed our README to the following:

curl -sSLO https://raw.githubusercontent.com/openfaas/faasd/master/cloud-config.txt
multipass launch --cloud-init cloud-config.txt  --name faasd

I had the pleasure of speaking to the engineering manager and PM for multipass on Zoom today and they told me you can even pass the cloud-init file as a URL, so our tutorial becomes even simpler:

multipass launch --cloud-init https://raw.githubusercontent.com/openfaas/faasd/master/cloud-config.txt  --name faasd

From there, you have a full Linux system with a working version of containerd and Container Network Initiative (CNI) plugins running, and most importantly faasd is up and running.

faas-cli list
faas-cli store deploy figlet
faas-cli invoke figlet <<< "faasd"

Whilst I've not used it yet, I'm told you can also mount folders from your base system to synchronise your GOPATH. So you could write code using VSCode and have the built-in terminal pane running "multipass exec /bin/bash" or an ssh session.

I recently wrote a post on One last trip down memory lane with the Raspberry Pi Zero where I tried to port faasd to a Raspberry Pi Zero. The Go build for faasd was taking so long that I gave up, opened multipass and cross-compiled it. That whole process was quicker than waiting for the poor little armv6 service to finish its work.

multipass for Kubernetes

You can also use multipass to run other workloads, I tried to deploy Kubernetes with microk8s, but ran into some issues with the default limits.

First of all: there was not enough RAM alloacted, then there was not enough disk, finally there were not enough vCPUs. After working all that out I came up with the following:

multipass launch --name microk8s -m 8G -c 2 -d 80G

Now it's not that much of a surprise that we never ran into that issue, because faasd is designed to be ridicuously lean. It even runs on a Raspberry Pi 3 which only has 1GB of RAM.

Wrapping up

faasd and containerd are written in Go. Checkout my new book Everday Go for practical tips and hands-on examples.

multipass has been useful for us whenever we need to access a Linux VM from a Mac. It could even be used for running a Kubernetes cluster, but I would usually prefer to deploy Kubernetes in a Docker container using either KinD or k3d for the sheer speed and efficiency of it.

multipass is a much leaner alternative to tooling like VirtualBox and Vagrant. The team are looking for feedback and are already planning for a way to launch custom images. Think: multipass launch openfaas or multipass launch gitlab for instance.

We see multipass being an important part of enabling collaboration with users from all-over the world, whether they use Linux, MacOS or Windows on their desktop.

Around 20 people have contributed to faasd directly, and many more indirectly. It worked very well for us and we believe that multipass deserves more attention.

Go and try it out, let them know what works for you and where it can be improved for your workflow.

Users have already suggested using qemu on MacOS, and the new Virtualization.Framework introduced in Big Sur could also have some impact on the future roadmap and M1 support.

What about faasd vs. OpenFaaS on Kubernetes?

faasd now fills a nice gap where Docker Swarm used to live in the OpenFaaS ecosystem. If you're working your way up to production with Kubernetes, or already have experience then you may benefit from a cluster and installing OpenFaaS to it.

If you're wanting to run a few functions, start small, keep costs down, then faasd may be a better fit. Managing and keeping up with Kubernetes versions can be its own challenge. So if you want to deploy code for a customer and barely give it another thought, then package faasd as a VM or cloud-init script and be done with it.

faasd can be run for 5-10 USD on a cloud VPS, or on your Raspberry Pi for free using an inlets tunnel to get it a public IP address.

<![CDATA[One last trip down memory lane with the Raspberry Pi Zero]]>https://blog.alexellis.io/memory-lane-raspberry-pi-zero/5fca33e8759e7300016c1805Fri, 04 Dec 2020 17:08:40 GMTOne last trip down memory lane with the Raspberry Pi Zero

The Raspberry Pi Zero and original Raspberry Pi both have a 32-bit ARM architecture which many projects have dropped support for. So when I saw that containerd recently merged a fix for building containerd on armv6, started to think what I could do with it.

Earlier that month, a user of the faasd project which runs very well on VPSes, RPi 3 and RPi4 asked me whether it could work on the Zero and I told him it was not practical or worth his time. After all, an RPi4 with 2GB of RAM, 4 Cores and much faster I/O is only 25GBP/EUR/USD.

faasd is OpenFaaS, but for a single node, and designed for those who don't want to pay for and manage an entire Kubernetes cluster, GitOps, service-mesh, and IngressController just to deploy a few simple functions.

My past experiences told me that it would be a challenge to get the zero working. It has such a paltry amount of memory available and its I/O is really slow. Still, sometimes it's fun to do things that we shouldn't. Had I been "nerd sniped"?

One last trip down memory lane with the Raspberry Pi Zero > The RPi Zero originally released in 2015.

In this post I'll show you what you need to do to deploy functions through the OpenFaaS CLI and be able to invoke them, including multi-arch builds. At the end of the post I'll explain what the limitations are and whether we should leave the Raspberry Pi Zero with those fond memories we have of it when it was released in 2015, 5 years ago.



You will need:

  • Raspberry Pi Zero
  • Ethernet adapter and USB <> USB A micro shim
  • 16-32GB class 10 SD card
  • MicroUSB power adapter

Flash the latest version of Raspberry Pi OS Lite to your SD card, and then create an ssh file in the /boot/ folder.

Power up the Raspberry Pi and ssh to it with ssh pi@raspberrypi.local.

Install go

You can install Go from a package manager, but the version is likely to be rather old. Fortunately the Go team still ship a binary for armv6:

export ARCH="armv6l"
echo "Downloading Go"

curl -SLsf https://dl.google.com/go/go1.13.15.linux-$ARCH.tar.gz --output /tmp/go.tgz
sudo rm -rf /usr/local/go/
sudo mkdir -p /usr/local/go/
sudo tar -xvf /tmp/go.tgz -C /usr/local/go/ --strip-components=1

export GOPATH=$HOME/go/
export PATH=$PATH:/usr/local/go/bin/

You now have Go installed, run go version to see it working.

Build containerd

faasd uses containerd rather than Docker, but there are no official binaries for any of the ARM CPUs, so we have to build them from source.

For armv7 and ARM64, you can use my repo alexellis/containerd-arm. It's my hope to see ARM support upstreamed, but having spoken on several occasions to the containerd team, it seems very unlikely that we will see this happen.

On my first attempt of building I got the following error:

+ bin/containerd
# github.com/containerd/containerd/cmd/containerd
/usr/local/go/pkg/tool/linux_arm/link: running gcc failed: fork/exec /usr/bin/gcc: cannot allocate memory
make: *** [Makefile:188: bin/containerd] Error 2

The RPi Zero and the largest RPi 1 only has 512MB of RAM and I slowly watched the RAM being eaten up with watch -n "free -h".. I should have remembered this.

Fortunately I had the battle scars and knew what I had to do:

sudo dd if=/dev/zero of=/swapfile bs=1024 count=1M &&\
  sudo mkswap /swapfile && \
  sudo swapon /swapfile

This creates 1GB of swap, on the SD card.. which already has terrible I/O, but it may just work.

The other trick you may have seen me talk about is adding gpu_mem=16 to /boot/config.txt. It seems pointless changing the split on an 8GB RPi4, but for our RPi Zero, we need every MB we can get.

Second time lucky?

No. It failed again due to some missing libraries. I reminded myself to read BUILDING.md from the containerd repo. After adding libseccomp and a few other packages, it started to move along again.

+ bin/ctr
+ bin/containerd
+ bin/containerd-stress
+ bin/containerd-shim
+ bin/containerd-shim-runc-v1
+ bin/containerd-shim-runc-v2
+ binaries

I don't know if it was 10 minutes or over an hour, but it was slow progress.

I then ran sudo make install and copied the systemd unit file into place with:

sudo cp containerd.service /etc/systemd/system/containerd.service
sudo systemctl enable containerd

I didn't want to start containerd at this time, to save on memory for the next task.

Build faasd

The faasd Makefile needed a patch because it was set to cross-compile to armv7, but we needed armv6.

.PHONY: dist
	CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=6 go build -mod=vendor -ldflags $(LDFLAGS) -a -installsuffix cgo -o bin/faasd-armhf

This took so long to build on the zero that I gave up and built it on my Intel NUC. Fortunately Go is very good as cross-compiling, especially when linking into C/C++ libraries is disabled with CGO_ENABLED=0.

I then ran scp to copy the binary to the Raspberry Pi.

sudo cp faasd-armhf /usr/local/bin/faasd
cd go/src/github.com/openfaas/faasd

The final step was to install faasd which creates two systemd unit files:

  • faasd - for the OpenFaaS core services
  • faasd-provider - for the provider that supports CRUD and Invoke operations
sudo faasd install

One last trip down memory lane with the Raspberry Pi Zero

Unexpected issues

I tried to log in with faas-cli login, but it didn't work. I saw a number of errors in the logs sudo systemctl journalctl -u faasd:

Dec 03 23:18:49 zero-dns default:basic-auth-plugin[13218]: standard_init_linux.go:207: exec user process caused "exec format error"
Dec 03 23:18:50 zero-dns containerd[12169]: time="2020-12-03T23:18:50.338589900Z" level=info msg="starting signal loop" namespace=default path=/run/containerd/io.containerd.runtime.v2.task/default/gateway pid=13324
Dec 03 23:18:52 zero-dns default:gateway[13335]: standard_init_linux.go:207: exec user process caused "exec format error"

It turned out that out of nats, prometheus, and the openfaas gateway, the openfaas basic-auth plugin only nats was available for armv6. It seems that projects really have moved on and left armv6 behind.

Then I had an idea

What if we didn't run the whole OpenFaaS stack at all, but just the faasd-provider? It would allow this tiny device to support all the CRUD operations on functions, and invocations, but nothing more.

The provider seemed to be running well, and responding to faas-cli commands:

One last trip down memory lane with the Raspberry Pi Zero

We'd lose the queue and async invocations, lose the UI, lose the metrics, but we could still deploy functions and invoke them.

It was already very late by this point so I decided to sleep on it.

In the morning

I powered up the RPi zero and tried deploying a container directly to the faasd endpoint, and it worked.

One last trip down memory lane with the Raspberry Pi Zero

Now, I had to make a few changes. Rather than building with the OpenFaaS watchdog in the container, I just used a Dockerfile and a plain Go HTTP server.

Here's what I ended up with:


FROM --platform=${BUILDPLATFORM:-linux/amd64} golang:1.13-alpine3.11 as build
RUN apk --no-cache add git


RUN mkdir -p /go/src/handler
WORKDIR /go/src/handler
COPY . .

# Run a gofmt and exclude all vendored code.
RUN test -z "$(gofmt -l $(find . -type f -name '*.go' -not -path "./vendor/*" -not -path "./function/vendor/*"))" || { echo "Run \"gofmt -s -w\" on your Golang code"; exit 1; }


    go build --ldflags "-s -w" -a -installsuffix cgo -o handler .
RUN GOOS=${TARGETOS} GOARCH=${TARGETARCH} go test ./... -cover

FROM --platform=${TARGETPLATFORM:-linux/amd64} alpine:3.12
# Add non root user and certs
RUN apk --no-cache add ca-certificates \
    && addgroup -S app && adduser -S -g app app \
    && mkdir -p /home/app \
    && chown app /home/app

WORKDIR /home/app

COPY --from=build /go/src/handler/handler    .

RUN chown -R app /home/app

USER app

CMD ["./handler"]


package main

import (

func main() {
	s := &http.Server{
		Addr:           fmt.Sprintf(":%d", 8080),
		MaxHeaderBytes: 1 << 20, // Max header of 1MB

		w.Write([]byte("Hello world"))



version: 1.0
  name: openfaas
    lang: dockerfile
    handler: ./http
    image: alexellis2/http:0.1.4
      GO111MODULE: on

I ran this on my PC with the following:

faas-cli publish --platform linux/arm/v6

faas-cli deploy

Using the PC means we can cross-compile the Go HTTP server and push a multi-arch image to the container registry. The new faas-cli publish command comes in handy for this and can accept a list of architectures.

The deploy command talks to the remote machine over REST and tells it to deploy a function. The server-side handler deploy.go pulls the image into containerd and starts executing it as a container.

One more thing

When I say that it worked. I should clarify that I'd forgotten to deploy CNI - the networking layer required for faasd and containerd.

I saw this error by running the following:

sudo journalctl -u faasd

Not that long ago, I'd documented all the steps required for a manual installation, but somehow forgot this was needed.

export ARCH=arm
export CNI_VERSION=v0.8.5

sudo mkdir -p /opt/cni/bin
curl -sSL https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz | sudo tar -xz -C /opt/cni/bin

# Make a config folder for CNI definitions
sudo mkdir -p /etc/cni/net.d

# Make an initial loopback configuration
    "cniVersion": "0.3.1",
    "type": "loopback"

Just changing "ARCH" to "arm" was enough to make it work. So the CNI project are still building the plugins against armv6 for the time being.

Final attempt

Then it was a case of simply running faas-cli deploy again, and finally, it actually worked.

An initial benchmark with hey identified that the GetFunction() code was taking around 300ms. On a PC this is so fast that it's not noticable in everyday use.

The code gets the container for the name being invoked, gets its task, reads its status, and then looks up its IP address. To improve this, I'd probably optimize the code, use a cache, or build the container status/IP information outside of the data-path.

Wrapping up

So I now have a Raspberry Pi Zero running just the faasd-provider code and systemd unit file.

It can create, update, delete, and list containers and their associated secrets. It can invoke functions, but not a lot more.

Earlier on Twitter a user asked me whether there was a lightweight way of being able to deploy microservices from a GitHub Action, where he was just running Docker on a single node. I think faasd, or faasd's provider could be a useful option here. It provides authentication and a REST API for deploying containers with HTTP servers inside them.

I found the experience of working on the RPi zero slow and frustrating. I can remember when they were first introduced, and I even went as far as running Docker Swarm on them.

One of the benefits of running software on vastly under-powered hardware, is that it can show bottlenecks in the code, which aren't always noticeable during everyday usage on a newer machine. To some extent, I worry about us writing code on lightning fast Apple M1 machines, only to deploy it to a t1.micro and find it chokes during light usage.

You can see the write-up for that project here

I think it's time for the container community to leave behind armv6, but I would still recommend buying an RPi Zero for something like an embedded project, or for reading values from a sensor and sending them to MQTT or InfluxDB.

I hope you found my little adventure into Raspberry Pi Zero and armv6 interesting. If you have one of your own, why don't you see if you can deploy your own microservices or functions?

If you have a Raspberry Pi 3 or 4 to hand, then you can run the fully-functioning, faster version of faasd: faasd - lightweight Serverless for your Raspberry Pi

You may also like

We're celebrating the 4th OpenFaaS Birthday on 18th December, register and join us.

Related blog posts:

<![CDATA[Running Falco and k3s at the edge with 64-bit ARM]]>https://blog.alexellis.io/falco-at-the-edge-arm64/5f85ce5c3363a50001ef0f97Tue, 13 Oct 2020 17:01:03 GMT

Falco is an open source tool that peeks into the internals of your system to detect and report on syscall and network activity. It was originally built at Sysdig and later donated to the CNCF where it's doing well as an Incubator project with an active community.

My understanding of Falco from Sysdig's CTO is that it's designed to be the last-line of defence in systems where the focus is usually put on preventative measures. Falco stands in to report on anomalies and unexpected behaviour.

When running k3s at an edge location, it may be difficult to gain physical access to servers, and there may be partial connectivity available.


Pictured above: a net-booted Raspberry Pi cluster running on a purpose-designed industrial unit "BitScope Cluster Blade" by BitScope Designs

By default there are three possible event sources for Falco:

  • The kernel and a number of syscalls
  • eBPF - faster than kernel access, but not available everywhere
  • Userland using PTRACE(2) - for potential use with SaaS products like AWS Lambda

These events can be sent to a webhook such as an OpenFaaS function for remediation at the edge, or alerting and decision-making in a centralised action-centre in the cloud. As a generic event-sink, data can be ingested directly from the Kubernetes API, but a smart device such as a door-lock could also send data such as the current temperature, or unexpected motion picked up from an accelerometer.


Conceptual architecture: event detection from the Kernel, eBPF, the k3s API with alerting and remediation via OpenFaaS.

In this guide I'll show you how to configure Falco to run at the edge with k3s on an ARM64 host such as the Ampere eMAG® provided by Equinix Metal (Equinix provide credits for open source projects hosted by the CNCF), an AWS Graviton instance, or a Raspberry Pi 4.


We'll start off by flashing Ubuntu Linux to the SD card, then adding k3s, finally we'll install Falco and configure k3s to send it events.

Download the Operating System

You will need a Raspbery Pi 4. I am using a device with 4GB of RAM, but 2GB will also work.

Download the Ubuntu 20.04.01 image using the link for the Raspberry Pi 4.

You can get the image here: https://ubuntu.com/download/raspberry-pi

Now use Etcher.io or dd to write the server OS image to your Raspberry Pi's SD card.

First boot

Note: Unlike with RaspiOS, SSH is enabled by default.

Boot up the machine and find it on your network using nmap:

nmap -sP

Log in and change the password from ubuntu to whatever you like:

ssh ubuntu@

Next, copy over your SSH key:

ssh-copy-id ubuntu@

Prepare for k3s

Edit /firmware/boot/cmdline.txt and add to the first line, don't add any line-breaks:

cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

This enables cgroups which are required to set limits for containers. The step is taken from my tutorial for Raspbian: Will it cluster? k3s on your Raspberry Pi

Reboot your RPi:

sudo reboot

Install k3s with k3sup

Use my k3sup tool to your workstation to install k3s over SSH. Do not run this on the RPi itself.

curl -sLS https://get.k3sup.dev | sudo sh

Now install via k3sup install:

k3sup install --ip --user ubuntu \
  --channel latest

The --channel command installs Kubernetes 1.19.

Install Falco and its dependencies

Log back into the RPi via SSH.

A fork of the upstream project is required to run on 64-bit ARM at present, but I expect these changes to be merged back in soon. A maintainer has provided a .deb file:

sudo apt-get -y install linux-headers-$(uname -r) dkms

DKMS is required to build a Kernel module. This is one of the ways that Falco can instrument your system.

You can also build from source using the build/aarch64 branch.

Install Falco's binary and let it build its kernel module:

wget https://fs.fntlnz.wtf/falco/aarch64-builds/falco-0.26.1-48%2Bf82d905-aarch64.deb

sudo dpkg -i ./falco-0.26.1-48+f82d905-aarch64.deb

Silence some errors in the falco config file.

Edit /etc/falco/falco.yaml and replace the following:

    - log
    - alert
  rate: .03333
  max_burst: 10


    - ignore
  rate: .03333
  max_burst: 10

This turns off some noise that I was seeing from the ARM-64 version.

Enable it upon boot-up:

sudo systemctl enable falco \
  && sudo systemctl start falco

You can find the various pre-loaded rules for falco at /etc/falco/rules/

ls /etc/falco/rules/


Simulate a Falco event

Before configuring Kubernetes to send events to Falco, we can send a sample event that will trigger one of the built-in alerts.

cat >> sample-event.json <<EOF

Invoke the event via Falco's REST API:

curl --data-binary @sample-event.json -H "Content-Type: application/json"

Trigger an event from k3s

An additional source we can add is the events from the Kubernetes audit log. This feature has to be turned on via the k3s systemd definition.

Create a folder for audit events:

sudo mkdir -p /var/lib/rancher/audit

Download audit-policy.yaml to /var/lib/rancher/audit

wget https://raw.githubusercontent.com/falcosecurity/evolution/master/examples/k8s_audit_config/audit-policy.yaml
sudo cp audit-policy.yaml /var/lib/rancher/audit/

Create a webhook config file:

export IP=

cat << EOF | sudo tee /var/lib/rancher/audit/webhook-config.yaml
apiVersion: v1
kind: Config
- name: falco
    server: http://$IP:8765/k8s-audit
- context:
    cluster: falco
    user: ""
  name: default-context
current-context: default-context
preferences: {}
users: []

Edit /etc/systemd/system/k3s.service and add these lines to the end:

        '--kube-apiserver-arg=audit-log-path=/var/lib/rancher/audit/audit.log' \
        '--kube-apiserver-arg=audit-policy-file=/var/lib/rancher/audit/audit-policy.yaml' \
        '--kube-apiserver-arg=audit-webhook-config-file=/var/lib/rancher/audit/webhook-config.yaml' \

Now reload:

sudo systemctl daemon-reload && \
 sudo systemctl restart k3s

My favourite use-case from the docs was detecting AWS secret keys stored in ConfigMaps. Of course, we all know confidential data belongs inside Vault or Kubernetes Secrets which can be encrypted at rest.

Let's try it out?

Run this from your laptop:

kubectl create configmap aws-creds \
  --from-literal aws_access_key_id=AKES20LNOA

Now check the logs from Falco:

sudo journalctl -u falco --lines 100

Oct 13 16:55:42 ubuntu falco[89818]: 16:55:24.207932928: Warning K8s configmap with private credential (user=system:admin verb=create configmap=aws-creds-1 config={"aws_access_key_id":"AKES20LNOA"})

You can find this example under "Kubernetes Audit Rules" in the Falco docs.

Expand the k3s cluster

You can add additional nodes into the k3s cluster by installing Ubuntu as we did for the first node, then running the k3sup join command.

k3sup join --ip $IP -user ubuntu \

k3s also supports High-Availability and fail-over of the master nodes using a SQL backend or an embedded version of etcd.


Full disclosure: Sysdig is a client of OpenFaaS Ltd

We now have Ubuntu 20.04.01, Kubernetes with k3s and Falco all built for 64-bit ARM running on our Raspberry Pi.

It's now over to you to explore Falco more, and to fine-tune your Kubernetes rules.

I also want to give a shout out to my friend fntlnz who is the Falco maintainer that helped me get a working build of Falco on 64-bit ARM. It took us several days to get here.


It took me quite a while to figure out how to get this to work, but it was nice when everything clicked into place. So be warned, the Falco documentation still refers to "Dynamic Auditing" which was removed and deprecated in Kubernetes and the upstream instructions don't cover k3s yet, or any release newer than 1.13.

I hope to see the team at Sysdig spend some time updating the documentation and examples for newer Kubernetes releases and for an official 64-bit ARM binary to be made available soon.

Taking it further

The easier way to get security and other insights, just like the open source Falco project is to use Sysdig's paid product. You can find a step-by-step tutorial by Dan here: K3s + Sysdig: Deploying and securing your cluster… in less than 8 minutes!

You may also like my walk-through video: Installing Ubuntu and k3s to my Raspberry Pi 4

Find out more about the tools:

<![CDATA[inlets PRO launch day]]>https://blog.alexellis.io/inlets-pro-launch/5f34fee53363a50001ef0f1dThu, 13 Aug 2020 10:38:11 GMTinlets PRO launch day

inlets PRO is a secure, commercial TCP tunnel, built for connecting services in a Cloud Native way. Learn what it can do for you, and how it was built from OSS roots. At the end of the post, you'll be able to view the new landing page with both personal and commercial pricing, along with comparisons to other networking solutions.

inlets PRO launch day

What is inlets?

inlets is an open-source network tunnel project that I built out in late 2018/2019. After a long history of working for enterprise companies, and facing troubling network conditions on the road (captive portals), and restrictive firewalls, I sat down to write some code and try and solve the problem.

To start with, the code (written in Go) was rudimentary, and serialised raw HTTP request and response objects over the wire. I'd decided to pick websockets for the transport, because I knew that a websocket could penetrate almost any network conditions by initiating an outbound call from a client to a well-known port on a HTTP server - 80 or 443.

inlets PRO launch day

After the project gained traction, the proof of concept had served its purpose and I rewrote the code to use off the shelf libraries like Gorilla's websocket library.

Simply put, if you had a HTTP service on your laptop, Raspberry Pi, VM, Docker image, Windows PC, Mac, or Kubernetes cluster, using inlets you could expose it on another network, and that network was usually the Internet.

inlets 1.0 and 2.0 were released and attracted many comments on Hacker News and the project acquired is own community of personal and commercial users. As of today, it has 6.5k GitHub stars and can be used via a Docker image or a static binary.

inlets gains automation-tooling

Despite running in conditions where tools like Ngrok were completely banned, or non-functional, the barrier to entry was still too high for some users. They were used to simply running a single command and connecting to a SaaS tunnel.

SaaS tunnels are notoriously difficult to access from corporate networks, and are hamstringed by silly limits like being active for only 7 hours, or limiting connections per minute, even to those who pay for the service.

The route I took for this was to create inletsctl - a tool that could bootstrap a tunnel server or (exit-server) within a few moments on your favourite cloud.

inletsctl has a provision package which can set up new hosts on DigitalOcean, Packet, Civo, Scaleway, AWS EC2, GCE, Azure, Linode, and Hetzner. There are five different contributors to this code, and you can think of it a little bit like a basic version of Terraform.

Here's how it works:

inletsctl create --provider digitalocean \
 --region lon1 \
 --access-token-file ~/do-access-token

Then after a few moments, you'll get a "connection string" for your server.

Using provider: digitalocean
Requesting host: relaxed-brahmagupta1 in lon1, from digitalocean
2020/08/13 10:02:19 Provisioning host with DigitalOcean
Host: 203721867, status: 
[1/500] Host: 203721867, status: new

[31/500] Host: 203721867, status: active
Inlets OSS exit-node summary:
  Auth-token: sO7tIBsy7FaVXb0HiVKXejsBvt6GEUGiSHbcMlrEo6d928cUIIjgPj7nJxnwFvM6

  export UPSTREAM=
  inlets client --remote "ws://" \
	--token "sO7tIBsy7FaVXb0HiVKXejsBvt6GEUGiSHbcMlrEo6d928cUIIjgPj7nJxnwFvM6" \
	--upstream $UPSTREAM

To Delete:
	inletsctl delete --provider digitalocean --id "203721867"

How much does it cost? As little as possible, around 5 USD / mo, and you can of course, run more than one tunnel server there if you wish.

inlets integrates directly with Kubernetes

One of the other common complaints of the incumbant SaaS tunnels are that they do not integrate natively with Kubernetes, and when they do, they are still subjected to the same limits that we talked about above.

Enter the inlets-operator - the tagline is "Get Kubernetes LoadBalancers from within your private network."

It works by detecting any Service in your cluster of type "LoadBalancer" and then it acts just like inletsctl, setting up an exit-server on your favourite cloud. However, this time, rather than you running "inlets client" from your command-line, a Pod is deployed into the cluster, which connects a private service in your cluster to the new server with a public IP.

This works extremely well, and is much faster than getting an ALB for instance on Amazon's managed Kubernetes product, EKS.

inlets PRO launch day

See the tutorial: Expose Your IngressController and get TLS from LetsEncrypt

You can even bring your own IngressController along, since inlets PRO works at the TCP-level.

Walk-through: Get LetsEncrypt certificates and public IPs for your private IngressController

Like a PRO

Shortly after introducing inlets, inletsctl, and the inlets-operator, I developed a need for more than just a HTTP tunnel and started planning and thinking how to go about this. I also wanted to make sure that the websocket itself had TLS configured automatically, with the OSS version of inlets, it was up to the operator to add it with a reverse proxy.

I wanted to deploy cert-manager, ingress-nginx and then to get TLS certificates from LetsEncrypt, just like I would do on a public cluster. This didn't work, since the OSS version of inlets terminated TLS on the exit-server. I needed a tunnel that could pass-through TCP traffic.

inlets PRO launch day

I did something a little different with inlets PRO. I didn't open-source it, I worked in private, tested with a couple of trusted friends from the community, and then released only a binary.

The binary requires a JWT license code to work, and so I set up a "start a trial page" and sat back.

As the founder of other open-source projects like OpenFaaS, I was used to the following conversation with companies: "We will use your stuff for our production applications, and ask you for help for free, and no we won't sponsor, contribute to the project, or hire you for support."

With inlets PRO, the conversation changed. "We think we need your thing, we're almost certain of that. How much does it cost? A proof of concept, that sounds great, what will that cost?"

As I mentioned on the Google Kubernetes Podcast this week, companies don't pay for free. It's only when a pricing-gate is put in place, that value can be captured for the maintainer or vendor of software.

There's been dozens of trials issued already, and several large corporations that have expressed an interest, but not for what I expected.

Here are a couple of examples of how inlets PRO can be used:

Most enterprise companies, it turns out have some legacy deployed on premises, and some deployed on a public cloud. They need to securely tunnel services from the private cluster to the public one, but they do not want to expose them on the Internet. Sometimes these services are also running without encryption, think of an ancient MSSQL service, that's full of key data for an API.

inlets PRO launch day

inlets PRO provided a way for them to replace VPNs, which were complex to set up and manage. It also provided a way for the data to be encrypted over the wire, and only tunneled to the remote network, not being exposed. Think of it like a private or headless tunnel, maybe a "hybrid service mesh."

inlets PRO works much like the original open source version and shares some code. The inletsctl and inlets-operator projects both have switches to use the PRO version of inlets.

Launch day

This journey started out as building an open-source tunnel to help developers get past the impasse of using severely-limited SaaS tunnels. The tooling that has developed subsequently, with help from the community, has made inlets even more useful and easy to use.

Today is about making inlets more sustainable as an option over the long-haul. It's about providing clear business value to companies and teams that have hybrid cloud, are tired of VPNs, or that simply need to work with customers and clients on other networks.

With the launch of the landing page for inlets PRO you'll find a personal pricing option that gives unlimited use, a commercial option, and enterprise. The commercial option is not billed per month, because to keep inlets PRO completely private, there is no tracking included. Users are trusted to buy the right license for what they are using, and in return, no usage or tracking data will be collected.

inlets PRO launch day

You can also get inlets SWAG - your own hoodie, t-shirt or mug for the home office in the OpenFaaS store.

<![CDATA[Expose your private Grafana dashboards with TLS]]>https://blog.alexellis.io/expose-grafana-dashboards/5e8750da1e3bb600011eeb14Fri, 03 Apr 2020 16:33:34 GMTExpose your private Grafana dashboards with TLS

Grafana dashboards are great for getting quick feedback on critical metrics, or just for browsing interesting statistics about user behaviour on a production system.

In this post I'll show you how to use inlets PRO as a secure tunnel to expose your private Grafana dashboards on your laptop, private datacenter or your local network. inlets PRO will use TLS to provide link-level encryption to prevent snooping on tunnel traffic. Since Grafana doesn't come with TLS by default, we'll use Caddy to obtain a certificate for Grafana by tunnelling out port 80 and 443 instead of the plaintext HTTP port 3000.

Expose your private Grafana dashboards with TLS

Your Grafana dashboard may be deployed inside Kubernetes, or as a Docker container. I'm going to show you how to set it up with a sample dashboard, you can follow along and then adapt it for any existing deployments which you may have.

The instructions here are not strictly specific to Grafana, so if you have a flask app running on port 5000, just replace that port wherever you see 3000. Likewise, if you are running OpenFaaS on port 8080, replace 3000 with 8080 and so on.


Here's the workflow for the tutorial:

  • Run Grafana in Docker or point at an existing installation on our network
  • Install inlets-pro and inletsctl
  • Create a tunnel and an exit-server with a public IP
  • Get a domain or add a new sub-domain using our public IP
  • Install Caddy
  • Run Caddy and get a nice green lock for our dashboard

Then you get to enjoy your dashboard from anywhere.

I'll also show you how to access multiple dashboards or services from a single tunnel.

You'll need an inlets-pro license, you can buy one at store.openfaas.com for personal use or get a free trial using the link in the inlets documentation

Quick-start for Grafana

  • Install Docker
  • Run a Grafana container
docker run -d -p 3000:3000 \
  --name grafana \
  --restart=always \

Taken from the Grafana docs

Now you have a dashboard at, however you cannot access this over the Internet, and it has no TLS to secure it.

Make sure that you change the default password from admin/admin before going any further.

Expose your private Grafana dashboards with TLS

Get a public IP

The issue with running a private dashboard is that it's just not routeable, and it's unlikely that your administrator will open a port for you. If you are at home and want to do this with your ISP and port-forwarding, think again, you could be leaking your personal information and have your home network targeted.

Let's get a public IP with no strings attached.

Download inletsctl and use it to get the inlets-pro client:

curl -sSLf https://inletsctl.inlets.dev | sudo sh

sudo inletsctl download --pro

# or run without sudo, and move the binary after

Now create an exit-server (aka exit-node) for your favourite cloud. I am going to use DigitalOcean, which happens to be the fastest and cheapest. Other supported providers include: packet, ec2, scaleway, civo, gce and azure.

Run the following command, and make sure you have an access token from your dashboard saved as ~/access-token.txt

Take note of the --remote-tcp flag, it tells the inlets client where to send traffic.

inletsctl create  \
  --provider digitalocean \
  --access-token-file ~/access-token.txt \
  --region lon1 \

When the program completes running you'll be presented with a command you can run to connect to the tunnel and start proxying traffic.

You need to set the --tcp-ports flag to a comma-delimited list of ports to punch out of the tunnel, or a single port. In our example we won't expose Grafana directly, but we'll expose Caddy (a reverse proxy) which will have TLS enabled via LetsEncrypt.

Set export TCP_PORTS= to 80,443 which are the two ports required for plain HTTP to serve the ACME challenge, and 443 to serve the encrypted traffic afterwards.

export TCP_PORTS="80,443"
export LICENSE=""
inlets-pro client --connect "wss://" \
  --token "67462dccc691fd58c485ab5e631c6b69af594572" \
  --license "$LICENSE" \
  --tcp-ports $TCP_PORTS

Here's my client process running:

Expose your private Grafana dashboards with TLS

Get a domain ready for the dashboard

We will need a domain for this step, and they cost as little as 1 USD, so no complaining please. Head over to namecheap.com and get something fun, you can re-use it and it won't hurt your pocket.

If you already have a domain, we'll use a sub-domain.

Use the IP address from the exit-server and create a sub-domain with a DNS A record:


If you're a DigitalOcean user, you can use doctl to create this address with: doctl compute domain create --ip-address $IP $SUB_DOMAIN.

Install a reverse proxy

We need a reverse proxy that can obtain a TLS certificate. I find that Caddy 1.0 is quick and easy enough for us to use. Caddy 2 is also available, but you will need to learn the new Caddyfile format if you decide to use it.

Download the binary for MacOS, Linux or Windows: v1.0.4

Unzip or un-tar the downloaded file, you'll find caddy or caddy.exe.

  • Create a Caddyfile

proxy / {

The first line tells Caddy which domain to use when getting the TLS certificate

The second code-block says that any requests send to this domain should be proxied to our local machine on port 3000.

Try it out

We're almost there, all we need to do is to run Caddy and then open up our URL.

  • Run caddy

If you're on MacOS, you'll need to run as sudo to access port 80, 443 on the local host.

sudo ./caddy -disable-tls-alpn-challenge

Note, you will be asked for your email for the domain, enter it then continue.


Check the green lock:

Expose your private Grafana dashboards with TLS

Going further

Now that we have a single dashboard up, with its own domain, we can add additional dashboards and services using the same approach and our Caddy server.

Run a second dashboard on a different port:

docker run -d -p 3001:3000 \
  --name grafana \
  --restart=always \

Edit Caddyfile and restart caddy:

dashboard1.domain.com {
  proxy / {

dashboard2.domain.com {
  proxy / {

There's no need to restart the tunnel or the tunnel server.

Putting things on hold

When you no-longer need the tunnel, simply stop the inlets-pro client and caddy, and start them again whenever you need. The exit server can remain provisioned, or you can delete it completely with the command given by inletsctl create.

Wrapping up

You now have a secure tunnel since inlets-pro uses TLS for its control-plane, with additional security for the Grafana dashboard with a TLS certificate served by Caddy.

Next, why don't you try running another Docker container, and putting that behind your Caddy proxy, using a different DNS name, what about `openfaas.example.com?

Can I get some Kubernetes with that?

If you like your DevOps more automated than this, you should check out the Kubernetes operator for Inlets, which can expose your IngressController and use cert-manager to obtain TLS certs automatically.