As we start to develop more containers that are being run in Kubernetes, we encounter non-http workloads. I came across a workload that represents a non-http processor for queued events. In .NET, I used the IHostedService offerings to run a simple service in a container to do this work.
However, when it came time to deploy to Kubernetes, I quickly realized that my standard liveness/health checks would not work for this container. I searched around, and the HealthChecks libraries are limited to ASP.NET Core. Not wanting to bloat my image, I looked for some alternatives. My Google searches led me to Bruce Lee.
No, not Bruce Lee the actor, but Bruce Lee Harrison. Bruce published a library called TinyHealthChecks, which provides the ability to add lightweight endpoints without dragging in the entire ASP.NET Core libraries.
While it seems a pretty simple concept, it solved an immediate need of mine with minimal effort. Additionally, there was a sample and documentation!
Why call this out? Many developers use open source software to solve these types of problems, and I feel as though they deserve a little publicity for their efforts. So, thanks to the contributors to TinyHealthCheck, I will certainly watch this repository and contribute as I can.
Moving my home lab to GitOps and ArgoCD has been, well, nearly invisible now. With the build pipelines I have in place, I’m able to work on my home projects without much thought to deploying changes to my clusters.
My OCD, however, prevents me from running old versions. I really want to stay up-to-date when it comes to the tools I am using. This resulted in a periodic update of all of my Helm repositories to search for new versions.
After getting tired of this “search and replace,” I wrote a small Powershell script which automatically updates my local Helm repositories, and then searches through the current directory (recursively) for chart.yaml files. If there are dependencies in that Chart.yaml, it will upgrade them to the latest version and save the Chart.yaml file.
It does not, at the moment, automatically commit these changes, so I do have a manual step to confirm the upgrades. However, it is a much faster way to get changes staged for review and committing.
I have been doing some work to build up some standard processes for Kubernetes. ArgoCD has become a big part of that, as it allows us to declaratively manage the state of our clusters.
After recovering from a small blow-up in the home lab (post coming), I wanted to migrate my cluster tools to utilize the label selector feature of the ApplicationSet’s Cluster Generator. Why? It allows me to selectively manage tools in the cluster. After all, not every cluster needs the nfs-external-provisioner to provide a StorageClass for pod file storage.
As part of this, I wanted to deploy to tools to the local Argo cluster. In order to do that, the local cluster needs a secret. I tried to follow the instructions, but when I clicked to view the cluster details, I got a 404 error. I dug around the logs, and my request wasn’t even getting to the Argo application server container.
When I looked at the Ingress controller logs, it showed the request looked something like this:
Notice the encoding: My Nginx reverse proxy (running on a Raspberry Pi outside my main server) was decoding the request before passing it along to the cluster.
The question was, why? A quick Google search lead right to their documentation:
If proxy_pass is specified with URI, when passing a request to the server, part of a normalized request URI matching the location is replaced by a URI specified in the directive
If proxy_pass is specified without URI, a request URI is passed to the server in the same form as sent by a client when processing an original request
What? Essentially, it means the slash at the end will dictate whether Nginx does anything with the request.
## With a slash at the end, the client request is normalizedlocation/name/ { proxy_pass http://127.0.0.1/remote/;}## Without a slash, the request is as-islocation/name/ { proxy_pass http://127.0.0.1;}
Removing the slash, the Argo UI was able to load the cluster details correctly.
Just a quick note: It would seem that somewhere between Friday, April 29, 2022 and Monday, May 2, 2022, Azure DevOps pipelines changed their handling of newlines in YAML literal blocks. The change caused our pipelines to stop executing with the following error:
While scanning a literal block scalar, found extra spaces in first line
What caused it? Multi-line, inline block definitions.
- powershell: | Write-Host "Azure Fails on this now"displayName: Bad Script- powershell: | Write-Host "Azure works with this"displayName: Good Script
I have been using Nginx as a reverse proxy for some time. In the very first iteration of my home lab, it lived on a VM and allowed me to point my firewall rules to a single target, and then route traffic from there. It has since been promoted to a dedicated Raspberry Pi in my fight with the network gnomes.
My foray into Kubernetes in the home lab has brought Nginx in as an ingress controller. While there are many options for ingress, Nginx seems the most prevalent and, in my experience, the easiest to standardize on across a multitude of Kubernetes providers. As we drive to define what a standard K8 cluster looks like across our data centers and public cloud providers, Nginx seemed like a natural choice for our ingress provider.
Configurable to a fault
The Nginx Ingress controller is HIGHLY configurable. There are cluster-wide configuration settings that can be controlled through ConfigMap entries. Additionally, annotations can be used on specific ingress objects to control behavior on individual ingress.
As I worked with one team to setup Duende’s Identity Server, we started running into issues with the identity server endpoints using http instead of https in its discovery endpoints (such as /.well-known/openid-configuration. Most of our research suggest that the X-Forwarded-* headers needed to be configured (which we did), but we were still seeing the wrong scheme in those endpoints.
It was a weird problem: I had never run into this issue in my own Identity Server instance, which is running in my home Kubernetes environment. I figured it had to do with an Nginx setting, but had a hard time figuring out which one.
One blog post pointed me in the right direction. Our Nginx ingress install did not have the use-forwarded-headers setting configured in the ConfigMap, which meant the X-Forwarded-* headers were not being passed to the pod. A quick change of our deployment project, and the openid-configuration endpoint returned the appropriate schemes.
For reference, we are using the ingress-nginx helm chart. Adding the following to our values file solved the issue:
controller:replicaCount: 2service:... several service settingsconfig:use-forwarded-headers: "true"
Investigation required
What I do not yet know is, whether or not I randomly configured this at home and just forgot about it, or if it is a default of the Rancher Kubernetes Engine (RKE) installer. I use RKE at home to stand up my clusters, and one of the add-ons I have it configure is ingress with Nginx. Either I have settings in my RKE configuration to forward headers or it’s a default of RKE…. Unfortunately, I am at a soccer tournament this weekend, so the investigation will have to wait until I get home.
Update:
Apparently I did know about use-forwarded-headers earlier: it was part of the options I had set in my home Kubernetes clusters. One of many things I have forgotten.
With a push to driving better documentation, it is worth remembering that Visual Studio Code has a variety of extensions that can help with linting/formatting of all types of files, including your README.md files.
Markdown All in One and markdownlint are my current extensions of choice, and they have helped me clean up my README.md files in both personal and professional projects.