Matt Gerega

Tag: Prometheus

  • Pulling metrics from Home Assistant into Prometheus

    I have setup an instance of Home Assistant as the easiest front end for interacting with my home automation setup. While I am using the Universal Devices ISY994 as the primary communication hub for my Insteon devices, Home Assistant provides a much nicer interface for my family, including a great mobile app for them to use the system.

    With my foray into monitoring, I started looking around to see if I was able to get some device metrics from Home Assistant into my Grafana Mimir instance. Turns out, there is an a Prometheus integration built right in to Home Assistant.

    Read the Manual

    Most of my blog posts are “how to” style: I find a problem that maybe I could not find an exact solution for online, and walk you through the steps. In this case, though, it was as simple as reading the configuration instructions for the Prometheus integration.

    ServiceMonitor?

    Well, almost that easy. I have been using ServiceMonitor resources within my clusters, rather than setting up explicit scrape configs. Generally, this is easier to manage, since I just install the Prometheus operator, and then create ServiceMonitor instances when I want Prometheus to scrape an endpoint.

    The Home Assistant Prometheus endpoint requires a token, however, and I did not have the desire to dig in to configuring a ServiceMonitor with an appropriate secret. For now, it is a to-do on my ever-growing list

    What can I do now?

    This integration has opened up a LOT of new alerts on my end. Home Assistant talks to many of the devices in my home, including lights and garage doors. This means I can write alerts for when lights go on or off, when the garage door goes up or down, and, probably the best, when devices are reporting low battery.

    The first alert I wrote was to alert me when my Ring Doorbell battery drops below 30%. Couple that with my Prometheus Alerts module for Magic Mirror, and I now get a display when the battery needs changed.

    What’s Next?

    I am giving back to the community. The Prometheus integration for Home Assistant does not currently report cover statuses. Covers are things like shades or, in my case, garage doors. Since I would like to be able to alert when the garage door is open, I am working on a pull request to add cover support to the Prometheus integration.

    It also means I would LOVE to get my hands on some automated shades/blinds… but that sounds really expensive.

  • Getting Synology SNMP data into Prometheus

    With my new cameras installed, I have been spending a lot more time in the Diskstation Manager (DSM). I always forget how much actually goes on within the Synology, and I am reminded of that every time I open the Resource Monitor.

    At some point, I started to wonder whether or not I could get this data into my metrics stack. A quick Google search brought me to the SNMP Exporter and the work Jinwoong Kim has done to generate a configuration file based on the Synology MIBs.

    It took a bit of trial and error to get the configuration into the chart, mostly because I forgot how to make a multiline string in a Go template. I used the prometheus-snmp-exporter chart as a base, and built out a chart in my GitOps repo to have this deployed to my internal monitoring cluster.

    After grabbing one of the community Grafana dashboards, this is what I got:

    Security…. Not just yet

    Apparently, SNMP has a few different versions. v1 and v2 are insecure, utilizing a community string for simple authentication. v3 has added a more complex authentication and encryption, but this makes configuration more complex.

    Now, my original intent was to enable SNMPv3 on the Synology, however, I quickly realized that meant putting the auth values for SNMP v3 in my public configuration repo, as I haven’t figured out how to include that configuration as a secret (instead of a configmap). So, just to get things running, I settled on just enabling V1/V2 with the community string. These SNMP ports are blocked from external traffic, so I’m not too concerned with it, but my to-do list now includes a migration to SNMPv3 for the Synology.

    When you have a hammer…

    … everything looks like a nail. The SNMP Exporter I am currently running only has configuration for Synology, but the default if_mib module that is packaged with the SNMP Exporter can grab data from a number of products which support it. So I find myself looking for “nails” as it were, or objects from which I can gather SNMP data. For the time being, I am content with the Synology data, but don’t be surprised if my network switch and server find themselves on the end of SNMP collection. I would say that my Unifi devices warrant it, but the Unifi Poller I have running takes care of gathering that data.

  • Kubernetes Observability, Part 2 – Collecting Metrics with Prometheus

    This post is part of a series on observability in Kubernetes clusters:

    Part 1 – Collecting Logs with Loki
    Part 2 – Collecting Metrics with Prometheus (this post)
    Part 3 – Building Grafana Dashboards
    Part 4 – Using Linkerd for Service Observability
    Part 5 – Using Mimir for long-term metric storage

    “Prometheus” appears in many Kubernetes blogs the same way that people whisper a famous person’s name as they enter a crowded room. Throughout a lot of my initial research, particularly with the k8s-at-home project, I kept seeing references to Prometheus in various Helm charts. Not wanting to distract myself, I usually just made sure it was disabled and skipped over it.

    As I found the time to invest in Kubernetes observability, gathering and viewing metrics became a requirement. So I did a bit of a deep dive into Prometheus and how I could use it to view metrics.

    The Short, Short Version

    By no means am I going to regurgitate the documentation around Prometheus. There is a lot to ingest around settings and configuration, and I have not even begun to scratch the surface around what I can do with Prometheus. My simple goal was to get Prometheus installed and running on my clusters, gathering metrics for my clusters. And I did this by installing Prometheus on each of my clusters using Bitnami‘s kube-prometheus Helm chart. Check out the cluster-tools folder in my Argo repository for an example.

    The Bitnami chart includes kube-state-metrics and node-exporter, in addition to settings for Thanos as a sidecar. So, with the kube-prometheus install up and running, I was able to create a new data source in Grafana and view the collected metrics. These metrics were fairly basic, consisting mainly of the metrics being reported by kube-state-metrics.

    Enabling ServiceMonitors

    Many of the charts I use to install cluster tools come with values sections that enabled metrics collection via a ServiceMonitor CRD. A ServiceMonitor instance instructs Prometheus how to discover and scrape services within the cluster.

    For example, I use the following Helm charts with ServiceMonitor definitions:

    So, in order to get metrics on these applications, I simply edited my values.yaml file and enabled the creation of a ServiceMonitor resource for each.

    Prometheus as a piece of the puzzle

    I know I skipped a LOT. In a single cluster environment, installing Prometheus should be as simple as choosing your preferred installation method and getting started scraping. As always, my multi-cluster home lab presents some questions that mirror questions I get at work. In this case, how do we scale out to manage multiple clusters?

    Prometheus allows for federation, meaning one Prometheus server can scrape other Prometheus servers to gather metrics. Through a lot of web searching, I came across an article from Vikram Vaswani and Juan Ariza centered on creating a multi-cluster monitoring dashboard using Prometheus, Grafana, and Thanos. The described solution is close to what I would like to see in my home lab. While I will touch on Grafana and Thanos in later articles, the key piece of this article was installing Prometheus in each cluster, and, in as described, creating a Thanos sidecar to aid in the implementation of Thanos.

    Why Bitnami?

    I have something of a love/hate relationship with Bitnami. The VMWare-owned property is a collection of various images and Helm charts meant to be the “app store” for your Kubernetes cluster. Well, more specifically, it is meant to be the app store for your VMWare Tanzu environments, but the Helm charts are published so that others can use them.

    Generally, I prefer using “official” charts for applications. This usually ensures that the versions are the most current, and the chart is typically free from the “bloat” that can sometimes happen in Bitnami charts, where they package additional sub-charts to make things easy.

    That is not to say that I do not use Bitnami charts at all. My WordPress installation uses the Bitnami chart, and it serves its purpose. However, being community-driven, the versions can lag a bit. I know the 6.x release of WordPress took a few weeks to make it from WordPress to Bitnami.

    For Prometheus, there are a few community-driven charts, but you may notice that Helm is not in the list of installation methods. This, coupled with the desire to implement Thanos later, led me to the kube-prometheus chart by Bitnami. You may have better mileage with one of the Prometheus Community charts, but for now I am sticking to the Bitnami chart.