Matt Gerega

Tag: Grafana

  • Centralizing Telemetry with Linkerd Multi-Cluster: Simplifying Internal Routing

    Running multiple Kubernetes clusters is great until you realize your telemetry traffic is taking an unnecessarily complicated path. Each cluster had its own Grafana Alloy instance dutifully collecting metrics, logs, and traces—and each one was routing through an internal Nginx reverse proxy to reach the centralized observability platform (Loki, Mimir, and Tempo) running in my…

  • Automating Grafana Backups

    After a few data loss events, I took the time to automate my Grafana backups. A bit of instability It has been almost a year since I moved to a MySQL backend for Grafana. In that year, I’ve gotten a corrupted MySQL database twice now, forcing me to restore from a backup. I’m not sure…

  • Re-configuring Grafana Secrets

    I recently fixed some synchronization issues that had been silently plaguing some of the monitoring applications I had installed, including my Loki/Grafana/Tempo/Mimir stack. Now that the applications are being updated, I ran into an issue with the latest Helm chart’s handling of secrets. Sync Error? After I made the change to fix synchronization of the…

  • Synced, But Not: ArgoCD Differencing Configuration

    Some of the charts in my Loki/Grafana/Tempo/Mimir stack have an odd habit of not updating correctly in ArgoCD. I finally got tired of it and fixed it… I’m just not 100% sure how. Ignoring Differences At some point in the past, I had customized a few of my Application objects with ignoreDifferences settings. It was…

  • Maturing my Grafana setup

    I may have lost some dashboards and configuration recently, and it got me thinking about how to mature my Grafana setup for better persistence. Initial Setup When I first got Grafana running, it was based on the packaged Grafana Helm chart. As such, my Grafana instance was using SQLite database file stored in the persistent…

  • Speed. I.. am.. Speed.

    “One winner, 42 losers. I eat losers for breakfast. Breakfast? Maybe I should have had breakfast. A little brekie could be good for me….” My quest for speed lead to using 1TB of data every day…

  • A Lesson in Occam’s Razor: Configuring Mimir Ruler with Grafana

    Occam’s Razor posits “Of two competing theories, the simpler explanation is to be preferred.” I believe my high school biology teacher taught the “KISS” method (Keep It Simple, Stupid) to convey a similar principle. As I was trying to get alerts set up in Mimir using the Grafana UI, I came across an issue that…

  • Kubernetes Observability, Part 5 – Using Mimir for long-term metric storage

    This post is part of a series on observability in Kubernetes clusters: Part 1 – Collecting Logs with Loki Part 2 – Collecting Metrics with Prometheus Part 3 – Dashboards with Grafana Part 4 – Using Linkerd for Service Observability Part 5 – Using Mimir for long-term metric storage (this post) For anyone who actually…

  • Getting Synology SNMP data into Prometheus

    With my new cameras installed, I have been spending a lot more time in the Diskstation Manager (DSM). I always forget how much actually goes on within the Synology, and I am reminded of that every time I open the Resource Monitor. At some point, I started to wonder whether or not I could get…

  • Kubernetes Observability, Part 3 – Dashboards with Grafana

    This post is part of a series on observability in Kubernetes clusters: Part 1 – Collecting Logs with Loki Part 2 – Collecting Metrics with Prometheus Part 3 – Dashboards with Grafana (this post) Part 4 – Using Linkerd for Service Observability Part 5 – Using Mimir for long-term metric storage What good is Loki’s…