Matt Gerega

Tag: Grafana

  • Automating Grafana Backups

    After a few data loss events, I took the time to automate my Grafana backups. A bit of instability It has been almost a year since I moved to a MySQL backend for Grafana. In that year, I’ve gotten a corrupted MySQL database twice now, forcing me to restore from a backup. I’m not sure…

  • Re-configuring Grafana Secrets

    I recently fixed some synchronization issues that had been silently plaguing some of the monitoring applications I had installed, including my Loki/Grafana/Tempo/Mimir stack. Now that the applications are being updated, I ran into an issue with the latest Helm chart’s handling of secrets. Sync Error? After I made the change to fix synchronization of the…

  • Synced, But Not: ArgoCD Differencing Configuration

    Some of the charts in my Loki/Grafana/Tempo/Mimir stack have an odd habit of not updating correctly in ArgoCD. I finally got tired of it and fixed it… I’m just not 100% sure how. Ignoring Differences At some point in the past, I had customized a few of my Application objects with ignoreDifferences settings. It was…

  • Maturing my Grafana setup

    I may have lost some dashboards and configuration recently, and it got me thinking about how to mature my Grafana setup for better persistence. Initial Setup When I first got Grafana running, it was based on the packaged Grafana Helm chart. As such, my Grafana instance was using SQLite database file stored in the persistent…

  • Speed. I.. am.. Speed.

    “One winner, 42 losers. I eat losers for breakfast. Breakfast? Maybe I should have had breakfast. A little brekie could be good for me….” My quest for speed lead to using 1TB of data every day…

  • A Lesson in Occam’s Razor: Configuring Mimir Ruler with Grafana

    Occam’s Razor posits “Of two competing theories, the simpler explanation is to be preferred.” I believe my high school biology teacher taught the “KISS” method (Keep It Simple, Stupid) to convey a similar principle. As I was trying to get alerts set up in Mimir using the Grafana UI, I came across an issue that…

  • Kubernetes Observability, Part 5 – Using Mimir for long-term metric storage

    This post is part of a series on observability in Kubernetes clusters: Part 1 – Collecting Logs with Loki Part 2 – Collecting Metrics with Prometheus Part 3 – Dashboards with Grafana Part 4 – Using Linkerd for Service Observability Part 5 – Using Mimir for long-term metric storage (this post) For anyone who actually…

  • Getting Synology SNMP data into Prometheus

    With my new cameras installed, I have been spending a lot more time in the Diskstation Manager (DSM). I always forget how much actually goes on within the Synology, and I am reminded of that every time I open the Resource Monitor. At some point, I started to wonder whether or not I could get…

  • Kubernetes Observability, Part 3 – Dashboards with Grafana

    This post is part of a series on observability in Kubernetes clusters: Part 1 – Collecting Logs with Loki Part 2 – Collecting Metrics with Prometheus Part 3 – Dashboards with Grafana (this post) Part 4 – Using Linkerd for Service Observability Part 5 – Using Mimir for long-term metric storage What good is Loki’s…

  • Kubernetes Observability, Part 2 – Collecting Metrics with Prometheus

    This post is part of a series on observability in Kubernetes clusters: Part 1 – Collecting Logs with Loki Part 2 – Collecting Metrics with Prometheus (this post)Part 3 – Building Grafana DashboardsPart 4 – Using Linkerd for Service ObservabilityPart 5 – Using Mimir for long-term metric storage “Prometheus” appears in many Kubernetes blogs the…