Matt Gerega

  • Replacing ISY with Home Assistant – Part 3 – Movin’ In!

    This is the continuation of a short series on transitioning away from the ISY using Home Assistant.

    Having successfully gotten my new Home Assistant instance running, move in day was upon me. I did not have a set plan, but things were pretty simple.

    But first, HACS

    The Home Assistant Community Store (HACS) is a custom component for Home Assistant that enables UI management of other custom components. I have a few integrations that utilize custom components, namely Orbit B-Hyve and GE Home (SmartHQ).

    In my old HA instance, I had simply copied those folders in to the custom_components folder under my config directory, but HACS gives me the ability to manage these components from the UI, instead of via SSH. I followed the setup and configuration instructions to the letter, and was able to install the above custom components with ease.

    The Easy Stuff

    With HACS installed, I could tackle all the “non-major.” I am classifying major as my Insteon and Z-Wave devices, since those require some heavier lifting. There were lots of little integrations with external services that I could pretty quickly setup in the new instance and remove from the old. This included things like:

    • Orbit B-Hyve: I have an irrigation system in the backyard for some potted plants, and I put an Orbit Smart Hose timer on it. The B-Hyve app lets me set the schedule, so I don’t really need to automate that every day, but I do have it setup to enable the rain delay via NodeRED.
    • MyQ: I have a Chamberlain garage door open which is connected to MyQ, so this gives me the status of the door and the ability to open/close it.
    • GE Home: Not sure that I need to be able to see what my oven is doing, but I can.
    • Rheem Econet: I can monitor my hot water heater and set the temperature. It is mostly interesting to watch usage, and it is currently the only thing that allows me to track its power consumption.
    • Ring: This lets me get some information from my Ring doorbell, including its battery percentage.
    • Synology: The Synology integrate lets me monitor all of my drives and cameras. There is not much to control, per say, but it collects a lot of data points that I then scrape into Prometheus for alerting.
    • Unifi: I run the Unifi Controller for my home network, and this integration gives me an entity for all the devices on my network. Again, I do not use much of the control aspect, but I definitely use the data being collected.

    Were these all easy? Definitely. I was able to configure all of these integrations on the new instance and then delete them from the old without conflict.

    Now it’s time for some heavy lifting.

    Z-Wave Migration

    I only have 6 Z-Wave devices, but all were on the Z-Wave network controlled by the ISY. To my knowledge, there is no easy migration. I set up the Z-Wave JS add-on in Home Assistant, selecting my Z-Wave antenna from the USB list. Once that was done, I had to drop each device off of the ISY and then re-add it to the new Home Assistant instance.

    Those steps were basically as follows:

    1. Pick a device to remove.
    2. Select “Remove a Z-Wave Device” from the Z-Wave Menu in the ISY.
    3. While it is waiting, put the device in “enroll/un-enroll” mode. It’s different for every device. On my Mineston plugs, it was ‘click the power button three times quickly.’
    4. Wait for the ISY to detect the removal.
    5. In Home Assistant, under the Z-Wave integration, click Devices. Click the Add Device button, and it will listen for devices.
    6. Put the device in “enroll/un-enroll” mode again.
    7. If prompted, enter the device pin. Some devices require them, some do not. Of my 6 devices, three had pins, three did not.
    8. Home Assistant should detect the device and add it.
    9. Repeat steps 1 through 8 for all your Z-Wave devices.

    As I said, I only have 6 devices, so it was not nearly as painful. If you have a lot of Z-Wave devices, this process will take you some time.

    Insteon Migration

    Truthfully, I expected this to be very painful. It wasn’t that bad. I mentioned in my transition planning post that I grabbed an XML list of all my nodes in the ISY. This is my reference for all my Insteon devices.

    I disconnected the ISY from the PLM and connected it to the Raspberry Pi. I added the Insteon integration, and entered the device address (in my case, it showed up as /dev/ttyUSB1). At that point, the Insteon integration went about finding all my devices. They showed up with their device name and address, and the exercise was to look up the address in my reference and rename the device in Home Assistant.

    Since scenes are written to the devices themselves, my scenes came over too. Once I renamed the devices, I could set the scene names to a friendly name.

    NodeRED automation

    After flipping the URL in my new Home Assistant instance to be my old URL, I went into NodeRED to see the damage. I had to make a few changes to get things working:

    1. I had to generate a new long-lived token in Home Assistant, and update NodeRED with the new token.
    2. Since devices changed, I had to touch every action and make sure I had the right devices selected. Not terrible, just a bit tedious.

    ALEXA!

    I use the ISY Portal for integration with Amazon Alexa, and, well, my family have gotten used to doing some things with Alexa. Nabu Casa provides Home Assistant Cloud to fill this gap.

    It is not worth much space here, other than to say their documentation on installation and configuration was spot on, so check it out if you need integration with Amazon Alexa or Google Assistant.

    Success!!!

    My ISY is shut down, and my Home Assistant is running the house, including the Insteon and Z-Wave devices.

    I did notice that, on reboot, the USB address of the Z-Wave and PLM device swapped. I hope that isn’t a recurring thing. The solution was to re-configure the Insteon and Z-Wave integrations with the new address. Not hard, I just hope it is not a pattern.

    My NodeRED integrations are much more stable. Previously, NodeRED was calling Home Assistant, which was trying to use the ISY to control the devices. This was fraught with errors, mostly because the ISY’s APIs can be dodgy. With Home Assistant calling the shots directly, it’s much more responsive.

    I have to work on some of my scenes and automations for Insteon: While I had previously moved most of my programs out of the ISY and into NodeRED, there were a few stragglers that I need to setup on NodeRED. But that will take about 20 minutes.

    At this point, I’m going to call this venture successful. That said, I will now focus my attention on my LED strips. I have about 6 different LED strips with some form of MiLight/MiBoxer controller. I hate them. So I will be exploring alternatives. Who knows, maybe my exploration will generate another post.

  • Replacing ISY with Home Assistant – Part 2 – A New Home

    This is the continuation of a short series on transitioning away from the ISY using Home Assistant.

    Getting Started, again

    As I mentioned in my previous post, my plan is to run my new instance of Home Assistant in parallel with my old instance and transfer functionality in pieces. This should allow me to minimize downtime, and through the magic of reverse proxy, I will end up with the new instance living at the same URL as the old instance.

    Part of the challenge of getting started is simply getting the Raspberry Pi setup in my desired configuration. I bought an Argon One M.2 case and an M.2 SSD card to avoid running Home Assistant on an SD Card. However, that requires a bit of prework, particularly for my older Pi.

    New Use, New Case

    I ordered the Argon One M.2 case after a short search. I was looking for a solution that allowed me to mount and connect an M.2 SSD. In this sense, there were far too many options. There are a number of “bare board” solutions, including one from Geekworm and another from Startech.com. The price points were similar, hovering around $25 per board. However, the bare board required me to buy a new case, and most of the “tall” cases required for both the Pi and the bare board ran another $15-$25, so I was looking at around $35-$45 for a new board and case.

    My Amazon searches kept bringing up the Argon One case, so I looked into it. It provided both the case and the SSD support, and added some thermal management and a sleek pinout extension. And, at about $47, the price point was similar to what I was going to spend on a board and new case, so I grabbed that case. Hands on, I was not disappointed: the case is solid and had a good guide for installation packaged with it.

    Always read ahead…

    When it comes to instructions, I tend to read ahead. Before I put everything in the case, I wanted to make sure I was going to be ready before I buttoned it up. As I read through the waveshare.com guide for getting the Argon One case running with a boot to SSD, I noticed steps 16 and 17.

    The guide walked through the process of using the SD Card Copier to move the image from the SD card to the SSD card. However, I am planning on using the Home Assistant OS image, which means I’ll need to image the SSD from my machine with that image. Which means I have to get the SSD connected to my machine…

    Yet another Franken-cable

    I do not have a USB adapter for SSD cards, because I do not flash them often enough to care. So how do I use the Raspberry Pi Imager to flash Home Assistant OS onto my SSD? With a Franken-cable!

    I installed the M.2 SSD in the Argon One’s base, but did not put the PI on it. Using the bare base, I installed the “male to male” USB U adapter in the M.2 base, and used a USB extension cable to attach the other end of the U Adapter to my PC. It showed up as an Argon SSD, and I was able to flash the SSD with the Home Assistant OS.

    Updated Install Steps

    So, putting all this together, I did the following to get Home Assistant running on Raspberry Pi / Argon One SSD:

    1. Install the Raspberry Pi in the Argon One case, but do not attach the base with the SSD.
    2. From this guide, follow steps 1-15 as written. Then shutdown the system and take out the SSD.
    3. Install the SSD in Argon One base, and attach it to your PC using the USB Male to Male U adapter (included with the Argon) and a USB extension cable.
    4. Write the Home Assistant OS for RPI4 to the SSD using the Raspberry Pi Imager utility.
    5. Put the Argon One case together, and use the U adapter to connect the SSD to the RPI.
    6. Power on the RPI

    At this point, Home Assistant should boot for the first time and begin its setup process.

    Argon Add-ons

    Now, the Argon case has a built-in fan and fan controller. When using Raspbian, you can install the controller software. Home Assistant OS is different, but thankfully, Adam Outler wrote add-ons to allow Home Assistant to control the Argon fan.

    I followed the instructions, but then realized that I needed to enable I2C in order to get it to work. Adam to the rescue: Adam wrote a HassOS configurator add on for both I2C and Serial support. I installed the I2C configurator and ran it according to its instructions.

    Running on Empty

    My new Home Assistant instance is running. It is not doing anything, but it is running. Next steps will be to start migrating my various integrations from one instance to another.

  • Replacing ISY with Home Assistant – Part 1a – Transition Planning

    This is the continuation of a short series on transitioning away from the ISY using Home Assistant.

    The Experiment

    I mentioned in my previous post that I had ordered some parts to run an experiment with my PowerLinc Modem (PLM). I needed to determine that I could use my existing PLM (an Insteon 2413S, which is the serial version) with Home Assistant’s Insteon plugin.

    Following this highly-detailed post from P. Lutus, I ordered the following parts:

    • A USB Serial Adapter -> I used this one, but I think any USB to DB9 adapter will work.
    • A DB9 to RJ45 Modular Adapter -> The StarTech.com one seems to be popular in most posts, and it was easy to use.

    While I waited for these parts, I grabbed the Raspberry Pi 4 Model B that I have tasked for this and got to work installing a test copy of Home Assistant on it. Before you even ask, I have had this Raspberry Pi 4 for a few years now, prior to the shortages. It has served many purposes, but its most recent task was as a driver for MagicMirror on my office television. However, since I transitioned to a Banana Pi M5 for my network reverse proxy, well, I had a spare Raspberry Pi 3 Model B hanging around. So I moved MagicMirror to the RPi 3, and I have a spare RPi 4 ready to be my new Home Assistant.

    Parts Arrived!

    Once my parts arrived, I assembled the “Frankencable” necessary to connect my PLM to the Raspberry Pi. It goes PLM -> Standard Cat 5 (or Cat 6) Ethernet Cable -> RJ45 to DB9 Adapter -> Serial to USB Adapter -> USB Port on the Pi.

    With regard to the RJ45 to DB9 Adapter, you do need the pinout. Thankfully, Universal Devices provides one as part of their Serial PLM Kit. You could absolutely order their kit and it would work. But their Kit is $26: I was able to get the Serial to USB adapter for $11.99, and the DB9 to RJ45 for $3.35, and I have enough ethernet cable lying around to wire a new house, so I got away for under $20.

    Before I started, I grabbed an output of all of my Insteon devices from my ISY. Now, P. Lutus’ post indicates using the ISY’s web interface to grab those addresses, but if you are comfortable with Postman or have another favorite program for making Web API calls, you can get an XML document with everything. The curl command is 

    curl http://<isy ip address>/rest/nodes -u "<isy user>:<isy password>"

    I used Postman to make the call and stored the XML in a file for reference.

    With everything plugged in, I added the Insteon integration to my test Home Assistant installation, selected the PLM Serial option, and filled in the device address. That last one took some digging, as I had to figure out which device to use. The easiest way to do it is to plug in the cable, then use dmesg to determine where in /dev the device is mounted. This linuxhint.com post gives you a few options for finding out more about your USB devices on Linux systems.

    At that point, the integration took some time to discover my devices. As mentioned in P. Lutus’ post, it will take some time to discover everything, and the battery-operated devices will not be found automatically. However, all of my switches came in, and each device address was available.

    What about Z-Wave?

    I have a few Z-Wave devices that also use the ISY as a hub. To move off of the ISY completely, I need a Z-Wave alternative. A colleague of mine runs Z-Wave at home to control EVERYTHING, and does so with a Z-Wave antenna and Home Assistant. I put my trust in his work and did not feel the need to experiment with the Z-Wave aspect. I just ordered an antenna.

    With that out of the way, I declared my experiment a success, and starting working on a transition plan.

    Raspberry Pi à la Mode

    Raspberry Pi is great on its own, but everything is better with some ice cream! In this case, my “ice cream” is a new case and a new M.2 SSD. Why? Home Assistant is chatty, and will be busy running a lot of my Home Automation. And literally every channel I’ve seen on the Home Assistant Discord server says “do not run on an SD card!”

    The case above is an expandable system that not only lets me add an M.2 SATA drive to the Pi, but also adds some thermal management for the platform in general. Sure, it also adds and HDMI daughter board, but considering I’ll be running Home Assistant OS, dual screen displays of the command line does not seem like a wise display choice.

    With those parts on order, I have started to plan my transition. It should be pretty easy, but there are a number of steps involved. I will be running two instances of Home Assistant in parallel for a time, just to make sure I can still turn the lights on and off with the Amazon Echo… If I don’t, my kids might have a fit.

    1. Get a good inventory of what I have defined in the ISY. I will want a good reference.
    2. Get a new instance of Home Assistant running from the SSD on the RPI 4. I know there will be some extra steps to get this all working, so look for that in the next post.
    3. Check out voice control with Home Assistant Cloud. Before I move everything over, I want to verify the Home Assistant Cloud functionality. I am currently paying for the ISY portal, so I’ll be switching from one to the other for the virtual assistant integration.
    4. Migrate my Z-Wave devices. Why Z-Wave first? I only have a few of those (about 6), and they are running less “vital” things, like lamps and landscape lighting. Getting the Z-Wave transferred will allow me to test all of my automation before moving the Insteon Devices
    5. Migrate my Insteon Devices. This should be straight forward, although I’ll have to re-configure any scenes and automations in Node Red.

    Node Red??

    For most of my automation, I am using a separate installation of Node Red and the Home Assistant palette.

    Node Red provides a great drag-and-drop experience for automation, and allows for some pretty unique and complex flows. I started moving away from the ISY programs over the last year. The only issue with it has been that the ISY’s API connectivity is spotty, meaning Home Assistant sometimes has difficulty talking to the ISY. Since Node Red goes through Home Assistant to get to the ISY, sometimes the programs look like they’ve run correctly when, in fact, they have not.

    I am hoping that removing the ISY will provide a much better experience with this automation.

    Next Steps

    When my parts arrive, I will start into my plan. Look for the next in the series in a week or so!

  • Replacing ISY with Home Assistant – Part 1 – Preparation

    This will hopefully be a short series on migrating away from my ancient ISY994.

    They killed it!

    I have had an ISY994 since early 2018, and it has served me well. It is the core communicator with my Insteon and Z-Wave devices. However, Universal Devices is killing it in favor of their eisy device.

    Now, I have to be very clear: as a software engineer, I absolutely understand their decision. Innovating software while having to support old hardware is a painful process. However, the cost to move is making me look elsewhere.

    When I originally purchased the ISY, I paid about $400 for the unit and the Serial PowerLinc Modem (PLM). Considering it has been running for 5 years, $80 a year is not bad at all. But to move to the eisy, I need to buy:

    • eisy – $290
    • Serial Adapter for my PLM – $26
    • Z-Matter USB – $126

    So I am looking at about $450 for an upgrade. But some more “recent” developments make me wonder if I can do it better.

    Enter Home Assistant

    I do not have an exact date, but I have been running Home Assistant for a few years, and I prefer it over the ISY. The interface is newer, and the open source nature makes it a bit more reactive to new technology. Now, Home Assistant has an integration with the ISY, but the ISY’s APIs are, well, flaky. I find myself having to remove/re-add the ISY to Home Assistant, reboot the Home Assistant, and/or reboot the ISY to get it back.

    With the ISY being retired, can I just replace it with the Home Assistant? Well, that’s what the prep work is about.

    Requirements Gathering

    Like any good project, I started by outlining some basic requirements:

    1. Insteon Support -> I have a lot of Insteon devices, mostly hardwired switches. Supporting those is non-negotiable. I have a Serial PLM, it would be nice to re-use that for communication with my Insteon devices.
    2. Z-Wave Support -> I have a few Z-Wave devices, mostly some plug-in outlets and a relay. These are currently supported via my ISY, but the antenna is weak and therefore the Z-Wave is less reliable.
    3. Standalone -> I am running Home Assistant as a Kubernetes node in my production cluster. Sure, it works, and it makes upgrades easier. Having a critical system in lab components makes me nervous, so I want to move Home Assistant to its own hardware.

    Experimentation

    Right now, I am in experimentation mode. I have ordered some parts to connect my PLM directly to a Raspberry Pi, and have started the process of installing Home Assistant on the Pi. I am also shopping Z-Wave dongles.

    The next few weekends will involve some experimentation. I’m sure everyone in the house will be thrilled when Alexa no longer controls the lights…

  • Tech Tip – Not all certificates are the same

    I have been trying to build a model in Azure to start modernizing one of our applications. Part of that is configuring an application gateway correctly and getting end-to-end SSL configured. As it turns out, not all certificates are good certificates, at least to Azure.

    Uploading the Cert

    I have a wildcard certificate for a test domain, so I exported it into a full chain PFX that I could upload into Azure where I needed. The model I’m building is “hand built” for now, so I am not terribly concerned about uploading the certificate in a few places just to get things moving.

    I was able to upload the certificate into Key Vault, as well as to the Azure Application Gateway I created. But, when I went to use the certificate for a custom domain in an Azure App Service, well, it was fighting me.

    Legacy Only??

    As it turns out, App Services has some very specific requirements for its certificates. My method to export was “too new” to work. Thankfully, I came across a StackOverflow question that solved the issue.

    For everyone’s reference, I had to import the certificate into Windows, and then export another PFX with the proper encryption.

    From the Stack Overflow post above, in Powershell, import the existing PFX:

    Import-PfxCertificate -FilePath "pfx file path" -CertStoreLocation Cert:\LocalMachine\My -Password (ConvertTo-SecureString -String 'MyPassword' -AsPlainText -Force) -Exportable

    Grab the thumprint (you’ll need it), and then export the certificate in Powershell:

    Export-PfxCertificate -Cert Microsoft.PowerShell.Security\Certificate::LocalMachine\My\B56CE9B122FB04E29A974A4D0DB3F6EAC2D150C0 -FilePath 'newPfxName.pfx' -Password (ConvertTo-SecureString -String 'MyPassword' -AsPlainText -Force)

    The newly generated PFX can be used in Azure App Services!

  • You can’t trust anyone…

    Did I “hack” a major online website. You could call it that. If using the browser’s built-in developer tools to change an element id so that I could submit my order… well, then, yes. I “hacked” the site so I could buy something….

    “Refresh your cache” and other tall tales

    I was helping someone I know sign up for a service through a major online website. The process was, well, long, as there were several different options and a number of hoops to jump through. However, upon getting to the page where they could accept their agreements and submit the order, the site blocked us.

    The problem was pretty evident to me: there were 6 different checkboxes that were required to accept various terms, conditions, and agreements. Two of those checkbox elements had the same text. When I clicked on the second of the two matching checkboxes, it selected (or unselected) the first instance, and not the second. I hit F12 and looked at the HTML source. Sure enough, those two instances shared the same name and id attribute values.

    In an effort to be a good citizen, I reached out through the chat window to see if they could resolve this problem on their side. There answers were “Delete your cache,” “close and reopen your browser,” and, my favorite, “restart the entire process.” At this point in the day, I had to leave the house, so I figured I would try again later.

    Second Attempt

    For my second attempt, I used the private browsing mode, which should alleviate cookies and other cache values. I went through the process again, and the same issue presented it self on the submit screen.

    Technical Support tried to get me to go to the Sales department, but it was after hours, and I was getting a little frustrated. So I opened the Developer Tools in the browser and examined the checkbox elements.

    This is where things got weird. It turns out, for the name and id attributes, they were using the actual page text value for those. I’ve NEVER seen a site which uses the user text to populate the id attribute. That generally seems like bad practice. But, what do I know? The best part is, the developer tools window gave me a warning because there were two checkboxes with the same aria and label values.

    My “Hack”

    Since I couldn’t check the second instance of the duplicate, the submit button was not enabled. My “hack” was to use the developer tools to modify the name and id values to be unique, which I did by just adding an extra space somewhere in the value. Once I did that, I was able to check the second checkbox, which then enabled the Submit button, and I was able to proceed in placing the order.

    Let me be VERY clear: I modified the client HTML so that I was able to accept all the terms of the service being ordered, and then submitted said order. There was no malicious activity: I got nothing for free (no additional services or deals) and the same action would have been undertaken by picking up the phone and calling a salesperson.

    With this in mind, I will be reporting this to the vendor so that they may correct their software.

    Validate your input!!!

    For all the web developers out there: validate your client input. The rise of Single Page Applications (SPAs) that run in Javascript on the client have blinded us to the need to handle all incoming data as if it could be different than what we expect. We just assume the user will click through what we give them. But the F12 Developer Tools built in to modern browsers give everyone the ability to change the HTML on your page. Sure, I did it so I could submit my order without talking to a human being. But malicious actors could use that functionality for far more nefarious purposes.

  • When browsers have a mind of their own…

    I recently implemented Cloudflare on my domains to allow me some protection on my domains. There were some unintended side effects, and it seems to have to do with my browser’s DNS.

    Internal Site Management

    All of my mattgerega.com and mattgerega.org sites are available outside of my network, so putting Cloudflare in front as a WAF was not a problem. However, I have a few sites hosted on mattgerega.net which, through security in the reverse proxy, are not accessible outside of my home network.

    My internal DNS has an entry to push the mattgerega.net sites directly to my reverse proxy, and should bypass the WAF. Non-browser traffic, like a simple curl command, works just fine. However, in the browser, I was getting 403 errors most of the time. Oddly, incognito/private modes sometimes fixed it.

    My Browser has a DNS Cache

    This is where I learned something new: modern browsers have their own DNS cache. Want to see yours? In Chrome, navigate to chrome://net-internals/#dns. There is a lookup function as well as an ability to clear the DNS cache.

    Using the nslookup in my browser, mattgerega.net resolved to some Cloudflare IPs. Even though my machine is using my internal DNS, the browser has other ideas.

    No Proxy For You!

    This was all, well, rather annoying. Because mattgerega.net was being proxied, the browser DNS seemed to be preferring the Cloudflare DNS over my local one. Perhaps because it is secure, even though I turned off the requirement for secure DNS in the browser.

    The solution was to stop proxying mattgerega.net in Cloudflare. This allowed my local DNS entries to take over, and my access was fixed without me having to open up to other IP addresses.

    Now I just have to do better about making the mattgerega.net sites internal-only.

  • Collecting WCF Telemetry with Application Insights

    I got pulled back in to diagnosing performance issues on an old friend, and it lead me into some dead code with a chance for resuscitation.

    Trouble with the WCF

    One of our applications has a set of WCF Services that serve the client. We had not instrumented them manually for performance metrics, although we found that System.ServiceModel is well decorated for trace, so adding a trace listener gives us a bevy of information. However, it’s all in *.svclog files (using an XML Trace Listener), so there is still a lot to do in order to find out what’s wrong. A colleague asked if Application Insights could work. And that suggested started me down a path.

    I found the lab

    I reached out to some contacts at Microsoft, and they pointed me to the Application Insights SDK Labs. In that lab is a library that instruments WCF services and applications. The problem: it hasn’t been updated in about 5 years.

    I figured, our application is based on technology about that old, I suppose I could try it. So I followed the instructions, and I started getting telemetry in my Application Insights instance! However, I did notice a few things:

    1. The library as published relies on an old Application Insights version (2.5.0, I believe). The Application Insights libraries are up to 2.21, which means we may not see everything we can.
    2. The library is based on .Net Framework 4.5, not 4.6.2, which is what we are using.

    So I did what any sensible developer does…. fork it!

    I’ll do it myself!

    I forked the repository, created a branch, and got to work upgrading. Since this is just for work (right now), I did not bother to setup CI/CD yet, and took some liberties in assuming we would be running .Net Framework 4.6.2 for a while.

    I had to wade through the repository setup a bit: There is a lot of configuration around Nuget and versioning, and, frankly, I wasn’t looking to figure that out right now. However, I did manage to get new library built, including updating the references to Microsoft.Application insights.

    Again, in the interest of time and test, I manually pushed the package I built to our internal Nuget feed. I’m going to get this pushed to a development environment so that I can get some better telemetry than just the few calls that get made in my local environment.

    Next Steps?

    If this were an active project, I would probably have made a little more effort to do things the way they were originally structured and “play in the sandbox.” However, with no contribution rules or guidelines and no visible build pipeline, I am on my own with this one.

    If this turns out to be reasonably useful, I will probably take a look at the rest of the projects in that repository. I may also tap my contacts at Microsoft for some potential “next steps” with this one: while I do not relish the thought of owning a public archive, perhaps I could at least get a copy of their build pipeline so that I can replicate it on my end. Maybe a GitHub Actions pipeline with a push to Nuget?? Who knows.

  • Aging into fun

    It is obvious that my posts have been, well, tech-centric for the better part of the last year. I do my best to not post specifics about my kids because, well, their life is theirs to live, not mine to post. But I have to credit them with introducing me to an activity that keeps me active and lets me have a little fun: soccer.

    Fitness was an issue…

    My introduction to soccer was, as I recall, 11v11, full field games somewhere between ages 8 and 11. Genetics dealt me a Mac truck, not a Ferrari, and so chasing a ball and a bunch of Ferraris around a field just was not going to appeal to me. I ended up playing baseball until high school, middle school football, and high school volleyball, but never returned to soccer.

    Side Note: Based on my size alone, I get comments pretty much weekly as to if/when I played football. I probably could have been good, but my passive personality did not lend itself well to that sport. But, as I tell my brother, I have no chronic knee, back, shoulder pain from a contact sport…. So I have no regrets.

    Having a kid in your early 20’s is a stressful affair, and that stress didn’t exactly help my weight problems. With two kids, and approaching thirty well north of three hundred pounds, I decided I should make some changes if I want to see my kids grow up. I started running and watching what I eat. I ended up losing about seventy pounds, and felt pretty good.

    Becoming a soccer player

    My oldest played youth soccer from U6 through U18. I spent a lot of time at the soccer fields, and have fond memories of even coaching a few U8 seasons with him. As I was introduced to that community, I found that there were some co-ed leagues in our area. The first one I joined was a 7v7 league which was played on either a U10 or a U12 field (I do not remember the exact size).

    I had a lot of fun in that league, and was introduced to the game in a way where I was not required to run full-field sprints to recover. I learned the basics of the game, and became a serviceable field player.

    Graduating to full field

    At some point, I thought it might be fun to play in the Over 30 adult league. This league was a little more formal, with actual home and away games, real kits (or at least a nice jersey), and a full referee team.

    Did I run more than I ever had in my life? Yes. Was it fun? Absolutely. In spite of some of the attitude issues (we aren’t playing for the World Cup, but you’d think we were), I had a great time with it. My personal and professional life was getting busy, and that, coupled with the attitude issues, led me to walk away from that team for a while.

    When opportunity knocks..

    A lot happened in the next few years. Got divorced, then COVID, then I got married again (well, married during COVID, as if we needed a bigger challenge), all the while working through a few different roles at work. I never actively looked for a team: there was enough to do without introducing another activity. A chance encounter with an old board member from the club changed that.

    I came to find out that the community club my kids play started fielding adult teams a few years ago. As of now, we have an an over 30 team and an over 40 team. So I joined on, officially on the over 40 team. Sure, I can fill in on the over 30 team, and I have a few times. But that only made me realize how painfully out of shape I’ve become. Over 40 is a little more my speed. Still 11v11, full field, but 20 minute quarters instead of 45 minute halves. More breaks… I love it.

    What are you getting at?

    This post turned into something of “Matt’s life sport story.” The point is this: you are never too old to try something new. Soccer is a great way to stay active, and gives me just another reason to stay fit and healthy. And this new “adventurous” spirit has led me to try other new things, including scuba diving and skiing. Scuba I love, and my comfort in the water gives me a jump on a lot. Skiing…. Well, I had a lesson, and let’s say I can do it, but I need a lot more practice.

    So, try something new. You never know what will happen.

  • Maturing my Grafana setup

    I may have lost some dashboards and configuration recently, and it got me thinking about how to mature my Grafana setup for better persistence.

    Initial Setup

    When I first got Grafana running, it was based on the packaged Grafana Helm chart. As such, my Grafana instance was using SQLite database file stored in the persistent volume. This limits me to a single Grafana pod, since the volume is not setup to shared across pods. Additionally, that SQL database file is to the lifecycle of the claim associated with the volume.

    And, well, at home, this is not a huge deal because of how the lab is setup for persistent volume claims. Since I use the nfs-subdir-external-provisioner, PVCs in my clusters automatically generate a subfolder in my NFS share. When the PVC is deleted, the subdir gets renamed with an archive- prefix, so I can usually dig through the folder to find the old database file.

    However, using the default Azure persistence, Azure Disks are provisioned. When a PVC gets deleted, so to does the disk, or, well, I think it does. I have not had the opportunity to dig in to the Azure Disk PVC provisioning to understand how that data is handled when PVCs go away. It is sufficient to say that I lost our Grafana settings because of this.

    The New Setup

    The new plan is to utilize MySQL to store my Grafana dashboards and data stores. The configuration seems simple enough: add the appropriate entries in the grafana.ini file. I already know how to expand secrets, so getting the database secrets into the configuration was easy using the grafana.ini section of the Helm chart.

    For my home setup, I felt it was ok to run MySQL as another dependent chart for Grafana. Now, from the outside, you should be saying “But Matt, that only moves your persistence issues from the Grafana chart to the MySQL chart!” That is absolutely true. But, well, I have a pretty solid backup plan for those NFS shares, so for a home lab that should be fine. Plus I figured out how to backup and restore Grafana (see below).

    The real reason is that, for the instance I am running in Azure at work, I want to provision an Azure MySQL instance. This will allow me to have much better backup retention that inside the cluster, but the configuration at work will match the configuration at home. Home lab in action!

    Want to check out my home lab configuration? Check out my ops-internal infrastructure repository.

    Backup and Restore for Grafana

    As part of this move, I did not want to lose the settings I had in Grafana. This mean finding a backup/restore procedure that worked. An internet search lead me to the Grafana Backup Tool. The tool provides backup and restore capabilities through Grafana’s APIs.

    That said, it is written in Python, so my recent foray into Python coding served me well to get this tool up and running. Once I generated an API Key, I was off and running.

    There really isn’t much to it: after configuring the URL and API Token, I ran a backup to get a .tar.gz file with my Grafana contents. Did I test the backup? No. It’s the home lab, worst that could happen is I have to re-import some dashboards and re-create some others.

    After that, I updated my Grafana instance to include the MySQL instance and updated Grafana’s configuration to use the new MySQL service. As expected, all my dashboards and data sources disappeared.

    I ran the restore function using my backup, refreshed Grafana in my browser, and I was back up and running! Testing, schmesting….

    What’s Next?

    I am going to take my newfound learnings and apply them at work:

    1. Get a new MySQL instance provisioned.
    2. Backup Grafana.
    3. Re-configure Grafana to use the new MySQL instance.
    4. Restore Grafana.

    Given the ease with which the home lab went, I cannot imagine I will run into much issue.