Matt Gerega

Author: Matt

  • Replacing ISY with Home Assistant – Part 1 – Preparation

    This will hopefully be a short series on migrating away from my ancient ISY994.

    They killed it!

    I have had an ISY994 since early 2018, and it has served me well. It is the core communicator with my Insteon and Z-Wave devices. However, Universal Devices is killing it in favor of their eisy device.

    Now, I have to be very clear: as a software engineer, I absolutely understand their decision. Innovating software while having to support old hardware is a painful process. However, the cost to move is making me look elsewhere.

    When I originally purchased the ISY, I paid about $400 for the unit and the Serial PowerLinc Modem (PLM). Considering it has been running for 5 years, $80 a year is not bad at all. But to move to the eisy, I need to buy:

    • eisy – $290
    • Serial Adapter for my PLM – $26
    • Z-Matter USB – $126

    So I am looking at about $450 for an upgrade. But some more “recent” developments make me wonder if I can do it better.

    Enter Home Assistant

    I do not have an exact date, but I have been running Home Assistant for a few years, and I prefer it over the ISY. The interface is newer, and the open source nature makes it a bit more reactive to new technology. Now, Home Assistant has an integration with the ISY, but the ISY’s APIs are, well, flaky. I find myself having to remove/re-add the ISY to Home Assistant, reboot the Home Assistant, and/or reboot the ISY to get it back.

    With the ISY being retired, can I just replace it with the Home Assistant? Well, that’s what the prep work is about.

    Requirements Gathering

    Like any good project, I started by outlining some basic requirements:

    1. Insteon Support -> I have a lot of Insteon devices, mostly hardwired switches. Supporting those is non-negotiable. I have a Serial PLM, it would be nice to re-use that for communication with my Insteon devices.
    2. Z-Wave Support -> I have a few Z-Wave devices, mostly some plug-in outlets and a relay. These are currently supported via my ISY, but the antenna is weak and therefore the Z-Wave is less reliable.
    3. Standalone -> I am running Home Assistant as a Kubernetes node in my production cluster. Sure, it works, and it makes upgrades easier. Having a critical system in lab components makes me nervous, so I want to move Home Assistant to its own hardware.

    Experimentation

    Right now, I am in experimentation mode. I have ordered some parts to connect my PLM directly to a Raspberry Pi, and have started the process of installing Home Assistant on the Pi. I am also shopping Z-Wave dongles.

    The next few weekends will involve some experimentation. I’m sure everyone in the house will be thrilled when Alexa no longer controls the lights…

  • Tech Tip – Not all certificates are the same

    I have been trying to build a model in Azure to start modernizing one of our applications. Part of that is configuring an application gateway correctly and getting end-to-end SSL configured. As it turns out, not all certificates are good certificates, at least to Azure.

    Uploading the Cert

    I have a wildcard certificate for a test domain, so I exported it into a full chain PFX that I could upload into Azure where I needed. The model I’m building is “hand built” for now, so I am not terribly concerned about uploading the certificate in a few places just to get things moving.

    I was able to upload the certificate into Key Vault, as well as to the Azure Application Gateway I created. But, when I went to use the certificate for a custom domain in an Azure App Service, well, it was fighting me.

    Legacy Only??

    As it turns out, App Services has some very specific requirements for its certificates. My method to export was “too new” to work. Thankfully, I came across a StackOverflow question that solved the issue.

    For everyone’s reference, I had to import the certificate into Windows, and then export another PFX with the proper encryption.

    From the Stack Overflow post above, in Powershell, import the existing PFX:

    Import-PfxCertificate -FilePath "pfx file path" -CertStoreLocation Cert:\LocalMachine\My -Password (ConvertTo-SecureString -String 'MyPassword' -AsPlainText -Force) -Exportable

    Grab the thumprint (you’ll need it), and then export the certificate in Powershell:

    Export-PfxCertificate -Cert Microsoft.PowerShell.Security\Certificate::LocalMachine\My\B56CE9B122FB04E29A974A4D0DB3F6EAC2D150C0 -FilePath 'newPfxName.pfx' -Password (ConvertTo-SecureString -String 'MyPassword' -AsPlainText -Force)

    The newly generated PFX can be used in Azure App Services!

  • You can’t trust anyone…

    Did I “hack” a major online website. You could call it that. If using the browser’s built-in developer tools to change an element id so that I could submit my order… well, then, yes. I “hacked” the site so I could buy something….

    “Refresh your cache” and other tall tales

    I was helping someone I know sign up for a service through a major online website. The process was, well, long, as there were several different options and a number of hoops to jump through. However, upon getting to the page where they could accept their agreements and submit the order, the site blocked us.

    The problem was pretty evident to me: there were 6 different checkboxes that were required to accept various terms, conditions, and agreements. Two of those checkbox elements had the same text. When I clicked on the second of the two matching checkboxes, it selected (or unselected) the first instance, and not the second. I hit F12 and looked at the HTML source. Sure enough, those two instances shared the same name and id attribute values.

    In an effort to be a good citizen, I reached out through the chat window to see if they could resolve this problem on their side. There answers were “Delete your cache,” “close and reopen your browser,” and, my favorite, “restart the entire process.” At this point in the day, I had to leave the house, so I figured I would try again later.

    Second Attempt

    For my second attempt, I used the private browsing mode, which should alleviate cookies and other cache values. I went through the process again, and the same issue presented it self on the submit screen.

    Technical Support tried to get me to go to the Sales department, but it was after hours, and I was getting a little frustrated. So I opened the Developer Tools in the browser and examined the checkbox elements.

    This is where things got weird. It turns out, for the name and id attributes, they were using the actual page text value for those. I’ve NEVER seen a site which uses the user text to populate the id attribute. That generally seems like bad practice. But, what do I know? The best part is, the developer tools window gave me a warning because there were two checkboxes with the same aria and label values.

    My “Hack”

    Since I couldn’t check the second instance of the duplicate, the submit button was not enabled. My “hack” was to use the developer tools to modify the name and id values to be unique, which I did by just adding an extra space somewhere in the value. Once I did that, I was able to check the second checkbox, which then enabled the Submit button, and I was able to proceed in placing the order.

    Let me be VERY clear: I modified the client HTML so that I was able to accept all the terms of the service being ordered, and then submitted said order. There was no malicious activity: I got nothing for free (no additional services or deals) and the same action would have been undertaken by picking up the phone and calling a salesperson.

    With this in mind, I will be reporting this to the vendor so that they may correct their software.

    Validate your input!!!

    For all the web developers out there: validate your client input. The rise of Single Page Applications (SPAs) that run in Javascript on the client have blinded us to the need to handle all incoming data as if it could be different than what we expect. We just assume the user will click through what we give them. But the F12 Developer Tools built in to modern browsers give everyone the ability to change the HTML on your page. Sure, I did it so I could submit my order without talking to a human being. But malicious actors could use that functionality for far more nefarious purposes.

  • When browsers have a mind of their own…

    I recently implemented Cloudflare on my domains to allow me some protection on my domains. There were some unintended side effects, and it seems to have to do with my browser’s DNS.

    Internal Site Management

    All of my mattgerega.com and mattgerega.org sites are available outside of my network, so putting Cloudflare in front as a WAF was not a problem. However, I have a few sites hosted on mattgerega.net which, through security in the reverse proxy, are not accessible outside of my home network.

    My internal DNS has an entry to push the mattgerega.net sites directly to my reverse proxy, and should bypass the WAF. Non-browser traffic, like a simple curl command, works just fine. However, in the browser, I was getting 403 errors most of the time. Oddly, incognito/private modes sometimes fixed it.

    My Browser has a DNS Cache

    This is where I learned something new: modern browsers have their own DNS cache. Want to see yours? In Chrome, navigate to chrome://net-internals/#dns. There is a lookup function as well as an ability to clear the DNS cache.

    Using the nslookup in my browser, mattgerega.net resolved to some Cloudflare IPs. Even though my machine is using my internal DNS, the browser has other ideas.

    No Proxy For You!

    This was all, well, rather annoying. Because mattgerega.net was being proxied, the browser DNS seemed to be preferring the Cloudflare DNS over my local one. Perhaps because it is secure, even though I turned off the requirement for secure DNS in the browser.

    The solution was to stop proxying mattgerega.net in Cloudflare. This allowed my local DNS entries to take over, and my access was fixed without me having to open up to other IP addresses.

    Now I just have to do better about making the mattgerega.net sites internal-only.

  • Collecting WCF Telemetry with Application Insights

    I got pulled back in to diagnosing performance issues on an old friend, and it lead me into some dead code with a chance for resuscitation.

    Trouble with the WCF

    One of our applications has a set of WCF Services that serve the client. We had not instrumented them manually for performance metrics, although we found that System.ServiceModel is well decorated for trace, so adding a trace listener gives us a bevy of information. However, it’s all in *.svclog files (using an XML Trace Listener), so there is still a lot to do in order to find out what’s wrong. A colleague asked if Application Insights could work. And that suggested started me down a path.

    I found the lab

    I reached out to some contacts at Microsoft, and they pointed me to the Application Insights SDK Labs. In that lab is a library that instruments WCF services and applications. The problem: it hasn’t been updated in about 5 years.

    I figured, our application is based on technology about that old, I suppose I could try it. So I followed the instructions, and I started getting telemetry in my Application Insights instance! However, I did notice a few things:

    1. The library as published relies on an old Application Insights version (2.5.0, I believe). The Application Insights libraries are up to 2.21, which means we may not see everything we can.
    2. The library is based on .Net Framework 4.5, not 4.6.2, which is what we are using.

    So I did what any sensible developer does…. fork it!

    I’ll do it myself!

    I forked the repository, created a branch, and got to work upgrading. Since this is just for work (right now), I did not bother to setup CI/CD yet, and took some liberties in assuming we would be running .Net Framework 4.6.2 for a while.

    I had to wade through the repository setup a bit: There is a lot of configuration around Nuget and versioning, and, frankly, I wasn’t looking to figure that out right now. However, I did manage to get new library built, including updating the references to Microsoft.Application insights.

    Again, in the interest of time and test, I manually pushed the package I built to our internal Nuget feed. I’m going to get this pushed to a development environment so that I can get some better telemetry than just the few calls that get made in my local environment.

    Next Steps?

    If this were an active project, I would probably have made a little more effort to do things the way they were originally structured and “play in the sandbox.” However, with no contribution rules or guidelines and no visible build pipeline, I am on my own with this one.

    If this turns out to be reasonably useful, I will probably take a look at the rest of the projects in that repository. I may also tap my contacts at Microsoft for some potential “next steps” with this one: while I do not relish the thought of owning a public archive, perhaps I could at least get a copy of their build pipeline so that I can replicate it on my end. Maybe a GitHub Actions pipeline with a push to Nuget?? Who knows.

  • Aging into fun

    It is obvious that my posts have been, well, tech-centric for the better part of the last year. I do my best to not post specifics about my kids because, well, their life is theirs to live, not mine to post. But I have to credit them with introducing me to an activity that keeps me active and lets me have a little fun: soccer.

    Fitness was an issue…

    My introduction to soccer was, as I recall, 11v11, full field games somewhere between ages 8 and 11. Genetics dealt me a Mac truck, not a Ferrari, and so chasing a ball and a bunch of Ferraris around a field just was not going to appeal to me. I ended up playing baseball until high school, middle school football, and high school volleyball, but never returned to soccer.

    Side Note: Based on my size alone, I get comments pretty much weekly as to if/when I played football. I probably could have been good, but my passive personality did not lend itself well to that sport. But, as I tell my brother, I have no chronic knee, back, shoulder pain from a contact sport…. So I have no regrets.

    Having a kid in your early 20’s is a stressful affair, and that stress didn’t exactly help my weight problems. With two kids, and approaching thirty well north of three hundred pounds, I decided I should make some changes if I want to see my kids grow up. I started running and watching what I eat. I ended up losing about seventy pounds, and felt pretty good.

    Becoming a soccer player

    My oldest played youth soccer from U6 through U18. I spent a lot of time at the soccer fields, and have fond memories of even coaching a few U8 seasons with him. As I was introduced to that community, I found that there were some co-ed leagues in our area. The first one I joined was a 7v7 league which was played on either a U10 or a U12 field (I do not remember the exact size).

    I had a lot of fun in that league, and was introduced to the game in a way where I was not required to run full-field sprints to recover. I learned the basics of the game, and became a serviceable field player.

    Graduating to full field

    At some point, I thought it might be fun to play in the Over 30 adult league. This league was a little more formal, with actual home and away games, real kits (or at least a nice jersey), and a full referee team.

    Did I run more than I ever had in my life? Yes. Was it fun? Absolutely. In spite of some of the attitude issues (we aren’t playing for the World Cup, but you’d think we were), I had a great time with it. My personal and professional life was getting busy, and that, coupled with the attitude issues, led me to walk away from that team for a while.

    When opportunity knocks..

    A lot happened in the next few years. Got divorced, then COVID, then I got married again (well, married during COVID, as if we needed a bigger challenge), all the while working through a few different roles at work. I never actively looked for a team: there was enough to do without introducing another activity. A chance encounter with an old board member from the club changed that.

    I came to find out that the community club my kids play started fielding adult teams a few years ago. As of now, we have an an over 30 team and an over 40 team. So I joined on, officially on the over 40 team. Sure, I can fill in on the over 30 team, and I have a few times. But that only made me realize how painfully out of shape I’ve become. Over 40 is a little more my speed. Still 11v11, full field, but 20 minute quarters instead of 45 minute halves. More breaks… I love it.

    What are you getting at?

    This post turned into something of “Matt’s life sport story.” The point is this: you are never too old to try something new. Soccer is a great way to stay active, and gives me just another reason to stay fit and healthy. And this new “adventurous” spirit has led me to try other new things, including scuba diving and skiing. Scuba I love, and my comfort in the water gives me a jump on a lot. Skiing…. Well, I had a lesson, and let’s say I can do it, but I need a lot more practice.

    So, try something new. You never know what will happen.

  • Maturing my Grafana setup

    I may have lost some dashboards and configuration recently, and it got me thinking about how to mature my Grafana setup for better persistence.

    Initial Setup

    When I first got Grafana running, it was based on the packaged Grafana Helm chart. As such, my Grafana instance was using SQLite database file stored in the persistent volume. This limits me to a single Grafana pod, since the volume is not setup to shared across pods. Additionally, that SQL database file is to the lifecycle of the claim associated with the volume.

    And, well, at home, this is not a huge deal because of how the lab is setup for persistent volume claims. Since I use the nfs-subdir-external-provisioner, PVCs in my clusters automatically generate a subfolder in my NFS share. When the PVC is deleted, the subdir gets renamed with an archive- prefix, so I can usually dig through the folder to find the old database file.

    However, using the default Azure persistence, Azure Disks are provisioned. When a PVC gets deleted, so to does the disk, or, well, I think it does. I have not had the opportunity to dig in to the Azure Disk PVC provisioning to understand how that data is handled when PVCs go away. It is sufficient to say that I lost our Grafana settings because of this.

    The New Setup

    The new plan is to utilize MySQL to store my Grafana dashboards and data stores. The configuration seems simple enough: add the appropriate entries in the grafana.ini file. I already know how to expand secrets, so getting the database secrets into the configuration was easy using the grafana.ini section of the Helm chart.

    For my home setup, I felt it was ok to run MySQL as another dependent chart for Grafana. Now, from the outside, you should be saying “But Matt, that only moves your persistence issues from the Grafana chart to the MySQL chart!” That is absolutely true. But, well, I have a pretty solid backup plan for those NFS shares, so for a home lab that should be fine. Plus I figured out how to backup and restore Grafana (see below).

    The real reason is that, for the instance I am running in Azure at work, I want to provision an Azure MySQL instance. This will allow me to have much better backup retention that inside the cluster, but the configuration at work will match the configuration at home. Home lab in action!

    Want to check out my home lab configuration? Check out my ops-internal infrastructure repository.

    Backup and Restore for Grafana

    As part of this move, I did not want to lose the settings I had in Grafana. This mean finding a backup/restore procedure that worked. An internet search lead me to the Grafana Backup Tool. The tool provides backup and restore capabilities through Grafana’s APIs.

    That said, it is written in Python, so my recent foray into Python coding served me well to get this tool up and running. Once I generated an API Key, I was off and running.

    There really isn’t much to it: after configuring the URL and API Token, I ran a backup to get a .tar.gz file with my Grafana contents. Did I test the backup? No. It’s the home lab, worst that could happen is I have to re-import some dashboards and re-create some others.

    After that, I updated my Grafana instance to include the MySQL instance and updated Grafana’s configuration to use the new MySQL service. As expected, all my dashboards and data sources disappeared.

    I ran the restore function using my backup, refreshed Grafana in my browser, and I was back up and running! Testing, schmesting….

    What’s Next?

    I am going to take my newfound learnings and apply them at work:

    1. Get a new MySQL instance provisioned.
    2. Backup Grafana.
    3. Re-configure Grafana to use the new MySQL instance.
    4. Restore Grafana.

    Given the ease with which the home lab went, I cannot imagine I will run into much issue.

  • Why I am not using Resharper anymore

    I have been a subscriber to JetBrains Resharper for a number of years. Additionally, my employer carries a number of licenses for Resharper, and some teams still use it. But I recently decided to cancel my subscription, and it might be worth a few words as to why.

    IDEs are uniquely personal

    To developers, IDEs are kind of like jeans: you shop around for a brand you like, then buy every different style of that brand to have a little variety in a known quantity. When I started my development journey, IDEs were tied to the framework you were using: Visual Studio for Microsoft stacks like VB6, VC++, and the .Net Framework, Eclipse for Java, etc. These tools have branched out, and new IDEs have come around for good multi-language support. Techrepublic has a good overview of 12 strong players in the space.

    The point is, once you find one you like, you stick with it. Personally, I have always been a fan of Visual Studio and it’s baby brother, Visual Studio Code. The former is my go to when I need a full blown IDE, the latter is great for my “text-only” work, like Powershell scripting, Hashicorp Terraform-ing, and even Python development when I must.

    Adding Resharper

    I was first introduced to Resharper by a company I consulted. They used it heavily, and, at the time, it had some features that I got used to, so much so that I bought a subscription.

    • Ctrl-T search: Resharper’s search functionality made it quick and easy to find classes and enums using Ctrl-T. Sure, a Ctrl-Shift-F search was similar, but the pop-up of Resharper made it less disruptive.
    • Refactoring: Resharper’s refactoring capability was head and shoulders above Visual Studio. The refactor preview was helpful in understanding what all you were changing before you did it. This refactoring is probably the feature I would miss the most if I were still working through legacy code. More on that later.
    • Code styling and standards: Resharper added a lot of functionality to standardizing code styling, including several rules.

    Why stop using Resharper?

    Yes, Resharper added a lot of functionality. My choice to stop using it boils down to a combination of a change in needs on my end and the industry playing catch up.

    My own changing needs

    As a software developer digging in to legacy production code, I spent a lot of time being careful not to break anything. Resharper’s careful refactoring made that easier. Additionally, when Resharper’s styling rules could be enforced through group tools like Sonarqube, it made sense to align on a ruleset for the team.

    As I migrated to a higher level technical role, my coding turned into less production and more proof of concept. With that, I need to move more quickly, and I found that Resharper was quite literally slowing me down. There have always been complaints about Resharper affecting Visual Studio speed, and I definitely notice an improvement when Resharper is not installed.

    The Industry is catching up

    Microsoft has not sat idly by. Visual Studio continues to improve its own search and refactor capabilities. Roslyn allows for code styling and standards to be embedded in your project and checked as part of compilation. So? Well, embedding styling checks in the compiler means we can fail builds when styling checks fail. No more external tools for enforcing code styling.

    Additionally, tools like Sonarlint have extended Roslyn to add additional checks. So we can use third party tools to extend our standards without adding much to the build process.

    Resharper in a professional environment

    I have mixed feelings on using Resharper in a professional environment. I can see the benefits that it provides to individual developers with regard to refactoring, search, and styling. However, its somewhat more difficult to enforce styling across the team, and most code quality tools (like Sonarqube) do not have direct integrations with Resharper.

    Resharper provides a new set of CLI tools for executing code analysis and inspections in build pipelines, and it looks like they are trying to bridge the gap. But the relative simplicity of setting up Sonarlint and Sonarqube together allow for tracking of overall code quality across multiple projects, something that I simply cannot see with Resharper today.

    The Experiment

    Before deciding on whether or not to renew my subscription, I ran an experiment: I disabled Resharper in Visual Studio. I figured I would pretty quickly find where I was missing Resharper’s functionality and be able to determine whether or not it was worth the money to continue my subscription.

    To my great surprise, in a 3 month period, not once did I run into a situation where I tried to use a Resharper function that I was missing. My day-to-day coding was unaffected by Resharper being disabled. I would have expected to hit Ctrl-T once and wonder why it nothing came up. It would seem my muscle memory for Resharper was not as strong as I though it was. With that, I cancelled my subscription.

    Perhaps I’ll use that extra $100 a year on something new and interesting…

  • Tech Tip – Configuring RKE2 Nginx Ingress using a HelmChartConfig Resource

    The RKE2 documentation is there, but, well, it is not quite as detailed as I have seen in other areas. This is a quick tip for customizing your Nginx Ingress controllers when using RKE2

    Using Nginx Ingress in RKE2

    By default, an RKE2 cluster deploys the nginx-ingress Helm chart. That’s great, except that you may need to customize that chart. This is where the HelmChartConfig resource is used.

    RKE2 uses HelmChartConfig custom resource definitions (CRDs) to allow you to set configuration options for their default Helm deployments. This is pretty useful, and seemed straightforward, except I had a hard time figuring out HOW to set the options.

    Always easier than I expect

    The RKE2 documentation points you to the nginx-ingress chart, but it took me a bit to realize that the connection was as simple as setting the valuesContent value in the HelmChartConfig spec to whatever values I wanted to pass in to Nginx. 

    apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      config:
        use-forwarded-headers: "true"
        proxy-buffer-size: "256k"
        proxy-buffer-number: "4"
        large-client-header-buffers: "4 16k"
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true
          additionalLabels:
            cluster: nonproduction

    The above sets some configuration values in the controller AND enables metrics collection using the ServiceMonitor object. For Nginx, valid values for valuesContent are the same as values in the chart’s values.yaml file.

    Works with other charts

    RKE2 provides additional charts that can be deployed and customized with similar methods. There are charts which are deployed by default, and they provide instructions on disabling them. However, the same HelmChartConfig method above can be used to customize the chart installs as well.

  • Literally every time I go away…

    It really truly seems like every time I go away, something funny happens in the lab and stuff goes down. That pattern continued last week.

    Loads of Travel

    A previously planned Jamaican getaway butted up against a short-notice business meeting in Austin. Since I could not get a direct flight to either location, I ended up on 8 flights in 8 days.

    The first four were, for lack of a more descriptive word, amazing! My wife and I took a short trip to Couples Swept Away in Negril for a short holiday. While I knew it was an all-inclusive, I did not realize that included scuba diving. I took advantage of that and dove five times in three days. Thankfully, I was always back by noon for a few much needed beach naps.

    The Saturday that we were gone, I got a few text messages from myself indicating that my websites were down. Since I got the messages, well, that meant I had power to the Raspberry Pi that runs the monitoring software and my internet was up. I tried logging in, but nothing that was attached to the NAS was coming up… So the NAS was down.

    A quick troubleshooting session

    Since there was not much I could do from the beach, I put it away until I got home Tuesday night. Now, mind you, I had a flight out early Wednesday morning for Austin, but I really could not sleep until I figured out what was wrong.

    As it turns out, we must have had a power brownout/blackout. The NAS went down and did not reboot. I ended up having to unplug it to force a restart.

    That was, well, the extent of my troubleshooting: once the NAS came back up, Kubernetes took over and made sure everything was running again.

    Move to the cloud?

    My home lab is just a lab… When stuff goes down, well, no one should care but me. And that is true for everything except this site. I am torn between spending some money to host this site externally, or fortifying my current setup with some additional redundant systems.

    What kinds of fortifications, you might ask… Well, a proper UPS would be nice, avoiding the hard shutdowns that could wreck my systems. But, there’s no way I am going to pay for redundant internet to my house, so I will always be tied to that.

    For now, I am ok with hosting as I am today and having to live with some downtime. March was a rough month with a few days of outage, but I generally hit 98-99% uptime, based on the data from my StatusPage site. For a home lab, I would say that is pretty good.