Category: Software

  • Maturing my Grafana setup

    I may have lost some dashboards and configuration recently, and it got me thinking about how to mature my Grafana setup for better persistence.

    Initial Setup

    When I first got Grafana running, it was based on the packaged Grafana Helm chart. As such, my Grafana instance was using SQLite database file stored in the persistent volume. This limits me to a single Grafana pod, since the volume is not setup to shared across pods. Additionally, that SQL database file is to the lifecycle of the claim associated with the volume.

    And, well, at home, this is not a huge deal because of how the lab is setup for persistent volume claims. Since I use the nfs-subdir-external-provisioner, PVCs in my clusters automatically generate a subfolder in my NFS share. When the PVC is deleted, the subdir gets renamed with an archive- prefix, so I can usually dig through the folder to find the old database file.

    However, using the default Azure persistence, Azure Disks are provisioned. When a PVC gets deleted, so to does the disk, or, well, I think it does. I have not had the opportunity to dig in to the Azure Disk PVC provisioning to understand how that data is handled when PVCs go away. It is sufficient to say that I lost our Grafana settings because of this.

    The New Setup

    The new plan is to utilize MySQL to store my Grafana dashboards and data stores. The configuration seems simple enough: add the appropriate entries in the grafana.ini file. I already know how to expand secrets, so getting the database secrets into the configuration was easy using the grafana.ini section of the Helm chart.

    For my home setup, I felt it was ok to run MySQL as another dependent chart for Grafana. Now, from the outside, you should be saying “But Matt, that only moves your persistence issues from the Grafana chart to the MySQL chart!” That is absolutely true. But, well, I have a pretty solid backup plan for those NFS shares, so for a home lab that should be fine. Plus I figured out how to backup and restore Grafana (see below).

    The real reason is that, for the instance I am running in Azure at work, I want to provision an Azure MySQL instance. This will allow me to have much better backup retention that inside the cluster, but the configuration at work will match the configuration at home. Home lab in action!

    Want to check out my home lab configuration? Check out my ops-internal infrastructure repository.

    Backup and Restore for Grafana

    As part of this move, I did not want to lose the settings I had in Grafana. This mean finding a backup/restore procedure that worked. An internet search lead me to the Grafana Backup Tool. The tool provides backup and restore capabilities through Grafana’s APIs.

    That said, it is written in Python, so my recent foray into Python coding served me well to get this tool up and running. Once I generated an API Key, I was off and running.

    There really isn’t much to it: after configuring the URL and API Token, I ran a backup to get a .tar.gz file with my Grafana contents. Did I test the backup? No. It’s the home lab, worst that could happen is I have to re-import some dashboards and re-create some others.

    After that, I updated my Grafana instance to include the MySQL instance and updated Grafana’s configuration to use the new MySQL service. As expected, all my dashboards and data sources disappeared.

    I ran the restore function using my backup, refreshed Grafana in my browser, and I was back up and running! Testing, schmesting….

    What’s Next?

    I am going to take my newfound learnings and apply them at work:

    1. Get a new MySQL instance provisioned.
    2. Backup Grafana.
    3. Re-configure Grafana to use the new MySQL instance.
    4. Restore Grafana.

    Given the ease with which the home lab went, I cannot imagine I will run into much issue.

  • Why I am not using Resharper anymore

    I have been a subscriber to JetBrains Resharper for a number of years. Additionally, my employer carries a number of licenses for Resharper, and some teams still use it. But I recently decided to cancel my subscription, and it might be worth a few words as to why.

    IDEs are uniquely personal

    To developers, IDEs are kind of like jeans: you shop around for a brand you like, then buy every different style of that brand to have a little variety in a known quantity. When I started my development journey, IDEs were tied to the framework you were using: Visual Studio for Microsoft stacks like VB6, VC++, and the .Net Framework, Eclipse for Java, etc. These tools have branched out, and new IDEs have come around for good multi-language support. Techrepublic has a good overview of 12 strong players in the space.

    The point is, once you find one you like, you stick with it. Personally, I have always been a fan of Visual Studio and it’s baby brother, Visual Studio Code. The former is my go to when I need a full blown IDE, the latter is great for my “text-only” work, like Powershell scripting, Hashicorp Terraform-ing, and even Python development when I must.

    Adding Resharper

    I was first introduced to Resharper by a company I consulted. They used it heavily, and, at the time, it had some features that I got used to, so much so that I bought a subscription.

    • Ctrl-T search: Resharper’s search functionality made it quick and easy to find classes and enums using Ctrl-T. Sure, a Ctrl-Shift-F search was similar, but the pop-up of Resharper made it less disruptive.
    • Refactoring: Resharper’s refactoring capability was head and shoulders above Visual Studio. The refactor preview was helpful in understanding what all you were changing before you did it. This refactoring is probably the feature I would miss the most if I were still working through legacy code. More on that later.
    • Code styling and standards: Resharper added a lot of functionality to standardizing code styling, including several rules.

    Why stop using Resharper?

    Yes, Resharper added a lot of functionality. My choice to stop using it boils down to a combination of a change in needs on my end and the industry playing catch up.

    My own changing needs

    As a software developer digging in to legacy production code, I spent a lot of time being careful not to break anything. Resharper’s careful refactoring made that easier. Additionally, when Resharper’s styling rules could be enforced through group tools like Sonarqube, it made sense to align on a ruleset for the team.

    As I migrated to a higher level technical role, my coding turned into less production and more proof of concept. With that, I need to move more quickly, and I found that Resharper was quite literally slowing me down. There have always been complaints about Resharper affecting Visual Studio speed, and I definitely notice an improvement when Resharper is not installed.

    The Industry is catching up

    Microsoft has not sat idly by. Visual Studio continues to improve its own search and refactor capabilities. Roslyn allows for code styling and standards to be embedded in your project and checked as part of compilation. So? Well, embedding styling checks in the compiler means we can fail builds when styling checks fail. No more external tools for enforcing code styling.

    Additionally, tools like Sonarlint have extended Roslyn to add additional checks. So we can use third party tools to extend our standards without adding much to the build process.

    Resharper in a professional environment

    I have mixed feelings on using Resharper in a professional environment. I can see the benefits that it provides to individual developers with regard to refactoring, search, and styling. However, its somewhat more difficult to enforce styling across the team, and most code quality tools (like Sonarqube) do not have direct integrations with Resharper.

    Resharper provides a new set of CLI tools for executing code analysis and inspections in build pipelines, and it looks like they are trying to bridge the gap. But the relative simplicity of setting up Sonarlint and Sonarqube together allow for tracking of overall code quality across multiple projects, something that I simply cannot see with Resharper today.

    The Experiment

    Before deciding on whether or not to renew my subscription, I ran an experiment: I disabled Resharper in Visual Studio. I figured I would pretty quickly find where I was missing Resharper’s functionality and be able to determine whether or not it was worth the money to continue my subscription.

    To my great surprise, in a 3 month period, not once did I run into a situation where I tried to use a Resharper function that I was missing. My day-to-day coding was unaffected by Resharper being disabled. I would have expected to hit Ctrl-T once and wonder why it nothing came up. It would seem my muscle memory for Resharper was not as strong as I though it was. With that, I cancelled my subscription.

    Perhaps I’ll use that extra $100 a year on something new and interesting…

  • Tech Tip – Configuring RKE2 Nginx Ingress using a HelmChartConfig Resource

    The RKE2 documentation is there, but, well, it is not quite as detailed as I have seen in other areas. This is a quick tip for customizing your Nginx Ingress controllers when using RKE2

    Using Nginx Ingress in RKE2

    By default, an RKE2 cluster deploys the nginx-ingress Helm chart. That’s great, except that you may need to customize that chart. This is where the HelmChartConfig resource is used.

    RKE2 uses HelmChartConfig custom resource definitions (CRDs) to allow you to set configuration options for their default Helm deployments. This is pretty useful, and seemed straightforward, except I had a hard time figuring out HOW to set the options.

    Always easier than I expect

    The RKE2 documentation points you to the nginx-ingress chart, but it took me a bit to realize that the connection was as simple as setting the valuesContent value in the HelmChartConfig spec to whatever values I wanted to pass in to Nginx.

    apiVersion: helm.cattle.io/v1
    kind: HelmChartConfig
    metadata:
      name: rke2-ingress-nginx
      namespace: kube-system
    spec:
      valuesContent: |-
        controller:
          config:
            use-forwarded-headers: "true"
            proxy-buffer-size: "256k"
            proxy-buffer-number: "4"
            large-client-header-buffers: "4 16k"
          metrics:
            enabled: true
            serviceMonitor:
              enabled: true
              additionalLabels:
                cluster: nonproduction

    The above sets some configuration values in the controller AND enables metrics collection using the ServiceMonitor object. For Nginx, valid values for valuesContent are the same as values in the chart’s values.yaml file.

    Works with other charts

    RKE2 provides additional charts that can be deployed and customized with similar methods. There are charts which are deployed by default, and they provide instructions on disabling them. However, the same HelmChartConfig method above can be used to customize the chart installs as well.

  • Snakes… Why Did It Have To Be Snakes?

    What seems like ages ago, I wrote some Python scripts to keep an eye on my home lab. What I did not realize is that little introduction to Python would help me dive into the wonderful world of ETL (or ELT, more on that later).

    Manage By Numbers

    I love numbers. Pick your favorite personality profile, and I come out as the cold, calculated, patient person who needs all the information to make a decision. As I ramp up and improve my management skills after a brief hiatus as an individual contributor, I identified a few blinds spots that I wanted to address with my team. But first, I need the data, and that data currently lives in our Jira Cloud instance and an add-on called Tempo Timesheets.

    Now, our data teams have started to build out an internal data warehouse for our various departments to collect and analyze data from our various sales systems. They established ELT flow for this warehouse with the following toolsets:

    • Singer.io – Establish data extraction and loading
    • dbt – Define data transformations
    • Prefect – Used to orchestrate flows via Singer Taps & Targets and dbtCore transformations.

    Don’t you mean ETL?

    There are two primary data integration methods:

    • ETL – Extract, Transform, and Load
    • ELT – Extract, Load, Transform

    At their core, they have the same basic task: get data from one place to another. The difference lies in where data transformations are processed.

    In ETL, data is extracted from the source system, transformed, and then loaded into the destination system (data warehouse) where it can be analyzed. In this case, the raw data is not stored within the data warehouse: only the transformed data is available.

    ELT, on the other hand, loads raw data directly into the data warehouse, where transformations can be exercised within the data warehouse itself. Since the raw data is stored in the warehouse, multiple transformations can be run without accessing the source system. This allows for data activities such as cleansing, enrichment, and transformation to occur on the same data set with less strain on the source system.

    In our case, an ELT transition made the most sense: we will have different transformations for different departments, including the need to perform point-in-time transformations for auditing purposes.

    Getting Jira Data

    Jira data was, well, the easier part. Singer.io maintains a Jira tap to pull data our of Jira Cloud using Jira’s APIs. “Taps” are connectors to external systems, and Singer has a lot of them. “Targets” are ways to load the data from tap streams into other systems. Singer does not have as many official targets, however, there are a number of open source contributors with additional targets. We are using the Snowflake target to load data into a Snowflake instance.

    Our team built out the data flow, but we were missing Tempo data. Tempo presents some REST APIs, so I figured I could use the tap-jira code as a model to build out a custom tap for Tempo data. And that got me back into Python.

    Environment Setup

    I’m running WSL2 on Windows 11 with Ubuntu 22.04. I finished up my tap development, things seemed to be working fine using virtualenv to isolate, and I had finished testing on my Tempo tap. I wanted to test pushing the data into Snowflake, and upon trying to load the target-snowflake library, I got a number of errors about version incompatibility.

    Well hell. As it turns out, most of the engineers at work are using Windows 10/WSL2 with Ubuntu 20.04. With that, they are running Python 3.8. I was running 3.10. A few quick Google searches and I quickly realized that I need a better way to isolate my environments than virtualenv. Along comes a bigger snake…

    Anaconda

    My Google searches led me to Anaconda. First, I’m extremely impressed that they got that domain name. Second, Anaconda is way more than what I’m using it for: I’m using the environment management, but there is so much more.

    I installed Anaconda according to the Linuxhint.com guide and, within about 10 minutes, I had a virtual environment with Python 3.8 that I could use to build and test my taps and targets. The environment management is, in my mind, much easier than using virtualenv: the conda command can be used to list environments and search for available packages, rather than remembering where you stored your virtual environment files and how you activate them.

    Not changing careers yet

    Sure, I wrote a tap for Tempo data. Am I going to change over to a data architect? Probably not. But at least I can say I succeeded in simple tap development.

    A Note on Open Source

    I’m a HUGE fan of open source and look to contribute where I can. However, while this tap “works,” it’s definitely not ready for the world. I am only extracting a few of the many objects available in the Tempo APIs. I have no tests to ensure correctness. And, most importantly, I have no documentation to guide users.

    Until those things happen, this tap will be locked up behind closed doors. When I find the time to complete it, I will make it public.

  • My 15 pieces of flair… Cloudflare

    With parts of my home lab exposed to the internet for my own convenience, it is always good to add layers of protection to incoming traffic. At a colleague’s suggestion, I took Cloudflare up on their free WAF offering to help add some protection to my setup. As a bonus, I have a much better DNS for my domains, which made automating my SSL certificate renewals a snap.

    What’s a WAF?

    A Web Application Firewall, or WAF, protects your web applications by processing requests and monitoring for attacks. While not a comprehensive solution, it adds a layer of protection.

    Cloudflare offers cloud-based services which have a variety of price points. For my hobby/home lab sites, well, free as in beer is the best price point for me. Now, you may notice on the price sheet, that “WAF” is not actually included in the free version. The free edition does not let you define custom firewall rules and block, challenge, or log requests that match those rules.

    Well, that’s not 100% accurate: you get 5 active firewall rules with the free plan. Not enough to go crazy, but enough to test if you need it.

    And, well, I do not particularly care: For my home lab, the features of most interest to me are the DNS, Caching, DDoS Protection, and the Managed Ruleset.

    Basic Content Caching

    Cloudflare provides some basic caching on my proxied sites, which definitely helps with sites like WordPress. My PageSpeed insights scores are almost 100 ms faster on mobile devices (down from 310 ms), which is pretty good. While I have never paid too much attention to page load speeds, it is good to know that I can improve some things while adding a layer of protection

    DDoS and Managed Rulesets

    Truthfully, I have not read up on much of this, and have left the Cloudflare defaults pretty much intact. Cloudflare’s blog does a good job of explaining the Managed Rules, and their documentation covers the DDoS rulesets.

    Perhaps if I get bored, or need something to put me to sleep at night, I will start reading up on those rulesets. For now, they are in place, which gives me a little more protection than I had without them.

    Cloudflare DNS

    Truthfully, if Cloudflare did nothing else than manage my DNS in a way that allowed certbot to automatically renew my Let’s Encrypt certificates, I would have still moved everything over. Prior to the cutover, I was using GoDaddy’s DNS management and, well, it’s a pain. GoDaddy is very good at selling websites, but DNS management is clearly very low on their list. Cloudflare’s DNS, meanwhile, is simple to manage both through their portal and through the APIs.

    With my DNS moved over, I revisited the certificates on my internal reverse proxy. Following the instructions from Vineet Choudhary over at developerinsider.co, I updated certbot to renew using the Cloudflare plugin.

    Automagic Renewals?

    In the past, with certbot-auto, you would have to schedule a cron job to schedule automatic renewals. The new certbot snap, however, uses systemctl and timers to achieve the same. So, with my certificates renewed using the correct plugin, I ran a quick test:

    sudo certbot renew --dry-run

    The dry run succeeded without issue. So I checked the timers with the following command:

    systemctl list-timers

    Lo and behold, the certbot time is scheduled to run in the middle of the night.

    Restarting Nginx on Certbot renewals

    There is one small issue: even though I am using the --cert-only option to only get or renew certificates and not edit Nginx, I AM using Nginx as a reverse proxy. Therefore, I need a way to restart Nginx after certbot has done its thing. I found this short article and followed the instructions to edit the /etc/letsencrpyt/cli.ini file with a deploy hook.

    The article above noted that, to test, you can run the following:

    certbot renew --dry-run

    However, for me, this did NOT trigger the deploy hook. To force triggering the deploy hook, I needed to run this command:

    sudo certbot renew --dry-run --run-deploy-hooks

    This command executed the renewal dry run and successfully reloaded Nginx.

    Minimal Pieces of Cloudflare

    Sure, I have only scratched the surface of Cloudflare’s offerings by adding some free websites and proxying some content. But, as I mentioned, it adds a layer of protection that I did not have before. And, in this day and age, the wire coming into the house presents a bigger security threat than the front door.

  • Badges… We don’t need no stinkin’ badges!

    Well… Maybe we do. This is a quick plug (no reimbursement of any kind) for the folks over at Shields.io, who make creating custom badges for readme files and websites an easy and fun task.

    A Quick Demo

    License for spyder007/MMM-PrometheusAlerts
    Build Status for spyder007/MMM-PrometheusAlerts

    The badges above are generated from Shields.io. The first link looks like this:

    https://img.shields.io/github/license/spyder007/MMM-PrometheusAlerts

    My Github username (spyder007) and the repository name (MMM-PrometheusAlerts) are used in the Image URL, which generates the badge. The second one, build status, looks like this:

    https://img.shields.io/github/actions/workflow/status/spyder007/MMM-PrometheusAlerts/node.js.yml

    In this case, my Github username and the repository name remain the same, but node.js.yml is the name of the workflow file for which I want to display the status.

    Every badge in Shields.io has a “builder” page that explains how to build the image and even allows you to override styles, colors, and labels, and even add logos from any icon in the Simple Icons collection.

    Some examples of alterations to my build status above:

    “For the Badge” style, Bugatti Logo with custom color
    Flat style, CircleCI logo, Custom label

    Too many options to list…

    Now, these are live badges, meaning, if my build fails, the green “passing” will go to a red “failing.” Shields.io does this by using the variety of APIs available to gather data about builds, code coverage, licenses, chat, sizes and download counts, funding, issue tracking… It’s a lot. But the beauty of it is, you can create Readme files or websites which have easy to read visuals. My PI Monitoring repository‘s Readme makes use of a number of these shields to give you a quick look at the status of the repo.

  • Building Software Longevity

    The “Ship of Theseus” thought experiment is an interesting way to start fights with historians, but in software, replacing old parts with new parts is required for building software longevity. Designing software in ways that every piece can be replaced is vital to building software for the future.

    The Question

    The Wikipedia article presents the experiment as follows:

    According to legend, Theseus, the mythical Greek founder-king of Athens, rescued the children of Athens from King Minos after slaying the minotaur and then escaped onto a ship going to Delos. Each year, the Athenians commemorated this legend by taking the ship on a pilgrimage to Delos to honor Apollo. A question was raised by ancient philosophers: After several centuries of maintenance, if each individual part of the Ship of Theseus was replaced, one at a time, was it still the same ship?

    Ship of Theseus – Wikipedia

    The Software Equivalent

    Consider Microsoft Word. Released in 1983, Word is approaching its 40th anniversary. And, while I do not have access into its internal workings, I am willing to bet that most, if not all of the 1983 code has since been replaced by updated modules. So, while it is still called Word, its parts are much newer than the original 1983 iteration. I am sure if I sat here long enough, I could identify other applications with similar histories. The branding does not change, the core functionality does not change, only the code changes.

    Like the wood on the Ship of Theseus, software rots. And it rots fast. Frameworks and languages evolve quickly to take advantage of hardware updates, and the software that uses those must do the same.

    Design for Replacement

    We use the term technical debt to categorize the conscious choices we make to prioritize time to market over perfect code. It is worthwhile to consider software has a “half life” or “depreciation factor” as well: while your code may work today, chances are, without appropriate maintenance, it will rot into something that is no longer able to serve the needs of your customers.

    If I had a “one sized fits all” solution to software rot, I would probably be a millionaire. The truth is, product managers, engineers, and architects must be aware of software rot. Not only must we invest the time into fixing the rot, but we must design our products to allow every piece of the application to be replaced, even as the software still stands.

    This is especially critical in Software as a Service (SaaS) offerings, where we do not have the luxury of large downtimes for upgrades or replacements. To our customers, we should operate continuously, with upgrades happening almost invisibly. This requires the foresight to build replaceable components and the commitment to fixing and replacing components regularly. If you cannot replace components of your software, there will come a day where your software will no longer function for your customers.

  • D-N-S Ja!

    With all this talk of home lab cluster provisioning, you might be wondering if I am actually doing any software development at home. As a matter of fact, I am. Just because it is in support of my home lab provisioning does not mean it is not software development!

    Keeping the Lab Tidy

    One of the things that has bothered me in my home lab management is the DNS management. As I provision and remove Linux VMs, having appropriate DNS records for them makes it easy to find them. Generally it makes for a more tidy environment, as I have a list of my machines and their IPs in one place. I have a small Powershell module that uses the DnsServer module in Windows. What I wanted was an API that would allow me to manage my DNS.

    Now, taking a cue from my Hyper-V wrapper, I created a small API that uses the DnsServer module to manage DNS entries. It was fairly easy, and works quite well on my own machine, which has the DnsServer module installed because I have the Remote Server Administrative Toolset installed.

    Location, Location, Location

    When I started looking at where I could host this service, I realized that I could not host it on my hypervisor as I did with the Hyper-V service. My server is running Windows Server 2019 Hyper-V edition, which is a stripped down version of Windows Server meant for hypervisors. That means I am unable to install the DNS Server role on it. Admittedly, I did not try installing RSAT on it, but I have tendency to believe that would not work.

    Since the DnsServer module would be installed by default on my domain controller, I made the decision to host the DNS API on that server. I went about creating an appropriate service account and installed it as a service. Just like the Hyper-V API, the Windows DNS API is available on Github.

    Return to API Management

    At this point, I have API hosted on a few different machines plus the APIs hosted in my home lab clusters. This has forced me to revisit installing an API Management solution at home. Sure, no one else uses my lab, but that is not the point. Right now, I have a “service discovery” problem: where are my APIs, how do I call them, what is their authentication mechanism, etc. This is part of what API Management can solve: I can have a single place to locate and call my APIs. Over the next few weeks I may delve back into Gravitee.io in an effort to re-establish a proper API Management service.

    Going Public, Going Github

    While it may seem like I am “burying the headline,” I am going to start making an effort to go public with more of my code. Why? Well, I have a number of different repositories that might be of use to some folks, even as reference. Plus, well, it keeps me honest: Going public with my code means I have to be good about my own security practices. Look for posts on migration updates as I get to them.

    Going public will most likely mean going Github. Yes, I have some public repositories out in Bitbucket, but Github provides a bit more community and visibility for my work. I am sure I will still keep some repositories in Bitbucket, but for the projects that I want public feedback on, I will shift to Github.

    Pop Culture Reference Section

    The title is a callout to Pitch Perfect 2. You are welcome.

  • Installing Minio on a Synology Diskstation with Nginx SSL

    In an effort to get rid of a virtual machine on my hypervisor, I wanted to move my Minio instance to my Synology. Keeping the storage interface close to the storage container helps with latency and is, well, one less thing I have to worry about in my home lab.

    There are a few guides out there for installing Minio on a Synology. Jaroensak Yodkantha walks you through the full process of setting up the Synology and Minio using a docker command line. The folks over at BackupAssist show you how to configure Minio through the Diskstation Manager web portal. I used the BackupAssist article to get myself started, but found myself tweaking the setup because I want to have SSL communication available through my Nginx reverse proxy.

    The Basics

    Prep Work

    I went in to the Shared Folder section of the DSM control panel and created a new shared folder called minio. The settings on this share are pretty much up to you, but I did this so that all of my Minio data was in a known location.

    Within the minio folder, I created a data folder and a blank text file called minio. Inside the minio file, I setup my minio configuration:

    # MINIO_ROOT_USER and MINIO_ROOT_PASSWORD sets the root account for the MinIO server.
    # This user has unrestricted permissions to perform S3 and administrative API operations on any resource in the deployment.
    # Omit to use the default values 'minioadmin:minioadmin'.
    # MinIO recommends setting non-default values as a best practice, regardless of environment
    
    MINIO_ROOT_USER=myadmin
    MINIO_ROOT_PASSWORD=myadminpassword
    
    # MINIO_VOLUMES sets the storage volume or path to use for the MinIO server.
    
    MINIO_VOLUMES="/mnt/data"
    
    # MINIO_SERVER_URL sets the hostname of the local machine for use with the MinIO Server
    # MinIO assumes your network control plane can correctly resolve this hostname to the local machine
    
    # Uncomment the following line and replace the value with the correct hostname for the local machine.
    
    MINIO_SERVER_URL="https://s3.mattsdatacenter.net"
    MINIO_BROWSER_REDIRECT_URL="https://storage.mattsdatacenter.net"

    It is worth noting the URLs: I want to put this system behind my Nginx reverse proxy and let it do SSL termination, and in order to do that, I found it easiest to use two domains: one for the API and one for the Console. I will get into more details on that later.

    Also, as always, change your admin username and password!

    Setup the Container

    Following the BackupAssist article, I installed the Docker package on to my Synology and opened it up. From the Registry menu, I searched for minio and found the minio/minio image:

    Click on the row to highlight it, and click on the Download button. You will be prompted for the label to download, I chose latest. Once the image is downloaded (you can check the Image tab for progress), go to the Container tab and click Create. This will open the Create Wizard and get you started.

    • On the Image screen, select the minio/minio:latest image.
    • On the Network screen, select the bridge network that is defaulted. If you have a custom network configuration, you may have some work here.
    • On the General Settings screen, you can name the container whatever you like. I enabled the auto-restart option to keep it running. On this screen, click on the Advanced Settings button
      • In the Environment tab, change MINIO_CONFIG_ENV_FILE to /etc/config.env
      • In the Execution Command tab, change the execution command to minio server --console-address :9090
      • Click Save to close Advanced Settings
    • On the Port Settings screen, add the following mappings:
      • Local Port 39000 -> Container Port 9000 – Type TCP
      • Local Port 39090 -> Container Port 9090 – Type TCP
    • On the Volume Settings Screen, add the following mappings:
      • Click Add File, select the minio file created above, and set the mount path to /etc/config.env
      • Click Add Folder, select the data folder created above, and set the mount path to /mnt/data

    At that point, you can view the Summary and then create the container. Once the container starts, you can access your Minio instance at http://<synology_ip_or_hostname>:39090 and log in with the password saved in your config file.

    What Just Happened?

    The above steps should have worked to create a Docker container running on Synology on your Minio. Minio has two separate ports: one for the API, and one for the Console. Reviewing Minio’s documentation, adding the --console-address parameter in the container execution is required now, and that sets the container port for the console. In our case, we set it to 9090. The API port defaults to 9000.

    However, I wanted to run on non-standard ports, so I mapped ports 39090 and 39000 to port 9090 and 9000, respectively. That means that traffic coming in on 39090 and 39000 get routed to my Minio container on ports 9090 and 9000, respectively.

    Securing traffic with Nginx

    I like the ability to have SSL communication whenever possible, even if it is just within my home network. Most systems today default to expecting SSL, and sometimes it can be hard to find that switch to let them work with insecure connections.

    I was hoping to get the console and the API behind the same domain, but with SSL, that just isn’t in the cards. So, I chose s3.mattsdatacenter.net as the domain for the API, and storage.mattsdatacenter.net as the domain for the Console. No, those aren’t the real domain names.

    With that, I added the following sites to my Nginx configuration:

    storage.mattsdatacenter.net
      map $http_upgrade $connection_upgrade {
          default Upgrade;
          ''      close;
      }
    
      server {
          server_name storage.mattsdatacenter.net;
          client_max_body_size 0;
          ignore_invalid_headers off;
          location / {
              proxy_pass http://10.0.0.23:39090;
              proxy_set_header Host $host;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-proto $scheme;
              proxy_set_header X-Forwarded-port $server_port;
              proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
    
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection $connection_upgrade;
    
              proxy_http_version 1.1;
              proxy_read_timeout 900s;
              proxy_buffering off;
          }
    
        listen 443 ssl; # managed by Certbot
        allow 10.0.0.0/24;
        deny all;
    
        ssl_certificate /etc/letsencrypt/live/mattsdatacenter.net/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/mattsdatacenter.net/privkey.pem; # managed by Certbot
    }
    s3.mattsdatacenter.net
      map $http_upgrade $connection_upgrade {
          default Upgrade;
          ''      close;
      }
    
      server {
          server_name s3.mattsdatacenter.net;
          client_max_body_size 0;
          ignore_invalid_headers off;
          location / {
              proxy_pass http://10.0.0.23:39000;
              proxy_set_header Host $host;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-proto $scheme;
              proxy_set_header X-Forwarded-port $server_port;
              proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
    
              proxy_set_header Upgrade $http_upgrade;
              proxy_set_header Connection $connection_upgrade;
    
              proxy_http_version 1.1;
              proxy_read_timeout 900s;
              proxy_buffering off;
          }
    
        listen 443 ssl; # managed by Certbot
        allow 10.0.0.0/24;
        deny all;
    
        ssl_certificate /etc/letsencrypt/live/mattsdatacenter.net/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/mattsdatacenter.net/privkey.pem; # managed by Certbot
    }

    This configuration allows me to access the API and Console via domains using SSL terminated on the proxy. Configuring Minio is pretty easy: set MINIO_BROWSER_REDIRECT_URL to the URL of your console (In my case, port 39090), and MINIO_SERVER_URL to the URL of your API (port 39000).

    This configuration allows me to address Minio for S3 in two ways:

    1. Use https://s3.mattsdatacenter.net for secure connectivity through the reverse proxy.
    2. Use http://<synology_ip_or_hostname>:39000 for insecure connectivity directly to the instance.

    I have not had the opportunity to test the performance difference between option 1 and option 2, but it is nice to have both available. For now, I will most likely lean towards the SSL path until I notice degradation in connection quality or speed.

    And, with that, my Minio instance is now running on my Diskstation, which means less VMs to manage and backup on my hypervisor.

  • Using SonarCloud for Open Source

    My last few posts have centered around adding some code linting and analysis to C# projects. Most of this has been to identify some standards and best practices for my current position.

    During this research, I came across SonarCloud, which is Sonarqube’s hosted instance. SonarCloud is free for open source projects, and given the breadth of languages it supports, I have decided to start adding my open source projects to SonarCloud. This will allow some extra visibility into my open source code and provide me with a great sandbox for evaluating Sonarqube for corporate use.

    I added Sonar Analysis to a GitHub actions pipeline for my Hyper-V Info API. You can see the Sonar analysis on SonarCloud.io.

    The great part?? All the code is public, including the GitHub Actions pipeline. So, feel free to poke around and see how I made it work!