Proxy Down!

A simple package updates seems to have adversely affected the network on my Banana Pi proxy server. I wish this was a post on how I solved it, but, alas, I haven’t solved it.

It was a simple upgrade…

That statement has been uttered prior to massive outages a time or two. I logged in to my proxy server, ran apt update and apt upgrade, and restarted the device.

Now, this is a headless device, so I tried re-connecting via SSH a few times. Nothing. I grabbed the device out of the server closet and connected it to a small “workstation” I have in my office. I booted it up, and, after a pretty long boot cycle, it came up, but with the network adapter down.

Why’s the network down?

Running ip addr show displayed eth0 as being down. But, well, I have no idea why. And, since this is something of a production device, I manually enabled as follows:

# enable the link
sudo ip link set eth0 up
# restart networking
sudo systemctl restart systemd-networkd
# restart nginx
sudo systemctl restart nginx

This all worked and the network was back up and running. But what happened?

Digging through the syslog file, I came across this:

Jan  8 14:09:28 m5proxy systemd-udevd[2204]: eth0: Failed to query device driver: Device or resource busy

A little research yielded nothing really fruitful. I made sure that everything was setup correctly, including cloud-init and netplan, but nothing worked.

What Now?

I am in a weird spot. The proxy is working, and it’s such a vital piece of my current infrastructure that I do not currently have the ability to “play with it.”

What I should probably do is create a new proxy machine, either with one of my spare Raspberry PIs or just a VM on the server. Then I can transfer traffic to the new proxy and diagnose this one without fear of downtime.

Before I do that, though, I’m going to do some more research into the error above and see if I can tweak some of the configuration to get things working on a reboot. Regardless, I will post a solution when one comes about.