Modernizing the Gateway: From NGINX Ingress to Envoy Gateway

As with any good engineer, I cannot leave well enough alone. Over the past week, I’ve been working through a significant infrastructure modernization across my home lab clusters – migrating from NGINX Ingress to Envoy Gateway and implementing the Kubernetes Gateway API. This also involved some necessary housekeeping with chart updates and a shift to Server-Side Apply for all ArgoCD-managed resources.

Why Change?

The timing couldn’t have been better. In November 2024, the Kubernetes SIG Network and Security Response Committee announced that Ingress NGINX will be retired in March 2026. The project has struggled with insufficient maintainer support, security concerns around configuration snippets, and accumulated technical debt. After March 2026, there will be no further releases, security patches, or bug fixes.

The announcement strongly recommends migrating to the Gateway API, described as “the modern replacement for Ingress.” This validated what I’d already been considering – the Gateway API provides a more standardized, vendor-neutral approach with better separation of concerns between infrastructure operators and application developers.

Envoy Gateway, being a CNCF project built on the battle-tested Envoy proxy, seemed like a natural choice for this migration. Plus, it gave me an excuse to finally move off Traefik, which was… well, let’s just say it was time for a change.

The Migration Journey

The migration happened in phases across my ops-argo, ops-prod-cluster, and ops-nonprod-cluster repositories. Here’s what changed:

Phase 1: Adding Envoy Gateway

I started by adding Envoy Gateway as a cluster tool, complete with its own ApplicationSet that deploys to clusters labeled with spydersoft.io/envoy-gateway: "true". The deployment includes:

GatewayClass and Gateway resources: Defined a main gateway that handles traffic routing
EnvoyProxy configuration: Set up with a static NodePort service for consistent external access
ClientTrafficPolicy: Configured to properly handle forwarded headers – crucial for preserving client IP information through the proxy chain

The Envoy Gateway deployment lives in the envoy-gateway-system namespace and exposes services via NodePort 30080 and 30443, making it easy to integrate with my existing network setup.

Phase 2: Migrating Applications to HTTPRoute

This was the bulk of the work. Each application needed its Ingress resource replaced with an HTTPRoute. The new Gateway API resources are much cleaner. For example, my blog (www.mattgerega.com) went from an Ingress definition to this:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: wp-mattgerega
  namespace: sites
spec:
  parentRefs:
    - name: main
      namespace: envoy-gateway-system
  hostnames:
    - www.mattgerega.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: wp-mattgerega-wordpress
          port: 80

Much more declarative and expressive than the old Ingress syntax.

I migrated several applications across both production and non-production clusters:

Gravitee API Management
ProGet (my package management system)
n8n and Node-RED instances
Linkerd-viz dashboard
ArgoCD (which also got a GRPCRoute for its gRPC services)
Identity Server (across test and stage environments)
Tech Radar
Home automation services (UniFi client and IP manager)

Phase 3: Removing the Old Guard

Once everything was migrated and tested, I removed the old ingress controller configurations. This cleanup happened across all three repositories:

ops-prod-cluster:

Removed all Traefik configuration files
Cleaned up traefik-gateway.yaml and traefik-middlewares.yaml

ops-nonprod-cluster:

Removed Traefik configurations
Deleted the RKE2 ingress NGINX HelmChartConfig (rke2-ingress-nginx-config.yaml)

The cluster-resources directories got significantly cleaner with this cleanup. Good riddance to configuration sprawl.

Phase 4: Chart Maintenance and Server-Side Apply

While I was in there making changes, I also:

Bumped several Helm charts to their latest versions:
- ArgoCD: 9.1.5 → 9.1.7
- External Secrets: 1.1.0 → 1.1.1
- Linkerd components: 2025.11.3 → 2025.12.1
- Grafana Alloy: 1.4.0 → 1.5.0
- Common chart dependency: 4.4.0 → 4.5.0
- Redis deployments updated across production and non-production
Migrated all clusters to use Server-Side Apply (ServerSideApply=true in the syncOptions):
- All cluster tools in ops-argo
- Production application sets (external-apps, production-apps, cluster-resources)
- Non-production application sets (external-apps, cluster-resources)

This is a better practice for ArgoCD as it allows Kubernetes to handle three-way merge patches instead of client-side strategic merge, reducing conflicts and improving sync reliability.

Lessons Learned

Gateway API is ready for production: The migration was surprisingly smooth. The Gateway API resources are well-documented and intuitive. With NGINX Ingress being retired, now’s the time to make the jump.

HTTPRoute vs. Ingress: HTTPRoute is more expressive and allows for more sophisticated routing rules. The explicit parentRefs concept makes it clear which gateway handles which routes.

Server-Side Apply everywhere: Should have done this sooner. The improved conflict handling makes ArgoCD much more reliable, especially when multiple controllers touch the same resources.

Envoy’s configurability: The EnvoyProxy custom resource gives incredible control over the proxy configuration without needing to edit ConfigMaps or deal with annotations.

Multi-cluster consistency: Making these changes across production and non-production environments simultaneously kept everything aligned and reduced cognitive overhead when switching between environments.

Current Status

All applications across all clusters are now running through Envoy Gateway with the Gateway API. Traffic is flowing correctly, TLS is terminating properly, and I’ve removed all the old ingress-related configuration from both production and non-production environments.

The clusters are more standardized, the configuration is cleaner, and I’m positioned to take advantage of future Gateway API features like traffic splitting and more advanced routing capabilities. More importantly, I’m ahead of the March 2026 retirement deadline with plenty of time to spare.

Now, the real question: what am I going to tinker with next?