Synced, But Not: ArgoCD Differencing Configuration

Some of the charts in my Loki/Grafana/Tempo/Mimir stack have an odd habit of not updating correctly in ArgoCD. I finally got tired of it and fixed it… I’m just not 100% sure how.

Ignoring Differences

At some point in the past, I had customized a few of my Application objects with ignoreDifferences settings. It was meant to tell ArgoCD to ignore fields that are managed by other things, and could change from the chart definition.

Like what, you might ask? Well, the external-secrets chart generates it’s own caBundle and sets properties on a ValidatingWebhookConfiguration object. Obviously, that’s managed by the controller, and I don’t want to mess with it. However, I also don’t want ArgoCD to report that chart as Out of Sync all the time.

So, as an example, my external-secrets application looks like this:

project: cluster-tools
source:
  repoURL: 'https://github.com/spydersoft-consulting/ops-argo'
  path: cluster-tools/tools/external-secrets
  targetRevision: main
destination:
  server: 'https://kubernetes.default.svc'
  namespace: external-secrets
syncPolicy:
  syncOptions:
    - CreateNamespace=true
    - RespectIgnoreDifferences=true
ignoreDifferences:
  - group: '*'
    kind: '*'
    managedFieldsManagers:
      - external-secrets

And that worked just fine. But, with my monitor stack, well, I think I made a boo-boo.

Ignoring too much

When I looked at the application differences for some of my Grafana resources, I noticed that the live vs desired image was wrong. My live image was older than the desired one, and yet, the application wasn’t showing as out of sync.

At this point, I suspected ignoreDifferences was the issue, so I looked at the Application manifest. For some reason, my monitoring applications had an Application manifest that looked like this:

project: external-apps
source:
  repoURL: 'https://github.com/spydersoft-consulting/ops-internal-cluster'
  path: external/monitor/grafana
  targetRevision: main
  helm:
    valueFiles:
      - values.yaml
    version: v3
destination:
  server: 'https://kubernetes.default.svc'
  namespace: monitoring
syncPolicy:
  syncOptions:
    - RespectIgnoreDifferences=true
ignoreDifferences:
  - group: "*"
    kind: "*"
    managedFieldsManagers:
    - argocd-controller
  - group: '*'
    kind: StatefulSet
    jsonPointers:
      - /spec/persistentVolumeClaimRetentionPolicy
      - /spec/template/metadata/annotations/'kubectl.kubernetes.io/restartedAt'

Notices the part where I am ignoring managed fields from argocd-controller. I have no idea why I added that, but, it looks a little “all inclusive” for my tastes, and it was ONLY present in the ApplicationSet for my LGTM stack. So I commented it out.

Now We’re Cooking!

Lo and behold, ArgoCD looked at my monitoring stack and said “well, you have some updates, don’t you!” I spent the next few minutes syncing those applications individually. Why? There are a lot of hard working pods in those applications, I don’t like to cycle them all at once.

I searched through my posts and some of my notes, and I honestly have no idea why I decided I should ignore all fields managed by argocd-controller. Needless to say, I will not be doing that again.