Some of the charts in my Loki/Grafana/Tempo/Mimir stack have an odd habit of not updating correctly in ArgoCD. I finally got tired of it and fixed it… I’m just not 100% sure how.
Ignoring Differences
At some point in the past, I had customized a few of my Application
objects with ignoreDifferences
settings. It was meant to tell ArgoCD to ignore fields that are managed by other things, and could change from the chart definition.
Like what, you might ask? Well, the external-secrets
chart generates it’s own caBundle
and sets properties on a ValidatingWebhookConfiguration
object. Obviously, that’s managed by the controller, and I don’t want to mess with it. However, I also don’t want ArgoCD to report that chart as Out of Sync
all the time.
So, as an example, my external-secrets
application looks like this:
project: cluster-tools
source:
repoURL: 'https://github.com/spydersoft-consulting/ops-argo'
path: cluster-tools/tools/external-secrets
targetRevision: main
destination:
server: 'https://kubernetes.default.svc'
namespace: external-secrets
syncPolicy:
syncOptions:
- CreateNamespace=true
- RespectIgnoreDifferences=true
ignoreDifferences:
- group: '*'
kind: '*'
managedFieldsManagers:
- external-secrets
And that worked just fine. But, with my monitor stack, well, I think I made a boo-boo.
Ignoring too much
When I looked at the application differences for some of my Grafana resources, I noticed that the live vs desired image
was wrong. My live image was older than the desired one, and yet, the application wasn’t showing as out of sync.
At this point, I suspected ignoreDifferences
was the issue, so I looked at the Application
manifest. For some reason, my monitoring applications had an Application
manifest that looked like this:
project: external-apps
source:
repoURL: 'https://github.com/spydersoft-consulting/ops-internal-cluster'
path: external/monitor/grafana
targetRevision: main
helm:
valueFiles:
- values.yaml
version: v3
destination:
server: 'https://kubernetes.default.svc'
namespace: monitoring
syncPolicy:
syncOptions:
- RespectIgnoreDifferences=true
ignoreDifferences:
- group: "*"
kind: "*"
managedFieldsManagers:
- argocd-controller
- group: '*'
kind: StatefulSet
jsonPointers:
- /spec/persistentVolumeClaimRetentionPolicy
- /spec/template/metadata/annotations/'kubectl.kubernetes.io/restartedAt'
Notices the part where I am ignoring managed fields from argocd-controller
. I have no idea why I added that, but, it looks a little “all inclusive” for my tastes, and it was ONLY present in the ApplicationSet
for my LGTM stack. So I commented it out.
Now We’re Cooking!
Lo and behold, ArgoCD looked at my monitoring stack and said “well, you have some updates, don’t you!” I spent the next few minutes syncing those applications individually. Why? There are a lot of hard working pods in those applications, I don’t like to cycle them all at once.
I searched through my posts and some of my notes, and I honestly have no idea why I decided I should ignore all fields managed by argocd-controller
. Needless to say, I will not be doing that again.