Argo CD Synchronization is BROKEN! It Should Switch to Eventual Consistency!

We dive into the challenges faced with Argo CD's current synchronization strategy. Despite its wide adoption for managing Kubernetes deployments, there are cracks in the system's design, particularly when it comes to handling sync operations. We'll explore why Argo CD's insistence on strong consistency is causing more problems than it solves and discuss how an eventual consistency model could be the game-changer that Argo CD needs to streamline deployment workflows and minimize outages.

▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Sponsor: Twingate
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

#ArgoCD #DevOps #Kubernetes #GitOps

▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬

▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬

▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬

▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬

▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Intro to Kubernetes Synchronization
02:12 Twingate (sponsor)
02:56 Deployment Order with kubectl
06:50 Deployment Order with Argo CD
14:48 Deployment Order with Crossplane
19:06 Closing Thoughts

[DevOpsToolkit]

Are you struggling with Argo CD synchronization order?

[mihaigalos]

I love this. We specify resources declaratively, but still do imperative steps to help convergence.

[mirceanton]

Great video! I fully agree. This is something that had me confused in the beginning when learning Flux, as I was expecting it to behave exactly like you are describing here.

[DennisHaney]

I totally agree... ArgoCD without adding ServerApply and a couple of Retries is much harder to work with. But with those, it works like wonders.

[IvanRizzante]

Thanks for another great video! I definitively agree on eventual consistency for ArgoCD for resources creation, but there's also the other face of the medal: deletion order.
There, sync waves and other mechanisms like the ones you described are needed.
After all, that's also why Crossplane introduced the (still in alpha) Usages concept which was for a long time requested to address issues like these.

[Artazar]

Good point with disaster recovery. I did undergo the exercise of bootstrapping a fresh cluster after setting up a full FluxCD repository with all cluster components. Had to define some dependencies for CRDs and a bit of ordering as well. Eventually full reconciliation of a repo to a cluster should succeed from point 0, that's an essential basis for GitOps approach.

[gunnarsinn]

I experienced this already long time ago. I had to define sequence in my bootstrap process. I was always surprised that on the ArgoCD videos it was mentioned that all k8s resources should always reconcile till everything is running, waiting for their dependencies to become ready. Never happened on my cluster. It had a complex setup with ArgoCD, running 20+ controllers with inter-dependencies, even v-cluster and applications on top controlled by same ArgoCD instance. Managed secrets already imposed challenges (first CNI, then secret controller, then ArgoCD as it used secrets, then mesh, then ...). When bootstrapping I used a bash script (I know, there are better ways :-)) to control sequence, ended up a bit messy (as expected). But the fact that in bootstrapping sequence always mattered was frustrating and caused (is causing) massive additional complexity. If crossplane can manage that, that logic should also be used in a gitops tool. The woly grail for a solution which can manage bootstrapping and ongoing lifecycle management with one seamless integrative logic does not yet seem to be out there..great, some room for improvement!

[Ardentcraft]

I've noticed this issue and I agree 100% percent.
Atm it is just a small inconvenience for me, however if they fix this in a near future that's going to be great improvement.

[spenhouet]

Funny coincidence. We just today shared some frustration about Argo CD and Argo WF in our team. We just recently updated from an old version to the current one, and it seems everything got worse. The application is instable and fails for some team members. A lot of IT issues. We were disappointed to find that literally None of the things we wished to improve in Argo were addressed over the long period of time. The UI is buggy as hell and a UX nightmare. UI elements that glitch out. Namespace drop downs which don't actually show the available namespaces but instead store everything you type in and provide that as options next time without any UI option to clear these. Everything is persisted across tabs. You can't use Argo in multiple tabs at all. And I can go on with issues we face. On top of all that, the newest version now completely crashes for some of our team members with Chromium based browser. Oh and on the topic of Argo CD sync, we had services failing to sync because of a property that apparently Argo added on the fly, which we did not specify, and it then detected as change. So it tried to sync something that wasn't there. I know, we shouldn't complain and rather contribute to fix these issues...

[javisartdesign]

I kinda agree, it's difficult to express dependencies between applications since it could have circular dependencies as you pointed. So that means that it should be another mechanism like Terraform to generate a dependency graph to apply the changes in order or a recursive operation that it constantly retries in a preemptive way, so it does not throw a dead lock executing the task.

[marianferenc]

I would like to also add to this, that Argo Workflows are also broken. The most complex part when it comes to work with Argo Workflows is creation of Argo Events trigger, so that Argo Workflow will be automatically triggered when new changes are being pushed into the app repository.

[roganl]

Seems like a no brainer... Great preso. I like the terminal overlays, they feel less distracting than the audio bubbles. my 2¢ - comb the mustache before the camera comes on. Would seem that ArgoCD could easily retry its resource creation at least as many times as there were resources being instantiated, so that a worst case of complete (persistent) failure would then notify, or alternatively watch for forward progress in its work queue, and if none occurred then throw the failure notification. I've seen this sequencing problem arise with chef as well... making idempotent configuration is hard.

[TheProcinger]

Sounds like someone who had the same working week as me 😅 I have been using Argo Tools for 2 years now and love them. They do what they are supposed to do. But with complex projects, especially when entire infrastructures are built (ArgoCD + Crossplane / CRDs) it quickly becomes a pain. It would be nice if Argo CD would simply try again "later" if a CRD is missing ... I'm just too old to worry about pre/post hooks / wave sync 👴

and last but not least! once you have built your infra, always destroy and re-sync to see if the order is correct🤞

[brk]

Good video. Also to add, there seems to be a variety of scenarios where a sync won't bounce a pod/deployment (e.g. configmap or secret changes content but not name). This means once ArgoCD syncs you still have to go in and manually scale the deploy down and up to restart it. Please let me know if Im wrong on this because it's very annoying. Essentially the selling point of GitOps / ArgoCD is that I push to Git and it takes care of the rest. In reality this is nowhere near the case.

[WernerWohlfahrter]

Great Video!! I totally agree with what you have said. My silly suggestion: Wouldn't it be amazing if you (Crossplane/Upbound) and ArgoCD join forces to fix this issue as you seem to already have a working solution and not being a direct competitor? It would help improve things quickly for the whole k8s community - do you agree?

[fafgracas]

But ArgoCD is more similar to kubectl command behavior; if you have ready-to-go Kubernetes manifest files in the folder and start to apply them with kubectl one by one in the order of your file system show you, you will probably face some issues because of the dependencies among them. So, I would recommend all ArgoCD users to have helm templates for your deployments since helm templates render all manifest in the correct order to be deployed(here you will face some issues if your application helm template mixes with operators' CRDs, which helm doesn't understand the proper order of them). So, having plain k8s manifest files in a folder is not the best way to use Argo for deployments, I would say.

[autohmae]

Perfection has not been reached... but luckily progress is being made.

[Bill_the_Red_Lichtie]

Hi Viktor, where Argo CD fails is exactly where Flux CD (V2) wins. It repeats the apply attempts repeatedly until it works. It also reports missing requirements. As pretty as Argo CD may be, I personally find Flux CD (V2) to be the better solution. Does Flux CD need a nice "standard" UI, the same way as Argo CD? Yes, 100% yes! But is Crossplane the only solution doing eventual consistency?

[DerJoe]

It looks like eventual consistency is easier to work with, better, and even simpler and I agree with your complaints. Now I wonder why Argo does it differently and if that has some advantages, too.
EC might possibly make it easier to create disasters by mistake (e.g. when renaming/moving stuff causes unwanted deletion of resources, etc), but that is just a feeling.

[Zoe-eyme]

Great video! I had a recent issue with syncing where the application gets stuck at presync running even after the job completes successfully and I have to terminate the running sync operation and start another manually! Still can't figure out why this is happening