LinkedIn recently replaced Argo CD with a custom GitOps engine that deploys across more than 250 Kubernetes clusters and handles over 100,000 deployments a week. The scale is unusual, but the underlying friction is not. Teams running dozens of clusters against one Argo CD instance eventually hit the same walls: reconciliation lag, app sprawl, and webhooks that stop telling the truth.

By the end of this article you will know which limits you will actually hit, how to push them back, and when a custom controller is genuinely justified.

Where vanilla Argo CD strains first

Argo CD watches Git repositories and reconciles cluster state against them. The strain shows up in three predictable places as you grow:

  • Application count. Each Application object is a controller reconcile target. Past a few thousand, the controller's reconcile queue slows down and drift detection lags behind commits.
  • Webhook fan-out. One commit to a shared config repo can touch hundreds of Applications. The GitHub or GitLab webhook updates them, but the controller processes them serially enough that perceived sync time grows.
  • API server load. Every Application, every Refresh annotation, every status write lands on the cluster's etcd through the Kubernetes API. This couples your delivery system's health to your control plane's headroom.

None of these are bugs. They are the cost of a generic reconciler that treats every Application the same.

Measure before you migrate

Most teams that think they outgrew Argo CD have outgrown their repository layout instead. Before anything else, measure it:

# How many Applications does the controller actually manage? kubectl get applications.argoproj.io -A --no-headers | wc -l # Reconciliation health: look for long queues in the controller logs kubectl logs -n argocd deploy/argocd-application-controller \ | grep -i "reconcile" | tail -20

If drift detection runs minutes behind a merged PR, note the delta. If it is under your team's tolerance, you have a config problem, not a platform problem.

The AppProject and ApplicationSet fixes

The biggest win costs nothing: split monorepos. When one commit touches a manifest shared by 800 services, Argo CD re-evaluates all of them. Splitting per-team or per-service repos shrinks the blast radius of every push.

Then use ApplicationSet to generate Applications from templates instead of hand-authoring thousands:

apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: payments namespace: argocd spec: generators: - list: elements: - cluster: eu-west env: prod - cluster: us-east env: prod template: metadata: name: 'payments-{{cluster}}' spec: project: payments source: repoURL: https://git.example.com/platform/payments.git targetRevision: HEAD path: 'deploy/{{cluster}}' destination: server: https://kubernetes.default.svc syncPolicy: automated: prune: true selfHeal: true

This is the same idea LinkedIn's engine implements natively: treat a deployment as a rendered output over services and clusters, not a pile of individual objects.

Sharding and what it buys you

If one controller still can't keep up, Argo CD supports sharding the Application controller across clusters. Each shard reconciles a subset, so reconcile throughput scales roughly with shard count.

# In the argocd-cmd-params-cm ConfigMap controller.shard.replicas: "4" controller.sharding.algorithm: "consistent-hashing"

Sharding fixes throughput, not sprawl. A shard still holds every Application's status in memory for its clusters, and you now run a real distributed system for your CD tool.

When a custom engine is actually justified

LinkedIn's move makes sense because their requirements are unusual: multi-tenant deployment isolation, org-wide policy enforcement, and progress reporting integrated with internal tooling across hundreds of clusters. A generic tool fights you at that point.

Use this rough decision table:

Situation Approach
Slow reconciliation, few clusters Repo layout plus tuned controller settings
Thousands of Applications, one control plane ApplicationSets plus controller sharding
Hard multi-tenant or compliance requirements baked into deploy flow Consider a custom reconciler
Occasional webhook delays Check Git provider delivery, not Argo CD

If you do build your own, keep the GitOps contract: Git is the source of truth, the controller converges on it, and drift is reported rather than silently corrected. You can rewrite the engine without rewriting the workflow.

Key takeaways

  • Argo CD's limits at scale are reconcile throughput and API load, not correctness; measure both before concluding you need a replacement.
  • Splitting repositories and using ApplicationSets removes most of the fan-out pain at a fraction of the cost of a custom engine.
  • Controller sharding scales reconcile throughput across clusters but turns your CD tool into a distributed system you must operate.
  • A custom GitOps engine is justified when deployment policy, isolation, and reporting requirements are core product features, as they are at LinkedIn's scale.

Source

LinkedIn Replaces Argo CD with a Custom GitOps Engine for Kubernetes Deployments — this article adds the failure modes, measurement steps, and configuration-level mitigations the announcement leaves out.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD · TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY