GitOps With Argo CD: What the Getting-Started Guide Skips
Most Argo CD tutorials show you how to deploy a single application from a Git repository. That's the easy part. The hard part is designing a GitOps workflow that handles 50+ microservices, multiple environments, secrets that can't live in Git, and promotions that need human approval before hitting production.
I've operated Argo CD at organizations with 100+ applications across four environments. The patterns here reflect what worked after multiple iterations of getting it wrong.
Repository Structure
The first architectural decision is whether to use a monorepo or separate repos for application code and deployment manifests. I've used both, and I strongly prefer separate repos.
With separate repos, your application repo contains source code, Dockerfiles, and CI pipelines that produce container images. Your deployment repo (often called the "config repo" or "gitops repo") contains Kubernetes manifests, Helm values files, or Kustomize overlays. A CI pipeline in the application repo builds the image, pushes it to a registry, and then updates the image tag in the deployment repo. Argo CD watches the deployment repo and syncs changes.
# Deployment repo structure
gitops-config/
├── apps/ # Argo CD Application manifests
│ ├── api-server.yaml
│ ├── frontend.yaml
│ └── worker.yaml
├── base/ # shared base manifests
│ ├── api-server/
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ └── kustomization.yaml
│ └── frontend/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
└── overlays/
├── dev/
│ ├── api-server/
│ │ ├── kustomization.yaml
│ │ └── patches.yaml
│ └── frontend/
├── staging/
└── production/Why Kustomize Over Helm for GitOps
When Argo CD manages the deployment, I prefer Kustomize over Helm for the deployment repo. Helm charts are great for distributing reusable packages, but for your own applications, the rendered output is what matters — and with Kustomize, the rendered output is transparent. You can look at a Kustomize overlay and see exactly what differs from the base. With Helm, you need to run helm template to see what you're actually deploying.
Argo CD supports both, but debugging sync failures is easier with Kustomize because you can read the manifest files directly. With Helm, you're debugging the template rendering layer on top of the actual Kubernetes resources.
Application of Applications Pattern
When you have many services, you don't want to create each Argo CD Application resource manually. The "app of apps" pattern uses a parent Application that points to a directory of Application manifests:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: production-apps
namespace: argocd
spec:
project: production
source:
repoURL: https://git.internal/gitops-config.git
targetRevision: main
path: apps/production
destination:
server: https://kubernetes.default.svc
syncPolicy:
automated:
prune: true
selfHeal: trueThe apps/production directory contains one Application manifest per service. When you add a new service, you create its manifest in that directory. The parent Application detects the new file and creates the child Application. It's Argo CD managing Argo CD — recursive GitOps.
A newer alternative is ApplicationSets, which can generate Applications from templates. An ApplicationSet with a Git directory generator creates one Application for each directory in a path:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: production-services
namespace: argocd
spec:
generators:
- git:
repoURL: https://git.internal/gitops-config.git
revision: main
directories:
- path: overlays/production/*
template:
metadata:
name: '{{path.basename}}'
spec:
project: production
source:
repoURL: https://git.internal/gitops-config.git
targetRevision: main
path: '{{path}}'
destination:
server: https://kubernetes.default.svc
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: trueAdd a new directory under overlays/production/, and Argo CD automatically creates an Application for it. No manual Application manifest needed.
Environment Promotion Workflow
The promotion workflow determines how changes move from dev to staging to production. The simplest approach: each environment tracks a different branch or path in the deployment repo.
But branch-based promotion has a problem — merging the dev branch into staging means every change in dev goes to staging together. You can't promote a single service independently. Path-based promotion (where each environment has its own overlay and image tags are updated independently) gives you more control.
Our promotion workflow uses pull requests. CI builds the image and opens a PR that updates the image tag in the dev overlay. That PR gets auto-merged (dev deploys automatically). To promote to staging, a separate PR updates the staging overlay with the same image tag. That PR requires one approval. Production requires two approvals from the platform team.
# CI script for image tag update
IMAGE_TAG="sha-$(git rev-parse --short HEAD)"
cd gitops-config
# Update dev automatically
kustomize edit set image api-server=registry.internal/api-server:$IMAGE_TAG
git commit -am "deploy: api-server $IMAGE_TAG to dev"
git push
# Create staging PR
git checkout -b promote/api-server-$IMAGE_TAG-staging
cd overlays/staging/api-server
kustomize edit set image api-server=registry.internal/api-server:$IMAGE_TAG
git commit -am "promote: api-server $IMAGE_TAG to staging"
gh pr create --title "Promote api-server $IMAGE_TAG to staging" --base mainSync Waves and Hooks
When deploying a service that depends on database migrations, you need the migrations to run before the new code deploys. Argo CD sync waves handle this ordering:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migration
annotations:
argocd.argoproj.io/sync-wave: "-1"
argocd.argoproj.io/hook: PreSync
spec:
template:
spec:
containers:
- name: migrate
image: registry.internal/api-server:latest
command: ["./migrate", "up"]
restartPolicy: NeverThe sync-wave: "-1" runs this Job before wave 0 (where your Deployment lives by default). The hook: PreSync annotation tells Argo CD to run this as a pre-sync hook — it executes before the main sync begins, and the sync only proceeds if the Job succeeds.
Handling Secrets
Secrets can't live in Git. But GitOps means everything comes from Git. This tension has spawned several solutions. I've used two that work well.
The External Secrets Operator syncs secrets from an external provider (AWS Secrets Manager, Vault, GCP Secret Manager) into Kubernetes Secrets. You commit an ExternalSecret resource to Git, and the operator creates the actual Secret:
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api-secrets
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: api-secrets
data:
- secretKey: DATABASE_URL
remoteRef:
key: production/api-server/database-urlThis resource lives in Git. The actual secret value lives in AWS Secrets Manager. Argo CD syncs the ExternalSecret, the operator reads the value from AWS, and creates a standard Kubernetes Secret. Your pods reference the Secret as usual.
The second option is Sealed Secrets, which encrypts secrets with a public key so the encrypted version can live in Git. Only the Sealed Secrets controller in the cluster can decrypt them. This is simpler to set up but harder to rotate — changing a secret requires re-encrypting and committing the new sealed version.
Multi-Cluster GitOps
When you have multiple Kubernetes clusters - production in two regions, staging, and dev - Argo CD can manage all of them from a single control plane. Register external clusters using their kubeconfig, and Applications can target any registered cluster.
The multi-cluster setup introduces a question: should you run one Argo CD instance that manages all clusters, or one per cluster? I've done both. A single centralized instance is simpler to manage and gives you one dashboard for everything. But it creates a single point of failure - if the Argo CD cluster goes down, no cluster can receive deployments.
My recommendation: one Argo CD per environment tier. One instance manages all dev/staging clusters. A separate instance manages production clusters. This limits the blast radius - a misconfiguration in the dev Argo CD doesn't affect production deployments.
Drift Detection and Self-Healing
Argo CD's selfHeal setting automatically reverts any manual changes made to cluster resources. If someone runs kubectl edit deployment and changes the replica count, Argo CD detects the drift and reverts it to match the Git source of truth within 3 minutes (the default sync interval).
This is usually what you want - it enforces GitOps discipline. But it can cause problems during incidents. If you need to temporarily scale up a service or modify a configuration to mitigate an incident, self-heal will revert your changes. For incident response, either disable auto-sync temporarily on the affected Application before making emergency changes, then commit the fix to Git and re-enable auto-sync, or better - make the emergency change in Git directly. Push to the deployment repo, and Argo CD applies it within seconds.
Notifications and Status Integration
Argo CD Notifications (built into Argo CD 2.6+) sends alerts when sync status changes. I configure it to send Slack messages on sync failures and successful production deployments. For production deployments, I also send a notification to a deployment tracking channel with the image tag, the Git commit that triggered the deployment, and a link to the Argo CD UI. This creates an audit trail that's easier to search than Git history, and it gives oncall engineers immediate context when they're investigating an incident that started after a deployment.
Performance at Scale
With 100+ Applications, Argo CD's default configuration starts showing strain. The application controller reconciles all applications in a single loop, and with many applications the reconciliation cycle can exceed the sync interval, causing sync delays.
Tune these settings for large installations: increase the controller's sharding count to distribute applications across multiple controller pods. Set the reconciliation timeout to prevent slow syncs from blocking the loop. Increase the repo server cache TTL to reduce redundant Git clones. And run the repo server with multiple replicas behind a headless service - it's the component that does the actual manifest rendering, and it becomes a bottleneck before the controller does.