Sync, Waves, Hooks and Drift
14. What is the difference between sync status and health status?
Sync status says whether live state matches Git: Synced, OutOfSync or Unknown. Health status says whether the resources work: Healthy, Progressing, Degraded, Suspended, Missing or Unknown. They are independent.
An Application can be Synced and Degraded, when the Deployment matches Git but its pods crash-loop, or OutOfSync and Healthy, when someone scaled a Deployment by hand and it is serving traffic.
Health comes from built-in checks for core kinds and from Lua health scripts for CRDs, either bundled or set in resource.customizations.health.<group>_<kind>.
A Deployment is Progressing until its rollout finishes; a paused Argo Rollout reports Suspended.
Two recent changes are good probes: since 3.0 per-resource health is no longer persisted in the Application CR by default, so scripts that parsed .status.resources[].health broke, and since 3.4 an Application shows Missing only when all of its resources are missing.
15. How do sync waves order a sync?
Set the argocd.argoproj.io/sync-wave annotation to an integer.
Per the sync waves docs, resources default to wave 0, waves can be negative, and within a sync Argo CD sorts by phase, then wave (lowest first), then kind (namespaces, then other Kubernetes kinds, then custom resources), then name.
Argo CD applies the first wave that contains an out-of-sync or unhealthy resource, waits for that wave to become healthy, then moves on.
A 2-second delay between waves, set by ARGOCD_SYNC_WAVE_DELAY, gives other controllers time to update status so Argo CD does not read a stale "healthy" object.
# wave -1: the CRD and the namespace-scoped config the app needs
metadata:
annotations:
argocd.argoproj.io/sync-wave: "-1"
The consequence a strong candidate names: a resource that never becomes healthy blocks every later wave, so a missing custom health check for a CRD in wave 0 can stall the whole Application. Waves work only inside one Application.
Ordering across Applications needs an app of apps with waves on the child Application objects, or an ApplicationSet progressive sync.
16. Which resource hooks exist, and how are they cleaned up?
Hooks are resources, usually Jobs, annotated with argocd.argoproj.io/hook. The phases are PreSync, Sync, PostSync, SyncFail, PreDelete, PostDelete and Skip.
PreSync runs before any manifest is applied, PostSync runs once all resources are healthy, SyncFail runs when a sync fails, and PreDelete and PostDelete run around Application deletion. PreDelete hooks arrived in 3.3.0.
Cleanup is set by argocd.argoproj.io/hook-delete-policy:
HookSucceeded: delete after it succeeds.
HookFailed: delete after it fails.
BeforeHookCreation: delete the old instance before creating the new one. This is the default when no policy is set.
Hooks also take sync waves, so two PreSync Jobs can run in order.
Helm chart hooks map onto Argo CD hooks, for example helm.sh/hook: pre-install becomes PreSync, but Argo CD does not distinguish install from upgrade, so a chart that relies on a post-install-only hook runs it on every sync.
A candidate who has run database migrations as hooks will say the migration must be safe to run twice.
17. Find the bug: a migration Job that breaks the second deploy.
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
annotations:
argocd.argoproj.io/sync-wave: "-1"
spec:
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/payments:1.42.0
args: ["migrate", "up"]
The first sync works. When the next release bumps the image tag, the sync fails with a "field is immutable" error, because a Job's pod template cannot be changed after creation, and the Job is a normal resource that Argo CD tries to patch.
Even when it does not fail, a completed Job that stays in the cluster does not run again.
The fix is to make it a hook: add argocd.argoproj.io/hook: PreSync so it runs before the Deployment changes, and rely on the default BeforeHookCreation policy, or set it explicitly, so the old Job is deleted before the new one is created.
Some teams add HookSucceeded too, and keep failed Jobs for their logs.
A strong candidate also asks what happens when the migration succeeds and the new Deployment then fails health: the schema has moved forward while the old pods keep running, so migrations must be backward compatible with the previous release.
18. What do prune and selfHeal do, and what are the risks of each?
Both are off by default under spec.syncPolicy.automated.
Per the automated sync docs, prune: true deletes live resources that were removed from Git, selfHeal: true re-syncs when the live state drifts, after a default 5-second wait (--self-heal-timeout-seconds), and allowEmpty: true permits a prune that leaves the Application with no resources.
Automated sync runs once per commit SHA and parameter set, and a failed sync is not retried against the same pair unless a retry policy is set.
The risks are what make this a senior question:
- Prune plus a wrong
path or a Kustomize overlay that renders nothing deletes the app; that is why allowEmpty defaults to false.
- Self-heal fights any other controller that writes the same field, such as an HPA on
replicas or a mutating webhook (question 19).
- Self-heal also reverts the on-call engineer's emergency
kubectl fix within seconds, so incident runbooks need a step to disable auto-sync.
- Rollback through Argo CD is blocked while automated sync is on; the rollback is a Git revert.
Individual resources can opt out of pruning with argocd.argoproj.io/sync-options: Prune=false, which suits PersistentVolumeClaims and namespaces.
19. Find the bug: an HPA and Argo CD fighting over replicas.
spec:
ignoreDifferences:
- group: apps
kind: Deployment
jsonPointers:
- /spec/replicas
syncPolicy:
automated:
prune: true
selfHeal: true
The Deployment manifest in Git sets replicas: 3, and an HPA scales it to 12 under load. The Application now stays Synced, which looks correct.
But the next sync triggered by any other change, an image bump for example, applies the whole manifest, including replicas: 3, and the service drops to three pods at peak.
ignoreDifferences only changes the comparison. To make the sync skip the field too, add the RespectIgnoreDifferences=true sync option.
The cleaner fix is to remove replicas from the manifest so there is nothing to apply, which Kubernetes recommends anyway when an HPA owns scaling.
A strong candidate mentions jqPathExpressions for fields inside lists, and the system-level resource.customizations.ignoreDifferences in argocd-cm for patterns such as webhook-injected CA bundles.
Argo CD 3.6 also accepts an ignore-differences annotation on individual resources.
20. Which sync options do you reach for, and why?
Sync options go in spec.syncPolicy.syncOptions for the whole Application, or in the argocd.argoproj.io/sync-options annotation for one resource. The ones that come up in production:
ServerSideApply=true: applies with Kubernetes server-side apply. It handles objects too large for the last-applied-configuration annotation, such as big CRDs; the 3.3 ApplicationSet CRD needed it (question 29). Since 3.6 it also turns on Server-Side Diff.
CreateNamespace=true: creates the destination namespace if missing.
PruneLast=true: prunes after all other waves are healthy, so the old ConfigMap survives until new pods are up.
ApplyOutOfSyncOnly=true: applies only changed resources, which cuts sync time for Applications with thousands of objects.
Replace=true: uses kubectl replace or create instead of apply; a last resort.
FailOnSharedResource=true: fails the sync if another Application already tracks the resource.
A strong answer pairs each option with the failure it fixes.
21. How do ApplicationSet progressive syncs roll a change across clusters?
The RollingSync strategy syncs generated Applications in steps selected by labels, waiting for each step to become healthy before the next.
Per the progressive syncs docs, the feature is beta since 3.3.0 and off by default; enable it with applicationsetcontroller.enable.progressive.syncs: "true" in argocd-cmd-params-cm.
spec:
strategy:
type: RollingSync
rollingSync:
steps:
- matchExpressions:
- key: env
operator: In
values: [staging]
- matchExpressions:
- key: env
operator: In
values: [prod]
maxUpdate: 25%
deletionOrder: Reverse
Details that separate candidates: RollingSync forces automated sync off on the generated Applications, because the controller has to decide when each one syncs; Applications that match no step are never synced by the rollout; maxUpdate defaults to 100% of the step; and deletionOrder: Reverse (added in 3.2) removes prod before staging when the ApplicationSet shrinks.
In 3.6 the controller also refreshes all owned Applications and waits for them to reconcile before progressing, which can slow large sets.