Flux restarting pods again after rolling restart

[jwerre]

I'm running into a strange issue where if I do a rolling restart on a deployment the kustomize-controller dispatches an event (after 10 or so minutes ) and restarts that same deployment. For example:

kubectl rollout restart deployment -n MY_NS MY_DEP

flux logs --since 30m --kind Kustomization

[...]

2022-05-20T12:54:57.993Z info Kustomization/flux-system.flux-system - server-side apply completed 
2022-05-20T12:54:59.038Z info Kustomization/flux-system.flux-system - server-side apply completed
2022-05-20T12:54:59.058Z info Kustomization/flux-system.flux-system - Reconciliation finished in 2.033319648s, next run in 10m0s

2022-05-20T13:05:00.044Z info Kustomization/flux-system.flux-system - server-side apply completed
2022-05-20T13:05:01.152Z info Kustomization/flux-system.flux-system - server-side apply completed
2022-05-20T13:05:01.156Z info Kustomization/flux-system.flux-system - Dispatching event: Deployment/MY_NS/MY_DEP configured
2022-05-20T13:05:01.177Z info Kustomization/flux-system.flux-system - Reconciliation finished in 2.114664112s, next run in 10m0s
2022-05-20T13:05:01.178Z info Kustomization/flux-system.flux-system - Dispatching event: Reconciliation finished in 2.114664112s, next run in 10m0s

[... MY_DEP restarts here]

2022-05-20T13:15:02.471Z info Kustomization/flux-system.flux-system - server-side apply completed
2022-05-20T13:15:04.037Z info Kustomization/flux-system.flux-system - server-side apply completed
2022-05-20T13:15:04.090Z info Kustomization/flux-system.flux-system - Reconciliation finished in 2.910387883s, next run in 10m0s

After Kustomization reconciliation finishes at 13:05:01 MY_DEP preforms another rolling restart. Any idea why this could be happening or how to fix it? I've restarted all my flux deployments and rerun flux bootstrap which doesn't seem to resolve the issue. Below is all the information on my cluster:

flux check
► checking prerequisites
✔ Kubernetes 1.21.9-eks-0d102a7 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.21.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.22.1
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.18.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.25.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.23.5
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.24.4
✔ all checks passed

flux --version
flux version 0.30.2

kubectl describe -n flux-system deployments.apps kustomize-controller
Name:                   kustomize-controller
Namespace:              flux-system
CreationTimestamp:      Thu, 27 Jan 2022 09:29:58 -0700
Labels:                 app.kubernetes.io/instance=flux-system
                        app.kubernetes.io/part-of=flux
                        app.kubernetes.io/version=v0.30.2
                        control-plane=controller
                        kustomize.toolkit.fluxcd.io/name=flux-system
                        kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:            deployment.kubernetes.io/revision: 6
Selector:               app=kustomize-controller
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=kustomize-controller
  Annotations:      prometheus.io/port: 8080
                    prometheus.io/scrape: true
  Service Account:  kustomize-controller
  Containers:
   manager:
    Image:       ghcr.io/fluxcd/kustomize-controller:v0.25.0
    Ports:       8080/TCP, 9440/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      --events-addr=http://notification-controller.flux-system.svc.cluster.local./
      --watch-all-namespaces=true
      --log-level=info
      --log-encoding=json
      --enable-leader-election
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:      100m
      memory:   64Mi
    Liveness:   http-get http://:healthz/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  http-get http://:healthz/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      RUNTIME_NAMESPACE:   (v1:metadata.namespace)
    Mounts:
      /tmp from temp (rw)
  Volumes:
   temp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   kustomize-controller-784bd54978 (1/1 replicas created)
Events:          <none>

---

EDIT:

Perhaps this is a different issue but I noticed there is one error in my logs regarding the kustomize-controller:

2022-05-14T15:54:57.203Z error Kustomization/flux-system.flux-system - Reconciliation failed after 32.983979ms, next try in 10m0s failed to download artifact from http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/SOME_HASH.tar.gz, status: 404 Not Found

This is about the time all this started and I wonder if it has something to do with it. So I saw issue #2307 and thought it could be a pod to pod communication issue (100% not an overload issue) so I create and logged into a pod in the flux-system namespace and did a curl on this url:

$ curl -vvv http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/SOME_HASH.tar.gz

*   Trying 172.20.153.209:80...
* Connected to source-controller.flux-system.svc.cluster.local (172.20.153.209) port 80 (#0)
> GET /gitrepository/flux-system/flux-system/SOME_HASH.tar.gz HTTP/1.1
> Host: source-controller.flux-system.svc.cluster.local
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 19 May 2022 22:51:14 GMT
< Content-Length: 19
<
404 page not found
* Connection #0 to host source-controller.flux-system.svc.cluster.local left intact

It looks like it connect successfully to the host:

Connected to source-controller.flux-system.svc.cluster.local (172.20.153.209) port 80 (#0)"

... but failed to find SOME_HASH.tar.gz and returned a page not found error.

EDIT:

[stefanprodan]

Flux does a rollback of any kubectl edits in-cluster, kubectl rollout restart deployment adds an annotation that Flux removes, so Kubernetes sees the removal and restarts the deployment a 2nd time. All manual edits must be using kubectl --field-manager=flux-client-side-apply docs here: https://fluxcd.io/docs/faq/#why-are-kubectl-edits-rolled-back-by-flux

[stefanprodan]

Not that new, the kubectl rollback feature was released in late March.

[beasteers]

Does it have to be literally --field-manager=flux-client-side-apply or can it be any custom manager value (e.g. the name of an operator)?

I'm submitting a bug report to an operator about this because flux keeps removing their Lock annotations, but the idea of having them implement Flux-specific functionality doesn't seem like a simple ask.

[stefanprodan]

Yes it can be some other controller name, but not kubectl.

[beasteers]

In the meantime, does flux have any resource-side ways of controlling this?

For example, I just found ArgoCD docs describing preserveFields letting you whitelist annotations.

https://argo-cd.readthedocs.io/en/stable/operator-manual/applicationset/Controlling-Resource-Modification/#preserving-changes-made-to-an-applications-annotations-and-labels

[matheuscscp]

Maybe you're looking for this: https://fluxcd.io/flux/components/kustomize/kustomizations/#controlling-the-apply-behavior-of-resources