kustomize-controller in CrashLoopBackOff failed to get API group resources (i/o timeout)

[kertzi]

In one of our k8s clusters (running in AWS EKS) kustomization-controller and almost all other flux related controllers are in CrashLoopBackOff stage.

Here is error logs from kustomization controller pod:

$ kubectl logs -n flux-system kustomize-controller-f8697f65f-gqh5p
{"level":"info","ts":"2024-10-02T12:06:51.363Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"error","ts":"2024-10-02T12:07:21.366Z","logger":"setup","msg":"unable to create controller","controller":"kustomize-controller","error":"failed setting index fields: failed to get API group resources: unable to retrieve the complete list of server APIs: kustomize.toolkit.fluxcd.io/v1: Get \"https://172.20.0.1:443/apis/kustomize.toolkit.fluxcd.io/v1\": dial tcp 172.20.0.1:443: i/o timeout"}

So looks like it can't connect some api.

I'm not sure when this happen because haven't touched cluster for a while.
All apiservices looks to be available stage:

$ kubectl get apiservices
NAME                                     SERVICE                       AVAILABLE   AGE
v1.                                      Local                         True        349d
v1.acme.cert-manager.io                  Local                         True        132d
v1.admissionregistration.k8s.io          Local                         True        349d
v1.apiextensions.k8s.io                  Local                         True        349d
v1.apps                                  Local                         True        349d
v1.authentication.k8s.io                 Local                         True        349d
v1.authorization.k8s.io                  Local                         True        349d
v1.autoscaling                           Local                         True        349d
v1.batch                                 Local                         True        349d
v1.cert-manager.io                       Local                         True        132d
v1.certificates.k8s.io                   Local                         True        349d
v1.coordination.k8s.io                   Local                         True        349d
v1.crd.projectcalico.org                 Local                         True        132d
v1.discovery.k8s.io                      Local                         True        349d
v1.events.k8s.io                         Local                         True        349d
v1.flowcontrol.apiserver.k8s.io          Local                         True        132d
v1.kustomize.toolkit.fluxcd.io           Local                         True        132d
v1.networking.k8s.io                     Local                         True        349d
v1.node.k8s.io                           Local                         True        349d
v1.notification.toolkit.fluxcd.io        Local                         True        132d
v1.operator.tigera.io                    Local                         True        132d
v1.policy                                Local                         True        349d
v1.rbac.authorization.k8s.io             Local                         True        349d
v1.scheduling.k8s.io                     Local                         True        349d
v1.source.toolkit.fluxcd.io              Local                         True        132d
v1.storage.k8s.io                        Local                         True        349d
v1alpha1.crd.k8s.amazonaws.com           Local                         True        349d
v1alpha1.elbv2.k8s.aws                   Local                         True        132d
v1alpha1.external-secrets.io             Local                         True        132d
v1alpha1.networking.k8s.aws              Local                         True        132d
v1alpha1.vpcresources.k8s.aws            Local                         True        349d
v1beta1.elbv2.k8s.aws                    Local                         True        132d
v1beta1.external-secrets.io              Local                         True        132d
v1beta1.image.toolkit.fluxcd.io          Local                         True        132d
v1beta1.kustomize.toolkit.fluxcd.io      Local                         True        132d
v1beta1.notification.toolkit.fluxcd.io   Local                         True        132d
v1beta1.source.toolkit.fluxcd.io         Local                         True        132d
v1beta1.vpcresources.k8s.aws             Local                         True        349d
v1beta2.image.toolkit.fluxcd.io          Local                         True        132d
v1beta2.kustomize.toolkit.fluxcd.io      Local                         True        132d
v1beta2.notification.toolkit.fluxcd.io   Local                         True        132d
v1beta2.source.toolkit.fluxcd.io         Local                         True        132d
v1beta3.flowcontrol.apiserver.k8s.io     Local                         True        132d
v2.autoscaling                           Local                         True        349d
v2beta1.helm.toolkit.fluxcd.io           Local                         True        132d
v3.projectcalico.org                     calico-apiserver/calico-api   True        132d

Flux version 2.1.2, I'm also tried to upgrade it to latest 2.4.0 but I think it's not applied correctly when these controllers are not very healthy.

There is also one kustomization in failed state but haven't found a way to force flux to pick up commit which fixes:

$ flux get kustomization
NAME            REVISION                SUSPENDED       READY   MESSAGE                                                                                                                                                                                                                                 
flux-system     master@sha1:99ebd8da    False           False   GitRepository/flux-system/app dry-run failed: failed to prune fields: failed add back owned items: failed to convert merged object at version source.toolkit.fluxcd.io/v1: .spec.gitImplementation: field not declared in schema

Any help how to get flux "alive state" in cluster would be appreciated!

Thank you!

[stefanprodan]

Looks like the EKS API server is no longer accessible from the nodes where Flux runs. Cannot say what’s causing it, call AWS support or look into CNI/CoreDNS/VPC.

[kertzi]

hmm, wierd. Then I check security group stuff etc.
Thanks!

[kertzi]

But it also looks that endpoint kustomization controller tries to connect to is not correct. At least it doesn't match kubernetes endpoint

$ kubectl get endpoints 
E1003 10:30:23.774139 3912630 memcache.go:265] couldn't get current server API group list: Get "https://37B8065B16B81F8B234F62AB4AA6682F.gr7.eu-central-1.eks.amazonaws.com/api?timeout=32s": dial tcp 3.122.220.100:443: i/o timeout
NAME                     ENDPOINTS                                                        AGE
kubernetes               10.0.1.102:443,10.0.41.23:443                                    350d

When it tries to connect 172.20.0.1:443 which doesn't match any endpoints I can find from cluster.

[kertzi]

Looks like this get fixed just by re-creating nodes in cluster (terminating old and let autoscaling group to launch new).
So for somehow networking was broken