Question: sharding on kustomize controller only

[masterphenix]

Hello,
One of our k8s cluster has more than 300 namespaces ; we are using flux to manage content of this cluster, by means of:

• 1 gitRepository source, that points to a unique repository, formatted like this:

<cluster_name>
  |_<namespace1>
  |_<namespace2>
    |_ kustomization and resources

• 1 kustomization (kustomize.toolkit.fluxcd.io/v1) per namespace
• a webhook to trigger reconcilation when a push is made on the git repo

Today, we are facing some delays when pushing changes to git, because the kustomize controller will trigger a reconcile on every kustomization (more than 300) ; as a result, for a small change on a single namespace, it can take more than 5 minutes to be applied on the cluster, depending on the order in which reconciliations are done.

We considered using the sharding capability of the kustomize controller, but it appears that, since flux kustomization are depending on the GitRepository source, the latter should also be sharded: this is an issue for us, since there is only 1 gitRepository for 300 kustomization.

Did we understand something wrong ? Would you by any chance have any recommendations to help us lower these delays ?

[stefanprodan]

You can shard only kustomize-controller, but before doing this, I suggest you increase the --concurrent and limits, see https://fluxcd.io/flux/installation/configuration/vertical-scaling/.

A single kustomize-controller instance can do 1K reconciliation in about 3m, see https://github.com/fluxcd/flux-benchmark/blob/main/RESULTS.md

[masterphenix]

Thank you for your quick answer ! I'll look deeper into that, thanks a lot.

Regarding sharding, I did try to shard only kustomize-controller, but all resources were handled by the shard instead of the main controller, which made me believe that source controller was supposed to be sharded also. But I just found out, while replying to you, that we activated sharding while keeping the following flag: --enable-leader-election, which, of course, is kinda incompatible with sharding. I tried without it, and sharding seems to work better ;-)

[stefanprodan]

The --enable-leader-election is not incompatible with sharding, each shard elects a primary.

[masterphenix]

That's odd, as soon as I removed leader election, both controllers started behaving normally ; before that, I noticed that both the main controller and the shard tried to acquire the same lease, the main controller failed to win the election, the shard behaved as the main controller and totally disregarded the flag --watch-label-selector=sharding.fluxcd.io/key and handled all resources.

I just tried to reproduce that, but with no luck