flux diff with remote cluster [.spec.kubeConfig]

[milas]

Currently, if I have a Kustomization that targets another cluster using .spec.kubeConfig, when running flux diff, it seems to ignore that and pretend it's being applied to the current cluster.

I realize full client-side multi-cluster diff is likely impractical, but does anyone have any tips here? Using --ignore-paths is one option, though it causes the diff to state that those resources are being deleted, which is a bit scary/confusing. [I've been using flux diff to post a comment to PRs, which has been helpful for devs as a quick sanity check/preview ahead-of-time.]

IMO, Ideally, such resources could be skipped & perhaps show something like:

[!] ks-foo uses Secret/my-kubeconfig, diff not supported

Happy to try to work on a PR for something along those lines if it seems feasible?

But I'm also interested in any creative solutions/workarounds from others, and of course I'd love to find out I'm holding it wrong and Flux already supports this.

[stefanprodan]

I agree that we should error out, maybe the message could be diff not supported for remote clusters

[milas]

Thanks - opened a PR @ #5071 , can continue impl discussion there

[maboehm]

I was also looking into diffing Kustomizations that are applied to a different cluster, and came up with this proposal:
main...maboehm:flux2:diff-kubeconfig
(I did not consider the recursive case just yet)

In my approach, the CLI tries to use the secret from spec.kubeConfig when diffing, but also offers a --target-kubeconfig for cases where the in-cluster kubeconfig only works from inside the cluster.

Would that be something worth adding? It's still not perfect, but still an improvement IMO.