In-cluster decryption with sops-age caused a catastrophic failure of my cluster. Please help me figure out how/why.

[2fst4u]

So I've been using flux for many months in my cluster. I have nearly every yaml relevant to my cluster in my private repo, all nicely organised into folders where relevant. Everything was working perfectly.

I was using Longhorn for storage in my cluster until recently. I got fed up with failures it has and decided to go back to old faithful rook now that it supports arm64 natively (yay!).

I restored all the backups I had, rebuilt everything that didn't, got it all running and started work on my next project, putting encrypted secrets into my repo with sops and age. Previously I'd just kept my secrets in a file elsewhere out of my repo for security obviously. But I want to edit them and manage them the same way as everything else in my cluster in vscode. In comes sops and age.

I actually followed TechnoTim's video on the subject on YouTube, and it was super helpful in getting sops with age up and running on my computer. But getting it working in flux is the next step that I spent a while figuring out. I followed the docs (which, no hate, but I feel like there are a lot of unexplained settings in the guide for this subject that you're just expected to understand. That's for a different time) and got to the point of:

Create a kustomization for reconciling the secrets on the cluster:

flux create kustomization my-secrets \
--source=my-secrets \
--path=./clusters/cluster0 \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-gpg

But I changed "source" for my existing repo I bootstrapped from, and "path" to "clusters/production" in my repo. And "sops-gpg" for "sops-age" obviously. I had put my encrypted secrets right in "clusters/production" where all my other yamls are, not deeper.

It reconciled, looked like everything worked fine, and then after about half an hour I decided to check on it. I just HAPPENED to see an event in the flux-system namespace saying something like "my-secrets created/changed [every single file in my repo listed out]". My heart dropped, I checked my pods, slowly every single one was deleted. Their PVCs, the data on them, every namespace referenced in my repo. All gone. Many hung due to CRD finalizers etc and many needed tidying.

Weirdly, the flux components ALSO got removed, including the flux-system namespace itself. They hung of course, because it was trying to reconcile, and bootstrapping again just made it delete everything again until the kustomizations had been fully deleted. My flux components aren't referenced in my repo other than the default bootstrapping created though.

Everything not referenced in my repo, like metallb which gets spun up directly from kubctl apply -f [gitrepo] stayed put, so it's clear flux caused this (through my fault, not blaming anyone other than myself).

So, the actual question: what did I do wrong? Does this sound like what I did caused the issues? What should I do differently and how should I have added in-cluster decryption without flux thinking everything in my repo is a new object? Would it have been fine if my secrets were in their own path and I reference that? If this is human error, is there something flux should potentially do to prevent this? Is there something missing from the docs that could have made this clearer? I'm coming from a position of no blame, I'm simply wanting to use my failure as a learning opportunity.

Sorry for the novel, I'm just hoping to put in words what I messed up and hopefully get to the bottom of how to prevent it myself and in others. I couldn't find a more appropriate place to write this out and ask this other than directly where the specialists are.

[stefanprodan]

Only the root Flux Kustomization named flux-system should have the cluster path --path=./clusters/cluster0. In the cluster0 dir there shouldn’t be anything else but Flux Kustomizations. Please see the example repo on how to structure your files https://github.com/fluxcd/flux2-kustomize-helm-example

[2fst4u]

Thank god so I'm not going crazy then. Ok, I will do my best to try again without following the way the tutorial says then.

Now, you touched on repo structure before and it's got me confused about that too. In the docs under https://fluxcd.io/flux/installation/ it says:

After running bootstrap you can place Kubernetes YAMLs inside a dir under path e.g. clusters/staging/my-app

Which to me reads as, in my repo under clusters/production, I can put any yamls directly under production. This is what I have been doing until now. Is this correct?

But in https://github.com/fluxcd/flux2-kustomize-helm-example that you linked earlier it has the following:

Repository structure

The Git repository contains the following top directories:

    apps dir contains Helm releases with a custom configuration per cluster
    infrastructure dir contains common infra tools such as ingress-nginx and cert-manager
    clusters dir contains the Flux configuration per cluster

├── apps
│   ├── base
│   ├── production 
│   └── staging
├── infrastructure
│   ├── configs
│   └── controllers
└── clusters
    ├── production
    └── staging

(Note I'm not using helm. Does this matter for this?)
Now in this diagram it shows your apps are outside your cluster directory. Is this a requirement? I haven't been doing this so now I'm confused about how this has been working.

├── apps
│      ├── base
│      ├── production 
│      └── staging
├── infrastructure
│      ├── configs
│      └── controllers
└── clusters
         ├── production
         │      └─# I have all my yamls here
         └── staging

Should they not be here?

[stefanprodan]

Your YAMLs shouldn’t be in the cluster dir if you have more than one cluster. As a general rule, best is to place app definitions outside of the clusters dir, this gives you the flexibility to patch them per environment, e.g. different ingress domain on dev vs production.

[2fst4u]

Ok I've done a lot of reading on that part and I'm even more confused about how you make it work but that's a topic for another day. In any case the answer to my original question, it comes down to the command I used breaking the cluster, correct?

[2fst4u]

Just coming back to this because I feel like I didn't really close this off to the point that the question is fully answered. If I were to try this again, should I be using a different repo? If I follow the docs and put my secrets in a different repo, will this work as expected?

[2fst4u]

@stefanprodan hoping you can shed some light on this. Is there an updated command that I need to use to bootstrap using the same repo? My flux install is getting very out of date and I'd like to update it but I can't if I don't know the right bootstrap command that won't break everything.

[crazyelectron-io]

Sorry for reviving this old thread, but I too have some struggles with the current documentation around secrets decryption.
As already mentioned, the example in the documentation only shows how to use a separate repo for secrets, but not how to modify the repo that is already created during bootstrap. I have manually added a decryption section (with age key) to gotk-sync.yaml

spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-age

but I don't think that is the correct way. How should I add this decryption definition to the cluster?

[stefanprodan]

Any of these Flux Kustomizations can have decryption enabled:

• https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/clusters/production/apps.yaml
• https://github.com/fluxcd/flux2-kustomize-helm-example/blob/main/clusters/production/infrastructure.yaml

To enable it, see the docs here: https://fluxcd.io/flux/components/kustomize/kustomizations/#decryption

Example for apps:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-keys
  interval: 10m0s
  dependsOn:
    - name: infra-configs
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  wait: true
  timeout: 5m0s

[2fst4u]

I've gotta say it's incredibly frustrating to get these canned responses. I feel like you're not understanding my concerns.

The example yaml you just provided me looks exactly like what I had in my root already, which you just told several of us we explicitly can't do. Why is this any different to just editing the one that already existed for the cluster?

[2fst4u]

Honestly it doesn't matter how many times I read the example of how to structure your repo, the more I read it the more confused I get.

Your example kustomization for example:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  decryption:
    provider: sops
    secretRef:
      name: sops-keys
  interval: 10m0s
  dependsOn:
    - name: infra-configs
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps/production
  prune: true
  wait: true
  timeout: 5m0s

How does this get made? What makes flux look for this apps location to know it should be reconciled? This information is completely missed from all of the docs, nothing tells you how to create this kustomization or where it should be placed. And if the docs do state this then I'm at a loss as to where that information is located.

What happens when you do a new bootstrap to upgrade the cluster? Doesn't this reference to other directories in the repo get wiped?

These are the questions we're confused about. These are the things I think you're not quite understanding that we're confused on.

So I go back to my original question. How and where do I implement secret decryption the correct way. What commands specifically am I supposed to run if I want my secrets to be in the same repo?

[stefanprodan]

The flux bootstrap command overrides the clusters/<cluster name>/flux-system dir content. All the Flux Kustomizations such as the apps and infrastructure from the example repo are left untouched.

What commands specifically am I supposed to run if I want my secrets to be in the same repo?

The ones from the https://fluxcd.io/flux/guides/mozilla-sops/#encrypting-secrets-using-age

[2fst4u]

There is no explanation for where your secrets should be located or what any of the commands do. The closest that page gets is

"flux create <kind> --export > manifest.yaml."

What do I put for "kind"? What other parameters do I need to make it decrypt secrets? Do I just manually edit the file it outputs? That doesn't seem right.

When I have this other magical file, how does flux know where it is from clusters/<cluster name>/flux-system? If I bootstrap everything there disappears. How do the other files get referenced?

Can you see my concern at all or am I just a complete moron? I'm not the only user in here confused, others have said they put theirs in the same place as me and none of us understand it better.