Backing up resources that are not managed by Flux

[rombert]

Hi,

I am wondering, what are others doing for backup? The functionality that I am looking for is to back up resources not managed by Flux, mainly persistent volumes. Cert-manager certificates are a nice-to-have.

I think the most used tool is velero, but it looks like the most prevalent use case is a full backup-restore of the full cluster. Since I am using flux, I am more concerned with restoring persistent volume data, either on a case-by-case basis, or all of them altogether.

I have not yet managed to convince velero (with restic) to back up just the persistent volumes, I seem to need to include the pods and then the persistent volumes are discovered...

At any rate, I am curious about what others are doing for backing up non-flux-managed objects in their Kubernetes deployments.

Thanks!

[boedy]

@rombert A couple of months have passed since your post. Have you managed to find a workable solution?

I'm also trying to figure out how to best combine Flux with Velero. Specifically how to restore a cluster running stateful applications in case of disaster recovery.

[rombert]

@boedy - I did not make progress yet unfortunately. I've recently moved all my SQL workloads ( well, a couple of MariaDB instance ) to a MariaDB instance hosted on a VM, which already has a backup process in place. I don't have a good solution for persistent volumes though.

[Nils98Ar]

@rombert Any progress?

[rombert]

Nope, I did not look to much into it. Right now I am manually taking snapshots of the persistent volumes manually, by executing commands directly on the storage server, while the workloads are scaled down.

[juliafmorgado]

@rombert - If you're still exploring options you might find Veeam Kasten a useful alternative (not open source). Kasten is purpose-built for Kubernetes data management, offering seamless backup, restore, and mobility capabilities, including for persistent volumes and stateful applications. Additionally, Kasten integrates with open-source projects like Kanister, which provides a framework for data operations, enabling more advanced workflows such as custom data transformations during backup and restore. This combination could give you more control over selective data protection and recovery in Kubernetes environments, tailored to your needs. It might be worth exploring!

[rombert]

Thanks for the node @juliafmorgado . Looking at Kasten I noted that

Currently, on-premises Veeam Kasten deployments are only supported if storage is provisioned via the Container Storage Interface (CSI).

https://docs.kasten.io/latest/restrictions.html#on-premises-storage-providers

That is unfortunately not my case right now. But I'll keep a lookout for Kasten when I'm getting back to this topic.

[svengreb]

One popular OSS project (also used in commercial products like OpenShift) is VolSync. It is based on the concept of "movers" that are baked by large and stable projects like restic, rclone (used by restic), syncthing, and good ol' rsync.
It is based on CRDs where a backup is created with a ReplicationSource resource which can be restored with into a new volume with ReplicationDestination resource. The backup target is simply the name of the volume where, based on the used mover, some special configurations are possible, e.g. selecting only specific paths (sadly not yet possible with restic but other movers like rclone).

I used VolSync for all of my private clusters and the plan is to introduce it to clusters at my job too. Flux is used for all clusters already and the combo of both works like a charm 😄

Note that only the restic mover is a pure backup solution while all other movers can be used for backups, but are mainly used to continuously synchronize data between volumes, within the same cluster or across clusters.

[svengreb]

Another project is k8up which I also consider to test and maybe it even fits better for backup-only use cases while the basic way how the CRDs works are similar, but k8up comes with some additional features like "Pre-Backup Pods" that allow to run custom logic before a backups starts like creating database dump to ensure data consistency.

[svengreb]

Some more results from my research: stash. It's build on top of a freemium model where most features for homelab and private users are included in the community edition. It comes with one (in my opinion) great features that are (sadly) still missing (but planned) in VolSync and k8up: pattern to include/exclude specific files and directories!
I'll also give it a try since it looks like an advanced project that should be around in the future due to the (hopefully) financial base plan, even though k8up is also a CNCF "sandbox" project which is great for a possibly stable financial support in the future through CNCF.

[svengreb]

Update regarding stash

As of 2024-01-01 the "community" edition is not available anymore, but they "forgot" to update their GitHub repository README. So there goes another OSS project into the lawyer hell that stands on top of first-class OSS projects like restic, but turn into money grabbers as soon as their project gets larger. Welcome to the new reality where any large OSS project should better update their license to prevent any of these companies to never be able to use their code again for all times. Sorry for the rage, it's just a pure shame for humanity that individual maintainers of essential projects (restic, curl, etc.) are left behind while others suck on the money cow.

[rombert]

Thanks for the detailed info @svengreb . I'm not looking into this topic right now but your notes are very valuable.