Prerequisites for following along: the kubectl tool, the fluxcli tool, a Kubernetes cluster with FluxCD installed, and at least 2 Kustomizations.
A foreword on GitOps - a portmanteau of the words Git, the version control tool, and Operations - which describes a design paradigm by which a system is managed by looking at a Git repository as a source of truth. To make changes in the system, this is first reflected in the git repository, before being applied by a tool.
Some reasons for choosing this as a deployment mechanism are:

  • a single source of truth means it’s consistent and reproducible
  • eliminates configuration drift: if someone changes something manually, the operator service puts it back to exactly how it is in git next time it runs
  • version history: you can look back through the commits and see how something has evolved over time
  • rollback: equally, if something goes wrong, previous commits contain all the config to which you can simply roll back
    One such tool, FluxCD, or Flux, is an open-source tool that continually checks and updates Kubernetes resources against a provided configuration in the git repository. This is referred to at the top level as a Kustomization. This could include any native Kubernetes object, such as a Deployment or Service, or a Custom Resource such as an ExternalSecret (or any thing else really, as is the nature of Custom Resource Definitions.) A commonly deployed object is the or FluxCD Custom Resource, a HelmRelease. This takes a YAML manifest with some details about a [Helm] chart and deploys it, which in turn creates and manages a set of Kubernetes resources, such as a simple app with a deployment and a service.
    This is what our example HelmRelease would look like when applied under the apps Kustomization:
apps # the top level Kustomization)
└── my-simple-app # a HelmRelease object
    └── simple-app # a Helm deployment of a Helm Chart
        ├── service.yaml # the Kubernetes Service object
        └── deployment.yaml # the Kubernetes Deployment object


My personal homelab setup is actually a small Kubernetes cluster (K3s, actually) which has Flux checking a GitHub repository and reconciling any changes, such as infrastructure changes, new apps, changes to images used, and more. I have multiple levels of Kustomizations:

  • a core Kustomization containing the absolute essentials to first hit the cluster - this contains the monitoring stack Prometheus, the storage management system Longhorn, the networking system metallb to provide my cluster with some inter-node and cluster networking - all very much basic blocks that are crucial to the cluster itself being operable
  • an infra[structure] Kustomization containing some augmentation services, such as cert-manager to provide SSL Certificates, external-dns to manage DNS records, external-secrets to allow the abstraction of sensitive information to a cloud service, and an oauth2-proxy to restrict my personal-only services to be access only from inside my network. In other words, all very useful services that add a lot of functionality, but not important enough to the running of the cluster that they were put into the core Kustomization. This Kustomization relies on the core Kustomization being applied successfully in order to start applying.
  • finally, an apps Kustomization. This depends on the previous Kustomizations both being applied successfully, and applies sequentially after core and infra. It deploys this website, for one, along with a Homer dashboard for quick access to some linked services.
    Let’s say I made a mistake (I know, right?), and I want to move something so it’s controlled by a different Kustomization? Well, I could just move it from one part of the git repo to the new place, but remember the chained dependencies I’ve created, requiring that core applies before infra does, and infra before apps? If I move a service up the “chain” of Kustomizations, e.g. from apps to infra, Flux will see that everything it wants to make already exists, and it will fail to apply. If I move the service down the chain, e.g. from core to infra, Flux will apply core, deleting the app*, and then create it in the infra Kustomization.
*This only happens if the Prune option is enabled at the top level - see below for more info⌗

But what if it’s a critical or stateful service, and I absolutely can’t have it being deleted by Flux? There’s git got to be a better way… And there is! This method leverages the fact that Prune is optional in the Kustomization, as well as a key label in the created resource that denotes ownership.

CORRECTION: So, I had a suspicion but had to confirm this first: you can actually ignore all the Prune commands for the purposes of moving a resource out of a Kustomization. The reason being that the crucial element in Flux’s control of a resource is the label, and we can change this as described below. As long as the label is changed before the changes are merged, Flux will cease to control that resource, and so will not remove it if/when the Kustomization is removed. Many Bothans died to bring us this information.

It is worth noting that if you don’t pause the Kustomizations, then as soon as you change the label on the resource, Flux will try to bring up another resource as it believes it no longer has a version of it. Equally, the new Kustomization will immediately detect it, but consider it to be out of config, and promptly delete it.

Just to touch on Prune - which is named so because having it enabled will “prune” anything that’s not in the current configuration - you don’t have to have this enabled all the time. However, leaving it off and having Flux in create-only mode means you have to clean up any old created resources by hand, which can get messy and/or time-consuming. It’s also teeechnically a violation of the GitOps principles, as it means that what’s in git is not being mirrored by what’s deployed.
I’ll walk through an example where I move an InfluxDB HelmRelease from the apps to the infra Kustomization.

1. Suspend continuous deployment⌗

So, first things first, let’s pause the process both Kustomizations that we’re working with - otherwise as soon as we make a change, Flux will change it straight back to what’s in git.

❯ flux suspend kustomization infra
► suspending kustomization infra in flux-system namespace
✔ kustomization suspended

# N.B.: ks is short for kustomization!

❯ flux suspend ks apps
► suspending kustomization apps in flux-system namespace
✔ kustomization suspended

2. Turn off Prune, if it’s enabled You can disregard this on the basis of the above corrective Edit⌗

Let’s edit the Kustomization directly, but you could also use a patch file.

❯ kubectl edit kustomization apps
...
spec:
  prune: true --> false
...

3. Change ownership of the object you’re moving⌗

Let’s edit the HelmRelease directly again, altbough - again - you could use a patch file for this, and it would certainly save time if you’re doing it a lot. Although, if you were, that would raise some questions about why you’re doing it a lot, right?
Running kubectl edit helmrelease influxdb opens the server-side version of this object and displays the metadata and specification. We’re interested in the first label we can see here, kustomize.toolkit.fluxcd.io/name: apps.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  generation: 6
  labels:
    kustomize.toolkit.fluxcd.io/name: apps
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: influxdb
  namespace: assistant


I can just change this from apps to infra, like this:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  generation: 6
  labels:
    kustomize.toolkit.fluxcd.io/name: infra
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: influxdb
  namespace: assistant


Now I’ll save and close this editor and head back to the PR to merge my changes.

4. Make the necessary changes in git⌗

If you made a merge request or pull request, well done! You’re better than most of us. Everyone else can just commit straight to prod branch like the degenerates we are. Either way, as long as it’s in git and Flux has the latest code, it doesn’t matter. As long as the deployment controller is paused, this won’t go to the cluster until we manually resume things.

5.Resume the Kustomizations⌗

Now that we’ve done the the tweaks to the resource to change the Kustomization that owns the HelmRelease (according to the label), we can resume these without fear that anything will break.

❯ flux resume ks apps
► resuming kustomization apps in flux-system namespace
✔ kustomization resumed
◎ waiting for Kustomization reconciliation
✔ Kustomization apps reconciliation completed
✔ applied revision main@sha1:be6960451c8f928f68ab1b8c8c09e6ac6cdd0a48
...
► resuming kustomization infra in flux-system namespace
✔ kustomization resumed
◎ waiting for Kustomization reconciliation
✔ Kustomization infra reconciliation completed
✔ applied revision main@sha1:be6960451c8f928f68ab1b8c8c09e6ac6cdd0a48

6. Watch as everything works flawlessly⌗

That’s literally it! Congrats, you’ve moved a resource between Kustomizations.

7. Turn Prune back on⌗

If you turned Prune off in the git repo, don’t forget to turn it back on so that your cluster is perfectly GitOps-y. Otherwise, it’ll set itself back next time it reconciles.