Now that the hub-spoke network is provisioned, the next step in the AKS baseline reference implementation is preparing what your AKS cluster should be bootstrapped with.
Container registries often have a lifecycle that extends beyond the scope of a single cluster. They can be scoped broadly at organizational or business unit levels, or can be scoped at workload levels, but usually are not directly tied to the lifecycle of any specific cluster instance. For example, you may do blue/green cluster instance deployments, both using the same container registry. Even though clusters came and went, the registry stays intact.
- Azure Container Registry (ACR) is deployed, and exposed as a private endpoint.
- ACR is populated with images your cluster will need as part of its bootstrapping process.
- Log Analytics is deployed and ACR platform logging is configured. This workspace will be used by your cluster as well.
The role of this pre-existing ACR instance is made more prominant when we think about cluster bootstrapping. That is the process that happens after Azure resource deployment of the cluster, but before your first workload lands in the cluster. The cluster will be bootstrapped immedately and automatically after resource deployment, which means you'll need ACR in place to act as your official OCI artifact repository for required images and Helm charts used in that bootstrapping process.
We'll be bootstrapping this cluster with the Flux GitOps agent as installed as an AKS extension. This specific choice does not imply that Flux, or GitOps in general, is the only approach to bootstrapping. Consider your organizational familiarity and acceptance of tooling like this and decide if cluster bootstrapping should be performed with GitOps or via your deployment pipelines. If you are running a fleet of clusters, a GitOps approach is highly recommended for uniformity and easier governance. When running only a few clusters, GitOps might be seen as "too much" and you might instead opt for integrating that process into one or more deployment pipelines to ensure bootstrapping takes place. No matter which way you go, you'll need your bootstrapping artifacts ready to go before you start your cluster deployment so that you can minimize the time between cluster deployment and bootstrapping. Using the Flux AKS extension allows your cluster to start already bootstrapped and sets you up with a solid management foundation going forward.
In addition to ACR being deployed to support bootstrapping, this is where any other resources that are considered not tied to the lifecycle of an individual cluster is deployed. ACR is one example as talked about above. Another example could be an AKS Backup Vault and backup artifacts storage account which likely would exist prior to and after any individual AKS cluster's existance. When designing your pipelines, ensure to isolate components by their lifecycle watch for singletons in an architecture. These are typically resources like regional logging sinks, supporting global routing infrastructure, etc. This is in contrast to potentially transiently/replaceable components, like the AKS cluster itself. This implemention does not represent a complete seperation of stamp vs regional resources, but is fairly close. Deviations are strickly for ease of deployment in this walkthrough instead of as examples of guidance.
-
Create the AKS cluster resource group.
📖 The app team working on behalf of business unit 0001 (BU001) is looking to create an AKS cluster of the app they are creating (Application ID: 0008). They have worked with the organization's networking team and have been provisioned a spoke network in which to lay their cluster and network-aware external resources into (such as Application Gateway). They took that information and added it to their
acr-stamp.json,cluster-stamp.json, andazuredeploy.parameters.prod.jsonfiles.They create this resource group to be the parent group for the application.
# [This takes less than one minute.] az group create --name rg-bu0001a0008 --location eastus2 -
Get the AKS cluster spoke virtual network resource ID.
📖 The app team will be deploying to a spoke virtual network, that was already provisioned by the network team.
export RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE=$(az deployment group show -g rg-enterprise-networking-spokes -n spoke-BU0001A0008 --query properties.outputs.clusterVnetResourceId.value -o tsv) echo RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE: $RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE
-
Deploy the container registry and non-stamp resources template.
# [This takes about four minutes.] az deployment group create -g rg-bu0001a0008 -f acr-stamp.bicep -p targetVnetResourceId=${RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE} location=eastus2
-
Import cluster management images to your container registry.
Public container registries are subject to faults such as outages or request throttling. Interruptions like these can be crippling for a system that needs to pull an image right now. To minimize the risks of using public registries, store all applicable container images in a registry that you control, such as the SLA-backed Azure Container Registry.
# Get your ACR instance name export ACR_NAME_AKS_BASELINE=$(az deployment group show -g rg-bu0001a0008 -n acr-stamp --query properties.outputs.containerRegistryName.value -o tsv) echo ACR_NAME_AKS_BASELINE: $ACR_NAME_AKS_BASELINE # Import core image(s) hosted in public container registries to be used during bootstrapping az acr import --source ghcr.io/kubereboot/kured:1.12.0 -n $ACR_NAME_AKS_BASELINE
In this walkthrough, there is only one image that is included in the bootstrapping process. It's included as an reference for this process. Your choice to use Kubernetes Reboot Daemon (Kured) or any other images, including helm charts, as part of your bootstrapping is yours to make.
-
Update bootstrapping manifests to pull from your ACR instance. Optional. Fork required.
Your cluster will immedately begin processing the manifests in
cluster-manifests/due to the bootstrapping configuration that will be applied to it. So, before you deploy the cluster now would be the right time push the following changes to your fork so that it will use your files instead of the files found in the original mspnp repo which point to public container registries:- update the one
image:value inkured.yamlto use your container registry instead of a public container registry. See the comment in the file for instructions (or you can simply run the command below.)
⚠️ Without updating these files and using your own fork, you will be deploying your cluster such that it takes dependencies on public container registries. This is generally okay for exploratory/testing, but not suitable for production. Before going to production, ensure all image references you bring to your cluster are from your container registry (link imported in the prior step) or another that you feel confident relying on.sed -i "s:ghcr.io:${ACR_NAME_AKS_BASELINE}.azurecr.io:" ./cluster-manifests/cluster-baseline-settings/kured.yamlNote, that if you are on macOS, you might need to use the following command instead:
sed -i '' 's:ghcr.io:'"${ACR_NAME_AKS_BASELINE}"'.azurecr.io:g' ./cluster-manifests/cluster-baseline-settings/kured.yaml
Now commit changes to repository.
git commit -a -m "Update image source to use my ACR instance instead of a public container registry." git push - update the one
# run the saveenv.sh script at any time to save environment variables created above to aks_baseline.env
./saveenv.sh
# if your terminal session gets reset, you can source the file to reload the environment variables
# source aks_baseline.env