Kubernetes nodes cannot be provisioned any more in subnets tagged with sigs.k8s.io/cluster-api-provider-aws/association: secondary

[cellux]

/kind bug

What steps did you take and what happened:

Upgraded the CAPA provider to v2.7.1 and then tried to upgrade one of my AWS clusters to a newer Kubernetes version.

During the rolling update of MachineDeployments, CAPA v2.7.1 rejected creation of new EC2 instances saying subnet XXXX belongs to a secondary CIDR block which won't be used to create instances.

What did you expect to happen:

The new EC2 instances are provisioned - as this used to happen before the upgrade to v2.7.1.

Anything else you would like to add:

Downgrading CAPA provider to v2.6.1 resolved the issue.

The problem might be around this code block in pkg/cloud/services/ec2/instance.go:

			tags := converters.TagsToMap(subnet.Tags)
			if tags[infrav1.NameAWSSubnetAssociation] == infrav1.SecondarySubnetTagValue {
				errMessage += fmt.Sprintf(" subnet %q belongs to a secondary CIDR block which won't be used to create instances.", *subnet.SubnetId)
				continue
			}

Environment:

• Cluster-api-provider-aws version: v2.7.1
• Kubernetes version: (use kubectl version): v1.29.10-eks-7f9249a
• OS (e.g. from /etc/os-release): Ubuntu 20.04.6 LTS

We use four private subnets in AWS which are pre-provisioned by our IT team:

• two for Transit and NAT gateways, VPC endpoints, etc. - these are connected to the company network
• two for Kubernetes nodes and the pod network - nonrouted subnets sliced from 100.64.0.0/16

We followed the docs at https://cluster-api-aws.sigs.k8s.io/topics/eks/pod-networking#unmanaged-static-vpc:

• custom VPC CNI configuration
• secondary CIDR subnets tagged with sigs.k8s.io/cluster-api-provider-aws/association=secondary

We do not want to use the first two subnets for Kubernetes nodes as those are pretty small and could be easily exhausted when we scale out.

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.