Regression in behavior when setting load balancer source ranges for a Service

[elmiko]

What happened:

When setting both .spec.loadBalancerSourceRanges and the service.beta.kubernetes.io/load-balancer-source-ranges annotation on a Service object, the CCM will not reconcile the Service nor will it allow for its deletion. The following error is present:

E0612 16:22:37.193349       1 azure_loadbalancer.go:2952] "Failed to parse access control configuration for service" err="cannot set both spec.LoadBalancerSourceRanges and service annotation service.beta.kubernetes.io/azure-allowed-ip-ranges" logger="reconcileSecurityGroup" cluster="ci-op-mv04309s-04a70-6jszn" service="openshift-ingress/router-sourcerangesstatus" load-balancer="ci-op-mv04309s-04a70-6jszn" delete-lb=true

What you expected to happen:

Expected the Service to continue to be reconciled even if one of the source ranges is ignored.

How to reproduce it (as minimally and precisely as possible):

1. Create a service with both service.beta.kubernetes.io/load-balancer-source-ranges annotation and spec.LoadBalancerSourceRanges

Anything else we need to know?:

We (Red Hat) discovered this regression in our automated testing, and while the user behavior of setting both ranges may not be supported by the CCM, the error when reconciling makes it a possible upgrade blocker for users who may have (inadvertently or not) set both these range options.

There are 2 pull requests which appear to have combined to create this regression:

• Refine LoadBalancer service access control and consolidating security-rules #5164
• fix: allow space separated load balancer source ranges #5885

the effect of #5164 resulted in the following logic:

# If result is false then either not valid or no source ranges provided:
(spec.LoadBalancerSourceRanges OR service.beta.kubernetes.io/load-balancer-source-ranges) XOR (service.beta.kubernetes.io/azure-allowed-ip-ranges)

We do not have automation around the service.beta.kubernetes.io/azure-allowed-ip-ranges annotation, and as such we have not directly noticed this failure but we believe it exists.

when combined with #5885, the resulting logic becomes:

# If result is false then either not valid or no source ranges provided:
(spec.LoadBalancerSourceRanges) XOR (service.beta.kubernetes.io/load-balancer-source-ranges OR service.beta.kubernetes.io/azure-allowed-ip-ranges)

Related report in Red Hat bug tracking OCPBUGS-35757.

Environment:

This was tested on the upcoming release of OpenShift which uses Kubernetes version 1.30 and RHCOS 9.4 instances.

[elmiko]

cc @JoelSpeed @gcs278

[elmiko]

one thing that came up in discussion internally was whether we missed an announcement or release note about this change in behavior. is this just an organic regression or did we miss something about it?

[JoelSpeed]

@feiskyer @bridgetkromhout Can you please review this issue?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[JoelSpeed]

/remove-lifecycle stale

This is still an issue

[nilo19]

@zarvd could you help one this one? Do you think we should ignore spec.LoadBalancerSourceRanges if the annotation counterpart is set to support a smooth migration?