Kubernetes 1.24: Avoid Collisions Assigning IP Addresses to Services

Author: Antonio Ojea (Red Hat)

In Kubernetes, Services are an abstract way to expose an application running on a set of Pods. Services can have a cluster-scoped virtual IP address (using a Service of type: ClusterIP). Clients can connect using that virtual IP address, and Kubernetes then load-balances traffic to that Service across the different backing Pods.

How Service ClusterIPs are allocated?

A Service ClusterIP can be assigned:

dynamically

the cluster's control plane automatically picks a free IP address from within the configured IP range for type: ClusterIP Services.

statically

you specify an IP address of your choice, from within the configured IP range for Services.

Across your whole cluster, every Service ClusterIP must be unique. Trying to create a Service with a specific ClusterIP that has already been allocated will return an error.

Why do you need to reserve Service Cluster IPs?

Sometimes you may want to have Services running in well-known IP addresses, so other components and users in the cluster can use them.

The best example is the DNS Service for the cluster. Some Kubernetes installers assign the 10th address from the Service IP range to the DNS service. Assuming you configured your cluster with Service IP range 10.96.0.0/16 and you want your DNS Service IP to be 10.96.0.10, you'd have to create a Service like this:

apiVersion: v1
kind: Service
metadata:
 labels:
 k8s-app: kube-dns
 kubernetes.io/cluster-service: "true"
 kubernetes.io/name: CoreDNS
 name: kube-dns
 namespace: kube-system
spec:
 clusterIP: 10.96.0.10
 ports:
 - name: dns
 port: 53
 protocol: UDP
 targetPort: 53
 - name: dns-tcp
 port: 53
 protocol: TCP
 targetPort: 53
 selector:
 k8s-app: kube-dns
 type: ClusterIP

but as I explained before, the IP address 10.96.0.10 has not been reserved; if other Services are created before or in parallel with dynamic allocation, there is a chance they can allocate this IP, hence, you will not be able to create the DNS Service because it will fail with a conflict error.

How can you avoid Service ClusterIP conflicts?

In Kubernetes 1.24, you can enable a new feature gate ServiceIPStaticSubrange. Turning this on allows you to use a different IP allocation strategy for Services, reducing the risk of collision.

The ClusterIP range will be divided, based on the formula min(max(16, cidrSize / 16), 256), described as never less than 16 or more than 256 with a graduated step between them.

Dynamic IP assignment will use the upper band by default, once this has been exhausted it will use the lower range. This will allow users to use static allocations on the lower band with a low risk of collision.

Examples:

Service IP CIDR block: 10.96.0.0/24

Range Size: 2⁸ - 2 = 254
Band Offset: min(max(16, 256/16), 256) = min(16, 256) = 16
Static band start: 10.96.0.1
Static band end: 10.96.0.16
Range end: 10.96.0.254

Service IP CIDR block: 10.96.0.0/20

Range Size: 2¹² - 2 = 4094
Band Offset: min(max(16, 4096/16), 256) = min(256, 256) = 256
Static band start: 10.96.0.1
Static band end: 10.96.1.0
Range end: 10.96.15.254

Service IP CIDR block: 10.96.0.0/16

Range Size: 2¹⁶ - 2 = 65534
Band Offset: min(max(16, 65536/16), 256) = min(4096, 256) = 256
Static band start: 10.96.0.1
Static band ends: 10.96.1.0
Range end: 10.96.255.254

Get involved with SIG Network

The current SIG-Network KEPs and issues on GitHub illustrate the SIG’s areas of emphasis.

SIG Network meetings are a friendly, welcoming venue for you to connect with the community and share your ideas. Looking forward to hearing from you!