pickuma.
Infrastructure

What the kube-proxy IPVS removal means for you

Kubernetes plans to remove kube-proxy's ipvs mode in 1.40. Here's what that mode does, how to tell if you use it, and where to move.

6 min read

Kubernetes has put a version number on the end of one of the ways your cluster routes traffic. According to the Kubernetes blog post Scheduling the removal of the kube-proxy IPVS backend, kube-proxy’s ipvs backend has been deprecated since Kubernetes 1.35, and SIG Network now plans to remove it in Kubernetes 1.40. If your cluster runs kube-proxy in ipvs mode, 1.40 is the release you need to be ready for.

The background you need

A Kubernetes Service gives a set of pods one stable virtual IP address. Pods come and go, but clients keep talking to the same Service IP. The pods behind it at any moment are called endpoints.

Something on every node has to turn that virtual IP into the address of a real pod. On most clusters, that job belongs to kube-proxy. It is a program that runs on each node, watches the Kubernetes API for changes to Services and endpoints, and writes forwarding rules into the Linux kernel. kube-proxy does not move packets itself. The kernel does, by following the rules kube-proxy wrote.

The kernel has more than one system for holding those rules. Which one kube-proxy uses is called its mode or backend. On Linux there are three:

  • iptables: the long-standing Linux packet-filtering system, and the traditional kube-proxy default.
  • ipvs: IP Virtual Server, a load balancer built into the kernel. kube-proxy added it, starting in 2017 according to the post, as a faster alternative to iptables.
  • nftables: the newer Linux successor to iptables. According to Kubernetes, the nftables backend arrived in 1.29 and became GA (generally available, meaning stable and supported) in 1.33.

To see why ipvs existed, separate two kinds of speed. Data plane latency is how fast a new connection gets matched to a pod. Control plane latency is how fast kube-proxy can apply a change, such as a new pod or a deleted Service.

Picture a receptionist with a printed staff directory. The old iptables backend read the directory line by line for every visitor, so lookup time grew with the number of Services. That is what the post means by O(n). Worse, every time anyone changed desks, the old backend reprinted the whole directory. The post says that in very large clusters, kube-proxy would use 100% of one CPU just processing iptables updates. ipvs was like an indexed directory: a hash table lookup jumps straight to the right entry, and single entries get edited in place.

The analogy breaks in two places. First, ipvs cannot edit in bulk. The kernel IPVS interface has no batch updates, so a large wave of changes goes through one at a time. Second, iptables did not stay frozen. It has since learned to update only what changed, within the limits of its API.

One more term. Deprecated means a feature still works but prints a warning and is scheduled to go. Removed means it is gone, and the setting no longer works.

What actually changed

The deprecation itself is not new. ipvs has been deprecated since 1.35 and still runs today with a warning. What is new is a target release for removal, 1.40, and concrete migration guidance. In the post’s words: “All ipvs users should therefore be planning to migrate to either the nftables or iptables backends.”

The post gives several reasons ipvs is going:

  • It never fully matched Services. Kubernetes says the kernel IPVS API is a bad match for the Services API, and the backend never handled all Service edge cases correctly.
  • It cost more to maintain. It is architecturally different from the other two backends. The post also says its original authors moved on and new maintainers took a while to find.
  • It hit its own scaling limits. Without batch updates, ipvs could be slower than iptables when many changes happen at once. It also falls back on iptables for features beyond basic traffic distribution, so it inherits some iptables scaling problems.
  • It depends on iptables, which has a future problem on RHEL. The post cites Red Hat’s plan to remove the iptables API from Red Hat Enterprise Linux. That would make the ipvs backend unavailable there too.
  • Its schedulers mostly don’t help. IPVS offers several schedulers, which are strategies for picking an endpoint. But kube-proxy on each node doesn’t share scheduler state with other nodes. The post says even distributed-friendly options, such as the mh (Maglev hashing) scheduler, generally don’t beat the random selection that iptables and nftables use.

On performance, Kubernetes says nftables has very slightly better performance than ipvs. Its stated point is not that nftables is noticeably faster, only that it is not slower. It also describes iptables as much improved since 2017, though still not as good as ipvs.

What it means if you deploy and run apps

First, find out whether this affects you. On a kubeadm cluster, check the mode field in the kube-proxy ConfigMap. If you use a managed Kubernetes service, the provider usually sets this, so check its documentation or cluster settings. Some clusters use a networking plugin that replaces kube-proxy entirely. If yours does, this change does not apply directly to you. If you never deliberately chose ipvs, you are most likely not on it.

If you are on ipvs, pick a destination:

  • nftables is the recommendation if you run Kubernetes 1.36 or later and your nodes run Linux kernel 5.13 or later. Why 1.36 and not 1.33? The post says some performance and scaling fixes landed after 1.33 and were not backported. For kubeadm, the switch is changing mode: ipvs to mode: nftables in the kube-proxy ConfigMap and restarting the kube-proxy pods.
  • iptables is the fallback if your Kubernetes version or Linux distribution is too old for nftables. Kubernetes recommends rebooting each node after this change, not just restarting kube-proxy.

If you set ipvs-specific options, you will need to translate them to their nftables equivalents. If you picked a particular IPVS scheduler because you expected smarter balancing, the post suggests you were probably getting little benefit from it.

On RHEL, treat iptables as a stopgap. The post says the iptables backend has the same long-term support problem there as ipvs.

What to watch: KEP-5495, which the post says holds the full timeline and plan, and the release notes for 1.40. Also add a check to your upgrade checklist. Do not upgrade a cluster to 1.40 while kube-proxy is still set to ipvs.

What we don’t know yet

  • When 1.40 ships. The post names a version, not a calendar date.
  • Whether the plan holds. SIG Network says it plans to remove ipvs in 1.40. The post does not describe what could change that.
  • How managed providers handle it. The post does not say whether hosted Kubernetes services will migrate ipvs clusters automatically or leave it to you.
  • Which edge cases ipvs gets wrong. The post says it never handled all of them but does not list them, so you can’t easily tell whether your workloads hit any.
  • RHEL timing. The post cites Red Hat’s plan to remove the iptables API but gives no date.
  • Exact performance figures. The post’s latency comparison is a chart. The text only says nftables is not slower than ipvs.

FAQ

How do I know if my cluster uses ipvs mode?
On kubeadm clusters, look at the mode setting in the kube-proxy ConfigMap. On managed Kubernetes, check your provider's cluster settings or docs. If your cluster replaces kube-proxy with a networking plugin, this change does not apply directly.
Will switching to nftables make my apps slower?
According to Kubernetes, no. It says nftables has very slightly better performance than ipvs, and its main point is that nftables is not slower. Use Kubernetes 1.36 or later for the post-1.33 performance fixes.
Can I keep using ipvs for now?
Yes. It still works with a deprecation warning. Kubernetes plans to remove it in 1.40, so migrate before upgrading to that release.

Related tools

Some links above are affiliate links. We may earn a commission if you sign up. See our disclosure for details.

Related reading

See all Infrastructure articles →

Get the best tools, weekly

One email every Friday. No spam, unsubscribe anytime.