Backlog and Workers: Two Knobs on Two Layers

Working as a Customer Escalation Engineer on Linkerd, I get requests from companies with all kinds of setups, traffic, and workloads. Recently, several customers asked about connection timeouts they couldn't explain, and one wondered why Linkerd's CPU annotation didn't match real CPU usage. At first, these seemed like separate problems, but they actually have the same root cause: how the proxy queues incoming connections and how many workers are available to handle them.

If you run into problems in this area, what you’re likely to see is simply dropped connections. During a sudden traffic burst or a period of increased network latency, proxies may start logging connect timed out after 1s, but the server-side metrics remain unchanged – but every request to a meshed pod is measured and labeled, so how can dropped traffic not show up in the proxy metrics?

Sometimes the network itself is the problem, but in many cases what’s happening is that the connection is delayed in a part of the path where our usual metrics simply can’t see what’s happening. This article looks at two places where that can happen: the TCP listener backlog, where incoming connections wait to be accepted, and the proxy runtime workers, which process them once they are accepted. We’ll look at how both work, which settings control them, and how to determine which one is actually limiting your workload.

The TCP listener backlog: the kernel's queue in front of the Linkerd proxy

Before a connection reaches the Linkerd proxy, it first goes through the kernel. Depending on the phase, the kernel puts connections into two queues:

• the SYN queue, which holds half-open connections that are still waiting for the client’s ACK that’s part of TCP connection startup
• the accept queue, which holds fully established connections waiting for the application to process them. This is known as the TCP listener backlog.

If the accept queue is full, the kernel silently drops packets. This causes a connect timed out after 1s error on the client side. 

You won't see this in the server's proxy metrics because the proxy never saw the connection. The kernel tracks these events with two counters: TcpExtListenOverflows (when a completed handshake finds the accept queue full) and TcpExtListenDrops (all drops on listening sockets). You can check the queues and counters from inside the pod:

kubectl debug -it -n simple-app simple-app-v1-6bf978ccb4-m2m2q \ --image=nicolaka/netshoot --target=linkerd-proxy -- \ sh -c 'ss -ltn; nstat -az TcpExtListenOverflows TcpExtListenDrops'
State  Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0      128          0.0.0.0:4143      0.0.0.0:*
LISTEN 0      128          0.0.0.0:4191      0.0.0.0:*
LISTEN 0      128        127.0.0.1:4140      0.0.0.0:*
LISTEN 0      4096               *:5678            *:*
#kernel
TcpExtListenOverflows           0                  0.0
TcpExtListenDrops               0                  0.0

A quick note: the three sockets with a TCP backlog of 128 belong to the proxy. Port 4143 is the inbound listener, 4140 is the outbound one, and 4191 is the admin server. The application itself listens on 5678 and requests 4096 slots. But once the pod is meshed, iptables redirects all inbound traffic to the proxy's port 4143, so the queue in front of your service is now the proxy's 128. Even if your app sets its backlog to 4096, the proxy's setting matters. Both TcpExtListenOverflows and TcpExtListenDrops counters at zero means things are healthy; we'll see them change soon.

The muscles of the Linkerd proxy: the workers

After the kernel hands over a connection, the proxy takes over. The linkerd2-proxy's uses Tokio worker threads to actually do work. These are threads that accept connections from the TCP backlog, handle TLS handshakes, route requests, and more. The runtime can be single-threaded or multi-threaded, and the proxy announces which mode it uses in its first log lines.

kubectl logs -n simple-app    simple-app-v4-dbb8698ff-m9wlw -c linkerd-proxy
[     0.000040s]  INFO ThreadId(01) linkerd2_proxy: release 2.19.5 (ec0c3ea) by Buoyant, Inc. on 2026-03-07T01:32:27Z
[     0.001402s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime

or

kubectl logs -n simple-app    simple-app-v3-chb7148fd-f9gld -c linkerd-proxy
[     0.000040s]  INFO ThreadId(01) linkerd2_proxy: release 2.19.5 (ac2d6fb) by Buoyant, Inc. on 2026-03-07T01:32:27Z
[     0.001402s]  INFO ThreadId(01) linkerd2_proxy::rt: Using multi-threaded proxy runtime cores=3

In a multi-threaded runtime, each worker has its own local run queue with a fixed capacity of 256 tasks, and all workers can also use a shared queue. Two mechanisms help keep the workers balanced:

• If a worker's local queue fills up, it spills half of it into the shared queue for others to pick up.
• If a worker runs out of work, it steals tasks from a busier sibling.

Tuning the two layers: backlog size and worker count

Start with the workers, since their configuration has an important detail that's easy to overlook. The proxy sets its worker count at startup based on the CPU configuration it finds. If it doesn't find any, it uses proxy.runtime.workers.minimum, which defaults to 1. So, even on a 64-core node, if no limits are set, the proxy will use only one worker and ignore the other 63 cores. This setup might look generous, but it isn't.

If you set config.linkerd.io/proxy-cpu-limit: "2" you'll get two workers and a resources.limits.cpu will be set on the proxy container.

On clusters with different node sizes, though, application pods may be spread across nodes with wildly different available resources, so trying to assign specific CPU counts can be a non-starter. To help, the Linkerd team created the proxy.runtime.workers.maximumCPURatio Helm Chart value and the config.linkerd.io/proxy-cpu-ratio-limit annotation. These let you size the worker pool as a fraction of the node's cores. Suppose your EKS cluster has two node pools: one uses m5.xlarge instances (4 vCPUs) for general workloads, and the other uses m5.4xlarge instances (16 vCPUs) for heavier services. Since the same Deployment could run on either pool, setting a fixed proxy-cpu-limit of "2" would be too high for the smaller nodes and would not use most of the resources on the larger nodes. To handle this, add a ratio annotation to the workload or its namespace:

metadata:
 annotations:
   config.linkerd.io/proxy-cpu-ratio-limit: "0.25"

With this setup, the proxy sets its worker pool size at startup based on the node it runs on. For example, a pod on an m5.xlarge gets 1 worker (0.25 × 4), while the same pod on an m5.4xlarge gets 4 workers (0.25 × 16).

The backlog is simpler: it's 128 for both inbound and outbound unless you change it. Since BEL 2.20, you can configure the inbound/outbound listener's backlog with an environment variable, which you can set for the whole mesh from the chart:

helm upgrade linkerd-enterprise-control-plane \
  linkerd-buoyant/linkerd-enterprise-control-plane \
  --namespace linkerd \
  --reuse-values \
  --set-json 'proxy.additionalEnv=[
{"name":"LINKERD2_PROXY_INBOUND_TCP_LISTEN_BACKLOG","value":"1024"},
{"name":"LINKERD2_PROXY_OUTBOUND_TCP_LISTEN_BACKLOG","value":"1024"}
  ]'

One important thing to note is that LINKERD2_PROXY_INBOUND_TCP_LISTEN_BACKLOG and LINKERD2_PROXY_OUTBOUND_TCP_LISTEN_BACKLOG each control different listeners, and they work independently. If you only set one, the other will stay at its default value of 128. For example, later we set only LINKERD2_PROXY_INBOUND_TCP_LISTEN_BACKLOG. This increased the backlog for port 4143 (the inbound proxy path) and port 4191 (the inbound admin path), but port 4140, which handles outbound traffic, still used the default of 128. 

Overflow, busyness, and more: the metrics the proxy exposes

As mentioned earlier, every request to a meshed pod is measured and labeled. For workers, the proxy exports its runtime state in the tokio_rt metric family, which you can read directly from any meshed pod:

linkerd diagnostics proxy-metrics -n simple-app po/simple-app-v1-6bf978ccb4-m2m2q | grep tokio_rt

 linkerd diagnostics proxy-metrics -n simple-app po/simple-app-v1-6bf978ccb4-m2m2q | grep tokio_rt
tokio_rt_workers 1
tokio_rt_park_total 107542
tokio_rt_noop_total 12538
tokio_rt_steal_total 0
tokio_rt_steal_operations_total 0
tokio_rt_remote_schedule_total 4
tokio_rt_local_schedule_total 479172
tokio_rt_overflow_total 0
tokio_rt_polls_total 479175
tokio_rt_busy_seconds_total 15.114254765000024
tokio_rt_injection_queue_depth 0
tokio_rt_local_queue_depth 0
tokio_rt_budget_forced_yield_total 0
tokio_rt_io_driver_ready_total 89746...

Four of them are particularly important to understand what’s happening:

• tokio_rt_workers reports the worker count, and should match the cores= value from the startup log line. 
• tokio_rt_busy_seconds_total accumulates the time workers spent doing work rather than waiting for it. 
• tokio_rt_overflow_total counts the spill mechanism from earlier: each time a worker's 256-slot local queue fills and half of it moves to the shared queue, the counter goes up by one. 
• tokio_rt_steal_total tracks the opposite move: an idle worker taking tasks from a busy one.

Example

Now let's see this in action. We'll run three configurations against the same load and look at the results on the Grafana dashboard. This way, you can see where each symptom appears and which layer stays quiet.

The setup

A three-node k3d cluster running BEL 2.20.2 with two workloads:

• server: a meshed hashicorp/http-echo answering ok on port 5678, pinned to one node using a nodeSelector.
• client: an unmeshed Fortio instance, pinned to the other node using a nodeSelector.

Every case gets the same open-loop load of 20,000 requests per second across 600 threads (which will show different throughput), and each request opens a new TCP connection (so each request is one `accept()` on the proxy's inbound listener), over 60 seconds, via the following:

fortio load -qps 20000 -c 600 -uniform -jitter -nocatchup -sequential-warmup -keepalive=false -allow-initial-errors -timeout 500ms -t 60s http://server/

The overall topology is the following:

Case 1: default backlog, one worker

No configuration at all. The injector sets LINKERD2_PROXY_CORES=1, which makes the proxy using a single threaded worker, and the listener is created with the platform default backlog:

kubectl -n cpu-demo debug -it server-3b9a1c5e6d-2tk4h --image=alpine -- sh -c 'apk -q add iproute2 && ss -ltn && nstat -az TcpExtListenOverflows TcpExtListenDrops'
State  Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0      128        127.0.0.1:4140      0.0.0.0:*
LISTEN 0      128          0.0.0.0:4143      0.0.0.0:*
LISTEN 0      128          0.0.0.0:4191      0.0.0.0:*
#kernel
TcpExtListenOverflows           0                  0.0
TcpExtListenDrops               0                  0.0

Once the test completes, the bottleneck is clear: the worker. It accepts connections as fast as it can, around 17k per second, while the remaining work queues inside the runtime. The worker was effectively saturated, spending 99% of its time busy, and roughly 450 queued tasks account for most of the 15 ms of latency, since http-echo responds in microseconds.

The kernel queue in front of it mostly keeps up, but it overflows whenever arrivals bunch up while the worker is saturated. Each overflow means the kernel dropped a SYN before a connection was established. The client sees that as a connect timeout (688 requests failed with dial tcp 10.43.171.124:80: i/o timeout), while the proxy never sees it at all and reports a 100% success rate

‍Case 2: inbound backlog 1024, one worker

The only change is a deeper listen backlog, set through the Helm values so the injector adds it to every proxy:

proxy:
 additionalEnv:
   - name: LINKERD2_PROXY_INBOUND_TCP_LISTEN_BACKLOG
     value: "1024"

Which results in the following:

kubectl -n cpu-demo debug -it server-81c4e2a7f0-m9vwb --image=alpine -- sh -c 'apk -q add iproute2 && ss -ltn && nstat -az TcpExtListenOverflows TcpExtListenDrops'
State  Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0      128        127.0.0.1:4140      0.0.0.0:*
LISTEN 0      1024         0.0.0.0:4191      0.0.0.0:*
LISTEN 0      1024         0.0.0.0:4143      0.0.0.0:*
#kernel
TcpExtListenOverflows           0                  0.0
TcpExtListenDrops               0                  0.0

Even in this case, the worker is saturated, spending 99% of its time busy with the same 450-task runtime queue and a 15 ms p50 latency. However, this time, the larger backlog provides more buffer and absorbs the bursts. Connections that would previously have been dropped now wait their turn, so nothing is lost even while the single worker is busy. This is confirmed by the same throughput as before.

Case 3: default backlog, four workers

The backlog goes back to the default and the server deployment gets one annotation:

config.linkerd.io/proxy-cpu-limit: "4"

The injector turns that into a CPU limit of 4 on the proxy container and LINKERD2_PROXY_CORES_MAX=4, which results in the proxy using 4 workers.

Once the test is completed, we will see a drastic increase in throughput with 20k, with the worker having mode headroom being busy roughly 70%, a drop in latency: p50 0.5 ms and p99 1 ms, and a queue not exceeding 15 tasks.

Side by side

‍

Conclusion

The two layers solve different problems. The accept queue acts as a buffer; it absorbs bursts and, when full, drops SYNs that the proxy never sees. The workers provide capacity; they control how fast the queue drains. Case 2 shows that a deeper backlog can hide a burst but doesn't make things faster. Case 3 shows that more capacity makes the drops disappear, even with the default backlog of 128 and 2.8 proxy CPU cores. Which one you need depends on your arrival pattern, not your throughput.

FAQ

Why is Linkerd's proxy success rate 100% while clients report connection timeouts?

When the kernel's accept queue is full, it drops the SYN before the proxy ever sees the connection. The client logs a connect timeout, but the proxy never measured that request, so its success rate metric stays at 100%. Check TcpExtListenOverflows instead. 

Should you raise the backlog or add workers to fix connection timeouts?

It depends on the problem. A deeper backlog buffers short traffic bursts so connections wait instead of getting dropped. More workers add sustained capacity so the queue drains faster. Use backlog for bursts and workers for a sustained ceiling. 

Why does a Linkerd proxy use only 1 core on a 64-core node?

Without an explicit CPU limit, the proxy falls back to proxy.runtime.workers.minimum, which defaults to 1 worker no matter how many cores the node has. Set config.linkerd.io/proxy-cpu-limit or proxy-cpu-ratio-limit to use more of the node.  

What is the TCP listener backlog in a meshed Kubernetes pod?

It's the kernel's accept queue in front of the proxy's inbound listener, where established connections wait to be processed. Once a pod is meshed, iptables redirects traffic to the proxy, so its backlog (default 128) governs the pod, not the app's own setting. 

How do you size Linkerd proxy workers across nodes with different CPU counts?

Use the config.linkerd.io/proxy-cpu-ratio-limit annotation instead of a fixed CPU limit. It sets worker count as a fraction of each node's cores, so a 0.25 ratio gives 1 worker on a 4 vCPU node and 4 workers on a 16 vCPU node.