From 87 to 1,400 Requests a Second on a Home-Lab k3s Cluster

One k6 scenario, run after every change, and a table of what the wall was each time. Kubernetes doubled the ceiling. A connection setting, a cache and one more core per node did the rest, and the query rewrites that looked like the big win did not move the ceiling at all.
Author

Benedict Thekkel

Published

October 14, 2026

The workload is an existing Django REST + React + Celery application that I did not write. What I built is the platform under it and the performance work on it: a three-node k3s cluster on Proxmox VMs provisioned with Pulumi, a hand-written Helm chart, the load tests, and the backend changes below. It is a home lab with synthetic load, and the numbers should be read as that.

The method is the only thing here that generalises, so it comes first: run the same load test after every single change, and write down what saturated. Not what you expected to saturate. What did.

The setup

  • Cluster: k3s, one server and two agents, each a small VM. Pinned k3s version, verified by sha256.
  • Outside the cluster, on their own VMs: Postgres 16 (with a streaming replica added later), Redis, an S3-compatible object store with a private image registry, an nginx edge, Prometheus + Grafana, and a dedicated k6 load generator.
  • In the cluster: gunicorn web pods, a Celery worker and beat, a migrate hook Job. HPAs scale web 2 to 6 and worker 1 to 3. Pods run as non-root with a read-only root filesystem, asserted by tests on the rendered chart.
  • Ingress: Traefik as a hostNetwork DaemonSet, with k3s’s ServiceLB switched off.
  • Monitoring: kube-prometheus-stack with Prometheus in agent mode, remote-writing to the Prometheus outside the cluster, with alert rules unit-tested by promtool.

The test is a k6 breakpoint scenario: each iteration is one GET against one of five public read endpoints chosen at random, starting at 40 requests per second and adding 10 every minute until something gives.

The table

Every row is the same scenario. Dates are 2026-10-12 to 2026-10-14.

Step Ceiling (req/s) What saturated
Single app VM, 2 vCPU, no Kubernetes ~87 app VM CPU at 97%
k3s, 3 nodes x 2 vCPU, HPA ~172 database VM CPU at 95%
+ CONN_MAX_AGE=60 ~260 k3s nodes at 98%
+ two query rewrites ~260 k3s nodes at 98% (database fell 65% to 28%)
+ 30 s read cache >586 nothing (hit the scenario’s 600 cap)
Pushed to the breaking point ~990 k3s nodes at 98%
Nodes resized to 3 vCPU ~1,400 k3s nodes at 98%, database at 1%

The final run served 1,440,980 requests with zero failures, knee around 1,330 req/s, p95 49 ms.

Kubernetes moved the wall, it did not remove it

Going from one VM to three nodes doubled the ceiling, from about 87 to about 172. Then the wall was somewhere else entirely: the database VM, at 95% CPU, while the cluster had room.

This is the most common way a “we need Kubernetes” project disappoints. Horizontal scaling of the stateless tier is exactly what it is good at, and if the stateless tier was the bottleneck you get a real win. Then the bottleneck becomes whatever you did not scale, usually the database, and adding pods makes that worse, not better.

CONN_MAX_AGE: +51% for one line

Django’s default is CONN_MAX_AGE = 0: open a new database connection for every request and close it at the end. At low load that is invisible. At 170 requests a second it means 170 TCP handshakes, authentications and backend process setups a second, on the machine that was already the bottleneck.

DATABASES["default"]["CONN_MAX_AGE"] = 60

Ceiling from ~172 to ~260. At a fixed 40 req/s, database CPU dropped from 22% to 10%, which says how much of its work had been connection churn rather than queries.

Two caveats that make this a setting to think about rather than paste. Persistent connections are per worker thread, so connections held = pods x workers, and that has to stay under Postgres’s max_connections at the HPA’s maximum, not its minimum. And the honest alternative at larger scale is a pooler such as PgBouncer; at this size the setting is enough.

The query rewrites that did not raise the ceiling

With the database still busy, pg_stat_statements said two statements were 97% of its time. Both were the same ORM shape: an annotation that counts or maximises over a join, which makes Postgres build the join and then group it back down.

One counted interactions per item by joining through them, grouping about 5,900 rows to return 21. The other took the latest related timestamp with a filtered Max, which ran its subqueries per child row instead of per parent. Both became correlated subqueries:

# before: join, then GROUP BY back down
qs.annotate(likes=Count("item__interaction",
                        filter=Q(item__interaction__type="like"), distinct=True))

# after: one small indexed lookup per returned row
likes = (Interaction.objects.filter(item=OuterRef("item"), type="like")
         .order_by().values("item").annotate(n=Count("*")).values("n"))
qs.annotate(likes=Coalesce(Subquery(likes), 0))

The heavier statement went from 11.6 ms to 0.9 ms idle, and from 36 ms to 1.5 ms under load. Postgres time for the two statements over the run fell from 1,103 s to 209 s. Database CPU at the ceiling fell from 65% to 28%.

And the ceiling did not move. It stayed at about 260, because CONN_MAX_AGE had already shifted the wall to the k3s nodes. The rewrite was real and worth doing: it bought headroom on the one component that cannot scale horizontally. But if I had only measured latency, I would have written “8x faster queries” and believed the system had got faster. The table is what says otherwise.

The read cache, and why it is only fair to call it a best case

The five endpoints are public reads. So they got a Redis cache with a 30-second TTL, with three rules that matter more than the cache itself:

  • Anonymous viewers only. Any Authorization header, session or per-user cookie bypasses it, so no one is ever served someone else’s view.
  • It sits after the throttles and permission checks, inside the view, so a cache hit never skips rate limiting.
  • Invalidation by generation number. Every key includes a generation counter, and any write bumps it in transaction.on_commit. No key enumeration, no stale read after a committed write.

Result: the scenario hit its own 600 req/s cap with nothing saturated and p95 at 15 ms. Then, uncapped, about 990.

The honest caveat is in the scenario. k6 sent anonymous GETs to five endpoints, so the hit ratio was 98.8%, which is the cache’s best case. Real traffic has logged-in users and long-tail URLs. The number says the platform can serve cached reads at this rate. It does not say the application can.

The last 40% was hardware

At about 990 req/s the k3s nodes were at 98% CPU and the database was idle. Resizing each node from 2 to 3 vCPU took the ceiling to about 1,400: +50% cores for about +40% throughput, which is roughly what you would expect once everything else is out of the way.

That is the right order to do it in. Buying cores at row two would have bought nothing, because the database was the wall.

The next wall is already visible in the numbers. At the ceiling, responses average about 95 KB, which is roughly 1 Gbit/s through the single nginx edge VM. That is untested, and it is the next row of the table.

What happens when a node dies

Throughput is half the question. Two drills, both at a steady 100 req/s:

  • Pod loss: delete a web pod mid-run. 0 of 64,636 requests failed, worst request 1.15 s.
  • Node loss: hard power-off one worker VM, restart it 4.5 minutes later. 149 of 59,566 failed, 0.25%, all in the first minute.

With default node-health timeouts the node was marked NotReady 57 seconds after the power cut. Until then the Service still routed to the dead pods: 128 of the failures were Traefik giving up after its 30-second dial timeout, and 21 were k6’s own 60-second timeout. Pods were never rescheduled, because the default eviction delay is five minutes and the node was back in four and a half.

So the real failover time on default settings is “however long until the node is declared dead”, and the fix is on the ingress side (retries with a short dial timeout) or a shorter node grace period. Neither is applied yet; both are written down as not done.

What this is not

Single control-plane node, so the cluster itself is not highly available. No persistent volumes: every stateful thing lives on a VM outside the cluster, deliberately. No TLS, no cert-manager, no GitOps. Synthetic load from one generator. The p95 below the knee varied about 2x between identical runs, which is a good reason to distrust any single latency number here, including mine.

Takeaway

Change Ceiling Lesson
Kubernetes 2x Scales the stateless tier, then exposes the database
CONN_MAX_AGE +51% Connection churn was half the database’s work
Query rewrites 0% Real 8x per-query win, but on a component that was no longer the wall
Read cache 3.8x Best case by construction; say so
More cores +40% Only useful once nothing else is the wall

The method is the deliverable: one scenario, run after every change, and a column for what saturated. Without that column the query rewrite reads as the biggest win in the project. With it, it reads as what it was: headroom.

Back to top