Status
Over the last few weeks, we moved the customer-facing managed services on our NKE service clusters and Deploio from ingress-nginx to HAProxy: Grafana, Loki, ArgoCD, the container registry, Alertmanager, Tempo, OpenSearch, the Object Storage endpoints and all Deploio applications. Some clusters alone had more than 200 Ingresses.
We have written about the motivation twice before. In December, we explained what the retirement of ingress-nginx meant and which migration paths were open. Our TechTalk #29 recap covered our evaluation of Traefik, Gateway API and HAProxy, and why HAProxy became our default for classic Ingress on NKE.
This post is about what happened after that decision.
The annotation mapping table in our documentation gets you a working Ingress on HAProxy in a few minutes. What it does not show is how the defaults shift underneath your workloads. Two ingress controllers can parse the same Ingress resource and still differ in how they talk to your backends, which status codes they send, how long they wait and how many connections they accept before refusing new ones.
We found most of these differences in production, after the switch. One of them broke a customer’s website asset delivery. We are sharing them so you can check your own setup before you migrate.
HTTP to HTTPS Redirects
While comparing responses from both controllers, we noticed that plain HTTP requests were redirected with 302 Found on HAProxy, whereas nginx returned 308 Permanent Redirect.
ingress-nginx sets http-redirect-code to 308 by default. haproxy-ingress defaults its corresponding ssl-redirect-code to a temporary 302. Browsers follow both, but API clients, search crawlers and intermediate caches treat them differently. A 302 allows clients to rewrite a POST into a GET on redirect, whereas a 308 preserves the request method and body.
A silent downgrade to temporary redirects can break API consumers, so we set ssl-redirect-code: "308" globally on our managed controllers to keep the previous behaviour.
Connection Limits
During peak traffic, requests to Deploio applications slowed down and some new connections were refused.
The cause was the connection limit. nginx and HAProxy calculate their ceilings differently:
- nginx multiplies
worker_processesbyworker_connections. Withingress-nginxsettingworker_processesto the node’s CPU count andworker_connectionsto16384, even a modest node can handle tens of thousands of concurrent connections. - HAProxy enforces a single global
maxconn, whichhaproxy-ingressexposes asmax-connectionsand sets to 2,000 per pod by default.
Once a pod reaches that ceiling, HAProxy stops accepting new connections. They then queue in the kernel’s accept backlog, which caused the slowdown. Once the backlog is full, they are dropped.
We raised max-connections to 20,000 across all managed controllers.
Timeouts
Shortly after we migrated Deploio, some customers saw requests fail with 504 Gateway Timeout.
haproxy-ingress defaults timeout-client and timeout-server to 50 seconds, while ingress-nginx waits 60 seconds. Ten seconds make no difference for typical web requests, but data exports or reports that reliably finished in 55 seconds suddenly failed with a 504.
To avoid cutting off requests that used to succeed, our managed controllers set both timeout-client and timeout-server to 60 seconds. You can still override the timeout per Ingress with haproxy-ingress.github.io/timeout-server in place of nginx.ingress.kubernetes.io/proxy-read-timeout.
Upstream HTTP Version
In July, we moved an S3-compatible Object Storage endpoint to HAProxy. A customer served website assets through their own nginx reverse proxy in front of that endpoint. Right after the cutover, every asset request failed with 400 Bad Request.
The cause: nginx’s proxy_pass talks to upstreams over HTTP/1.0 by default. ingress-nginx always spoke HTTP/1.1 to backends, regardless of the client’s protocol, and so it silently upgraded the customer’s HTTP/1.0 requests. HAProxy forwards the request with the protocol version the client sent. The Object Storage backend rejected HTTP/1.0 requests, which produced the 400 errors.
The fix was for the customer to add proxy_http_version 1.1 to their reverse proxy configuration:
location / {
proxy_pass http://storage-endpoint;
proxy_http_version 1.1;
}
If you run a reverse proxy in front of an NKE or Deploio Ingress, check which HTTP version it sends.
Basic Auth
After we had migrated most managed services, one of the controllers spiked to more than two full CPU cores while serving only 155 concurrent connections. Almost all traffic to that pod hit two endpoints protected by HTTP basic authentication, one of them our metrics ingestion pipeline.
HAProxy verifies basic authentication via crypt(3) synchronously on every request, without caching. A bcrypt hash with a cost factor of 10, the default in most standard libraries and in Helm’s htpasswd template function, takes about 50 ms of CPU time to verify. At 50 ms per check, twenty requests per second are enough to saturate a whole CPU core.
This also applies to nginx, as its auth_basic module verifies every request synchronously and caches nothing either. Because each nginx worker is a single-threaded event loop, a 50 ms verification stalls every connection on that worker. But workers are separate processes, so a controller with eight workers can verify eight passwords in parallel.
HAProxy uses the thread-safe crypt_r(3) only on glibc and FreeBSD. On every other platform, including the Alpine-based image that haproxy-ingress ships, it wraps the plain crypt(3) call in a single process-wide spinlock. Only one thread can verify a password at a time, which caps the whole pod at roughly twenty verifications per second, no matter how many cores it has. And because it is a spinlock, waiting threads do not sleep. They keep their cores busy until it is their turn. That is how a pod serving 155 connections ended up using more than two cores.
The HAProxy manual warns that hashed passwords “can quickly become a major factor in HAProxy’s overall CPU consumption” and notes that musl’s hash implementations are slower than glibc’s. An upstream request to cache successful basic authentication results has been open since 2019 in haproxy/haproxy#370.
For credentials we generate internally, we reduced the bcrypt cost to 4, where verification takes under one millisecond. This is safe because those passwords are long, machine-generated strings of 20 random characters, which makes brute-force attacks computationally infeasible regardless of the cost factor. Helm’s built-in htpasswd function always hashes with bcrypt’s default cost of 10 and offers no way to change it (Sprig source), so chart-generated credentials need to be hashed outside Helm.
The annotation syntax is simpler on HAProxy. Setting haproxy-ingress.github.io/auth-secret alone enables authentication, auth-type is deprecated. The secret format stays the same: an htpasswd file stored under the auth key.
X-Forwarded-For
On paper, this looks like a change in behaviour. In practice it is not one, but it does add a header.
ingress-nginx replaced any incoming X-Forwarded-For with the connecting IP unless the cluster-wide compute-full-forwarded-for option was enabled, and on our shared controllers it was off.
HAProxy’s option forwardfor on its own behaves differently: it appends the connecting IP, so if the client already sent an X-Forwarded-For, the backend receives both. Applications behind a CDN or reverse proxy would then see the whole proxy chain instead of the single value ingress-nginx gave them.
haproxy-ingress closes that gap. Its forwardfor key defaults to add, which deletes any incoming X-Forwarded-For before HAProxy inserts the connecting IP. Your application therefore sees the same as before: a single address, the one that connected directly to the ingress, for example your CDN.
What is new is one extra header. If the request already carried an X-Forwarded-For, haproxy-ingress copies its original value to X-Original-Forwarded-For before discarding it:
X-Forwarded-For: <cdn-or-proxy-ip>
X-Original-Forwarded-For: <whatever-the-client-sent>
On its own, X-Original-Forwarded-For is untrusted input. It is copied verbatim from whoever connected to the ingress, and anyone can put anything into it.
It becomes useful once your application checks who connected. X-Forwarded-For holds the address that opened the TCP connection to HAProxy, which the client cannot forge. If that address belongs to a CDN you trust, for example one of the published edge IPs of your provider, then X-Original-Forwarded-For is the header that CDN sent. Its rightmost entry is the address the CDN saw connecting to it.
HSTS Headers
haproxy-ingress defaults its Strict-Transport-Security header to max-age=15768000 (six months). ingress-nginx sent max-age=31536000 (one year).
RFC 6797 leaves the duration to the operator, and the two projects measure against different yardsticks. Six months is the minimum Qualys SSL Labs has required for an A+ grade since January 2014. HAProxy’s own documentation examples use a value just above it, max-age=16000000, described as “a bit more than 6 months”. When HAProxy published those examples, the HSTS preload list accepted a max-age shorter than one year. It has since raised the minimum to one year.
For standard web traffic, this has no functional impact. If your domain is on the HSTS preload list, however, six months fails validation. We therefore override the haproxy-ingress default and send max-age=31536000 (one year), the same value ingress-nginx did.
If you need a different duration, set spec.forProvider.hsts.maxAge on the IngressHAProxy resource (a duration, for example 8760h). For a single Ingress, set haproxy-ingress.github.io/hsts-max-age (in seconds).
Request Buffering
Under ingress-nginx, high-volume write endpoints produced warnings like this in the controller logs:
[warn] a client request body is buffered to a temporary file /tmp/nginx/client-body/0000000123
request: "POST /loki/api/v1/push HTTP/1.1"
A while back, one node in our service cluster ran at 98% CPU, with its nginx pod using 1.4 cores mainly to write and clean up temporary disk buffers. On nginx, the fix was to set nginx.ingress.kubernetes.io/proxy-request-buffering: "off" on write-heavy Ingresses such as Loki, Promscale and container registries.
HAProxy streams request bodies to backends by default. There is no equivalent buffering annotation, because HAProxy does not write incoming payloads to disk.
Where This Leaves You
If you still use our managed ingress-nginx on NKE: as announced, support ended on 1 October 2026. HAProxy is available for every NKE cluster, and our migration documentation is the place to start.
If you are unsure how one of these differences affects your workloads, write to us at support@nine.ch and we will look at it with you.
