The Inverted Rate Limiter

Our IP rate limiter had a bug that did more than fail to stop abuse. It put every honest visitor in the world into one shared budget, and handed every attacker a private one they could renew per request. The cause was a single array index.

Fourteen places in our codebase needed to know who was making a request. All fourteen asked the same way, and all fourteen were wrong in the same way.

req.headers.get("x-forwarded-for")?.split(",")[0]?.trim() ?? "unknown"

This is a common line. You will find it in tutorials, in Stack Overflow answers with four-figure vote counts, and in a great many production codebases. It looks like it reads the client's IP address. What it actually reads is a string the client chose.

That value went straight into rate-limit bucket keys: how many SerendAI messages you may send an hour, how many uploads, how many tuk-tuk price reports, how many password-reset emails. Every one of those ceilings was defeated by sending a different header.

That is the half of the bug you would guess. The other half is stranger, and it is the reason this is worth writing down.

Who actually writes X-Forwarded-For

X-Forwarded-For is a request header, which means anyone can put anything in it. Its value is a comma-separated chain, and the convention is that each proxy appends the address it saw as it passes the request along.

The word “appends” is the whole story. Our nginx appends with $proxy_add_x_forwarded_for, which is defined as the incoming header, plus $remote_addr — the peer nginx actually accepted the connection from. So the rightmost entry is written by our own infrastructure and cannot be forged. Everything to the left of it arrived from outside.

split(",")[0] takes the leftmost entry. It reaches past everything our systems know to be true and reads the value furthest from us — the single most attacker-controlled string in the entire request — and uses it as an identity.

The header nobody was setting

Reading the wrong end of a chain is bad. What made this genuinely inverted was the nginx config we ship in our own deployment guide:

location / {
    proxy_pass http://localhost:3000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
}

There is no proxy_set_header X-Forwarded-For line. nginx was not adding the header at all.

So for every real visitor — who has no reason to send that header — the application received nothing, fell through to the literal string "unknown", and hashed that into a bucket key. Every honest user on the planet shared one budget. Meanwhile anyone who did send the header got a private bucket, and could mint a fresh one on every request by changing a single character.

The limiter throttled the innocent and waved through the abuse.

every real visitor

no header
↓
"unknown"
↓
one shared budget

→ throttled

one attacker in a loop

forged, rotated
↓
1   1   1   …
↓
a fresh budget each time

→ never throttled

This is the part worth internalising. A missing security control is a known quantity: you have zero protection and you know it. A control that is silently inverted is worse, because it reports success. Nothing errors. The dashboards show 429s being issued. They were being issued to the wrong people.

Counting hops from the right

The fix is not “read the last entry”. That is right for our topology and wrong the moment a CDN appears in front. The correct model is to know how many proxies you control and count that many in from the right.

With n trusted hops, the client's real address sits at index length − n. Everything to the left of that index is caller input and is ignored by construction.

const xff = headers.get("x-forwarded-for");
if (xff) {
  const entries = xff.split(",").map((e) => e.trim()).filter(Boolean);

  // Count from the RIGHT. Clamp rather than fail: a chain shorter than
  // declared means the leftmost entry is the closest thing to a true
  // client we have, and it is no worse than the old behaviour.
  const idx = Math.max(0, entries.length - hops);
  const ip = normalizeIp(entries[idx]);
  if (ip) return ip;
}

And the other half of the fix, which is not code at all:

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Neither half works alone. The library without the nginx directives resolves nothing and every request shares a bucket again; the directives without the library still read the forged end of the chain. We now say this explicitly at the top of the module, because a future reader will otherwise assume the TypeScript is the fix.

Too high is the dangerous direction

The hop count is a configuration value, which means somebody will eventually set it wrong. Those two mistakes are not symmetrical, and it is worth being loud about which one bites.

Setting	Topology	Result
hops = 1	nginx only	Correct. The index lands on the address nginx appended.
hops = 2	CDN in front of nginx	Correct. One more hop back is the CDN's view of the client.
too low	—	Over-grouping. Users behind one CDN edge share a bucket. Throttles; does not expose.
too high	—	The index reaches into client-supplied entries and spoofing works again.

When a value is uncertain, we bias low. Over-grouping is a support ticket; over-trust is the original bug returning quietly. Our config comment says exactly this, in those terms, so nobody has to re-derive it under time pressure.

Three details that decide whether it actually works

A non-address is a free bucket

If a caller sends X-Forwarded-For: hello, a naive implementation happily uses "hello" as a bucket key — an infinitely renewable identity that never collides with anything. Every extracted value is now parsed as an IPv4 or IPv6 address and rejected if it is not one. Rejection returns null, which callers treat as a missing signal rather than an identity.

One client must be one bucket

203.0.113.9, ::ffff:203.0.113.9 and 203.0.113.9:51234 are the same visitor, and a proxy can hand you any of the three. Left unnormalised, one person gets three budgets. Normalising the IPv4-mapped form and stripping ports closes that.

A truncated hash of an IPv4 address is not anonymous

We hash addresses before they become Redis keys, which is right — but the IPv4 space is 2³². An unsalted digest is reversed by brute force in seconds on a laptop, so “we only store a hash” was not the privacy property we thought we had. The hash is now keyed with a server-side pepper, scoped per feature so a user's chatbot budget and their upload budget cannot be correlated from the keys alone.

When the address genuinely cannot be resolved, the key is the literal string "unresolved" rather than a hash. Misconfiguration should look different from a real client in your logs, not like one address that happens to be extremely busy.

Making it impossible to get wrong quietly

The failure mode here was silence. Fixing the code without fixing the silence would leave us one nginx rebuild away from the same outage-that-does-not-look-like-one. So two things shipped alongside the fix.

A health endpoint that can be checked in one command.

curl -H 'X-Forwarded-For: 1.2.3.4' https://example.com/api/health/client-ip

{ "resolved": "198.51.100.22",   ← your real address, not 1.2.3.4
  "trustedProxyHops": 1,
  "spoofIgnored": true,
  "healthy": true }

If resolved comes back as 1.2.3.4, the header is being trusted from the client and the hop count is too high. If it comes back null, nginx is not setting the header. Both diagnoses, in one request, without reading any code.

A regression test that fails the build.

The check asserts the behaviour — a prepended entry is ignored, extra forged entries only push the attacker further from the trusted index, a non-address is rejected, an unresolvable address is labelled rather than disguised. Then it asserts the shape: no file outside the resolver may read x-forwarded-for at all. That second assertion is the one that matters in a year, when someone adds route 353 by copying route 91.

Worth checking in your own stack

• grep for split(",")[0] Anywhere you derive an identity from X-Forwarded-For, check which end of the chain you are reading, and whether it is a bucket key.
• Read your proxy config, not your framework docs The application half of this fix is worthless if the proxy is not appending. Confirm the directive exists in the config that is actually deployed.
• Send yourself a forged header One curl against production tells you more than an afternoon of reading. If your own address comes back, you are fine.
• Look for the "unknown" bucket A fallback string used as an identity is a shared budget with a friendly name. If a single key dominates your limiter's keyspace, that is not your busiest user.
• Ask what a limiter does when it cannot identify the caller Silently grouping everyone together is a decision. Make it deliberately, and make it visible when it happens.

The whole change is about sixty lines of TypeScript, three lines of nginx config, and a health endpoint. The reason it took a while to find is that nothing was broken in a way anything reported — the limiter ran, returned 429s, and looked entirely healthy while doing the opposite of its job.

That is the lesson we took from it. For any control whose whole purpose is to distinguish one caller from another, “it is running” is not evidence that it works. Ask it to prove, out loud, that it can tell you apart from someone pretending to be you.