How Rate Limiting Actually Works

Rate limiting looks like one line of config: "100 requests per minute". Under that line there are a few algorithms, a store that has to stay correct under concurrency, and a decision about what happens when the limiter itself breaks.

What a rate limiter does

A rate limiter controls how fast a consumer can use a finite resource.

A client sends 1000 requests per second. The backend can handle 100. The limiter sits in front and rejects the extra requests with 429 Too Many Requests.

Limits protect different things, and the goal decides everything else (algorithm, key, what to do on failure):

• Capacity. Keep the backend alive.
• Cost. One request can cost 100x another (an LLM call, an export). Limit the expensive unit directly.
• Fairness. One noisy user should not slow everyone else.
• Abuse. Login and signup need low limits, because attackers guess.
• A promise. "10 free chats a day" is a product rule. It needs an exact count that survives restarts.

A limiter built for capacity (fast, approximate) is the wrong tool for a promised quota (exact, durable). Keep that in mind, it comes back later.

Fixed window

The simplest one. Pick a window, like one minute. Count requests in it. If the count passes the limit, reject until the next window.

In Redis this is one integer per user per window, moved with INCR. The window either follows the clock (1:00 to 1:01) or starts at the first request.

-- KEYS[1] = "rl:user101", ARGV[1] = window in ms
local count = redis.call('INCR', KEYS[1])
if count == 1 then
  redis.call('PEXPIRE', KEYS[1], ARGV[1])
end
return count

INCR and PEXPIRE run together in one Lua script on purpose. If the app crashes between two separate calls, you get a key that never expires and a user who is locked out forever.

The problem. A user waits until the last second of a window and sends a batch, then sends another batch right after the reset. With a limit of 5 per minute, that is 5 requests at 1:00:59 and 5 more at 1:01:00. Ten requests in about two seconds, all allowed.

So fixed window can allow up to 2x the limit around a boundary. For small limits that is fine. It is also the cheapest option, so it is a good default.

Sliding window log

Instead of fixed windows, look at the window that ends at the incoming request: the last 60 seconds from now.

Say the limit is 100 per minute. At 1:00:52 you have seen 97 requests in the last 60 seconds, so 3 more are allowed.

Redis stores this in a sorted set. Score is the timestamp, member is a request id. On every request: remove entries older than the window, count what is left, and add the new one if there is room.

-- KEYS[1] = key, ARGV = now_ms, window_ms, limit, request_id
redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, ARGV[1] - ARGV[2])
if redis.call('ZCARD', KEYS[1]) >= tonumber(ARGV[3]) then
  return 0
end
redis.call('ZADD', KEYS[1], ARGV[1], ARGV[4])
redis.call('PEXPIRE', KEYS[1], ARGV[2])
return 1

There is no boundary trick here, the count is exact. The cost is memory: every request in the window is stored, so a user with a limit of 100 holds up to 100 entries. Good for small, sensitive keyspaces like login and payments. Expensive for a high-traffic API.

Sliding window counter

A cheaper middle ground. Keep two fixed-window counters, the previous and the current, and estimate.

Limit is 10 per minute and it is 10:01:30. The last 60 seconds started at 10:00:30, so half of the previous window still counts. Previous window had 8 requests, current has 2:

Add the new request and you are at 7, under 10, so it passes.

It is an estimate, because it assumes the previous window's traffic was evenly spread. Cloudflare wrote about using this and checked it on 400 million requests from 270,000 sources: 0.003% of requests were wrongly allowed or limited, with an average 6% gap between the real and estimated rate. Two integers per user, and accurate enough.

Token bucket

This one counts differently. There is a bucket with a capacity, and tokens drip in at a fixed rate. Every request spends one token. Empty bucket means reject.

Capacity 5, refill 5 per minute: the bucket holds at most 5 tokens and gets a new one every 12 seconds.

You do not need a timer. Store the token count and the last refill time, and on each request compute how many tokens arrived since then.

-- KEYS[1] = key, ARGV = capacity, rate_per_ms, now_ms
local d = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens, ts = tonumber(d[1]), tonumber(d[2])
if tokens == nil then tokens, ts = tonumber(ARGV[1]), tonumber(ARGV[3]) end
tokens = math.min(tonumber(ARGV[1]), tokens + (tonumber(ARGV[3]) - ts) * tonumber(ARGV[2]))
local allowed = 0
if tokens >= 1 then tokens = tokens - 1; allowed = 1 end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', ARGV[3])
redis.call('PEXPIRE', KEYS[1], 120000)
return allowed

The point of a token bucket is that it allows bursts on purpose. A client that was quiet for a while has a full bucket and can send 5 requests at once. That is good for clients that batch work. The catch: the bucket capacity is the largest burst your backend will see, so pick a capacity it can take. 100 req/min with capacity 100 means someone can send 80 in one second.

Stripe says it uses a token bucket: every user has a bucket, each request takes a token, and tokens drip back in.

Leaky bucket

The name is shared by two different things, which causes confusion.

Shaping (a queue). Requests wait in a queue and are released to the backend at a fixed rate. If the queue is full, new requests are rejected. 50 requests enter a queue of 50 and leave at 10 per second, so the last one waits about 5 seconds. You get a smooth stream to the backend, and you pay in latency.

Policing (a meter). The bucket starts empty. Each accepted request adds to the level, and the level drains at a fixed rate. If a request would overflow the bucket, it is rejected right away. Nothing waits.

NGINX's limit_req uses the leaky bucket method, and its burst setting shows both behaviours:

• burst=5: up to 5 extra requests are delayed to fit the rate.
• burst=5 nodelay: the 5 extra requests pass immediately, then the rate is enforced.
• delay=N: the first N pass at once, the rest are delayed.

Use shaping for fragile downstreams and outbound webhooks, where late is fine but dropped is not. Use rejection when the user has stopped waiting anyway.

Picking one

A daily quota is a fixed window with a 24 hour TTL.

The limits real systems use

Numbers help more than theory, so here are some that are published. They change, so check the docs before you rely on them.

GitHub REST API layers its limits:

• 60 requests per hour for unauthenticated requests
• 5,000 requests per hour per user (15,000 for apps owned by an Enterprise Cloud org)
• a separate secondary limit: at most 100 concurrent requests, and 900 points per minute per REST endpoint

Stripe runs four different mechanisms, and only the first is a request-rate limit:

1. A request rate limiter (token bucket, N requests per second per user).
2. A concurrent requests limiter: "20 requests in progress at the same time", for CPU-heavy endpoints.
3. A fleet usage load shedder: a share of capacity is reserved for critical methods (like creating charges). Non-critical requests past their share get a 503.
4. A worker utilization load shedder: the last line of defense when workers back up. It sheds lower-priority traffic first.

Stripe also ran new limiters in a log-only mode first, and enforced them after tuning. That is a good habit.

AWS API Gateway throttles with a token bucket, per account per region, with a steady rate and a burst. AWS says throttles are best-effort and "should be thought of as targets rather than guaranteed request ceilings". Every distributed limiter behaves like this, so design for a limit that can overrun a little.

Notice the pattern. A request-rate limit is only one layer. Concurrency limits and load shedding handle things a rate cannot: slow requests, expensive endpoints, and a system that is already in trouble.

Choose the key

The limit is per something. That something matters as much as the number.

• User or account ID is the fair key after login. Authentication must run before the limiter, or you only have the IP.
• IP is the only choice before login. One office or mobile carrier can sit behind one IP, so keep IP limits generous.
• API key for machine clients.
• Route class. 10 search calls and 10 exports are not the same load, so expensive routes get their own lower limit.

For login, combine keys. Per-account alone lets a botnet try one account from many IPs. Per-IP alone lets one IP try many accounts.

What to do when the limiter breaks

The limiter is a single point of failure. If the Redis behind it dies, you pick one:

• Fail open: every request goes through. The backend is unprotected.
• Fail closed: every request is rejected. Your API is down.

Neither is right everywhere:

The better design is a hybrid. Use the shared Redis limiter while it is up. If it is down, fall back to a local in-memory limiter on each API server. Each server only knows its own count, so across the fleet the limit is approximate, but that beats fully open or fully closed.

There is a third bad outcome: the limiter throws an error and the user sees a 500, or the client library queues commands and requests hang. Test the outage. Do not guess.

Where the counters live

• In memory. Fast, no dependency. But each server has its own count, so the real limit is the limit times the number of instances. On serverless, a cold start also resets it. Fine for approximate limits, wrong for a promise.
• Redis. One shared count. Three rules: make check and increment atomic with Lua (a GET followed by an INCR lets concurrent requests all pass), take time from Redis or use TTLs instead of app server clocks, and watch hot keys, because one key lives on one node.

Telling the client

Return 429 and send Retry-After every time. Without it, clients retry immediately, and many clients retrying together make a second spike.

On the client side, back off with jitter:

Without the random part, every client comes back in the same second.

Also publish the limit so clients can slow themselves down. The RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset headers are the common pattern, and an IETF draft is moving toward standardizing them. It is still a draft.

My own limiters

I run two projects that both have rate limits, and they are good examples of two different choices. I read both codebases for this post. Edward at commit c69d9ee, Agentic Chat at e9d47f0.

Edward

Edward uses express-rate-limit with rate-limit-redis. That library counts in a fixed window, and the counters live in Redis, so every API server shares them. The key is the user ID, because auth runs before the limiter.

The interesting one is the daily chat quota. It is custom, and it is a promise to the user, so it counts differently:

1. chatRateLimiter allows 10 messages a minute.
2. dailyChatRateLimiter reads the user's count from Redis. At 10 or more, it returns 429.
3. The run goes through admission control (caps on active runs per user and per chat). That is a concurrency cap, not a rate limit.
4. Only when a run finishes with a real answer does Edward INCR the daily key and set a 24 hour TTL. A failed run does not cost the user a message.

Counting successes is fairer to the user. It also has a price: check and increment are separate steps, so a few concurrent requests can pass the check before any of them counts. In practice the quota can overshoot by about one.

Two more things I know about this design:

• The count lives only in Redis. There is no usage table in the database. If Redis is wiped, every user gets a fresh 10. For a free tier that is an acceptable trade. For paid credits I would make the database the source of truth and use Redis as the fast gate.
• When the daily check throws, the code logs it and lets the request through. That is fail open. But the Redis client is set up with maxRetriesPerRequest: null, which queues commands during an outage. So in an outage the request more likely hangs than fails open. I read this in the code and have not tested it.

Agentic Chat

Agentic Chat takes the opposite route. It is a sliding window log, in a plain Map inside the server process:

The key is userId:endpoint and the value is a list of timestamps. On each request it drops timestamps older than the window, and if the count is at the max it returns 429 with Retry-After set to oldest timestamp + window - now. Otherwise it stores the new timestamp.

Limits: chat 20 per minute, approval 10 per minute, deep research 5 per 5 minutes.

The algorithm is exact and the response is correct. The store is the weak part. The Map lives in one process. On a serverless host, each warm instance has its own Map, and a cold start empties it. So the limit is per instance and not global. Many authenticated routes also have no limit at all.

The fix is the same idea Edward already uses: move the timestamps to Redis.

The short version

• Pick the limit from what you protect: capacity, cost, fairness, abuse, or a promise.
• Fixed window is the cheap default. Use a sliding log when you need exact. Use a sliding counter when good enough is enough. Use a token bucket when clients burst on purpose.
• Rate is one layer. Add concurrency limits and load shedding.
• Make check and increment atomic.
• Decide fail open or fail closed per endpoint, and test the outage.
• Always return Retry-After.

Sources

• Stripe, "Scaling your API with rate limiters": https://stripe.com/blog/rate-limiters
• GitHub REST API rate limits: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api
• AWS API Gateway throttling: https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-request-throttling.html
• Cloudflare, "How we built rate limiting capable of scaling to millions of domains" (sliding window counter and its accuracy numbers): https://blog.cloudflare.com/counting-things-a-lot-of-different-things/
• NGINX ngx_http_limit_req_module: https://nginx.org/en/docs/http/ngx_http_limit_req_module.html
• RFC 6585 (429 status code): https://www.rfc-editor.org/rfc/rfc6585