Every service that survives contact with the real internet eventually needs rate limiting. It is the quiet mechanism that keeps an API from being drowned by a runaway client, a scraper, a retry storm, or an outright attack — and it is one of those topics that seems trivial until you try to implement it correctly across more than one server. This is a practical walk through how rate limiting actually works: the algorithms underneath it, the hard problem of enforcing limits in a distributed system, and the details that separate a robust implementation from one that quietly misbehaves.
Why rate limiting exists
At its simplest, rate limiting caps how many requests a client may make in a given window of time. The motivations are more varied than "stop attackers," though that is one of them. Rate limiting protects scarce resources — database connections, CPU, downstream services — from being exhausted by a single misbehaving caller. It enforces fairness, so one aggressive client cannot degrade the experience for everyone else. It contains the blast radius of bugs, like a client stuck in a tight retry loop. And it underpins commercial tiering, where different plans are allowed different throughput.
Crucially, rate limiting is a reliability feature as much as a security one. Many of the worst outages are self-inflicted: an internal service retries aggressively after a blip, the retries amplify the load, and the system collapses under its own traffic. A well-placed limiter turns a cascading failure into a contained, recoverable one.
The token bucket algorithm
The most widely used rate-limiting algorithm is the token bucket, and it is popular because it models something intuitive: a bucket that holds tokens, refilled at a steady rate up to a maximum capacity. Each request removes one token. If a token is available, the request proceeds; if the bucket is empty, the request is rejected or delayed.
The elegance of the token bucket is that it allows controlled bursts. Because the bucket can hold up to its capacity in tokens, a client that has been quiet can spend a burst of accumulated tokens all at once, then is throttled back to the refill rate. This matches how real traffic behaves — bursty, not perfectly smooth — far better than a rigid "one request every N milliseconds" rule. Two parameters define the behavior: the refill rate (sustained throughput) and the bucket size (how large a burst is tolerated). Tuning those two numbers covers a surprising range of real-world needs.
The leaky bucket and why smoothing matters
A close cousin is the leaky bucket, which inverts the emphasis. Imagine requests pouring into a bucket that leaks at a constant rate; the leak represents processing, and if requests arrive faster than they drain, the bucket fills and eventually overflows, rejecting excess. Where the token bucket permits bursts, the leaky bucket enforces a smooth, constant output rate.
The choice between them is a choice about what you are protecting. If a downstream system can absorb short spikes but needs a bounded average, token bucket's burst tolerance is friendlier to clients. If a downstream system is fragile and must be fed at a strict, even pace — a legacy database, a third-party API with its own hard limits — the leaky bucket's smoothing is safer. Understanding what sits behind your limiter determines which shape of traffic you should be shaping toward.
Fixed windows, sliding windows, and the boundary problem
A third family counts requests within time windows. The naive version, the fixed window, simply counts requests per calendar interval — say, per minute — and resets the counter when the clock ticks over. It is trivial to implement and trivially flawed: a client can send a full window's worth of requests at the very end of one minute and another full window at the start of the next, briefly achieving double the intended rate right at the boundary.
The sliding window fixes this. Instead of resetting on a hard boundary, it considers a rolling interval, weighting the previous window's count as it slides forward, so the limit is enforced continuously rather than in discrete jumps. This eliminates the boundary burst at the cost of slightly more bookkeeping. For any limit that genuinely matters, the sliding window's accuracy is usually worth the modest extra complexity, and it has become the default expectation for well-behaved public APIs.
The hard part: distributed rate limiting
Every algorithm above is straightforward on a single server, where one process holds the counter in memory. The moment you run behind a load balancer with several instances, the problem changes character entirely. If each server enforces the limit independently using its own local state, a client spread across ten servers effectively gets ten times the intended allowance, because no single server sees the whole picture.
The standard solution is shared state, most commonly a fast in-memory store such as Redis that every instance consults. Each request checks and updates a counter in the shared store, so the limit is enforced globally rather than per instance. This introduces the classic distributed-systems tension: correctness now depends on a network round-trip on the hot path, and the shared store becomes both a latency contributor and a potential single point of failure. Implementations mitigate this with atomic operations or server-side scripts to avoid race conditions, and with careful thought about what happens if the store is briefly unavailable — do you fail open and risk overload, or fail closed and risk rejecting legitimate traffic? These are the kinds of trade-offs where reliability engineering earns its keep, and where the performance cost is as real as it is on any latency-sensitive path, a theme explored in why web performance still decides whether a site succeeds.
Responding well when the limit is hit
An often-overlooked part of rate limiting is what happens at the moment of rejection, and this is where good and bad implementations diverge sharply. The established convention is to return an HTTP 429 "Too Many Requests" status, but the status code alone is not enough. A well-behaved limiter also tells the client how long to wait, via a Retry-After header, and ideally exposes the client's current standing through headers describing the limit, the remaining allowance, and when the window resets.
This matters because rate limiting is a conversation, not a wall. A client that receives clear signals can back off intelligently, schedule its retries, and stay within bounds. A client that receives a bare rejection with no guidance will often do the worst possible thing — retry immediately and aggressively — turning your protective mechanism into the trigger for exactly the retry storm it was meant to prevent. Pairing 429 responses with clear metadata, and encouraging clients to use exponential backoff with jitter, closes that loop.
Where to enforce it
Finally, rate limiting can live at several layers, and mature systems often use more than one. At the edge — a CDN, API gateway, or reverse proxy — a coarse limit cheaply absorbs the crudest abuse before it ever reaches your application, protecting everything behind it. Inside the application, finer-grained limits can be tied to authenticated users, specific endpoints, or particular operations that are expensive. Layering these gives you both a cheap outer shield and precise inner control.
The guiding principle is to reject unwanted load as early and as cheaply as possible, while keeping the nuanced, business-aware decisions close to the logic that understands them. Enforcing everything in one place is either too crude or too expensive; distributing enforcement across layers lets each do what it is best at.
Conclusion
Rate limiting looks simple and is not. The algorithms — token bucket for bursts, leaky bucket for smoothing, sliding windows for accuracy — are each a different answer to the question of what kind of traffic you want to allow. The genuine difficulty appears when you move from one server to many, where shared state, atomicity, and failure modes turn a tidy counter into a real distributed-systems problem. And the details of the response, from 429 status codes to Retry-After guidance, determine whether your limiter calms traffic or inflames it. Get these pieces right and rate limiting becomes what it should be: an invisible, dependable guardrail that keeps a service healthy under precisely the conditions that would otherwise take it down.
