Rate Limiting: The Art of Saying No Before You Die Trying to Say Yes
Rate limiting is what you do when you realize you can't handle all the requests coming at you, so you politely (or not so politely) tell some of them to go away. It's the bouncer at the club door, the TSA checkpoint at the airport, the reason you can't refresh the ticket sales page fast enough to get Taylor Swift tickets. It's essential for survival, but implementing it well is surprisingly difficult, and implementing it poorly means you've added overhead to your dying system instead of saving it.
Let's talk about why rate limiting is easy when you control the source, hard when you're the destination, and why the difference matters more than you think.
The Fundamental Asymmetry: Source vs. Destination
Rate limiting at the source is trivial. You control the rate at which you send requests, so you simply don't send more than you're allowed to. A client configured to make 100 requests per second will make exactly 100 requests per second. Done. No ambiguity, no overhead, no complexity.
Rate limiting at the destination is where things get messy. By the time you know you want to reject a request, you've already:
- Received the packet (consumed network bandwidth)
- Interrupted your CPU to process it
- Allocated memory for connection state (TCP handshake)
- Parsed the request headers
- Looked up rate limit rules
- Decided to reject it
- Constructed a rejection response
- Sent that response back (more bandwidth)
All of this costs resources. When you're under heavy load and trying to shed requests, you're spending precious CPU cycles deciding which requests to reject. If your rejection logic is expensive (database lookups, complex rules), you might be doing more work rejecting requests than you would have handling them. This is the rate limiting paradox: the act of saying no can be more expensive than saying yes.
The Cost of Being Polite
When you reject a request, you have three options:
Reject cleanly (HTTP 429, 503): Send a proper response with status code, headers, maybe even a helpful message about when to retry. This is polite. It's also expensive. You've done HTTP parsing, rate limit checking, response formatting, and transmission. If you're rejecting 10,000 requests per second, that's 10,000 complete HTTP transactions just to say no.
Drop silently: Just stop processing and drop the connection. No response, no acknowledgment, the packet disappears into the void. This is cheaper, you're not sending responses, but the client doesn't know why their request failed. They might retry immediately, making things worse. Also, if you've already done a TCP handshake, the client thinks the connection is established and will wait for a response until timeout, wasting their time and your connection state.
Reject at TCP level (RST): Send a TCP RST (reset) to immediately terminate the connection. Cheaper than a full HTTP response, more informative than a drop, but still costs a packet transmission and the client knows the connection was rejected but not why.
Each approach has trade-offs between resource consumption, client experience, and retry behavior. There's no perfect answer, just less-bad choices based on your situation.
The Fundamental Reasons for Rate Limiting
Defense Against Malicious Traffic
The obvious reason: preventing DoS attacks. If an attacker can send 100,000 requests per second and you can handle 1,000, you're dead. Rate limiting caps how much damage a single source can do. This works great against unsophisticated attacks (one source, easily identified) and poorly against distributed attacks (DDoS from 100,000 sources each sending 10 requests per second).
For DDoS, rate limiting per-source becomes less effective. Limiting each source to 100 req/s doesn't help when you have 10,000 sources. You need smarter approaches: geographic rate limiting (this region shouldn't generate this much traffic), behavioral analysis (these requests look automated), or just upstream filtering (block it before it reaches you).
Protecting Yourself From Yourself
Here's a scenario: your system crashes or gets restarted. Maybe it was a deployment, maybe an outage, maybe you just needed to bounce a service. When it comes back online, what happens?
Every client that was waiting for a response retries immediately. Every background job that was scheduled to run retries. Every load balancer health check that failed starts sending traffic again. You get hit with a thundering herd, a synchronized wave of requests that can exceed your steady-state capacity by an order of magnitude.
This is why systems die repeatedly after recovering from an outage. They come up, get overwhelmed by retry traffic, crash again, come up, overwhelmed again, crash. This failure loop can continue until someone manually throttles traffic or enough clients give up.
Rate limiting during startup provides breathing room. Limit request rate to a fraction of capacity initially, gradually increase as the system warms up (caches populate, connections establish, threads spawn). This is called graceful startup or ramp-up, and it's the difference between recovery and perpetual failure.
Resource Protection
Some operations are expensive. Database writes, complex queries, external API calls, file uploads. Even if your web servers can handle 10,000 req/s, your database might only handle 100 writes/s before latency explodes. Rate limiting writes protects your backend from your frontend's success.
This gets into the concept of backpressure: when a downstream system is overloaded, propagate that information upstream so clients slow down. Without backpressure, requests queue up, latency increases, clients timeout and retry, making things worse. With backpressure, clients get rejected early and can back off or try later.
Buffers, Drops, and the Latency-Throughput Tradeoff
When a request arrives and you're at capacity, you have three fundamental choices:
Buffer It (Queue)
Accept the request into a queue and process it when capacity becomes available. This maximizes throughput, you're not dropping anything, everything gets processed eventually. The cost is latency. As the queue grows, the time from request arrival to processing increases.
Buffering works well when:
- Load spikes are short-lived
- Clients can tolerate increased latency
- You have bounded queue sizes (you won't run out of memory)
Buffering fails when:
- Load is sustained above capacity (queue grows forever until you run out of memory and crash)
- Clients timeout before their requests are processed (you do work that's already useless)
- Queue latency exceeds acceptable bounds (200ms response time becomes 30 seconds)
The classic mistake is unlimited buffers. "We'll just queue requests and process them eventually!" Eventually your queue has 100,000 requests, you're out of memory, and you crash. Or you spend 10 minutes processing requests that the client gave up on 9 minutes ago. Unbounded buffers turn graceful degradation into catastrophic failure, just slower.
Drop It (Discard)
Silently discard the request. No processing, no response, it's gone. This preserves resources, you're not spending CPU on something you can't handle. The cost is that the client doesn't know what happened. They'll probably retry, and if every retry is also dropped, they see total failure.
Dropping is appropriate when:
- You're severely overloaded and need to conserve resources
- The traffic is malicious (you don't care about response codes)
- You're implementing probabilistic dropping (randomly drop N% of requests to reduce load)
The danger is that random dropping can look like network problems, leading to escalating retries and customer support tickets.
Reject It (Return Error)
Process the request just enough to return an error code (HTTP 429, 503). This provides feedback to the client, they know they were rate limited and can implement backoff strategies. The cost is that you're still doing some work per request (parsing, rate checking, response formatting).
Rejecting is ideal when:
- You want clients to behave well (respect rate limits, backoff appropriately)
- You have capacity to generate rejection responses
- Client experience matters (they should know why they're failing)
The problem is that under extreme load, even generating rejection responses can overwhelm you. If rejecting 50,000 requests per second costs more CPU than handling 5,000 requests per second, you're using your resources to say no instead of being useful.
Congestion Collapse: When Everyone Makes It Worse
Congestion collapse is what happens when increased offered load decreases useful throughput. Sounds impossible? It's not only possible, it's common.
Here's the scenario:
- System operates at capacity, handling 1,000 req/s successfully
- Load increases to 2,000 req/s
- System can't keep up, requests queue, latency increases
- Clients timeout waiting for responses
- Clients retry their timed-out requests
- Now the system is processing 2,000 original requests + retries
- More timeouts occur, more retries happen
- System spends all its resources processing requests that have already timed out at the client
- Useful throughput drops to near zero despite the system being "busy"
This is congestion collapse. The system is working harder and harder while accomplishing less and less. The solution requires:
- Early rejection: Reject requests before they timeout so clients know immediately
- Request cancellation: Detect and stop processing requests whose clients have disconnected
- Backpressure propagation: Tell clients to slow down before collapse happens
- Admission control: Limit concurrent in-flight requests to prevent queue buildup
The key insight: doing nothing is better than doing useless work. If a client has already given up, stopping processing immediately frees resources for requests that matter.
Shedding the Right Load
Not all requests are equal. When you're overloaded and must shed load, you want to shed the least valuable work and preserve the most valuable. But how do you identify value?
Priority-Based Shedding
Assign priorities to requests. API health checks are low priority, they're just testing connectivity. User-facing reads are higher priority. Critical writes are highest priority. Under load, shed low-priority requests first.
The challenge is defining priorities and ensuring they're meaningful. Every team thinks their requests are high priority. You need clear criteria and enforcement.
User-Based Shedding
Shed anonymous users before authenticated users, free-tier before paid-tier, new users before established users. This protects your revenue and your most important customers.
The ethical question: is it fair to give better service based on payment? The pragmatic answer: your paying customers literally pay for priority, and keeping them happy keeps you in business.
Cost-Based Shedding
Shed expensive operations before cheap ones. A request that requires three database queries and external API call should be shed before a request that's served from cache. Maximize the number of successful requests by preferring cheap work.
Age-Based Shedding
Drop old requests that have been queued too long. If a request has been waiting 30 seconds and the client timeout is 30 seconds, there's no point processing it. Drop it and free resources for newer requests that might succeed.
Implementing this requires tracking request age, which adds overhead. But the payoff is huge during congestion: you stop doing useless work automatically.
Rate Limiting Algorithms: Choosing Your Weapon
Token Bucket: The Classic
Token bucket is probably the most common rate limiting algorithm, and for good reason. It's simple, effective, and handles bursts gracefully.
The concept: imagine a bucket that holds tokens. Tokens are added at a constant rate (the refill rate). To process a request, you need to consume a token from the bucket. If the bucket is empty, the request is rejected or queued.
Bucket capacity: 100 tokens Refill rate: 10 tokens/second Time 0: Bucket has 100 tokens (full) Burst of 50 requests arrives: 50 tokens consumed, 50 remain Next second: Bucket refills to 60 tokens (50 + 10) Another 20 requests: 20 tokens consumed, 40 remain Next second: Bucket refills to 50 tokens (40 + 10)
Token bucket allows bursts up to the bucket capacity while enforcing an average rate equal to the refill rate. This is often what you want: handle brief spikes without rejecting them, but prevent sustained overload.
Implementation is straightforward:
class TokenBucket:
def __init__(self, capacity, refill_rate):
self.capacity = capacity
self.tokens = capacity
self.refill_rate = refill_rate # tokens per second
self.last_refill = time.now()
def consume(self, tokens=1):
self.refill()
if self.tokens >= tokens:
self.tokens -= tokens
return True
return False
def refill(self):
now = time.now()
elapsed = now - self.last_refill
new_tokens = elapsed * self.refill_rate
self.tokens = min(self.capacity, self.tokens + new_tokens)
self.last_refill = now
The choice of capacity and refill rate depends on your system. Large capacity allows bigger bursts but takes longer to refill. Small capacity limits bursts but refills quickly. There's no universal right answer, just trade-offs.
Leaky Bucket: Constant Output Rate
Leaky bucket is similar to token bucket but enforces a more rigid output rate. Instead of consuming tokens, requests are added to a queue that drains at a constant rate.
Queue capacity: 100 requests Drain rate: 10 requests/second Requests arrive and enter the queue (if space available) Queue drains at exactly 10 req/s regardless of input rate If queue is full, new requests are rejected
Leaky bucket provides perfectly smooth output, no bursts. This is useful when downstream systems can't handle bursts or when you want predictable load patterns. The downside is less flexibility, legitimate bursts are treated the same as sustained overload.
Fixed Window: Simple But Flawed
Fixed window is the naive approach: count requests in fixed time windows (e.g., per minute). Once the limit is reached, reject requests until the next window starts.
Limit: 100 requests per minute 00:00:00-00:00:59: 100 requests allowed 00:01:00-00:01:59: Counter resets, 100 requests allowed
The problem: boundary conditions. At 00:00:59, send 100 requests. At 00:01:00, send 100 more requests. You've sent 200 requests in 2 seconds, but each window shows 100. Fixed windows allow bursts at boundaries, double your intended rate.
Despite this flaw, fixed windows are popular because they're easy to implement and reason about. For many use cases, the boundary issue is acceptable.
Sliding Window: Fixed Window Done Right
Sliding window fixes the boundary problem by maintaining a rolling window. Instead of discrete 1-minute windows, track requests in the last 60 seconds continuously.
Every time a request arrives, count how many requests occurred in the previous 60 seconds. If under the limit, allow. If over, reject.
This prevents boundary gaming but requires more state. You need to track timestamps of recent requests (or aggregate counts at finer granularity), which costs memory.
A hybrid approach uses fixed windows but applies weighting:
Current window: 00:01:00-00:01:59, 50 requests so far Previous window: 00:00:00-00:00:59, 80 requests total At 00:01:30 (50% through current window): Estimated current rate = 50 + (80 * 0.5) = 90 requests If limit is 100, allow. If limit is 85, reject.
This approximation is cheaper than true sliding windows while avoiding the worst boundary issues.
Adaptive Rate Limiting: Fighting Back Intelligently
All the algorithms above use fixed limits: 100 req/s, 1000 req/minute. But what if your capacity varies? What if you're running on cloud infrastructure that auto-scales? What if backend performance degrades under load?
Adaptive rate limiting adjusts limits based on system health:
- Measure actual capacity: Monitor response times, error rates, CPU usage
- Adjust limits dynamically: If response times increase, reduce rate limits. If system is healthy, increase limits.
- Feedback loop: System health influences rate limits, which influence system health
This requires careful tuning to avoid oscillation (limits too aggressive, system underutilized, limits loosened, system overloaded, repeat). But when done well, adaptive limiting maximizes throughput while maintaining stability.
Google's "adaptive throttling" in SRE practices is an example: throttle requests when error rate exceeds a threshold, back off when error rate drops. Simple but effective.
Linux TC and eBPF: Rate Limiting at the Kernel Level
Before we talk about where to apply rate limits in your application architecture, let's talk about Linux's built-in traffic control capabilities, because why would you implement rate limiting in userspace when the kernel can do it faster?
Traffic Control (TC): The Swiss Army Knife You Forgot You Had
Linux's TC (Traffic Control) subsystem has been around since the 2.2 kernel (1999), and it's one of those powerful tools that most people never learn because the syntax looks like it was designed by someone who hates humans. But underneath the cryptic commands is a sophisticated system for shaping, scheduling, and policing traffic.
TC operates at the network interface level, intercepting packets before they leave (egress) or after they arrive (ingress). You can rate limit, prioritize, delay, drop, or redirect traffic with incredibly fine granularity.
Here's a simple example of rate limiting outgoing traffic to 1 Mbps:
# Create a token bucket filter on eth0 tc qdisc add dev eth0 root tbf rate 1mbit burst 32kbit latency 400ms # What this means: # - qdisc = queueing discipline (traffic shaping algorithm) # - tbf = Token Bucket Filter # - rate 1mbit = refill tokens at 1 megabit per second # - burst 32kbit = bucket capacity (allows short bursts) # - latency 400ms = how long packets can wait in queue
This implements token bucket at the kernel level, operating on every packet that tries to leave eth0. It's fast (kernel space, no context switches), it's transparent (applications don't know it's happening), and it works for all protocols (TCP, UDP, ICMP, everything).
But TC can do much more sophisticated things:
# Classify traffic and apply different limits
tc qdisc add dev eth0 root handle 1: htb default 30
# Create classes (bandwidth buckets)
tc class add dev eth0 parent 1: classid 1:1 htb rate 10mbit
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 5mbit # High priority
tc class add dev eth0 parent 1:1 classid 1:20 htb rate 3mbit # Medium priority
tc class add dev eth0 parent 1:1 classid 1:30 htb rate 2mbit # Low priority (default)
# Filter traffic into classes based on port
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 \
match ip dport 443 0xffff flowid 1:10 # HTTPS -> high priority
tc filter add dev eth0 protocol ip parent 1:0 prio 2 u32 \
match ip dport 22 0xffff flowid 1:20 # SSH -> medium priority
Now you have hierarchical rate limiting: HTTPS traffic gets 5 Mbps, SSH gets 3 Mbps, everything else gets 2 Mbps. This is traffic shaping at line rate, in the kernel, with zero application changes.
The TC Queueing Disciplines (qdiscs) Zoo
TC supports multiple queueing disciplines, each implementing different algorithms:
TBF (Token Bucket Filter): The token bucket algorithm we discussed earlier. Simple, effective, allows bursts.
HTB (Hierarchical Token Bucket): Token bucket with hierarchical classes. You can create bandwidth trees where child classes share parent bandwidth, with borrowing when others are idle.
SFQ (Stochastic Fairness Queueing): Ensures fair bandwidth distribution among flows using hash-based scheduling. Prevents one flow from monopolizing bandwidth.
FQ_CODEL (Fair Queue Controlled Delay): Combines fair queueing with active queue management. It's designed to minimize bufferbloat (excessive buffering that causes latency). FQ_CODEL is the default qdisc in modern Linux because it works well without tuning.
Cake (Common Applications Kept Enhanced): A modern qdisc that combines shaping, prioritization, and AQM (Active Queue Management). It's designed for home routers and handles common patterns (gaming, VoIP, bulk downloads) intelligently.
PIE (Proportional Integral controller Enhanced): Active queue management algorithm that drops packets before queues fill, preventing bufferbloat. Used in enterprise routers.
The choice of qdisc depends on your use case. For simple rate limiting, TBF works. For complex traffic shaping with priorities, HTB. For minimizing latency and bufferbloat, FQ_CODEL or Cake.
The Problem With TC: It's Egress-Focused
Here's TC's fundamental limitation: it works great on egress (outgoing traffic) because you control when packets leave. But on ingress (incoming traffic), packets have already arrived and consumed bandwidth before TC sees them. You can drop them, but the bandwidth is already spent.
This is the same source vs. destination problem we discussed earlier. TC on ingress can classify and redirect traffic (send it to different interfaces or processes), but it can't prevent traffic from arriving in the first place. For true ingress rate limiting, you need cooperation from the sender or upstream routers.
The workaround: use TC on ingress for accounting and classification, then use iptables or eBPF to drop traffic before it's fully processed.
eBPF: Traffic Control on Steroids
eBPF (extended Berkeley Packet Filter) is the modern way to do packet processing in the Linux kernel. While TC has been around for 25 years, eBPF is relatively new (mainlined in 2014, continuously expanded since). eBPF lets you write custom programs that run in the kernel, with safety guarantees (the verifier ensures your code can't crash the kernel).
For rate limiting, eBPF provides two major hooks:
XDP (eXpress Data Path): Runs at the earliest possible point, in the network driver before the packet is even allocated to the kernel's network stack. This is the closest you can get to hardware-level packet processing in software.
TC-BPF: eBPF programs that attach to TC hooks, combining TC's classification abilities with eBPF's programmability.
XDP for Extreme Rate Limiting
XDP programs can inspect packets and decide their fate in nanoseconds:
- XDP_PASS: Allow packet to continue up the network stack
- XDP_DROP: Drop the packet immediately, no further processing
- XDP_TX: Transmit the packet back out the same interface (packet reflection)
- XDP_REDIRECT: Send packet to a different interface
- XDP_ABORTED: Error, drop the packet and log
Here's a conceptual example of rate limiting with XDP:
// Simplified XDP program (actual code would be in C)
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, __u32); // Source IP
__type(value, struct rate_limit_state);
} rate_limits SEC(".maps");
SEC("xdp")
int rate_limit_by_ip(struct xdp_md *ctx) {
// Parse packet headers
void *data = (void *)(long)ctx->data;
void *data_end = (void *)(long)ctx->data_end;
struct ethhdr *eth = data;
struct iphdr *ip = data + sizeof(*eth);
// Bounds check (required by verifier)
if ((void *)(ip + 1) > data_end)
return XDP_PASS;
__u32 src_ip = ip->saddr;
// Look up rate limit state for this IP
struct rate_limit_state *state = bpf_map_lookup_elem(&rate_limits, &src_ip);
if (!state)
return XDP_PASS; // No limit configured for this IP
// Token bucket logic
__u64 now = bpf_ktime_get_ns();
__u64 elapsed = now - state->last_update;
// Add tokens based on elapsed time
state->tokens += (elapsed * state->rate) / 1000000000;
if (state->tokens > state->capacity)
state->tokens = state->capacity;
state->last_update = now;
// Check if we have tokens
if (state->tokens >= 1) {
state->tokens -= 1;
return XDP_PASS;
}
// No tokens, rate limit exceeded
return XDP_DROP;
}
This program inspects every incoming packet at the driver level, implements token bucket per source IP, and drops packets that exceed the rate limit. All in kernel space, with no system calls, no context switches, and no userspace involvement. Performance: millions of packets per second per core.
The Power and Limitations of eBPF Rate Limiting
Advantages:
- Speed: Operating at XDP means minimal overhead. You can process line-rate traffic (10/40/100 Gbps) on modern hardware.
- Efficiency: Drop unwanted packets before they consume CPU, memory, or bandwidth in the network stack.
- Flexibility: eBPF programs can implement arbitrary logic, not just predefined qdiscs.
- Safety: The eBPF verifier ensures programs can't crash the kernel, making it safer than kernel modules.
Limitations:
- Programming complexity: You're writing C code that runs in the kernel. Debugging is harder than userspace code.
- Limited context: XDP programs can't access full kernel data structures. They see raw packets and eBPF maps (hash tables, arrays).
- Verifier restrictions: The verifier enforces constraints (no unbounded loops, limited stack size, provable termination) that can be frustrating.
- State management: eBPF maps store state, but they're limited in size and have performance characteristics you must understand.
TC vs. eBPF: When to Use Which
Use TC when:
- You need egress traffic shaping (bandwidth limits, prioritization)
- You want simple rate limiting without custom code
- You need hierarchical bandwidth allocation (HTB)
- You're comfortable with TC's CLI syntax (or scripting it)
Use eBPF/XDP when:
- You need custom packet processing logic
- You're defending against DDoS (drop bad packets at line rate)
- You need per-flow or per-connection rate limiting with custom criteria
- Performance is critical (you need millions of packets/second)
- You want ingress rate limiting that actually prevents processing
Use both when:
- XDP drops bad traffic on ingress, TC shapes good traffic on egress
- You need comprehensive traffic management across the stack
Real-World Examples
Cloudflare's DDoS mitigation: Uses XDP extensively to drop attack traffic at the edge. Their systems inspect packets at 40+ Gbps per server and make drop decisions in microseconds. This is only possible with XDP.
Facebook's Katran: An XDP-based load balancer that makes forwarding decisions at line rate. It implements rate limiting alongside load balancing, all in eBPF.
Cilium: A Kubernetes networking solution that uses eBPF for everything, including rate limiting, network policy enforcement, and load balancing. It shows how eBPF can replace traditional iptables-based systems with better performance.
SRE at scale: Companies running high-traffic services use TC HTB for bandwidth guarantees between different service classes, ensuring critical services (API) get bandwidth even when bulk traffic (logs, metrics) is high.
The Practical Reality
Most engineers will never write eBPF code directly. It's complex, low-level, and requires deep networking and kernel knowledge. But tools built on eBPF (Cilium, Cloudflare's stack, Facebook's infrastructure) are becoming standard. You benefit from eBPF's performance without writing it yourself.
TC is more accessible but still cryptic. Fortunately, configuration management tools (Ansible, Terraform) and network automation can template TC configurations, so you don't need to memorize the syntax.
For most applications, you'll rate limit at the application layer (simpler, more context) and use TC or eBPF for infrastructure-level protection (DDoS mitigation, bandwidth guarantees). The kernel handles the heavy lifting of traffic control, your application handles business logic rate limits.
But when you need extreme performance, when you're defending against massive attacks, or when you're building infrastructure at scale, TC and eBPF are your tools. They let you say "no" at millions of packets per second, and sometimes that's exactly what you need.
Where to Apply Rate Limits
At the Edge (CDN, API Gateway)
The best place to rate limit is as early as possible. If you can reject requests at your CDN or API gateway, you never consume resources in your application servers, databases, or backend services.
Cloudflare, Fastly, AWS API Gateway, all provide rate limiting. They're distributed globally, can handle massive request rates, and reject requests before they reach your infrastructure. This is rate limiting at the source (relative to your systems).
The downside: your edge needs to know rate limit state. For per-user limits, this means distributed state synchronization or accepting some inaccuracy (each edge node enforces limits independently, so a user might get 100 req/s per node instead of 100 req/s total).
At the Load Balancer
Your load balancer is the next choke point. NGINX, HAProxy, and cloud load balancers all support rate limiting. This protects your application servers from overload and provides a centralized point of control.
The challenge is that your load balancer becomes a bottleneck. If it's doing complex rate limit logic for every request, you've just moved the scaling problem.
In the Application
Application-level rate limiting gives you the most control. You can make sophisticated decisions based on user state, request cost, system load, and business logic. You can also prioritize and shed load intelligently.
The cost is that requests reach your application before being rate limited, so you're still consuming network bandwidth, connection state, and parsing overhead. But for complex rate limiting requirements, this is often necessary.
Layered Approach
The best solution is usually layered:
- Edge: Coarse-grained limits to block obvious abuse (1000 req/s per IP)
- Load balancer: Medium-grained limits to protect infrastructure (100 req/s per user)
- Application: Fine-grained limits based on operation cost and business logic (10 writes/s per user, 100 reads/s)
Each layer provides defense in depth, and if one fails, others still provide protection.
The Communication Problem
When you rate limit a client, you should tell them:
- That they were rate limited (not a generic error)
- What their limit is
- When they can retry
- How many requests they have remaining
HTTP has standard headers for this:
HTTP/1.1 429 Too Many Requests X-RateLimit-Limit: 100 X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1678901234 Retry-After: 60
This tells the client: you hit your limit of 100 requests, you have 0 remaining, the limit resets at Unix timestamp 1678901234, and you should retry after 60 seconds.
Well-behaved clients respect these headers and back off appropriately. Misbehaving clients ignore them and retry immediately, making the problem worse. You can't fix bad clients, but you can make good clients better.
The Hard Truths
Rate limiting is fundamentally about saying no. Your system has finite capacity, and when demand exceeds that capacity, someone doesn't get served. Rate limiting is how you choose who.
There's no perfect algorithm. Token bucket is good for most use cases but allows bursts. Leaky bucket provides smooth output but penalizes legitimate bursts. Fixed windows are simple but have boundary issues. Sliding windows are accurate but expensive. Adaptive limiting is smart but complex to tune.
You will reject legitimate users. When you're under load and shedding requests, some percentage of good traffic gets dropped. That's the trade-off: reject some to save the system for everyone else. Without rate limiting, you reject everyone by crashing.
Rate limiting at the destination is expensive. You want to rate limit as early as possible, ideally at the source or at the edge. But sometimes you need application-level context to make good decisions, so you accept the overhead.
Clients will misbehave. Some ignore your rate limit headers and retry aggressively. Some will try to game your limits by spreading requests across IPs or user accounts. You need to detect and handle this, which adds complexity.
Implementation Considerations
State Management
Rate limiting requires state: how many requests has this user made in the current window? In a distributed system, this state must be shared. Options:
- In-memory (single node): Fast but not shared, each node enforces limits independently
- Redis/Memcached: Shared state, fast, but adds a dependency and network latency
- Database: Durable but slow, usually not fast enough for per-request checks
- Eventually consistent: Accept some inaccuracy in exchange for performance, each node synchronizes periodically
The choice depends on your consistency requirements and scale. For most applications, Redis is the sweet spot: fast enough, consistent enough, and widely supported.
Performance
Rate limit checks happen on the hot path, for every request. They must be fast, ideally sub-millisecond. This means:
- In-memory or distributed cache for state
- Efficient data structures (not iterating through lists)
- Minimal computation per request
- Avoiding locks where possible (optimistic concurrency)
A slow rate limiting implementation is worse than no rate limiting, you've added latency without improving capacity.
Testing
Rate limiting is hard to test because it involves timing and concurrency. You need to verify:
- Limits are enforced accurately
- Boundary conditions work correctly
- Concurrent requests don't leak past limits
- System performs well under rate limit pressure
- Limit resets happen as expected
Load testing is essential. You need to actually generate traffic at scale and verify your rate limiting works under realistic conditions.
The Anti-Patterns
Unlimited queues: "We'll just buffer everything!" No, you'll run out of memory and crash. Queues must be bounded.
No monitoring: If you don't measure how often you're rate limiting, you don't know if your limits are too strict or too loose. Instrument everything.
Same limits for everyone: Treating all users identically means your VIP customers get the same limits as free-tier users or attackers. Differentiate.
Rate limiting without backpressure: If you rate limit at the application but don't communicate to clients, they'll retry aggressively and make things worse.
Complex algorithms without reason: Token bucket is simple and works well. Don't implement adaptive sliding window with priority queuing unless you've proven you need it. Start simple.
The Philosophical Bit
Rate limiting is admission that your system has limits. This is uncomfortable because we like to think we can scale infinitely. Add more servers! More cloud! More money! But there's always a bottleneck, database writes, external API limits, physics.
Good rate limiting is about honesty: being honest about your limits, communicating them clearly, and failing gracefully instead of catastrophically. It's accepting that you can't serve everyone and choosing to serve some well rather than failing everyone equally.
It's also about economics. Your system costs money to run, and unlimited free usage is unsustainable. Rate limits let you offer free tiers while protecting paid tiers, making the business viable.
The best rate limiting is invisible. Users stay within limits naturally, experiencing fast responses and no errors. You only notice rate limiting when it's working badly or when someone's trying to abuse the system.
Conclusion: Learning to Say No
Rate limiting is one of those unglamorous but essential technologies. It's not exciting, it won't impress at conferences, but it's the difference between a system that degrades gracefully under load and one that collapses catastrophically.
Start with simple token bucket rate limiting at your edge or load balancer. Measure how often you're hitting limits. Tune based on actual usage patterns. Add per-user or per-tier limits as needed. Implement proper headers so clients can back off intelligently. Monitor and alert on rate limiting metrics.
When your system is melting down under unexpected load, you'll be grateful you implemented rate limiting. When you're debugging why performance degraded at 3 AM, you'll appreciate having clear rate limit metrics showing exactly where the problem started. When you're explaining to the CFO why your cloud bill didn't explode during that viral moment, rate limiting is your hero.
It's not glamorous, but it works. And in production systems, working is better than elegant every single time.
Now go forth and rate limit responsibly. Your future self, your users, and your infrastructure will thank you. Just remember: the best rate limit is the one you never hit, and the worst rate limit is the one you implement after your system is already on fire.