System Design: Building an API Gateway Platform

In modern distributed microservice architectures, allowing client applications to communicate directly with dozens of individual backend services introduces severe operational problems. It forces clients to manage complex service addresses, tightens coupling, increases attack surfaces, and makes enforcing global policies like authentication, security filtering, and rate limiting nearly impossible.

An API Gateway Platform acts as the single, strongly unified ingress point for all external client traffic. It decouples external clients from internal service topology and provides a central control plane to enforce cross-cutting policies.

This case study details the engineering architecture required to design a highly scalable, non-blocking, and resilient API Gateway Platform capable of processing millions of requests per second.

1. Requirements & Core Constraints

An edge API gateway must execute high-throughput request processing without introducing noticeable latency overhead.

Functional Requirements

• Dynamic Request Routing: The gateway must route incoming requests to correct upstream microservices based on URL paths, HTTP methods, headers, or query parameters.
• Canary & Traffic Gating: It must dynamically shift traffic weights (e.g. routing 5% of requests to a new v2 release) without server restarts.
• Edge Security & Auth: The gateway must verify client signatures, decrypt SSL/TLS sessions, validate JWT tokens, and strip malicious request parameters.
• Global Rate Limiting: It must enforce per-client API quotas using algorithms like Token Bucket or Sliding Window to prevent service abuse.

Non-Functional Requirements & SLAs

• Ultra-Low Latency Overhead: The gateway sits on the critical path of every request. Its internal processing time (routing, token validation, filter execution) must be under 3 milliseconds at P99.
• High-Throughput Connection Scaling: The system must handle an average of 1 Million concurrent requests per second (RPS), utilizing non-blocking asynchronous event loops.
• Resilience & Fault Isolation: Failures in one upstream microservice must never saturate gateway thread pools or cause cascading outages across unrelated services (using circuit breakers and bulkheading).
• Zero-Downtime Configurations: Routing maps and rate-limit parameters must update dynamically within seconds across the entire proxy fleet.

Back-of-the-Envelope Estimates

Let's calculate the system scaling bounds for an edge gateway fleet processing 1 Million requests per second (RPS):

• Ingress Network Bandwidth: If the average incoming request size (including headers, cookies, and payload body) is 3 Kilobytes, and the average response size is 8 Kilobytes: $$\text{Total Ingress Bandwidth} = 1,000,000 \text{ RPS} \times 3 \text{ KB} \approx 3.0 \text{ GB/sec} \approx 24 \text{ Gbps}$$ $$\text{Total Egress Bandwidth} = 1,000,000 \text{ RPS} \times 8 \text{ KB} \approx 8.0 \text{ GB/sec} \approx 64 \text{ Gbps}$$

• Proxy Server Fleet Sizing: Assume a high-performance, non-blocking asynchronous proxy node (such as Envoy or Netty) running on an 8-core CPU can process 25,000 RPS (including SSL decryption and JWT validation overhead). $$\text{Minimum Active Server Count} = \frac{1,000,000 \text{ RPS}}{25,000 \text{ RPS/node}} = 40 \text{ nodes}$$ Accounting for a $N+2$ disaster recovery redundancy and a 50% safety cushion during sudden traffic peaks: $$\text{Total Edge Proxy Fleet Sizing} = 40 \text{ nodes} \times 1.5 + 2 \approx 62 \text{ nodes}$$

2. API Design & Core Contracts

The gateway requires a structured, declarative configuration format to manage routing rules, upstream clusters, and filter pipelines. Below is the JSON schema representing a dynamic gateway routing policy.

Ingress Route Configuration Contract

POST /api/v1/gateway/routes

• Request Payload (JSON):

{
  "route_id": "route_checkout_service",
  "match": {
    "path_prefix": "/api/v1/checkout",
    "headers": {
      "X-Device-Type": "mobile"
    },
    "methods": ["POST", "PUT"]
  },
  "filters": [
    {
      "type": "jwt_auth",
      "config": {
        "issuer": "https://auth.codesprintpro.com",
        "audience": "payment_gateway"
      }
    },
    {
      "type": "rate_limiter",
      "config": {
        "rate_limit_key": "user_id",
        "replenish_rate": 100,
        "bucket_capacity": 200
      }
    }
  ],
  "upstream_cluster": {
    "service_name": "checkout-microservice",
    "connection_timeout_ms": 200,
    "max_connections": 1024,
    "circuit_breaker": {
      "failure_threshold_ratio": 0.05,
      "recovery_time_ms": 5000
    }
  }
}

• Response Payload (JSON):

{
  "route_id": "route_checkout_service",
  "status": "ACTIVE",
  "version": 42,
  "last_synced_at": "2026-05-22T12:00:00Z"
}

3. High-Level Design (HLD)

To handle extreme concurrent connections without thread exhaustion, the API Gateway avoids synchronous, blocking thread-per-connection architectures. Instead, it utilizes an asynchronous Reactive Event Loop Architecture (similar to Envoy Proxy or Zuul 2).

Edge Request Flow & Middleware Filter Pipeline

The diagram below deconstructs the request lifecycle from a client browser down through the gateway's internal filter chain to the upstream microservices.

graph TD
    Client([User Client]) -->|1. HTTPS Request| GSLB[Global Server Load Balancer]
    GSLB -->|2. Route to Regional DC| L4LB[Layer 4 Load Balancer: Maglev/IPVS]
    L4LB -->|3. Forward Packets| Gateway[Envoy API Gateway Node]
    
    subgraph Gateway Filter Pipeline
        Gateway -->|4. Terminate SSL & Match Route| Router[Routing Engine]
        Router -->|5. Validate JWT| AuthFilter[Auth Middleware]
        AuthFilter -->|6. Check Rate Limit| RateLimiter[Rate Limit Middleware]
        RateLimiter -->|7. Verify Token| Redis[(Redis Rate Limit Cluster)]
        
        RateLimiter -->|8. Fetch Upstream IP| Registry[Consul Service Registry]
    end
    
    Registry -->|9. Proxy Forward| Upstream[Upstream Microservices Fleet]

Dynamic Configuration Rollout Pipeline

To update routing and gating configurations without causing packet drops or server restarts, the gateway uses a decoupled control plane pattern.

graph LR
    Admin[Admin Console] -->|1. Update Route JSON| ConfigStore[(Consul / etcd Config Store)]
    ConfigStore -->|2. Trigger Watch Event| ControlPlane[Gateway Control Plane: xDS API Server]
    ControlPlane -->|3. Push Delta Route Maps via gRPC| GatewayNodes[Envoy Edge Gateway Nodes]
    GatewayNodes -->|4. Hot Reload Router Memory| GatewayNodes

4. Low-Level Design (LLD) & Data Models

The gateway requires storage structures to manage dynamic route mappings and track rate-limiting allocations without committing blocking operations to disk.

Relational Schema (PostgreSQL): Gateway Route Registry

-- Represents registered upstream service targets
CREATE TABLE upstream_services (
    service_id VARCHAR(64) PRIMARY KEY,
    service_name VARCHAR(255) NOT NULL UNIQUE,
    health_check_url VARCHAR(512) NOT NULL,
    max_connections INTEGER NOT NULL DEFAULT 1000
);

-- Represents dynamic routing path rules
CREATE TABLE gateway_routes (
    route_id VARCHAR(64) PRIMARY KEY,
    path_pattern VARCHAR(255) NOT NULL,
    service_id VARCHAR(64) REFERENCES upstream_services(service_id),
    is_active BOOLEAN NOT NULL DEFAULT TRUE,
    traffic_weight INTEGER NOT NULL DEFAULT 100, -- Used for canary splits (0-100)
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- Create index to enable sub-millisecond route matching lookups
CREATE INDEX idx_active_routes ON gateway_routes (is_active, path_pattern);

Compilable Java Implementation: Sliding Window Rate Limiter Filter

This compilable Java class simulates the internal filter execution of our API gateway. It implements a non-blocking sliding window rate limiter using a queue array of epochs to track and restrict client request rates safely without thread contention.

package com.codesprintpro.gateway;

import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.logging.Logger;

public class SlidingWindowRateLimiter {
    private static final Logger logger = Logger.getLogger(SlidingWindowRateLimiter.class.getName());
    
    private final int maxRequestsPerWindow;
    private final long windowSizeMs;
    
    // In-memory request tracking: Map{ClientID -> Queue{Request Timestamps}}
    private final ConcurrentHashMap<String, ConcurrentLinkedQueue<Long>> clientRequestWindows = new ConcurrentHashMap<>();

    public SlidingWindowRateLimiter(int maxRequestsPerWindow, long windowSizeMs) {
        this.maxRequestsPerWindow = maxRequestsPerWindow;
        this.windowSizeMs = windowSizeMs;
    }

    public boolean isAllowed(String clientId) {
        long now = System.currentTimeMillis();
        long windowStart = now - windowSizeMs;

        // Retrieve or compute a thread-safe sliding window queue for the client
        ConcurrentLinkedQueue<Long> requestTimestamps = clientRequestWindows.computeIfAbsent(clientId, 
            k -> new ConcurrentLinkedQueue<>()
        );

        // Evict expired timestamps outside the current sliding window
        while (!requestTimestamps.isEmpty() && requestTimestamps.peek() < windowStart) {
            requestTimestamps.poll();
        }

        // Lock-free check and request incrementation
        synchronized (requestTimestamps) {
            if (requestTimestamps.size() < maxRequestsPerWindow) {
                requestTimestamps.add(now);
                logger.info("Request allowed for client '" + clientId + "'. Current Window Count: " + requestTimestamps.size());
                return true;
            } else {
                logger.warning("Rate limit exceeded for client '" + clientId + "'. Requests Blocked!");
                return false;
            }
        }
    }

    // Direct local verification check
    public static void main(String[] args) throws InterruptedException {
        // Rate limit: Max 3 requests in a 1-second (1000ms) sliding window
        SlidingWindowRateLimiter limiter = new SlidingWindowRateLimiter(3, 1000);
        String testClient = "client_app_abc";

        // Send 3 rapid requests (should be allowed)
        assert limiter.isAllowed(testClient) == true;
        assert limiter.isAllowed(testClient) == true;
        assert limiter.isAllowed(testClient) == true;

        // Send 4th request (should be blocked)
        boolean fourthRequest = limiter.isAllowed(testClient);
        System.out.println("Fourth request allowed: " + fourthRequest + " (Expected: false)");

        // Wait 1.1 seconds for sliding window to shift
        Thread.sleep(1100);

        // Request should now be allowed
        boolean postWaitRequest = limiter.isAllowed(testClient);
        System.out.println("Request after window shift allowed: " + postWaitRequest + " (Expected: true)");
    }
}

5. Scaling Challenges & Bottlenecks

Operating a global edge gateway at scale requires removing standard database locks from the critical connection path.

Distributed Rate Limiting Latency and Race Conditions

Using a central database to track client quotas adds huge latency overhead. In contrast, using a simple local cache (in-memory map) fails if clients are routed across different gateway nodes.

• The Solution: We deploy a Redis Cluster in each availability zone. Bidding or client sessions execute a non-blocking Redis Lua Script to check and update quotas atomically using a Sliding Window algorithm. By executing the calculation entirely inside the local Redis shard, we avoid network roundtrips and guarantee that rate-limiting adds less than 1 millisecond of latency.

Configuration Reloads Without Packet Loss

Updating routing configurations by restarting proxy servers kills active TCP connections, causing service drops.

• The Solution: We decouple the Control Plane from the Data Plane using Envoy's xDS APIs. Routing rules are maintained in a distributed configuration store (Consul/etcd). The Control Plane detects updates and pushes them to the proxy fleet via dynamic gRPC streaming. The proxy nodes swap the routing memory arrays atomically, ensuring zero-downtime updates and zero dropped packets.

6. Technical Trade-offs & Compromises

Deciding how to allocate resources within our API gateway involves critical architectural balances.

Blocking Thread-Per-Connection vs. Non-Blocking Event Loops

                       ┌──────────────────────────────┐
                       │    Gateway Connection Model  │
                       └──────────────┬───────────────┘
                                      │
            ┌─────────────────────────┴─────────────────────────┐
            ▼                                                   ▼
┌─────────────────────────────────────┐             ┌─────────────────────────────────────┐
│    Blocking Thread-per-Connection   │             │     Non-Blocking Event Loops        │
├─────────────────────────────────────┤             ├─────────────────────────────────────┤
│ • 1 Thread allocated per client     │             │ • 1 Thread handles thousands of TCP │
│ • Easy to write and debug           │             │ • Complex reactive programming      │
│ • High memory overhead (1MB/thread) │             │ • Zero memory overhead (Event Epoll)│
│ • Crashes under high concurrency    │             │ • Scales linearly to 1M+ active conns│
└─────────────────────────────────────┘             └─────────────────────────────────────┘

• Blocking Thread-per-Connection: Easy to design and debug, but runs out of memory quickly because Linux OS threads require approximately 1 Megabyte of stack memory each.
• Non-Blocking Event Loops: Uses single threads to handle thousands of concurrent TCP sockets by registering events on an OS event queue (Epoll/Kqueue).
• Staff Verdict: We choose the Non-Blocking model. While it increases code complexity, it allows our gateway fleet to handle millions of active concurrent connections with a minimal memory footprint.

Centralized vs. Decentralized Authentication Verification

• Centralized Auth: The gateway calls a central auth microservice for every request. This keeps validation logic simple but introduces a single point of failure and adds 10ms of latency.
• Decentralized Cryptographic Auth: The gateway decodes and validates cryptographically signed JWT tokens locally using public keys. This bypasses the central auth database entirely, reducing check latencies to under 0.5ms. We accept the tradeoff that revoking a compromised token requires waiting for its TTL to expire, or checking a fast Redis blacklist cache.

7. Failure Scenarios & Operational Resiliency

API Gateways must isolate downstream service failures to prevent cascading cluster outages.

Cascading Outages & Thread Pool Saturation

If an upstream microservice (e.g. shipping-service) becomes slow, the gateway's outbound thread pools can become saturated while waiting for responses, blocking requests to healthy microservices (e.g. cart-service).

• Mitigation: We enforce strict Circuit Breakers and Bulkhead Isolation. If the error or latency rate of a downstream service exceeds 5%, the circuit breaker trips. The gateway immediately rejects subsequent requests to that service with an HTTP 503 error, bypassing the network hop entirely. Downstream services are isolated in separate, bounded bulkhead connection pools, protecting the gateway's core memory resources.

Mitigating DDoS Attack Floods

Under a large-scale distributed denial-of-service (DDoS) attack, the gateway must protect the internal network.

• Mitigation: We run Shielding Proxies (like Cloudflare) at the network edge to absorb volumetric DDoS attacks. The gateway fleet is hidden inside a private VPC, only accepting connections that present authenticated mTLS (Mutual TLS) certificates from the edge load balancer.

8. Candidate Verbal Script

Mock Interview Sequence

Interviewer: How would you design a highly scalable API Gateway that handles authentication, rate limiting, and request routing for thousands of microservices, without adding significant latency to the request path?

Candidate: "To design a high-throughput, low-latency API Gateway, I would build the data plane using a non-blocking asynchronous event proxy like Envoy or Netty. This allows a single worker thread to handle thousands of concurrent TCP sockets using OS-level event polling (like Linux Epoll), keeping memory consumption low.

To keep latency under 3 milliseconds, we must decouple the gateway from centralized database lookups.

First, for Authentication, instead of calling a central auth service for every request, we utilize cryptographically-signed JSON Web Tokens (JWTs). The gateway caches the public keys of the Auth Server locally and validates the JWT signature in-memory in less than 0.5ms.

Second, for Rate Limiting, we implement a distributed sliding window rate limiter. We run a high-speed Redis cluster in each availability zone. The gateway executes an atomic Redis Lua script that updates and checks the client's quota within 1ms, preventing cross-node race conditions.

Third, for Routing, we separate the Control Plane from the Data Plane. Routing configurations are maintained in Consul or etcd. When a routing rule is updated, the Control Plane pushes the changes to Envoy via gRPC. Envoy hot-reloads the routing maps atomically in-memory without restarting or dropping active connections. This design keeps the gateway extremely fast, highly available, and completely stateless."

Interviewer: What happens if the Redis rate-limiting cluster goes down? Does the gateway block all requests or fail-open?

Candidate: "A senior staff engineer always designs for Fail-Open Resilience. If the Redis rate-limiting cluster becomes unreachable, the rate limiter filter will catch the connection timeout exception, fire a high-priority alert to Prometheus, and instantly bypass the rate limit check to allow the requests through.

We prioritize Availability over absolute quota enforcement during outages, ensuring that a transient cache failure does not cause a total platform outage for our users."

Read Next

• System Design: Designing an Authorization Service (Zanzibar)
• Deep Dive: gRPC vs. REST API Protocols
• System Design: Distributed Rate Limiter (Redis Token Bucket)