Push, don't poll

Plenty of software does its real work in the background. Someone starts an analysis, a worker picks it up, and minutes later it finishes. Between those two moments there's a person with a browser tab open, waiting to find out.

Getting the news to them sounds trivial. I architected a notification microservice on Redis and WebSockets that delivers in under 100 milliseconds, and the interesting parts weren't Redis or WebSockets, which are both well understood. They were the problems that only appear when it has to work for a lot of people at once, all day, through deploys and bad networks.

1The cost of asking

The obvious design is polling. Every open tab asks the server, every few seconds, whether anything has happened. It's easy to build, easy to reason about, and it works perfectly in a demo.

It has two costs, and they pull in opposite directions. The first is latency: poll every T seconds and news waits on average T/2 before anyone asks for it, and up to T in the worst case. The second is load: a thousand tabs polling every two seconds is five hundred requests a second, and nearly all of them get the answer "nothing yet". Improve one and you worsen the other.

Pushing inverts this. Each browser opens one long-lived WebSocket, and the server sends a message when, and only when, there's something to say. Latency becomes one trip across the network. Load becomes proportional to things actually happening, rather than to the number of people waiting for them.

2One connection, many servers

The first problem appears as soon as there's more than one server. Connections are spread across gateway instances behind a load balancer, so each user's socket lives on one particular instance. The worker that finishes a job has no idea which one. Hand the event to the wrong instance and there's nobody there to receive it.

Redis pub/sub solves this neatly. Each gateway subscribes to a channel for every user connected to it, and unsubscribes when they leave. The worker publishes the event once, to that user's channel, and Redis delivers it to whichever gateway is listening. Nobody needs to know where anyone is connected.

In code, a gateway is little more than a map from users to sockets and one subscriber connection to Redis:

import redis.asyncio as redis

r = redis.Redis()
pubsub = r.pubsub()            # one subscriber connection per gateway
sockets = {}                   # user id -> this gateway's sockets for that user

async def on_connect(user_id, ws):
    sockets.setdefault(user_id, set()).add(ws)
    await pubsub.subscribe(f"user:{user_id}")

async def on_disconnect(user_id, ws):
    sockets[user_id].discard(ws)
    if not sockets[user_id]:
        del sockets[user_id]
        await pubsub.unsubscribe(f"user:{user_id}")

async def relay():             # runs for the life of the gateway
    await pubsub.subscribe("gateway:ping")       # listen() needs a subscription to start
    async for msg in pubsub.listen():
        if msg["type"] != "message":
            continue
        user_id = msg["channel"].decode().split(":", 1)[1]
        for ws in sockets.get(user_id, ()):
            await ws.send(msg["data"])

# and in the worker, when a job finishes:
#     await r.publish(f"user:{user_id}", json.dumps(event))

Inside a data centre, the publish and the hop through Redis take on the order of a millisecond. Nearly all of a 100 ms budget goes on the last stretch to the browser, which is exactly the stretch that polling makes longer. That changes where the effort should go. Shaving the server is rarely worth much; not adding a polling interval on top of the network is worth everything.

3When everyone reconnects at once

The second problem arrives with the first deploy. Restart the gateways and every connected browser drops at the same instant. Each one, quite reasonably, tries to reconnect. If they all retry immediately, the new instances meet a wall of simultaneous handshakes. Many fail, and every failure retries. A routine deploy turns into a self-inflicted outage.

Waiting longer between retries isn't enough by itself. Exponential backoff spaces out each client's attempts, but if every client starts at the same moment and follows the same schedule, they stay in lockstep and arrive in synchronised waves. What breaks the waves is randomness. With full jitter, each client waits a random time between zero and its current backoff, and the herd smears into a steady stream the servers can absorb.

function nextDelay(attempt) {
  const base = 500, cap = 30_000;                     // milliseconds
  const backoff = Math.min(cap, base * 2 ** attempt);
  return Math.random() * backoff;                     // full jitter: anywhere in [0, backoff)
}

4Pub/sub forgets

The third problem is quieter. Redis pub/sub is fire and forget: a message published while nobody is subscribed is simply gone. A browser that was mid-reconnect at the moment its job finished will never hear about it, and a notification system that silently drops notifications is worse than one that is merely slow.

So the fast path needs a safety net. Every event is also written somewhere durable with an increasing ID; a Redis stream works, and so does a database table. Each client remembers the last ID it saw and asks for anything newer when it reconnects. Pub/sub gives you speed, the log gives you the guarantee, and neither is enough without the other.

The order of those two steps on reconnect matters more than it looks. The natural version reads the log first, to catch up, and then subscribes for whatever comes next. But an event published after the read and before the subscription takes effect lands in neither, and it's as lost as if there were no log at all. The other way round works. Subscribe, wait for Redis to confirm it, and hold any live messages that arrive; only then read the log, send what it returns, and follow it with the held messages, skipping any whose ID was already sent. The overlap can now produce a duplicate, but never a gap, and a duplicate is easy to drop by its ID.

On the gateway it's a short function. The worker now writes each event to a stream first and publishes it with the stream's ID, and the relay parks a resuming user's live messages in a list instead of sending them:

holding = {}                              # user id -> live messages parked while that user resumes

async def resume(ws, user_id, last_id):
    held = holding[user_id] = []
    await subscribed(f"user:{user_id}")   # 1. subscribe, and wait for Redis to confirm it
    backlog = await r.xrange(f"log:{user_id}", min=f"({last_id}")   # 2. only then read the log
    for msg_id, fields in backlog:        # 3. catch up, oldest first
        await send(ws, msg_id, fields["event"])
        last_id = msg_id
    while held:                           # 4. then whatever arrived meanwhile,
        msg_id, event = held.pop(0)       #    minus anything the log already covered
        if newer(msg_id, last_id):
            await send(ws, msg_id, event)
            last_id = msg_id
    del holding[user_id]                  # 5. and from here on, straight through

The ( makes the range exclusive, so the read starts just after the last event the browser saw. Stream IDs only ever increase, which is what makes the duplicate check a single comparison.

5One slow reader

The relay above has a flaw that only shows under load. It sends to each socket in turn and waits for each send to finish, and a WebSocket send waits whenever that connection's outgoing buffer is full: that's how the library pushes back on a writer that's going too fast. Most of the time the buffer is empty and the wait is nothing. But put one phone on a weak signal behind the gateway, let its buffer fill, and the relay stops at that phone's next message. Everyone else on the gateway waits behind it.

The cure is to stop any one connection from holding up the rest. Each connection gets its own short queue and its own writer task, and the relay only ever drops a message into a queue, which never waits. A slow connection now delays nobody but itself. Its queue needs a limit, too, because buffering without end for a client that can't keep up only moves the problem into the gateway's memory. When a client falls too far behind, the gateway closes the connection with the close code for "try again later", and the browser reconnects and asks the log for the rest, at whatever pace it can manage.

class Conn:
    def __init__(self, ws, limit=256):
        self.ws, self.closing = ws, False
        self.queue = asyncio.Queue(maxsize=limit)
        self.writer = asyncio.create_task(self.drain())

    async def drain(self):
        while True:                           # only this connection ever waits on its socket
            await self.ws.send(await self.queue.get())

    def offer(self, data):                    # relay() calls this instead of send; it never waits
        if self.closing:
            return                            # on its way out; the log has the message
        try:
            self.queue.put_nowait(data)
        except asyncio.QueueFull:             # too far behind to catch up live
            self.closing = True
            self.writer.cancel()
            asyncio.create_task(self.ws.close(code=1013, reason="try again later"))

6Is anyone there?

A connection can die without either end being told. A phone moves from Wi-Fi to mobile data, a laptop lid closes, a router somewhere forgets the address mapping. No goodbye is sent, so the browser goes on waiting on a socket that will never deliver, and the gateway goes on holding a subscription and writing into a connection that leads nowhere. TCP does notice eventually, but on its own schedule: its keepalive, when it's switched on at all, waits for two hours of silence on Linux before it sends its first probe.

There's a more mundane reason to send something now and then, too. Load balancers and proxies close connections that stay quiet for too long, after sixty seconds by default on both AWS's Application Load Balancer and nginx, and a notification socket is quiet nearly all of the time.

Heartbeats deal with both. The gateway sends a small message every twenty seconds and the browser answers it, and that traffic alone keeps the proxies from closing the connection. It has to be an ordinary message rather than the WebSocket protocol's own ping frame: browsers answer those automatically, but page code never sees them, so it couldn't tell when they stopped. If an answer is late, the gateway closes the connection and drops its subscription. If the heartbeats stop arriving, the browser reconnects without waiting to be told, and asks the log for anything it missed.

function watch(ws) {                      // call when it opens, and on every message
  clearTimeout(ws.deadline);
  ws.deadline = setTimeout(() => {         // a heartbeat is ten seconds overdue
    ws.onclose = null;                     // don't wait for a goodbye that won't come
    ws.close();
    reconnect();                           // jittered backoff, then "anything after lastId?"
  }, 20_000 + 10_000);
}

7Deploys without a storm

Jitter makes a reconnect storm survivable; draining makes it rare. When a container is about to be replaced, the orchestrator sends it SIGTERM and waits before killing it, thirty seconds by default on both Kubernetes and ECS. That's long enough to leave politely. Once the load balancer has stopped sending it new connections, the gateway closes its existing ones a few at a time across most of that window, each with the close code for "service restart". The browsers reconnect through the load balancer to instances that are staying, and ask the log for anything they missed. Instead of four thousand clients in the same second, a deploy becomes a steady trickle that nobody notices.

8Boring, on purpose

None of these pieces is novel, and the best compliment a notification system can get is that nobody thinks about it. People see their result the moment it's ready. A phone on a weak signal slows down only itself. A connection that dies quietly is noticed within half a minute. Engineers deploy without a reconnect storm. And nothing important depends on a message arriving through the one path that's allowed to drop it.