From 4-minute timeouts to durable queue: production-grade MCP

There's a version of MCP server development that looks like this: write a function, wrap it in a tool definition, test it locally, ship it. The tool works. The integration test passes. The client times out in production at 4 minutes on Thursday.

This is the story of that Thursday, what it revealed, and what production-grade MCP actually requires when your server is handling a fleet of agents under concurrent load.

The incident

Two agents triggered overlapping recall operations for the same user within the same second. Both acquired write paths to the same memory record. Both started executing.

The shorter one completed. The longer one ran. It kept running — no timeout, no gate, no mechanism to stop it once a concurrent write had already landed. Four minutes in, the HTTP request timed out. The user saw a 504.

The memory record was valid. The data was fine. But from the user's perspective, something broke. From the monitoring perspective: two in-flight writes for the same user ID, one of them burning compute for four minutes before dying.

The root cause wasn't complex. Two missing constraints on every async path: a timeout, and a single-flight gate per user.

Why MCP makes this worse than it looks

MCP calls are synchronous from the client's perspective. The client sends a tool call; it waits for the response. If the server spawns an async task and that task runs for 4 minutes, the client waits 4 minutes — or times out, whichever comes first.

There's no native "we'll call you back when it's done" in the base MCP protocol. The server either responds or it doesn't, and the client's timeout is the backstop.

This is fine for fast operations — recall, read, simple lookup. It's a problem for anything that involves write, extraction, indexing, or any operation that runs LLM inference as a subroutine. Those operations can legitimately take seconds. In a fleet under load, seconds-long operations from multiple concurrent agents stack up.

The solution is to distinguish operations by their runtime characteristics and handle each class correctly.

The two constraints that go on every async path

Timeout — not just on the HTTP request, but on the underlying operation itself. The client timeout is a blunt instrument; it terminates the connection but doesn't terminate the server-side task. Without an operation-level timeout, the task continues burning resources after the client has given up.

Every async task in our stack now has a hard ceiling. Operations that can't complete within that ceiling are cancelled, not abandoned.

Single-flight gate per user — if a write operation is already in flight for a given user ID, subsequent writes for that user queue rather than start a concurrent path.

This is not a lock. A lock blocks and creates backpressure. The single-flight gate queues: the second write starts when the first completes. The user's request gets a valid response instead of a race condition. The result is deterministic; the order is preserved by the gate rather than by execution timing.

# Simplified single-flight gate pattern
_in_flight: dict[str, asyncio.Lock] = {}

async def gated_write(user_id: str, write_fn):
    if user_id not in _in_flight:
        _in_flight[user_id] = asyncio.Lock()
    async with _in_flight[user_id]:
        return await asyncio.wait_for(write_fn(), timeout=WRITE_TIMEOUT_SEC)

Every write path goes through the gate. The timeout wraps the write function. Two lines of change per async path eliminated the class of incident we hit.

Fast vs. slow vs. very slow: a practical split

Not every operation belongs in the same category.

Fast (< 500ms target): synchronous response. Recall, read, lookup, tag resolution. These should complete in well under a second and return directly in the MCP tool response. No queue, no polling, no complexity.

Slow (500ms–30s): accept fast, return task ID, client polls. Writes, extractions, anything that involves a small amount of LLM inference. The tool call returns immediately with a task ID. The client polls a status endpoint or subscribes via SSE. When the operation completes, the result is available.

Very slow (30s+): background job with notification. LLM extraction pipelines, skill synthesis, batch operations. These run outside the request cycle entirely. The client gets a job ID; completion is signaled via webhook or SSE subscription. Attempting to hold a synchronous MCP connection open for a 2-minute extraction job will not work reliably across all MCP clients.

The classification isn't fixed — it shifts as your server scales and as the fleet grows. An operation that's reliably fast at 10 concurrent users may drift into "slow" territory at 1,000. Instrument everything; the boundary between synchronous and queue-backed will become obvious from your p99 latency data.

Structured error codes

The last production gap: MCP clients can't distinguish between failure modes if every error looks the same.

A 500 from a timeout and a 500 from an auth failure are different problems with different recovery paths. If the client gets the same error shape for both, it can't make intelligent retry decisions, and the operator reading logs can't triage quickly.

Every tool response in our stack now returns a structured error body when something fails:

{
  "error": {
    "code": "WRITE_GATE_TIMEOUT",
    "message": "Write operation timed out after 30s. A prior write for this user may still be in progress.",
    "retryable": true,
    "retry_after_ms": 5000
  }
}

code is machine-readable. retryable tells the client whether to retry immediately or back off. retry_after_ms gives the client a floor. The operator has enough context to triage without reading source.

Signals your MCP server isn't production-ready

• Any async path that doesn't have an operation-level timeout (not just the HTTP timeout)
• Any write path without a per-user single-flight gate
• No structured error codes — all failures return the same shape
• Retry behavior undefined — clients have to guess whether to retry
• No telemetry on p99 latency per tool — you can't see the category boundaries shifting as load grows

None of these gaps are hard to close. They're also not things that show up in local testing or against a single MCP client in dev. They show up on Thursday, under fleet load, at 4 minutes.

What production-grade actually means

It means the constraints that matter — timeout discipline, write gating, structured errors — are not afterthoughts you add after the incident. They're the design surface for any operation that touches shared state under concurrent load.

MCP is the protocol; the server behavior is yours. The protocol doesn't enforce timeout discipline for you. The client doesn't enforce per-user gate semantics for you. Those are server-side properties, and they're the difference between a tool that works in testing and a server that works in a fleet.