Production

Nothing in capabilities changes when you scale. Only how the runtime starts.

Boot

const pool = new pg.Pool({ connectionString: DATABASE_URL });
pool.on('error', (error) => log.error(error));
await migrate(pool);
const runtime = createRuntime({ capabilities, store: createPostgresStore(pool), instance: process.env.HOSTNAME, observe: (e) => log.info(e) });
const workers = runtime.start();

async function shutdown() {
  server.close();
  await workers.stop(); // finish in-flight deliveries before releasing the pool
  await pool.end();
}
for (const signal of ['SIGINT', 'SIGTERM'] as const) process.once(signal, () => void shutdown());

Cursors are persisted, so whatever a previous crash left undelivered goes first.

Pools and workerStore

Give the workers their own pool, so delivery never queues requests behind it (same database):

const runtime = createRuntime({
  capabilities,
  store: createPostgresStore(apiPool),          // dispatches, queries and await
  workerStore: createPostgresStore(workerPool), // deliveries, timers, leases
});

Local commits still wake the workers at once.

Deploy layouts

Both are safe:

1. N identical instances: every instance serves HTTP and runs workers; each consumer runs in whichever instance holds its lease, and instances split the consumers fairly.
2. Split roles: several API instances with runtime.start({ workers: false }) behind the load balancer, plus one or more worker instances (a spare takes over on failure). Scale the API part freely.

At least one process must run workers, or views go stale, timers never fire, await: 'views' never completes in time, and edges (push, email) never send. Give each process a stable, readable instance name: it appears in atlas_leases and in traces.

The MallFeed backend exposes this as two environment variables mapped to createRuntime({ instance }) and start({ workers }): an instance name (default <hostname>-<pid>) and a workers switch (default on).

Leases and failover

start({ leaseMs: 30_000, renewMs: 10_000 }) are the defaults. A clean stop() releases leases (takeover within about renewMs); a crash frees them when they expire (up to leaseMs + renewMs). The next owner resumes from the stored cursor. Never call drain() next to running workers.

Retries and timeouts

Setting	Default	Where
Conflict retries	retry: { attempts: 5, baseMs: 10, maxMs: 200 }	createRuntime. Full jitter; the last conflict answers conflict { attempts }.
Await bound	timeoutMs: 2000	dispatch(cmd, { await, timeoutMs }); over HTTP awaitTimeoutMs.
Outbox check	pollMs: 250, idleMs: 1000	start.
Delivery batches	delivery: { projections: 100, reactions: 20, subscribers: 10 }	createRuntime.
Cascade fuse	maxDepth: 1000	createRuntime.
Trace memory	traceLimit: 1000	createRuntime; wire observe to your logs.

Outages

• A lost connection fails the operation (store-failed, 503) or retries the delivery with backoff; the process lives.
• An identify that throws anything but unauthenticated() answers 503, so clients keep their token.
• Poisoned commits settle as store-rejected instead of blocking a consumer.
• The causes of decide-failed, query-failed, upcast-failed and store-failed are in runtime.trace() and observe, never in the response.

Monitoring

• A consumer’s cursor may trail entries it does not take by up to 5 s.
• failed events with store-rejected or cascade-exceeded, and warning events (lease-lost, consumer-starts-at-zero), need an operator.
• atlas cursor list --db <url> shows every cursor and its capability.

Upgrade notes

Run migrate() on every boot; schema changes are additive. Read the migration notes of each version you cross in Releases. The short version for 0.4 -> 0.5:

• Throw unauthenticated() from identify for bad credentials (or pass identifyErrors: 'unauthenticated' while migrating).
• Conflicts now retry 5 times with a wait; createRuntime({ retry: { attempts: 3, baseMs: 0 } }) restores 0.4.
• Optional: workerStore, and N identical worker instances now share the load.