Appearance
Delivery semantics
Everything on this page is observable with an API key and curl.
At least once
Spooler delivers each message at least once, and a delivery may repeat. Write handlers so a repeat run is harmless: make the effect idempotent, or track handled message ids.
A repeat delivery is not a bug. It happens when:
- a lease times out while the handler is still working (renew prevents this),
- the consumer crashed before acking,
- an ack was sent but the connection dropped before the response,
- the spool's primary changed — in-flight leases are dropped and their messages become visible again immediately.
Leases
A receive does not remove the message — it leases it. While the lease is valid, no other receiver gets that message; a lease stops being valid when it expires, is settled, or predates a primary change (stale_token on the next call). Settling ends the delivery, one of six ways: ack removes the message, nack returns it for redelivery, release returns it without spending a retry, fail marks it failed now, discard removes it as dropped rather than processed (discardedTotal, not ackedTotal), and ack-and-send acks it while appending the next message (pipelines).
Renew is not a settle: it keeps the delivery leased and resets the expiry to now plus the queue's current lease timeout, and you keep presenting the same lease. A reset, not an extension: after a settings change the new expiry can land earlier than the old one, or go away. On a queue with a lease timeout, an unsettled lease times out and the message is redelivered; with none, a lease holds until it is settled or a primary change drops it.
Renew is also how a long-running worker learns its task was cancelled: a message discarded by handle while leased makes the next renew, like any settle, answer 410 stale_token. Renew on a schedule and stop on 410.
Nacks and lease timeouts increment the retry count (Spooler-Message-Retries on the next delivery) — an expired lease is a nack the server performs on the consumer's behalf (the reference calls this the auto-nack). After the queue's maxRetries the message is failed: it is in the failed state (what other queues call a dead-letter queue, here a state of the same message rather than a place it moves to) until it is recovered, discarded, or its retention runs out. A redelivery caused by a primary change repeats the delivery without incrementing the count: what a consumer does — or fails to do in time — moves a message toward failed; infrastructure events never do.
Release is the voluntary version of that: the delivery comes back as if the receive never happened. It asserts the message was not at fault — a shutdown, a wrong worker, a dependency briefly away. Nack is the error path: a handler that releases on every error never parks poison, because a release never moves a message toward failed. Fail is the opposite call — a message the consumer can tell is poison is failed at once, intact, for you to handle as a failed message.
The lease token is opaque. Never decode, compare, or construct one; the only valid use is presenting it back.
Failed messages
List them with state=failed&handle=true. Each returned message carries a handle: present it to inspect to read the payload, recover to return the message for delivery with its retries reset, or discard to drop it. A 410 means the message moved on since the listing — someone else recovered or discarded it, or it expired: list again.
What invalidates a handle
A handle names a delivery generation, not an exact state. It has no lifetime of its own: it is valid for as long as the message is, up to the plan's retention. It goes stale (410) when the message's generation moves on:
| Transition | Caused by |
|---|---|
| visible → pending | a receive |
| pending → delayed or visible | nack, release, a lease expiring |
| pending → failed | fail, retries exhausted |
| failed → delayed or visible | recover |
| any → gone | ack, ack-and-send, discard, retention expiry |
| any | a primary change |
It stays valid when the message only becomes more or less ready to deliver, and inspect then reports the current state:
| Transition | Caused by |
|---|---|
| delayed → visible | the delay elapsing |
| visible → unavailable → visible | a payload load failing on the server and being retried |
So a handle to a delayed message still inspects and discards it once it is visible: nobody has received it, and the message you looked at is the message you are cancelling. If a consumer receives it first, that receive is the first row of the first table, and your discard answers 410.
Cancelling a send
A send with handle=true returns a handle in Spooler-Handle alongside the id. Its one use is to take the message back before anyone receives it: present it to discard and the message is gone. Once a delivery has happened the handle is stale and discard answers 410 — the consumer has it, and the at-least-once rule applies as for any delivery.
This is what makes a delay an escape hatch rather than a fuse: send the reminder, the payment retry, or the escalation with a delay and keep the handle; when the condition it was waiting on is met first, discard it. It also withdraws a send whose surrounding work failed to commit. The handle is a bearer credential, so it is minted only on request.
Cancellation is a race with delivery, not a transaction around the send: a discard can lose to a receive and answer 410, because the receive moved the message to a new generation. The message is still cancellable — list it (state=pending&handle=true) for a fresh handle and discard that; the consumer's next renew or settle answers 410, which is how a long-running worker learns to stop. What no cancel undoes is the work the consumer did before it noticed. A delay gives the producer time to cancel before delivery; it does not make the send atomic with another transaction — a producer that crashes before committing and cancelling leaves a deliverable message, so a worker that needs committed data must verify it before acting. A dedup hit (200) returns no handle: the original send's handle, if one was minted, is the one that cancels the message.
Pipelines
A stage that consumes from one queue and produces into the next settles both sides in one call: ack-and-send acks the delivery and appends the new message atomically. There is no moment where the input is acked and the output missing, or the output present and the input still pending. To try it, create a queue named next-stage, then receive a message from test without acknowledging it (SPOOLER_API and SPOOLER_KEY as in the quickstart):
sh
curl -si -X POST "$SPOOLER_API/spools/default/ack-and-send" \
-H "Authorization: Bearer $SPOOLER_KEY" \
-H "Spooler-Lease: <the spooler-lease header value>" \
-H "Spooler-Queue: next-stage" \
--data-binary 'hello, next stage'http
HTTP/2 201
spooler-message-id: 1-2Spooler-Queue names the queue the new message lands in; the lease names the message being acked. 201 means both happened.
The move is exactly-once without a dedup key: the ack half consumes the delivery, so a retry of a committed call fails with stale_token and appends nothing. Both halves carry the spool's replication guarantee together: on synchronous copies a redelivery to another consumer means the first call never committed; with no synchronous copy a primary change can lose a committed call, input and output alike, and redeliver the input. What stays at-least-once in every case is the stage's own side effects before the call — the same rule as for any handler.
A dedup key on the new message is for a different job: folding many inputs into one output, or guarding against duplicates already upstream. A hit refuses the whole call with 409 kind dedup_claimed and acks nothing; the body names the message holding the key. Ack the input explicitly if folding into it was the intent.
Rate limit
A queue can cap how fast it hands out messages. recvRateLimit in the queue's settings is a token bucket: intervalMicros is the sustained pace, one message per interval, and burst is how many an idle queue may release at once before the pace takes over. The cap is enforced on the server and shared by every consumer of the queue, so pacing a stage to what a downstream system tolerates needs no limiter in each worker. A receive that arrives before the next message is due waits for it inside its waitSeconds window; without a wait, or once the window elapses, it returns nothing (204), the same as an empty queue.
This is a queue setting about delivery. The account's request rate limit is separate: it caps requests of any kind and answers 429 — see limits.
Ordering
Spooler is not a FIFO queue. Visible messages are offered oldest first — approximately send order — but delivery order is not a contract:
- a delayed message becomes visible after messages sent later,
- a redelivery returns after messages sent later were already consumed,
- concurrent consumers settle in whatever order they finish.
Work that must happen in order belongs in one message, or behind sequencing you own. In exchange, nothing blocks the queue head: a failing message retries and fails alone while everything behind it flows.
Durability
On the hosted service a 201 from send returns after the message is written and synced to disk and the copies the spool's replication settings require have confirmed it. A 204 from ack confirms the acknowledgement with the same durability. What that survives — a crash, a restart, the loss of a node or of its storage — is set by the spool's numSync and spelled out in replication: with synchronous copies no acknowledged write is lost; with none, a primary change can lose the writes acknowledged since the surviving copy last caught up, an acked message included. The local image keeps everything in memory and promises none of this.
What a primary change does not preserve is in-flight leases — their messages are redelivered (above). A 500 with kind operation_unconfirmed means the write was accepted but its durability is unknown; retry a send only on a queue with a dedup window, carrying the same key — see deduplication.
Retention
effectiveRetentionSeconds is the period the queue enforces: retentionSeconds bounded by the plan. For a new message the clock starts when it first becomes visible, so a send delay never eats into it.
A leased message is never expired while a consumer holds it, but its clock keeps running: a nack, a release or a lease timeout returns the message with the clock as it was, and one whose period has already passed expires at once. Failing a message starts a fresh period, and recovering it starts another from the moment it becomes visible again.
Failed messages expire too: recover or discard them before their period ends. Past its period a message is discarded and expiredTotal in the spool stats counts it. A growing rate means messages age out before consumers reach them.
Lifecycle
Everything a message can be, and every call that moves it — this page in one picture. Each edge is an operation, a Spooler-* header, or a queue setting.
diagram
[delay set]
send
ack-and-send
release
recover
┌────────────────┐
│ ▼
│ ┌─────────┐
│ │ delayed │
│ └─────────┘
│ │ renew
│ │ ┌───┐ ack
│ │ │ │ ack-and-send
│ send ▼ recv │ ▼ discard
[*]────────▶ ┌─────────┐ ──────────────▶ ┌─────────┐ ─────────────▶ gone
│ visible │ │ pending │
┌────────── └─────────┘ ◀────────────── └─────────┘
│ discard ▲ nack │ fail
│ expired │ release │ [retries exhausted]
│ │ lease expired │ nack
│ │ │ lease expired
│ │ │
│ │ │ discard
▼ │ ▼ expired
gone │ ┌────────┐ ─────────────▶ gone
│ recover │ failed │
└────────────────────── └────────┘One state is not drawn: unavailable — the payload failed to load and is retried on its own; the message returns to visible by itself, and a handle survives the trip.