Deduplication

Last updated Sep 22, 2026

Send the same message twice, store it once: on a queue with a dedup window, pass Spooler-Dedup-String on a send, and a repeat send with the same key inside the window appends nothing — it returns 200 with the original message's id (SPOOLER_API and SPOOLER_KEY as set in the quickstart).

Create a queue with a window:

curl -X PUT "$SPOOLER_API/spools/default/queues/orders" \
  -H "Authorization: Bearer $SPOOLER_KEY" \
  -H "Content-Type: application/json" \
  --data '{"dedupWindowSeconds": 60}'

Returns 201. Then send the same keyed message twice:

curl -si -X POST "$SPOOLER_API/spools/default/queues/orders/send" \
  -H "Authorization: Bearer $SPOOLER_KEY" \
  -H "Spooler-Dedup-String: order-42-created" \
  --data-binary 'order 42 created'

The first send appends and returns 201:

HTTP/2 201
spooler-message-id: 1-17

The second, identical send matches the first and returns 200 with its id:

HTTP/2 200
spooler-message-id: 1-17

The status tells you which case you hit: 201 appended a new message, 200 matched an existing one — Spooler-Message-Id names the original either way. An ack-and-send with a key behaves differently on a hit: it refuses the whole call with 409 kind dedup_claimed and acks nothing — see pipelines.

Why this exists: safe retries

A send can fail after the server accepted it — the connection drops, the response is a 500 (operation_unconfirmed says so explicitly, but any 500 may have landed), or a 502 or 504 comes back from the gateway. Without deduplication, retrying such a send can append the message twice. With a dedup window and a stable key, the retry is idempotent: either the first attempt landed and the retry returns 200 with its id, or it didn't and the retry appends it. Two conditions make that hold: assign the key before the first attempt and reuse it on every retry, and retry while the window still remembers it — after the window, the same key appends another message. Deduplication matches the key, not the payload: a different logical message needs a different key.

The other statuses — every 4xx, 503 and 507 — are answered before the append is attempted and are safe to retry without a key; the full split is in errors.

If your producer retries sends — and it should — give it a dedup key.

Keys

Any stable string identifying the logical message: a UUID, a ULID, or a business key like order-42-created. Keys are case-sensitive; length and character rules are in limits. The server matches keys by a fingerprint of the key, not the key itself — the recipe and the collision odds are on Send a message.

Empty or repeated Spooler-Dedup-String headers fail with 400.

The window

Deduplication is per-queue and off by default. Set dedupWindowSeconds when creating the queue (as above) or on an existing one (its bound is in limits):

curl -X PATCH "$SPOOLER_API/spools/default/queues/orders" \
  -H "Authorization: Bearer $SPOOLER_KEY" \
  -H "Content-Type: application/json" \
  --data '{"dedupWindowSeconds": 60}'

A key is remembered for the window's duration after the original send. The window answers "did this key land", not "is the message still there": a message cancelled through its send handle stays the window's original, and a resend with the same key inside the window is a duplicate of it. Sends with a key to a queue without a window fail with 400 kind dedup_disabled — silently not deduplicating would be worse than refusing.

While a keyed send is still settling, a concurrent send with the same key fails with 409 kind dedup_in_flight; retry after the first settles.

Changing dedupWindowSeconds forgets every remembered key at once, whether the window grows or shrinks; setting it to its current value changes nothing. A producer retrying across the change can therefore append twice: change the window while producers are idle, or accept that one exposure.

Precomputed fingerprints

Callers computing the fingerprint themselves send it verbatim as Spooler-Dedup-Hash (format and recipe on Send a message); the same recipe makes it interoperate with Spooler-Dedup-String senders. Combining both headers in one send fails with 400.

Notes

The mechanism corresponds to SQS's MessageDeduplicationId, with two differences: it works on any queue with a window configured (no FIFO queue required), and the window length is per-queue rather than fixed.