Deduplication

Last updated Aug 19, 2026

Send the same message twice, store it once: on a queue with a dedup window, pass Spooler-Dedup-String on a send, and a repeat send with the same key inside the window appends nothing — it returns 200 with the original message's id (SPOOLER_API and SPOOLER_KEY as set in the quickstart).

Create a queue with a window:

curl -X PUT "$SPOOLER_API/spools/default/queues/orders" \
  -H "Authorization: Bearer $SPOOLER_KEY" \
  -H "Content-Type: application/json" \
  --data '{"dedupWindowSeconds": 60}'

Returns 201. Then send the same keyed message twice:

curl -si -X POST "$SPOOLER_API/spools/default/queues/orders/send" \
  -H "Authorization: Bearer $SPOOLER_KEY" \
  -H "Spooler-Dedup-String: order-42-created" \
  --data-binary 'order 42 created'

The first send appends and returns 201:

http

HTTP/2 201

{"duplicate":false,"id":"1-17"}

The second, identical send matches the first and returns 200 with its id:

http

HTTP/2 200

{"duplicate":true,"id":"1-17"}

The status and body tell you which case you hit: 201 with "duplicate": false appended a new message; 200 with "duplicate": true matched an existing one and returns its id.

Why this exists: safe retries

A send can fail after the server accepted it — the connection drops, or the response is a 500 with kind operation_unconfirmed (the append was accepted but its durability is unknown). Without deduplication, retrying such a send can append the message twice. With a dedup window and a stable key, the retry is idempotent: either the first attempt landed and the retry returns 200 with its id, or it didn't and the retry appends it.

If your producer retries sends — and it should — give it a dedup key.

Keys

Any stable string identifying the logical message: a UUID, a ULID, or a business key like order-42-created. Keys are case-sensitive; length and character rules are in limits. The server matches keys by a fingerprint of the key, not the key itself — the recipe and the collision odds are on Send a message.

Empty or repeated Spooler-Dedup-String headers fail with 400.

The window

Deduplication is per-queue and off by default. Set dedupWindowSeconds when creating the queue (as above) or on an existing one (its bound is in limits):

curl -X PATCH "$SPOOLER_API/spools/default/queues/orders" \
  -H "Authorization: Bearer $SPOOLER_KEY" \
  -H "Content-Type: application/json" \
  --data '{"dedupWindowSeconds": 60}'

A key is remembered for the window's duration after the original send. Sends with a key to a queue without a window fail with 400 kind dedup_disabled — silently not deduplicating would be worse than refusing.

While a keyed send is still settling, a concurrent send with the same key fails with 409 kind dedup_in_flight; retry after the first settles.

Precomputed fingerprints

Callers computing the fingerprint themselves send it verbatim as Spooler-Dedup-Hash (format and recipe on Send a message); the same recipe makes it interoperate with Spooler-Dedup-String senders. Combining both headers in one send fails with 400.

Notes

The mechanism corresponds to SQS's MessageDeduplicationId, with two differences: it works on any queue with a window configured (no FIFO queue required), and the window length is per-queue rather than fixed.

Deduplication ​

Why this exists: safe retries ​

Keys ​

The window ​