[{"data":1,"prerenderedAt":29},["ShallowReactive",2],{"$fGlKWiYjpWtyO9sPkuEvuqUG029m7QQLXT--QGA7BOrs":3},{"slug":4,"title":5,"description":6,"date":7,"tags":8,"draft":13,"readingMinutes":14,"html":15,"related":16},"redis-streams","Redis Streams in Node: Consumer Groups Without the Surprises","Redis Streams with consumer groups in Node.js, checked November 2026: the worker loop, the pending list, claiming stuck messages, trimming, and Redis 8's new commands.","2026-11-06",[9,10,11,12],"redis","node","streams","background-jobs",false,5,"\u003Cp>Redis Streams are an append-only log inside Redis. You \u003Ccode>XADD\u003C/code> an entry, it gets an ID like \u003Ccode>1730851200000-0\u003C/code> (milliseconds, then a sequence number), and any number of readers can go through the log at their own pace. Add a consumer group and Redis hands each entry to exactly one worker in the group, remembers who has it, and keeps it on a pending list until that worker says \u003Ccode>XACK\u003C/code>.\u003C/p>\n\u003Cp>That last part is why I'd reach for Streams over Pub/Sub for anything I can't afford to lose. Pub/Sub forgets a message the moment it's sent: if your worker was restarting, the message is gone. A stream keeps it.\u003C/p>\n\u003Cp>Versions as of November 6, 2026: Redis 8.10.3 (released October 26), and \u003Ccode>redis\u003C/code> 6.3.1 on npm, the official Node client. Command details are from the \u003Ca href=\"https://redis.io/docs/latest/develop/data-types/streams/\">Redis Streams docs\u003C/a>.\u003C/p>\n\u003Ch2>Streams or BullMQ?\u003C/h2>\n\u003Cp>Quick answer first, because it decides whether the rest of this post is for you. If what you need is "run this job later, retry it three times with backoff, show me failed jobs in a dashboard", use \u003Ca href=\"/blog/bullmq\">BullMQ\u003C/a>. It gives you delays, retries, rate limits and priorities, and you'd end up rebuilding all of that badly on top of Streams.\u003C/p>\n\u003Cp>Streams fit when the data is an event log more than a to-do list: order events that three different services each need to see, webhook payloads you want to replay after a bug fix, an activity feed, change events from your app going to a search indexer. Several consumer groups can read the same stream independently, each at its own position. That's the thing a job queue doesn't give you.\u003C/p>\n\u003Ch2>The worker loop\u003C/h2>\n\u003Cp>Create the group once. \u003Ccode>$\u003C/code> means "only new entries from now on", \u003Ccode>0\u003C/code> means "start from the beginning", and \u003Ccode>MKSTREAM\u003C/code> creates the stream if it doesn't exist yet:\u003C/p>\n\u003Cpre>\u003Ccode>XGROUP CREATE orders billing $ MKSTREAM\n\u003C/code>\u003C/pre>\n\u003Cp>Then each worker runs a loop like this with node-redis:\u003C/p>\n\u003Cpre>\u003Ccode class=\"language-js\">import { createClient } from 'redis';\n\nconst redis = await createClient({ url: process.env.REDIS_URL }).connect();\nconst consumer = `billing-${process.pid}`;\n\nasync function readBatch(id) {\n  const res = await redis.xReadGroup('billing', consumer, { key: 'orders', id }, { COUNT: 50, BLOCK: 5000 });\n  return res?.[0]?.messages ?? [];\n}\n\n// 1) whatever this consumer took before it crashed, 2) then new entries\nlet backlog = true;\nwhile (true) {\n  const messages = await readBatch(backlog ? '0' : '>');\n  if (backlog && messages.length === 0) { backlog = false; continue; }\n  for (const { id, message } of messages) {\n    await handleOrder(message);          // must be safe to run twice\n    await redis.xAck('orders', 'billing', id);\n  }\n}\n\u003C/code>\u003C/pre>\n\u003Cp>Two details in there matter more than they look. Reading with \u003Ccode>>\u003C/code> gives you entries never delivered to anyone in the group. Reading with \u003Ccode>0\u003C/code> gives you your own pending history, the entries you were handed but never acknowledged. If a worker restarts with the same consumer name and only ever reads \u003Ccode>>\u003C/code>, its old pending entries sit there forever. The Redis docs' own example does the same thing: drain your history first, then switch to new entries.\u003C/p>\n\u003Cp>And \u003Ccode>handleOrder\u003C/code> has to be idempotent. Streams are at-least-once: if the worker dies after charging the card but before \u003Ccode>XACK\u003C/code>, the entry is still pending and will be delivered again. Store the stream ID with the result and skip if you've seen it.\u003C/p>\n\u003Ch2>Stuck messages: the part people skip\u003C/h2>\n\u003Cp>If a worker dies for good (the container is replaced and gets a new name), its pending entries belong to a consumer that will never come back. Nobody else will touch them until someone claims them:\u003C/p>\n\u003Cpre>\u003Ccode>XAUTOCLAIM orders billing billing-replacement 60000 0-0 COUNT 25\n\u003C/code>\u003C/pre>\n\u003Cp>That takes up to 25 entries idle for more than 60 seconds and moves them to the caller. It returns a cursor to continue from, so you loop until the cursor comes back as \u003Ccode>0-0\u003C/code>. I'd run it from every worker on a timer, every 30 seconds or so; with a sensible idle time it's cheap and you never have to think about dead consumers.\u003C/p>\n\u003Cp>\u003Cimg src=\"/blog/redis-streams/redis-stream-consumer-group.webp\" alt=\"A Redis stream drawn as a strip of entry IDs, with the billing group's last-delivered pointer, two consumers' pending entries, one entry idle for 90 seconds being claimed by another worker, and the trimmed tail\">\u003C/p>\n\u003Cp>Every claim and every history read bumps the entry's delivery counter, which you can see in \u003Ccode>XPENDING orders billing - + 10\u003C/code>. That's your poison message detector. An entry with a delivery count of 10 is not going to work on the 11th try. Copy it to a \u003Ccode>orders:dead\u003C/code> stream, acknowledge it, and alert someone.\u003C/p>\n\u003Cp>Redis 8.8 added \u003Ccode>XNACK\u003C/code> for exactly this, with three modes: \u003Ccode>SILENT\u003C/code> undoes the delivery (for a graceful shutdown), \u003Ccode>FAIL\u003C/code> hands it back to the group as it is, and \u003Ccode>FATAL\u003C/code> sets the counter to the maximum so it's marked as permanently failed. Nice, but it's new. If you're on Valkey or a managed Redis that's a few versions behind, check the command exists before you build on it. The same goes for everything in the next section.\u003C/p>\n\u003Ch2>Trimming, or your stream eats your RAM\u003C/h2>\n\u003Cp>A stream never shrinks on its own. Acknowledged entries stay in it, because another group might still want them. So you trim:\u003C/p>\n\u003Cpre>\u003Ccode>XADD orders MAXLEN ~ 100000 * order_id 8812 status paid\n\u003C/code>\u003C/pre>\n\u003Cp>The \u003Ccode>~\u003C/code> means "about": Redis trims in whole internal nodes instead of to the exact count, which is much cheaper. You can also trim by age with \u003Ccode>MINID\u003C/code>, since IDs are timestamps.\u003C/p>\n\u003Cp>The catch, before Redis 8.2: trimming deletes entries even if a consumer group hasn't processed them yet, and the pending list keeps pointing at entries that no longer exist. Redis 8.2 added a mode argument to \u003Ccode>XADD\u003C/code> and \u003Ccode>XTRIM\u003C/code> (and two new commands, \u003Ccode>XDELEX\u003C/code> and \u003Ccode>XACKDEL\u003C/code>): \u003Ccode>KEEPREF\u003C/code> is the old behaviour and still the default, \u003Ccode>DELREF\u003C/code> also removes the pending references, and \u003Ccode>ACKED\u003C/code> only removes entries every group has acknowledged. On Redis 8.2 or newer, \u003Ccode>XTRIM orders ACKED MAXLEN 100000\u003C/code> is the one I'd use when several groups read the same stream.\u003C/p>\n\u003Ch2>The Redis settings still apply\u003C/h2>\n\u003Cp>Everything I wrote about Redis for BullMQ holds here too, because a stream is just a key. \u003Ccode>maxmemory-policy\u003C/code> must be \u003Ccode>noeviction\u003C/code>, or Redis may evict the whole stream under memory pressure. Persistence must be on, AOF with \u003Ccode>everysec\u003C/code> being the usual choice, or a restart takes your unprocessed events with it. And \u003Ccode>XREADGROUP\u003C/code> is a write command, so it only runs on the primary, never on a read replica.\u003C/p>\n\u003Cp>If you're building an event pipeline in Node and want a second pair of eyes on the design, or someone to just build it, \u003Ca href=\"https://calendly.com/director-norsky/meeting\">book a free call\u003C/a>. All consultation is free.\u003C/p>\n",[17,21,25],{"slug":18,"title":19,"description":20},"bullmq","BullMQ in Production: The Settings That Bite","BullMQ works on the first try in development. These are the Redis and job settings that decide whether it still works in production, checked October 2026.",{"slug":22,"title":23,"description":24},"node-cron","node-cron in Production: Overlaps, Replicas and DST","node-cron 4 in Node.js, checked November 2026: noOverlap, running on one replica, the day-of-week AND rule, DST gaps, and when systemd or a queue is better.",{"slug":26,"title":27,"description":28},"valkey-vs-redis","Valkey vs Redis: Which One I'd Run in Late 2026","Valkey vs Redis in October 2026: licenses, Valkey 9.1 vs Redis 8.10, the data-file break after Redis 7.4, and ElastiCache's 20-33% Valkey discount.",1794095794447]