A bot that takes an order is a weekend project. I wrote my first one in two evenings and it worked fine.
Then a real business started running on it. The same bot now had to put every order into the client's CRM and hold on to all of them while that CRM sat dead for twenty minutes. Nothing changed on screen for the user, but underneath I rewrote most of it.
The gap is all the failures you never see on your laptop. The same webhook delivered twice. A CRM handing you 502s for ten minutes straight. Redis itself going away mid-request. A deploy landing in the middle of someone's checkout. Rate limits. A model quoting a price nobody computed. A payload shaped like nothing you planned for. In production you get all of it, every week.
So now I build the defenses in from day one. Code for each one below.
Everything here ran before I published it, and not once. I pulled the code blocks back out of this article, added stubs for the CRM and the Telegram client, and ran the result against a real Redis and a real Postgres: one assembled harness posting real webhooks through a real Express app end to end, 32 assertions, all green. Versions: Node 22.22, Redis 7.0.15, PostgreSQL 16.15, BullMQ 6.3.13, ioredis 6.0.0, opossum 10.0.0, pino 10.4.0, Express 5.3. The same assembled program ran again on the neighboring majors, BullMQ 5.81.5 with ioredis 5.11.1 and opossum 9.0.0, same result.
A handful of findings below needed more than that one script, because what they test is a process actually dying, an OS signal actually arriving, or the module refusing to boot at all, and you can't fake any of those honestly inside a single assertion script. Those got their own standalone process: fork a real worker, SIGKILL it, confirm the job it was holding still reaches the dead-letter queue. Point Redis at a closed port and time how long a real HTTP request to the webhook takes before it gives up. Send SIGTERM, SIGINT, and SIGHUP at a running child process and confirm all three drain cleanly. Boot the module with four different bad secrets and one good one. All of those are green too, and I'll give the numbers where they matter.
More things showed up from running this than I expected, and they're all in here. BullMQ refuses most job IDs containing a colon. An opossum timeout doesn't cancel the call it gave up on, and the one AbortController it hands you is shared by every call in flight. My own rate limiter let twice its limit through on a one-second boundary, and the group limiter was unreachable from the actual code path that calls it. My own error filter, read literally, stopped the circuit breaker from ever opening on a dead CRM. My unique index let three identical orders through. My price guard both blocked ordinary sentences and let invented prices past. A worker that only knew how to read message crashed outright on nine other shapes Telegram sends in production. queue.add() hung forever, not for a few seconds but forever, the one time I actually pointed it at a dead Redis instead of just reasoning about it. And a circuit breaker's fallback message looked exactly like a real success, right up until the customer asked why their order never showed up anywhere.
Setup, so the later listings have something to refer back to:
const express = require('express'); const crypto = require('crypto'); const Redis = require('ioredis'); const { Queue, Worker } = require('bullmq'); const CircuitBreaker = require('opossum'); const pino = require('pino'); // Fail at boot on a missing or malformed secret, not on the first request that // arrives after a bad deploy. Telegram's own rule for this token: 1-256 chars, // only A-Z a-z 0-9 _ and -. const SECRET = process.env.TG_WEBHOOK_SECRET; if (!/^[A-Za-z0-9_-]{1,256}$/.test(SECRET || '')) { throw new Error('TG_WEBHOOK_SECRET missing or invalid (1-256 chars, A-Za-z0-9_-)'); } const WANT_SECRET = Buffer.from(SECRET); // Two separate connections, not one shared between the queue and the worker. // maxRetriesPerRequest: null is required for the Worker (it issues blocking // commands); on the queue's connection it means "never give up," which is // exactly wrong when Redis is down and you need queue.add() to fail in // bounded time instead of hanging the request forever. More on this below. const workerConnection = new Redis(process.env.REDIS_URL, { maxRetriesPerRequest: null }); const redis = new Redis(process.env.REDIS_URL, { commandTimeout: 2000, enableOfflineQueue: false, retryStrategy: (times) => (times > 3 ? null : Math.min(times * 200, 2000)), }); // noeviction keeps Redis from throwing jobs away on its own, but without a // retention bound they just sit there forever and the queue grows until a // write finally fails. Bound both sides. const RETAIN = { removeOnComplete: { age: 86_400, count: 10_000 }, removeOnFail: { age: 604_800 } }; const queue = new Queue('update', { connection: redis, defaultJobOptions: RETAIN }); const deadLetter = new Queue('update-dead', { connection: redis, defaultJobOptions: RETAIN }); const log = pino({ level: process.env.LOG_LEVEL || 'info' }); const app = express(); const server = app.listen(process.env.PORT || 3000); // section 9 needs the handle Queue and Worker both take the queue name, and it has to be the same string, or you get a producer and a consumer that never meet. queue.add('update', ...) further down passes a job name, which matches nothing and doesn't need to.
Keep the handle app.listen returns. Section 9 closes it on shutdown, and that call is the only place you get it.
One Redis setting before any of this: maxmemory-policy noeviction. BullMQ's own production guide is blunt: the library "cannot work properly if Redis evicts keys arbitrarily," and noeviction "is the only setting that guarantees the correct behavior of the queues." On allkeys-lru you lose jobs with no log line anywhere.
1. Validate the webhook, then enqueue, then answer
Telegram retries a webhook when your endpoint answers anything other than 2xx. The API docs put it like this: "In case of an unsuccessful request (a request with response HTTP status code different from 2XY), we will repeat the request and give up after a reasonable amount of attempts." Note what that doesn't promise. The attempts are finite, the docs don't say how many, and a slow-but-200 answer isn't documented as a retry trigger at all. Payment callbacks retry much harder. Either way, your handler will see the same event twice, and if it writes an order, you get two.
Plenty of write-ups tell you to answer with a 200 immediately and process afterward. Careful with that order. If the process dies between the response and the write, Telegram has been told you got it and will never send it again. The event is gone. Enqueueing first costs a millisecond, so do that, then answer:
// An update this bot doesn't act on shouldn't crash the worker or get // retried forever. message and callback_query.message are the only two // shapes it reads from; everything else (edited_message, channel_post, // my_chat_member, pre_checkout_query, inline_query, poll_answer, // chat_member, message_reaction, a callback on an inline message with no // .message at all, and more Telegram adds over time) gets acknowledged // and dropped at the edge, not discovered by a crash three layers in. function isHandled(update) { return Boolean(update.message || update.callback_query?.message); } function chatOf(update) { return update.message?.chat ?? update.callback_query?.message?.chat; } app.post('/webhook', express.json({ limit: '1mb' }), async (req, res) => { if (!validSecret(req)) return res.sendStatus(403); const update = req.body; if (!Number.isInteger(update?.update_id)) return res.sendStatus(400); if (!isHandled(update)) { log.warn({ update_id: update.update_id, keys: Object.keys(update) }, 'unhandled update type'); return res.sendStatus(200); } try { // the job ID is the dedup: add() won't duplicate a job that's already there await queue.add('update', update, { jobId: `upd-${update.update_id}`, attempts: 3, backoff: { type: 'exponential', delay: 500 }, }); } catch (err) { log.error({ err }, 'webhook intake failed'); return res.sendStatus(500); // let Telegram deliver it again } // Best-effort, and deliberately outside the try above. More on why below. if (update.callback_query) { try { await answerCallbackQuery(update.callback_query.id); } catch (err) { log.warn({ err }, 'callback answer failed'); } } res.sendStatus(200); }); function validSecret(req) { const got = Buffer.from(req.get('X-Telegram-Bot-Api-Secret-Token') || ''); return got.length === WANT_SECRET.length && crypto.timingSafeEqual(got, WANT_SECRET); } Five things about that handler are deliberate, and most of them I only found by actually breaking the thing they protect against.
The enqueue comes before anything that can fail. My first version claimed a Redis dedup key up front and enqueued afterward. Run that, kill the process in between, and the update is marked as seen and never queued. It does not come back.
The unhandled-type check exists because a worker written against message and callback_query alone does not gracefully ignore everything else, it throws. I pushed nine other update shapes through mine and watched all nine either crash the job or compute a chatId of undefined and fail somewhere further in. A 500 there would have told Telegram to retry something the worker will never know how to read, forever, up to whatever retry budget Telegram gives it. Answering 200 for a shape you don't act on isn't being lenient, it's telling the truth: nothing was supposed to happen, and nothing did.
The callback answer moved outside the try around queue.add. It used to be inside it, which reads like the safe, conservative choice: if either step fails, the whole request fails and Telegram retries. It isn't. Once the job is durably enqueued, failing the whole request because the callback acknowledgment failed means Telegram redelivers the same callback_query, you call answerCallbackQuery on the same ID a second time, and Telegram answers that with its own 400, query is too old. That failure would land right back inside the same try, produce another 500, and the update would never stop coming back on its own loop, even though it was enqueued correctly the very first time. Moving the callback answer to its own best-effort block after the job is safely queued means a failed acknowledgment costs you a user who sees their button spin a beat longer, not a webhook that answers 500 for an event that already succeeded.
The job ID is worth a paragraph. My first version used a colon in it, and it died on the first call:
Error: Custom Id cannot contain : BullMQ builds its own Redis keys with colons and guards yours against colliding with them, but the rule is narrower than the message suggests. From job.js, the check is jobId.includes(':') && jobId.split(':').length !== 3, which is a carve-out for legacy repeatable jobs. So a:b throws, a:b:c:d throws, and a:b:c is accepted. A purely numeric ID throws too, with Custom Id cannot be integers. upd-123 sidesteps both. Same behavior on BullMQ 5 and 6.
Now be honest about what the job ID buys you. It dedups while the job exists, which covers the case that actually bites: two deliveries of the same update arriving seconds apart. It stops covering you the moment the job is gone, whether that's removeOnComplete, a cleanup job, or a flushed Redis. A late redelivery after that creates a brand-new job and runs your worker again from scratch.
Which is why the database, not Redis, is where the truth lives:
create table orders ( id bigserial primary key, chat_id bigint not null, idempotency_key text not null, amount numeric not null, created_at timestamptz not null default now() ); create unique index orders_idempotency on orders (chat_id, idempotency_key); not null on both columns, on purpose. I had them nullable at first, and three identical orders walked straight through the unique index, because in Postgres every NULL is distinct from every other one. If you can't make the columns not null, PostgreSQL 15 and later let you declare a unique index nulls not distinct, which makes one NULL collide with the next. For an idempotency key, not null is still the better answer.
Then there's the advice that says an insert violating the index is a duplicate you can ignore. That's wrong, and it costs you work that had nothing to do with it. A 23505 error aborts the whole transaction, so everything else you did in there dies with it, and the COMMIT turns into a ROLLBACK. You usually want the existing order's ID anyway, to tell the user what happened. Upsert and return:
insert into orders (chat_id, idempotency_key, amount) values ($1, $2, $3) on conflict (chat_id, idempotency_key) do update set chat_id = excluded.chat_id -- no-op touch so RETURNING fires returning id, xmax <> 0 as was_duplicate; do nothing gives you zero rows on a conflict, which leaves you nothing to answer with. The no-op update returns the ID either way, and xmax <> 0 tells you which branch you took: false when the row was inserted, true when it was already there. It's an idiom on a system column rather than a documented API, and it also reads true if another transaction has the row locked, so if you want it airtight, select the row by its key instead. I don't use was_duplicate to decide whether to send a confirmation anymore, for a reason that gets its own paragraph in section 2, but it's still exactly what you want for logging and for answering "did this already exist."
The upsert isn't free, either: every repeat writes a new row version, which on a hot stream of duplicates means dead tuples and work for autovacuum.
For Telegram events, the idempotency_key is the update_id. For payments, it comes from the client, generated when the checkout screen opens. That's the only way you can tell a retried payment is the same one.
One more failure mode lives in this handler, and it's not in the handler's code at all, it's in what queue.add does the moment Redis stops answering. I assumed commandTimeout: 2000 on the connection meant a worst case of two seconds before the catch above fired and the webhook answered 500. I pointed REDIS_URL at a closed port and timed a real request to find out. It did not take two seconds. It did not return at all, not in two seconds, not in thirty.
The actual mechanism is one layer up from the command itself. Queue.add() first awaits the connection's own readiness promise, which ioredis resolves on a 'ready' event or rejects on 'end', and nothing else. commandTimeout only bounds an individual command on an already-ready connection; it has no opinion about a connection that is still trying to become ready. ioredis's default retryStrategy retries forever and never reaches 'end', so against a Redis that's simply gone, that readiness promise hangs, and commandTimeout never gets a turn. A bounded retryStrategy, the one in the setup block above, is what actually closes this: after three attempts it returns null, ioredis gives up and transitions to 'end', and queue.add() finally rejects. With that in place, the same dead-Redis request that used to hang came back in about 1.2 seconds with a 500. The honest cost is that once the connection gives up, it does not try again on its own. This is a deliberate tradeoff between "fail fast" and "self-heal," and it means a sustained run of 500s on /webhook is your signal to restart the process, not something the process will quietly recover from by itself. Put it on a dashboard, not just in a log.
2. The handler enqueues; a worker does the work
The prototype shape:
// don't ship this bot.on('message', async (msg) => { const parsed = await openai.parse(msg.text); await crm.createLead(parsed); await bot.sendMessage(msg.chat.id, 'Got it'); }); The webhook times out, Telegram retries, you run it twice. A deploy kills in-flight work with no record it started. And none of this blocks the event loop, since it's all awaited I/O, but it does eat your own concurrency: the handler holds the request open for the whole chain, and under load you pile up in-flight requests, file descriptors, and database connections until something gives. Don't blame the socket pool for it, by the way: Node's http.globalAgent and https.globalAgent both default to maxSockets: Infinity, so there's no pool to exhaust unless you configured one.
const worker = new Worker('update', async (job) => { const update = job.data; const chat = chatOf(update); const chatId = chat.id; const chatType = chat.type; const parsed = await parseMessage(update); const lead = await breaker.fire(parsed); // breaker comes from section 5 // The breaker's fallback and a real success must not look the same to the // customer. More in section 5. if (lead?.parked) { await sendMessage(chatId, "We're processing your order and will confirm shortly.", { chatType }); return lead; } // Claim the notification before sending it, not after, so a retry of the // whole job can't send "Got it" twice. But if the send itself throws, // release the claim before rethrowing, so the NEXT retry can still try — // otherwise a customer whose first send attempt hit an ECONNRESET never // gets told their order went through, ever, even though it did. const claimed = await redis.set(`notify:${lead.id}`, '1', 'EX', 60 * 60 * 24, 'NX'); if (claimed === 'OK') { try { await sendMessage(chatId, 'Got it', { chatType }); } catch (err) { await redis.del(`notify:${lead.id}`).catch(() => {}); throw err; } } return lead; }, { connection: workerConnection, concurrency: 10, }); Notice what's missing compared to a naive first pass: there's no per-chat lock here anymore, and the confirmation isn't gated on lead.was_duplicate from section 1. Both of those were in an earlier version of this worker, and both turned out to be the wrong tool for this job, for two different reasons that are worth walking through separately, because they're both the kind of bug that only a retry finds.
was_duplicate answers "did this order already exist in the database," which is not the same question as "has this customer already been told." The two drift apart in exactly the case that matters most: the CRM write succeeds, the job then fails before the confirmation goes out — an ECONNRESET on the send, say — and BullMQ retries the whole job from the top. On that retry, crm.createLead runs again, the upsert in section 1 finds the row already there, and was_duplicate comes back true. A worker that reads that as "already told them, skip the send" now never sends a confirmation to a customer whose order is sitting in the CRM exactly as it should be. I found this by deliberately failing the first send attempt in a test and watching the retry go silent. The fix is a question asked about notification, not about CRM state: a notify:<lead id> key, claimed right before the send and released if the send fails, so a retry can tell "already confirmed" apart from "paid for, never told."
The lock is section 6's story, and it gets the detail it deserves there, but the short version: it used to wrap this entire function, including the rate-limited, possibly-retried send, and a lock held across a call that can legitimately take longer than the lock's own TTL is a lock that stops protecting anything. It's gone from here and now only guards the handful of lines that actually need serializing.
concurrency: 10 is why a lock exists at all, just not here. Ten jobs run at once, and two of them can belong to the same conversation.
stalledInterval and maxStalledCount are worth knowing about, and I've left them out of the snippet because BullMQ's defaults are already 30 seconds and 1. Section 3 covers what actually happens to a job that stalls.
Pick which layer owns which retry. A single external call that failed on a blip is retried in place, within seconds. A whole job that died with the process retries through the queue, with attempts and backoff. Set both without thinking and three tries quietly become nine. I keep job-level retries as the default and reach for in-call retries only where a sub-second blip is common and the call is cheap to repeat.
Retrying a POST is a different decision from retrying a GET. A 502 from an API gateway often means the write landed and the response didn't. Four tries, four leads. So either the destination takes an idempotency key from you, or you look before you create, or you don't retry that call at all and let the job fail into the dead-letter queue. There's no fourth option, and this is the most common way an otherwise careful pipeline starts producing duplicates.
3. Backoff, and the errors worth retrying
const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); // Telegram's own error shape differs across client libraries: telegraf puts // the code at err.response.error_code, grammY at err.error_code directly, // a request-lib-style client at err.response.statusCode. Normalize once, // here, instead of reading err.response.status directly in three places and // quietly missing two of the three libraries people actually use. function normalizeTgError(err) { const raw = err?.response?.status ?? err?.response?.statusCode ?? err?.response?.error_code ?? err?.error_code; const status = Number(raw); const retryAfter = err?.response?.parameters?.retry_after ?? err?.parameters?.retry_after ?? err?.response?.retry_after; return { status: Number.isFinite(status) ? status : undefined, retryAfter }; } // retry everything except the errors that are mine to fix function isMyFault(err) { const { status } = normalizeTgError(err); if (status === undefined) return false; // no status at all: a transport error, not mine if (status === 429 || status === 408 || status === 425) return false; return status >= 400 && status < 500; } const RETRY_AFTER_CAP = 30; // seconds. Beyond that, don't hold a worker slot open — let it requeue. async function withRetry(fn, { tries = 4, base = 500, cap = 8_000 } = {}) { let lastErr; for Source：https://dev.to/__3e3eb520f2bd/your-weekend-bot-will-lose-orders-what-i-add-before-a-real-business-runs-on-it-2kjpRelated posts
DeepSeek's Open-Source "Super-Agent" Codes Without Being Asked
All-in-one platform or a stack you assemble? Where do coaches and educators land?
I Wanted to Live on Coruscant. PAT Scared Me. Now My Alexa Lives on Mute.
Detect What You Can Name: PaliGemma 2 Object Detection with Transformers
When Should an LLM Router Stop Being Sticky? llm-d 1.0 Cache-Affinity Routing, Break-Even Math, and a Simulation
