Short answer: Pick cron for a bounded, repeatable sweep that can rediscover eligible rows; pick a queue when each cleanup item needs its own due time, retry history, or operator-visible state. The deciding constraint is time semantics, not whether the job happens every hour.

A queue is the better fit when each cleanup item has its own deadline, retry history, or operator-visible state; cron is the better fit when one trusted clock should periodically discover a bounded batch. The deciding constraint is time semantics, not whether the job happens every hour.

I build support automation in Python, and this distinction matters for a mundane task: removing expired conversation attachments and compacting stale metadata without holding a web request open. I first assumed a nightly sweep was enough; then I noticed that the same design has a very different shape once one 500-row page takes longer than the scheduling interval. A nightly sweep can be easy to understand. It can also silently turn a 20-minute cleanup into a 3-hour catch-up if the dataset grows. Before choosing an execution model, write down what “due” means for one record, what happens after a missed run, and how much lateness the support workflow can tolerate.

How should I pick queue versus cron for scheduled data cleanup?

Consider a table of attachment records with expires_at, deleted_at, and cleanup_attempts. A periodic process can ask for every eligible row:

from datetime import datetime, timezone def find_expired(records: list[dict], now: datetime) -> list[dict]: return [ record for record in records if record["deleted_at"] is None and record["expires_at"] <= now ] now = datetime.now(timezone.utc) expired = find_expired(records, now)

That is a cron-shaped question: “What is true at this instant?” It is a good fit when the database remains the source of truth and a run can safely rediscover missed work. The schedule is a trigger, not a record of every obligation.

A queue-shaped question is different: “Which cleanup obligation should be delivered now?” Each item carries its own identity and delivery state. That is useful when a record must be handled near a particular deadline, when work arrives in uneven bursts, or when operators need to inspect individual attempts. The queue is not inherently a clock, though. A producer still needs a timer, a database query, or another mechanism to create delayed work.

This is the first trap. Calling a queue “scheduled” can hide the component that actually decides when an item becomes ready.

The smallest experiment I would run first

I would start with a bounded cron-triggered sweep and measure its shape before introducing per-record delivery. The sweep claims a small batch, performs idempotent cleanup, and records the last observed watermark. The important part is the boundary: a run must not pretend it finished everything when it only processed page one.

from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class CleanupResult: examined: int succeeded: int failed: int next_cursor: str | None def run_cleanup_page( repository, storage, now: datetime, cursor: str | None, page_size: int = 500, ) -> CleanupResult: rows = repository.expired_page(now=now, cursor=cursor, limit=page_size) succeeded = 0 failed = 0 for row in rows: try: storage.delete_if_present(row.attachment_key) repository.mark_deleted(row.id, now=now) succeeded += 1 except Exception: repository.record_failure(row.id, now=now) failed += 1 next_cursor = rows[-1].id if rows else None return CleanupResult(len(rows), succeeded, failed, next_cursor)

The example deliberately leaves the repository and storage interfaces generic. In a real system, delete_if_present and mark_deleted need a clear idempotency contract, and the claim/update operation needs protection against two workers touching the same row. Those are design obligations regardless of the scheduler.

My evaluation constraint would be a short table of observations over several normal and abnormal runs: page duration, eligible rows, completion lag, duplicate attempts, failure age, and the number of pages left after a run. I would also force a missed trigger and a storage timeout in the harness. A single successful dry run proves very little.

When is cron the cleaner boundary?

Cron is a strong boundary when discovery is cheap, the database can answer the eligibility query efficiently, and lateness is acceptable within a defined window. It gives the team one obvious owner for the trigger. A lock or lease prevents overlapping sweeps, while a cursor or stable keyset pagination keeps a large table from becoming one enormous transaction.

The usual failure is overlap, not syntax. A run starts at 01:00, still works at 01:10, and the next invocation starts at 01:15. Without a lease, both runs can select the same records. Idempotent deletes reduce damage, but they do not remove load, duplicate audit events, or confusing metrics. A second failure is assuming the schedule itself is durable history. Traditional cron schedules are time-based entries; they do not automatically become a per-record ledger when a machine or service misses a firing. Cron is the wrong boundary when a missed individual deadline must remain visible or when one slow item should not delay unrelated work.

For support cleanup, I would keep the trigger narrow: one process asks for eligible rows, limits each page, and leaves a durable checkpoint or queryable state. That makes a missed run recoverable on the next pass without inventing an event for every attachment.

When does a queue earn its extra state?

A queue earns its place when “eventually” is too vague for the item. Suppose a customer requests deletion and policy requires the related data to be removed within a stated interval. A per-record due time, attempt count, visibility timeout, and dead-letter path can make that obligation inspectable. The work can be partitioned across consumers as volume changes, and one slow attachment does not have to block every other record.

The cost is conceptual state. You now need to define enqueue timing, duplicate delivery, expiration of delayed items, poison work, and reconciliation between the queue and the database. Delivery is commonly at least once in queue systems, so the cleanup operation still has to tolerate replay. A queue also does not make storage deletion atomic with database acknowledgement. A worker can delete the object and crash before marking the row, then receive the item again. This is the main trade-off: the queue improves per-item control while adding another state machine to operate. A queue is the wrong boundary when the database already provides a cheap, authoritative eligibility query and no per-record deadline needs tracking.

This is where I resist “just put it on a queue.” If the database query already expresses eligibility and a five-minute scheduling window is fine, a queue can add a second truth source without solving the real problem. If each record has a deadline, a retry trail, or a fan-out rate limit, the extra state is easier to justify.

A decision rule that survives growth

Use a periodic sweep when eligibility is derived from current database state, missed triggers can be repaired by rediscovery, and work can be bounded into pages. Use queued work when each item needs an independent due time or delivery history, when arrival is bursty, or when consumers must scale separately from discovery. A hybrid is reasonable: a periodic reconciler finds records missing from the work system, while workers handle individual obligations. The reconciler is a guardrail, not a third execution path to maintain casually.

Before shipping, I would put these questions in the design review:

  • What is the lateness budget for one cleanup item?
  • What durable state proves that an item is eligible, claimed, complete, or failed?
  • Can a retry repeat every external side effect safely?
  • What stops two invocations from processing the same page?
  • How does an operator find work that has been eligible too long?
  • Which metric shows backlog growth before support agents notice it?

The answer should be visible in the data model and the runbook. If the only evidence is a line in a scheduler configuration, the system has a trigger but not an operating model. That limitation is useful to name during review: neither execution model removes the need for idempotency, reconciliation, and an operator-visible failure path.

I would measure the cleanup lag distribution, page duration, retry age, duplicate rate, and storage API error rate for a week of representative traffic. Then I would replay a missed run and a partial failure. Choose the simplest boundary that passes those tests.

References