Resilient Actions

One night, Cron was generating jobs as expected. The queue was accepting them as expected. The Worker was running as expected.

Then a single Action got stuck.

As a result:

  • Cron continued generating new jobs;
  • the queue continued growing;
  • other Actions stopped executing;
  • the issue was resolved only after a restart.

This is a classic example of how system reliability is determined not by how it behaves in a perfect world, but by how it handles failures.

An Action Must Be Designed for Failure

Every Action operates in a hostile environment:

  • the network may stop responding;
  • the database may become slow;
  • an external API may hang;
  • the server may restart;
  • the action may be cancelled by the user.

For this reason, an Action should be designed as if failure is inevitable.

Always Use CancellationToken

Bad:

await http.GetStringAsync(url);

Good:

await http.GetStringAsync(url, ct);

Bad:

await Task.Delay(TimeSpan.FromMinutes(5));

Good:

await Task.Delay(TimeSpan.FromMinutes(5), ct);

If an operation does not support cancellation, the Worker cannot stop the Action gracefully.

Always Define Timeouts

Even when a CancellationToken is used.

Bad:

var http = new HttpClient();

Good:

var http = new HttpClient { Timeout = TimeSpan.FromSeconds(30) };

External resources are not required to respond quickly.

Sometimes they do not respond at all.

Log Execution Stages

Often the most useful log messages are the simplest ones:

ReportProgress(0, "Loading data"); 
ReportProgress(0, "Downloaded data"); 
ReportProgress(0, "Saving data"); 
ReportProgress(0, "Completed");

If an Action gets stuck, the last message immediately identifies where execution stopped.

For example:

Loading new

It becomes obvious that the problem occurred before the database save phase.

Make Actions Idempotent

After a failure, an Action may execute again.

If re-execution creates duplicate data, the system is not resilient.

Bad:

INSERT INTO Tokens (...)

Better:

INSERT ... WHERE NOT EXISTS (...)

or

MERGE

or

UNIQUE INDEX

A repeated execution should be safe.

Do Not Store State in Memory

Bad:

private static int Processed;

The value will be lost after a restart.

Instead, store state in:

  • a database;
  • a queue;
  • a persistent state store.

An Action should survive process restarts.

Break Large Operations into Smaller Actions

Bad:

DownloadAllHistory(); ProcessAllHistory(); SaveAllHistory();

Good:

download process aggregate publish

Smaller Actions are easier to:

  • cancel;
  • retry;
  • troubleshoot.

Verify Behavior After a Timeout

It is not enough to verify that an Action was cancelled.

You must prove that the queue continues to operate.

A good integration test is:

  1. The first job becomes stuck.
  2. A timeout occurs.
  3. The second job executes successfully.
  4. The queue continues processing.

This scenario closely matches real production behavior.

A Practical Rule

Whenever an Action contains an:

await

ask yourself:

What happens if this operation hangs forever?

If the answer is:

Nothing good.

Then the Action is not production-ready yet.

Summary

A simple, resilient Action has five characteristics:

  • supports CancellationToken;
  • defines reasonable timeouts;
  • logs key execution stages;
  • is idempotent;
  • safely handles re-execution.

In an ideal world, failures never happen.

In production, they happen every day.

A good Action is not one that never fails.

A good Action is one after which the system continues to operate. 🚀🐈‍⬛