The problem with a single uptime checker

Most uptime monitors are one process making one HTTP request from one location. That is enough to catch most outages, but it has two blind spots: the checker itself can go down, and regional network problems look like global outages. If my checker is in us-east and an ISP there has a routing issue, I could page my team at 3 AM for a problem that only a small share of users can see.

I built Sentinel to explore whether a small network of independent validator nodes, each checking the same target from a different region and each signing its report, could reach a verdict by majority agreement and avoid false alarms from single-region noise. Here is what I learned.

How it works

 node-us-east --+ node-eu-west --+ signed reports +-----------------------+ node-ap-south --+-------------------> | HUB | node-us-west --+ POST /reports | - validates sig | | - aggregates window | | - runs quorum | | - stores verdict | | - fires alerts | +-----------------------+ | PostgreSQL 

One report's lifecycle, end to end:

  1. Check (node/src/validator-node.ts). On every tick the node fetches its assignments from the hub, runs HttpHealthChecker.check(url, timeoutMs), and gets back { status, httpStatus, latencyMs, error }.

  2. Build and sign (node/src/report-builder.ts, shared/crypto.ts). A ReportBuilder assembles the payload, including the current window_id and a random UUID nonce. signReport() then serializes the payload in a fixed key order and signs it with the node's Ed25519 private key using Node's built-in crypto module.

  3. Ingest (hub/src/ingestion/report-ingestion-service.ts). The hub passes the signed report through a Chain of Responsibility of five validators: schema check, signature check, clock-skew check, replay-guard check, rate-limit check. Only then does it write to Postgres.

  4. Aggregate (hub/src/consensus/window-aggregator.ts). A background loop runs every second. Time is split into 30-second windows, and the window_id is the current time divided by the window length, rounded down, so every node computes the same integer for the same period. A window closes when all live nodes have reported or the deadline has passed. The hub then gathers that window's reports and passes them to ConsensusEngine.decide().

  5. Verdict and alert (hub/src/consensus/, hub/src/alerts/). The engine runs the configured strategy, stores the verdict, and feeds it to a TargetStateMachine. When the state machine sees enough consecutive verdicts of a new status, it fires an alert through an Observer chain.

Key design decisions

1. Quorum instead of trusting one node

Problem: a single node can be wrong because of its network path, its region's ISP, or the node process itself.

Decision: a verdict needs at least quorumK reports, and the winning status must hold a strict majority of the reports received.

// hub/src/consensus/simple-quorum-strategy.ts const quorumMet = reports.length >= context.quorumK; const hasMajority = agreeing.length > reports.length / 2; 

Trade-off: with N nodes I set quorumK to floor(N/2) + 1, which is 2 for 3 nodes. If at most f nodes are faulty and N >= 2f + 1, honest nodes outnumber faulty ones, so a majority of the reports still reflects the truth. With 3 nodes, the system tolerates 1 faulty node. This is not full Byzantine fault tolerance: it assumes the faulty nodes are a minority. If a majority lie, the verdict will be wrong.

2. Signatures prove identity, not truth

Problem: I needed to know that a report came from a registered node and was not tampered with in transit.

Decision: every report is signed with the node's Ed25519 private key over a canonical payload. The hub verifies the signature against the registered public key before accepting the report.

// shared/canonical.ts const PAYLOAD_KEYS = [ 'node_id', 'target_id', 'window_id', 'timestamp', 'nonce', 'status', 'http_status', 'latency_ms', 'error' ] as const satisfies readonly (keyof ReportPayload)[]; export function canonicalReportBytes(payload: ReportPayload): Buffer { const ordered: Record<string, string | number | null> = {}; for (const key of PAYLOAD_KEYS) ordered[key] = payload[key]; return Buffer.from(JSON.stringify(ordered), 'utf8'); } 

Trade-off: a signature only proves who sent the report, not that the result is accurate. A node running with FAULT_MODE=lie signs a correctly formed report with a flipped status. The quorum is what handles that: if one node lies and the rest agree, the verdict follows the majority.

3. INCONCLUSIVE instead of guessing

Problem: if only 1 of 3 nodes reports before the window deadline, what verdict should the hub emit?

Decision: if reports.length < quorumK, the verdict is INCONCLUSIVE. No alert fires and no state transition occurs.

Trade-off: this is an explicit "I don't know" rather than a guess. A test covers it:

// hub/src/consensus/window-aggregator.test.ts it('waits before deadline, then closes partial reports with an inconclusive verdict', async () => { const nodes = [node('n1','ACTIVE'), node('n2','ACTIVE'), node('n3','ACTIVE')]; expect((await build(nodes,[report('n1')],34_000).closeIfReady(target,'0')).closed).toBe(false); const result = await build(nodes,[report('n1')],35_000).closeIfReady(target,'0'); expect(result.closed).toBe(true); expect(result.verdict?.status).toBe('INCONCLUSIVE'); }); 

The cost is that when the hub cannot collect enough reports, the target looks "unknown" instead of triggering an alert. I chose that over a wrong alarm.

4. Region-aware quorum to avoid single-region false DOWN

Problem: if the us-east nodes see DOWN but eu-west and ap-south see UP, the target is probably fine globally and us-east has a routing problem.

Decision: the region-aware strategy applies the simple majority check and adds one rule: if the verdict would be DOWN but fewer than requiredRegions distinct regions agree, it becomes DEGRADED.

// hub/src/consensus/region-aware-quorum-strategy.ts decide(reports: VerifiedReport[], context: ConsensusContext): Verdict { const verdict = majorityVerdict(reports, context, this.name); if (verdict.status !== 'DOWN' || verdict.regionsAgreeing >= context.requiredRegions) return verdict; return { ...verdict, status: 'DEGRADED', reason: 'single-region failure' }; } 

Trade-off: strategies are chosen per target. A simple internal service may not need region awareness, while a globally used API probably does. A ConsensusStrategyFactory lets me register a new strategy without modifying the engine.

5. Windowed aggregation and heartbeat-based failure detection

Problem: nodes check on their own schedule, and the hub needs to know when a period's results are in without waiting forever.

Decision: the aggregator closes a window when all live nodes have reported, or when the deadline (window end plus a grace period) has passed. Node liveness is tracked through heartbeats and a State pattern (ActiveState, SuspectState, DeadState). Dead nodes are excluded from the expected reporters, so they do not stall aggregation.

Trade-off: dead nodes shrink the effective quorum. The aggregator caps quorumK at min(target.quorumK, live_nodes). If most nodes die, the system is left with a dangerously small quorum. I list this under limitations below.

6. Replay protection and idempotent ingestion

Problem: a node may retry a failed POST, and the hub could see the same report twice.

Decision: two guards in sequence. First, an in-memory nonce check catches exact replays. Second, a window-uniqueness check on node_id:target_id:window_id stops a node from submitting a second report for the same window even with a fresh nonce. Postgres UNIQUE constraints back both as the durable layer. A 409 response is treated as success by the node.

// hub/src/ingestion/validators.ts async claim(node: string, nonce: string, target: string, window: string): Promise<boolean> { const nonceKey = `${node}:${nonce}`, windowKey = `${node}:${target}:${window}`; if (this.nonces.has(nonceKey) || this.windows.has(windowKey)) return false; this.nonces.add(nonceKey); this.windows.add(windowKey); return true; } 

Trade-off: InMemoryReplayGuard does not survive a hub restart. After a restart, a replay is only caught when the Postgres constraint rejects the insert. Running multiple hub replicas would need a shared store such as Redis for the in-memory layer.

7. Hysteresis to prevent alert flapping

Problem: a target that alternates between UP and DOWN on every window would generate a storm of alerts.

Decision: TargetStateMachine requires hysteresisCount (default 2) consecutive verdicts of a new status before it triggers a transition. An INCONCLUSIVE verdict resets the counter without triggering anything.

// hub/src/alerts/target-state-machine.ts this.consecutive = this.candidate === verdict.status ? this.consecutive + 1 : 1; this.candidate = verdict.status; if (this.consecutive < this.hysteresisCount) return undefined; 

Trade-off: alerts are delayed by hysteresisCount windows (by default 2 x 30s = 60s) in exchange for filtering out transient noise. It is a trade between alert latency and false positives.

Design patterns in the code

Node liveness: State (shared/liveness.ts). ActiveState, SuspectState and DeadState each own their transition logic and answer countsTowardQuorum(), so the aggregator never inspects a status string directly.

// shared/liveness.ts export class ActiveState implements NodeLivenessState { readonly status = 'ACTIVE' as const; afterMissedHeartbeat(age: number, suspect: number, dead: number): NodeLivenessState { return age >= dead ? new DeadState() : age >= suspect ? new SuspectState() : this; } countsTowardQuorum(): boolean { return true; } } 

Circuit breaker: State (hub/src/alerts/webhook-circuit-breaker.ts). ClosedState, OpenState and HalfOpenState hold the retry timestamp and decide whether to allow a probe, so there is no central if (state === 'Open') block.

Validation: Chain of Responsibility (hub/src/ingestion/report-validator.ts). BaseReportValidator.validate() calls check() and then delegates to next. A test proves the short-circuit: the signature validator's verify throws 'must not run' if the schema check fails first.

Repository observability: Decorator (hub/src/ingestion/report-repository-decorators.ts). LoggingReportRepository and MetricsReportRepository wrap the real repository and add structured logging and counters without touching the core insert logic.

Testing with fault injection

Any node process can run in one of three fault modes through FAULT_MODE:

// node/src/validator-node.ts if (this.faultMode === 'offline') continue; if (this.faultMode === 'slow') await new Promise(r => setTimeout(r, target.timeoutMs * 2)); if (this.faultMode === 'lie') result.status = result.status === 'UP' ? 'DOWN' : 'UP'; 

The consensus unit tests cover the key scenarios: one liar among three nodes still produces UP, a single-region DOWN under region-aware produces DEGRADED, and fewer than quorumK reports produces INCONCLUSIVE. A mock-target service exposes POST /toggle to flip between 200 and 503, so I can watch the full pipeline produce a DOWN verdict and then recover.

Future scope: blockchain integration

The next step I want to explore is settling verdicts on a blockchain. Publishing each verdict on-chain would give a tamper-proof record of what the network decided, and adding staking and slashing would give node operators an economic reason to report honestly. That would make the system suitable for an open network where nodes are run by people who don't trust each other.

The extension point is already in the code: VerdictPublisher in shared/ports.ts. The current implementation, LoggingVerdictPublisher, writes the verdict to the structured log on state transitions. A blockchain publisher would be a new implementation of the same interface that serializes the verdict (batched, to keep gas costs down) and submits it to a smart contract. The consensus engine would not need to change.

// shared/ports.ts export interface VerdictPublisher { publish(verdict: Verdict): Promise<void> } 

Other planned additions on this path are stake-weighted consensus as a new ConsensusStrategy, and a reputation score per node.

Limitations and what I would do next

Current limitations:

  • The hub is a single point of failure. One process and one Postgres. If it goes down, aggregation stops.
  • InMemoryReplayGuard is not shared across restarts or replicas. The Postgres constraints are the real guard.
  • quorumK shrinks as nodes die. If 3 of 4 nodes die, quorumK becomes 1, and a single node can produce a verdict. There should be an absolute minimum quorum floor.
  • No TLS between nodes and hub. Signatures authenticate reports, but the transport is plaintext.
  • No authentication on the hub API. Anyone who can reach the hub can register a node or add a target.

What I would do next:

  1. Add TLS (or mTLS) between nodes and hub.
  2. Replace InMemoryReplayGuard with a Redis-backed implementation for multi-instance deployments.
  3. Add a minimum absolute quorum floor, independent of the live node count.
  4. Implement an on-chain VerdictPublisher (see Future scope above).
  5. Replace assignment polling with a push model (long-poll or SSE).

Takeaways

  • Signatures authenticate the sender, not the content. A lying node can sign a false report correctly. Only agreement across independent nodes handles that.
  • An explicit "I don't know" beats a guess. Returning INCONCLUSIVE and requiring consecutive verdicts before alerting kept the system from raising alarms it could not justify.
  • State machines compose well. Liveness, target status and the circuit breaker each own exactly one lifecycle, so each is easy to test on its own and their interactions stay small.

GitHub: https://github.com/shiven16/Sentinel