When building real-time clinical AI systems and temporal digital twin simulators, latency and resilience are not just performance metrics—they are hard reliability requirements. Every clinical evaluation or simulation needs sub-millisecond repeatability, but cloud network hiccups or unprovisioned tables should never crash your local environment or block clinician workflows.

Here is how we built a dual-tier caching and state store architecture in Node.js/TypeScript that provides < 2ms response times locally while seamlessly backing up state to Amazon DynamoDB.

The Problem: The Latency vs. Resilience Dilemma

Directly reading and writing every decision or multi-variable trajectory to DynamoDB introduces network round-trip overhead (typically 15ms–60ms). For repeated decision classification or quick "what-if" counterfactual testing, that latency quickly compounds.

On the flip side, relying only on in-memory storage loses state whenever server instances restart.

As Tien noted in "Cache Doesn't Make Your System Faster. It Just Moves the Complexity Somewhere Else", adding a cache introduces boundary complexity: What happens when the backing store is unreachable? What happens when repeated decision evaluations need immediate eviction?

The Architecture: Active In-Memory Tier + Resilient DynamoDB Persistence

Inspired by layered caching patterns highlighted by Kareem in "How We Built a 3-Tier Node.js Cache", we implemented a dual-mode service:

  1. Tier 1 (Bounded In-Memory Store): Fast Map storage for active sessions, digital twin simulation states, and decider cache records with TTL validation.
  2. Tier 2 (Amazon DynamoDB Document Client): Persistent partition-key storage (PK: SESSION#<id>, PK: DECIDER_CACHE#<hash>).
  3. Graceful Soft-Fallback: If AWS credentials are absent or the remote table is still provisioning, operations soft-fail over to the in-memory tier with a gentle warning—keeping the app fully operational.

The Implementation

Here is the core service pattern we implemented in TypeScript:

import { DynamoDBClient, DescribeTableCommand, } from '@aws-sdk/client-dynamodb'; import { DynamoDBDocumentClient, PutCommand, GetCommand, } from '@aws-sdk/lib-dynamodb'; export interface DeciderCacheRecord { cacheKey: string; decision: any; timestamp: string; ttl: number; } // Tier 1: In-Memory Bounded State Store const inMemoryDeciderCache = new Map<string, DeciderCacheRecord>(); const inMemorySessions = new Map<string, any>(); export class DynamoDBStateService { private client: DynamoDBClient | null = null; private docClient: DynamoDBDocumentClient | null = null; private tableName: string; constructor() { this.tableName = process.env.AWS_DYNAMODB_TABLE_NAME || 'amm-clinical-sessions'; const region = process.env.AWS_REGION || 'us-east-1'; const hasCredentials = Boolean( (process.env.AWS_ACCESS_KEY_ID && process.env.AWS_SECRET_ACCESS_KEY) || process.env.AWS_PROFILE ); if (hasCredentials) { try { this.client = new DynamoDBClient({ region }); this.docClient = DynamoDBDocumentClient.from(this.client); } catch (err) { console.warn('[DynamoDBStateService] Fallback to in-memory state store.', err); } } } // Sub-millisecond Repeat Decision Accelerator async cacheDecision(cacheKey: string, decision: any, ttlSeconds = 3600): Promise<void> { const record: DeciderCacheRecord = { cacheKey, decision, timestamp: new Date().toISOString(), ttl: Math.floor(Date.now() / 1000) + ttlSeconds, }; // Write to Tier 1 instantly inMemoryDeciderCache.set(cacheKey, record); // Asynchronously or safely persist to Tier 2 if (this.docClient) { try { await this.docClient.send( new PutCommand({ TableName: this.tableName, Item: { PK: `DECIDER_CACHE#${cacheKey}`, SK: 'CACHE', ...record, }, }) ); } catch { // Soft-fail: In-memory copy ensures the request never halts } } } async getCachedDecision(cacheKey: string): Promise<any | null> { const cached = inMemoryDeciderCache.get(cacheKey); if (cached) { if (cached.ttl > Math.floor(Date.now() / 1000)) { return cached.decision; } inMemoryDeciderCache.delete(cacheKey); } return null; } }

Non-Destructive Health Probes

To make monitoring simple across deployment environments, we added a non-destructive status probe that checks DynamoDB connectivity without polluting tables with test rows:

async probeStatus() { const startTime = performance.now(); if (this.docClient && this.client) { try { await this.client.send(new DescribeTableCommand({ TableName: this.tableName })); return { status: 'CONNECTED_AWS', mode: 'Amazon DynamoDB (Strands State Store)', latencyMs: Math.round(performance.now() - startTime), }; } catch (err: any) { return { status: 'PROVISIONING_REQUIRED', mode: 'DynamoDB Active In-Memory Fallback', latencyMs: Math.round(performance.now() - startTime), }; } } return { status: 'OPERATIONAL_LOCAL', mode: 'High-Speed Bounded In-Memory Cache (< 2ms)', latencyMs: Math.round(performance.now() - startTime), }; }

Key Takeaways

  1. Never let infrastructure status become a single point of failure during dev or transient outages: Falling back smoothly to bounded memory keeps local testing instantaneous and resilient.
  2. Cache invalidation by TTL: Adding an epoch TTL directly on the cache record avoids memory bloat.
  3. Decouple writes: Synchronous user requests read from Tier 1 in < 2ms, while Tier 2 guarantees durability across reboots.

Have you implemented similar multi-tier caching patterns for your AI agents or stateful APIs? What edge cases did you encounter?