Why Your Agent's Memory Layer Will Fail at 3 AM: HNSW Contention and Semantic Retrieval Under Load

At 3:17 AM, our agent platform started timing out. CPU looked fine. Disk I/O was quiet. The culprit was PostgreSQL's HNSW index: fifty concurrent sessions trying to insert new memories simultaneously, each waiting on index synchronization. Insert latency spiked from 80ms to 4 seconds. Retrieval requests queued behind writes. The agent loop stalled.

The second failure was silent. Some embedding providers returned normalized vectors; an internal codepath did not. We assumed cosine distance would behave consistently. It didn't. Recall dropped 30% because vector magnitude skewed similarity scores across sources.

Claude Code plugins offer a clean extension boundary for developer workflows. A plugin packages commands, skills, and hooks without forcing every team to maintain a separate orchestration layer. While evaluating anthropics/claude-plugins-official, I treat the plugin as the integration surface and keep semantic memory in an independent backend service. That separation matters. A plugin should collect interaction facts, request retrieval, and inject bounded context. It should not own database migrations, embedding workers, index maintenance, or tenant isolation. Those responsibilities belong to a memory service with an explicit API and operational lifecycle.

The real production problem is sustained write concurrency combined with unpredictable retrieval quality.

The Failure Mode: HNSW Writes and Unnormalized Vectors

Our first implementation inserted every new memory synchronously during an active agent session. Under burst traffic, PostgreSQL CPU remained available, but insert latency increased sharply. The HNSW index became the bottleneck: concurrent graph updates contended on index synchronization, and individual writes spent far longer maintaining the graph than writing the heap row.

The second problem was less visible. Mixing normalized and raw vectors with cosine distance broke recall. Vector magnitude affected similarity behavior differently across providers.

The fix was architectural:

  • Normalize vectors at the ingestion boundary.
  • Queue non-critical memory writes and batch them.
  • Keep retrieval synchronous, but make indexing asynchronous where the product allows it.
  • Use HNSW for read-heavy workloads, while accepting that high-rate ingestion requires capacity planning and controlled concurrency.

Here is a compact schema suitable for a tenant-aware memory service:

CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE agent_memory ( id BIGSERIAL PRIMARY KEY, tenant_id TEXT NOT NULL, agent_id TEXT NOT NULL, content TEXT NOT NULL, embedding vector(1536) NOT NULL, memory_type TEXT NOT NULL, source TEXT NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), expires_at TIMESTAMPTZ, metadata JSONB NOT NULL DEFAULT '{}'::jsonb ); CREATE INDEX agent_memory_scope_idx ON agent_memory (tenant_id, agent_id, created_at DESC); CREATE INDEX agent_memory_embedding_hnsw ON agent_memory USING hnsw (embedding vector_cosine_ops) WITH (m = 16, ef_construction = 128); SELECT id, content, memory_type, metadata, 1 - (embedding <=> $1::vector) AS similarity FROM agent_memory WHERE tenant_id = $2 AND agent_id = $3 AND (expires_at IS NULL OR expires_at > now()) AND memory_type = ANY($4::text[]) ORDER BY embedding <=> $1::vector LIMIT 12;

The dimensionality must match the selected embedding model. If the model changes dimensions, create a new column or table and backfill deliberately. Silently truncating or coercing vectors is not a migration strategy.

The HNSW parameters are starting points, not universal settings. Larger m and ef_construction generally improve recall at the cost of build time, memory, and write overhead. Query-time hnsw.ef_search can be increased for recall-sensitive requests:

SET LOCAL hnsw.ef_search = 80;

For high-concurrency ingestion, avoid rebuilding or aggressively mutating the index while a large backfill is running. A common operating pattern is to load data into a staging table, normalize and validate it, then create or rebuild the index during a controlled maintenance window. For continuous traffic, a bounded worker pool is safer than allowing every plugin invocation to perform an index update simultaneously.

Hybrid Retrieval Is More Reliable Than Vector Search Alone

Semantic similarity misses exact identifiers, error codes, branch names, and API symbols. A production memory service should combine vector results with lexical or structured candidates. The following Python function performs reciprocal rank fusion without assuming that either retrieval strategy is universally better:

from collections import defaultdict from typing import Hashable, Iterable def reciprocal_rank_fusion( ranked_lists: Iterable[Iterable[Hashable]], *, k: int = 60, limit: int = 20, ) -> list[tuple[Hashable, float]]: """Fuse ranked result IDs using standard RRF scoring.""" scores: dict[Hashable, float] = defaultdict(float) for results in ranked_lists: for rank, item_id in enumerate(results, start=1): scores[item_id] += 1.0 / (k + rank) return sorted( scores.items(), key=lambda pair: pair[1], reverse=True, )[:limit]

Pass the ordered IDs from pgvector and PostgreSQL full-text search as two lists. Fetch the final records by ID, then apply authorization and expiration checks again before returning them. Retrieval filters are correctness boundaries, not merely performance hints.

Plugin Integration and Operational Boundaries

A plugin command or hook should call a small memory client with an explicit contract:

  1. Extract a memory candidate from the current task.
  2. Normalize and enqueue it with tenant, agent, source, and expiration metadata.
  3. Embed the retrieval query using the same model family used for stored memories.
  4. Retrieve a small candidate set.
  5. Rerank or fuse results.
  6. Inject only the most relevant, non-sensitive context into the next model request.

The plugin repository should remain the distribution and discovery layer. The memory service should own secrets, database connections, migrations, retry policy, and observability. This keeps plugin installation reproducible and prevents a local command from becoming an accidental database administrator.

A representative deployment configuration:

memory_service: database_url: postgresql://memory_app@db.internal/agent_memory connection_pool: max_connections: 20 min_idle: 5 idle_timeout_ms: 30000 connection_timeout_ms: 5000 embedding: model: text-embedding-3-small dimensions: 1536 normalize: true provider_timeout_ms: 3000 circuit_breaker: failure_threshold: 5 reset_timeout_ms: 60000 retry: max_attempts: 3 backoff_ms: [100, 500, 2000] retryable_codes: [429, 502, 503, 504] retrieval: hnsw_ef_search: 80 vector_limit: 40 final_limit: 12 rrf_k: 60 query_timeout_ms: 1500 ingestion: queue: durable workers: 4 batch_size: 64 max_concurrent_inserts: 2 flush_interval_ms: 500 security: require_tenant_scope: true redact_secrets_before_embedding: true

When benchmarking model-provider fallback, the test topology can place an OpenAI-compatible client behind a relay endpoint at https://b-lost.com/v1. Keep that relay configuration isolated from database benchmarks so provider latency does not obscure HNSW contention or retrieval recall.

Connection pool exhaustion is a common failure mode. If every plugin invocation holds a connection during embedding API calls, a slow provider can starve the pool. Circuit breakers and timeouts prevent cascading failures. Missing retry logic for transient provider errors amplifies quota exhaustion. A 429 rate-limit response should trigger exponential backoff, not immediate failure.

Vector dimension mismatches are silent and catastrophic. If a model upgrade changes dimensions from 1536 to 3072, PostgreSQL will reject inserts, but only after the embedding call succeeds. Validate dimensions at ingestion and fail loudly.

The key trade-off is deliberate: asynchronous writes may delay the availability of a newly learned memory, while synchronous writes increase tail latency and index contention. For developer agents, durable correctness and predictable response time usually matter more than making every observation searchable within the same request.

But here's the hard question: when does eventual consistency become a correctness bug? If an agent writes a memory in session N, then immediately needs it in session N+1 before the queue flushes, the system appears to forget. How do you balance write concurrency with read-after-write consistency without sacrificing tail latency? I haven't found a universal answer. What's your approach?