TL;DR: A vector upsert dimension mismatch means the embedding array length does not equal the index's configured dimension. Stop the batch, measure the vectors in Node.js, compare that measurement with an immutable index contract, and quarantine every nonconforming record before any upsert. Do not pad, truncate, or retry the same payload. Those moves hide the contract violation and can corrupt retrieval quality.
For a fintech team answering questions over a folder of PDFs, the first concern is containment. One malformed page should not turn a 20,000-chunk filing import into a retry storm, and a model change should not silently mix incompatible vectors into the corpus. The useful question is not merely "how do I make this request pass?" It is "where did the vector shape change, and how do I prove it cannot reach storage again?"
| What you observe | Most likely boundary | Pick this response | Index-cost effect |
|---|---|---|---|
| Every vector has the same wrong length | Embedder configuration changed | Halt the run; restore the declared embedder or create a separately versioned index | Avoids paying to re-embed blindly |
| Only some vectors have the wrong length | Mixed producer, stale cache, or malformed transformation | Quarantine bad records and trace their producer metadata | Reprocesses only the rejected subset |
| Stored vectors fit, but queries fail | Query embedder differs from the document embedder | Enforce the same contract on both paths | Prevents a second corpus build that would not fix querying |
| Length is correct, values are invalid | Serialization or numeric validation failed | Reject non-finite values before storage | Avoids ingesting unusable vectors |
Why does a dimension mismatch appear during an upsert?
An embedding is a fixed-length numeric vector. A vector index is configured around that length, so an array with a different number of elements cannot be compared under the index's expected geometry. The upsert error is the guardrail doing its job.
In a bulk PDF pipeline, the mismatch usually enters before storage: one worker uses a different embedding configuration, cached vectors outlive the configuration that produced them, or a transformation accidentally nests, slices, or flattens an array. Another common split occurs when document chunks and user questions pass through different embedders. Retrieval-augmented generation depends on retrieving external information for generation; that retrieval path still requires documents and questions to inhabit a compatible vector space [1]. Equal array length is necessary, but it is not proof of semantic compatibility.
So make the producer identity observable. Record a contract version, embedder identifier, configured dimension, actual dimension, document fingerprint, chunk identifier, and ingestion run identifier with each attempt. Do not log the PDF text: financial documents can contain sensitive material, and the shape failure can be diagnosed without copying content into logs.
This is the key distinction: an HTTP error is an event; dimension drift is a pipeline state.
Stop there.
Pick the response that matches the evidence
Keep the existing index when its contract is authoritative and the embedding change was accidental. Pin the producer back to that contract, invalidate incompatible cached vectors, then re-embed only affected chunks. This is the least disruptive route when existing document and query vectors are already consistent.
Create a new versioned index when the embedding configuration changed intentionally. Build it beside the old index, write only vectors produced under the new contract, validate retrieval on a fixed evaluation set, and switch reads after acceptance. Never place two embedding spaces in one index merely because their arrays happen to have equal lengths.
Quarantine individual records when most of a run conforms and a small subset does not. Preserve identifiers and metadata, not sensitive page content. A dead-letter stream is useful here because it separates deterministic contract failures from transient transport failures. Retrying a 1,024-element vector against a 768-dimension contract cannot succeed without changing the input or the destination.
The choice depends on evidence. If all records fail identically, investigate configuration before individual PDFs. If failures cluster by worker, deployment, or cache generation, compare those cohorts. If just one chunk fails, inspect the transformation that produced its array. Fast triage starts with cardinality, not stack traces. The trade-off is operational: keeping an index avoids a migration but requires confidence that its existing vectors share one producer contract; creating a versioned index isolates the change but temporarily duplicates embedding work and storage; quarantining records preserves throughput only if the team can operate replay and reconciliation without losing document coverage.
Enforce the contract in Node.js before the network call
Treat vector shape as a runtime schema. TypeScript's number[] cannot express a fixed runtime length, and compile-time types cannot validate data returned by another process. The check belongs immediately after embedding and again at the batch boundary.
interface EmbeddingContract { version: string; dimension: number; embedderId: string; } interface VectorRecord { id: string; values: number[]; metadata: { documentHash: string; chunkIndex: number; contractVersion: string; }; } type ValidationResult = | { ok: true; record: VectorRecord } | { ok: false; id: string; reason: "dimension_mismatch" | "non_finite_value"; actualDimension: number; }; function validateVector( record: VectorRecord, contract: EmbeddingContract, ): ValidationResult { if (record.values.length !== contract.dimension) { return { ok: false, id: record.id, reason: "dimension_mismatch", actualDimension: record.values.length, }; } if (!record.values.every(Number.isFinite)) { return { ok: false, id: record.id, reason: "non_finite_value", actualDimension: record.values.length, }; } return { ok: true, record }; } Keep the contract in deployment configuration or a schema registry with controlled changes. Read the destination's configured dimension through its supported metadata mechanism during startup when one exists, then compare it with the application contract. Refuse readiness on disagreement. That turns a costly mid-run surprise into a deployment failure with a precise reason.
Next, validate the whole batch locally and send only a homogeneous group. The adapter below stays generic on purpose: storage-specific request shapes belong behind one interface.
interface VectorStore { upsert(records: VectorRecord[]): Promise<void>; } interface IngestTelemetry { count(name: string, value: number, attributes: Record<string, string>): void; histogram(name: string, value: number, attributes: Record<string, string>): void; } async function ingestBatch( records: VectorRecord[], contract: EmbeddingContract, store: VectorStore, telemetry: IngestTelemetry, ): Promise<{ accepted: number; rejected: ValidationResult[] }> { const checked = records.map((record) => validateVector(record, contract)); const valid = checked.filter( (result): result is Extract<ValidationResult, { ok: true }> => result.ok, ); const rejected = checked.filter( (result): result is Extract<ValidationResult, { ok: false }> => !result.ok, ); for (const result of checked) { telemetry.histogram( "ingest.vector.dimension", result.actualDimension ?? contract.dimension, { contract_version: contract.version, embedder_id: contract.embedderId, outcome: result.ok ? "accepted" : result.reason, }, ); } telemetry.count("ingest.vector.rejected", rejected.length, { contract_version: contract.version, embedder_id: contract.embedderId, }); if (rejected.length > 0) { return { accepted: 0, rejected }; } await store.upsert(valid.map((result) => result.record)); return { accepted: valid.length, rejected }; } Notice the all-or-nothing batch policy. It makes replay accounting clear and prevents an operator from assuming a rejected batch was partially stored. If the storage adapter explicitly reports per-record outcomes, partial acceptance can be safe, but checkpoint only confirmed records and retain the exact rejected identifiers. That added throughput is paid for with more complicated recovery logic.
There is one subtle bug in many validators: checking only the first vector. Don't. A mixed-worker batch can begin correctly and drift halfway through. Validate every record, while grouping compatible work before the network call so failures remain cheap.
Make drift visible before it becomes index spend
A useful dashboard has four signals: attempted chunks, accepted chunks, rejected chunks by reason, and observed dimension distribution by contract version. Keep identifiers such as run ID in logs or traces rather than metric labels; unbounded identifiers create high-cardinality telemetry. The OpenTelemetry specification distinguishes signals including traces, metrics, and logs, which makes it a suitable vocabulary even when the backend varies [2].
Alert on a nonzero contract-rejection count for a production run. Also alert when the observed-dimension distribution contains more than the declared value. The latter catches drift at the validation boundary, before the storage service returns its own error. A ratio can help large imports, but an absolute first-failure alert matters because a fresh deployment may fail every record before enough volume exists for a ratio threshold to become meaningful.
Use structured logs for the forensic trail: run ID, chunk ID, document hash, producer deployment, contract version, expected dimension, actual dimension, and reason. Use a counter for rate and a histogram for shape. Use a trace to connect PDF extraction, chunking, embedding, validation, and upsert. Diagrammed in words, the path is: PDF fingerprint to chunk to embedder contract to validator to batch writer, with a quarantine branch leaving the validator.
Then test the failure on purpose. Feed arrays of dimension - 1, dimension, and dimension + 1; include NaN and Infinity; mix two valid-looking contract versions in one batch; and verify that no storage call occurs after local rejection. A deployment check should also fail when the configured destination dimension and producer contract disagree. These tests are small. They protect the expensive boundary.
Limits of the dimension check
Matching lengths do not guarantee useful retrieval. Two embedding configurations may emit arrays of the same length while mapping language differently, so contract identity must include the producer configuration, not only an integer. Retrieval quality still needs an evaluation set built from representative questions and relevant PDF passages.
This approach has limits. It is not suitable as the only release gate because matching lengths say nothing about relevance, grounding, or whether the correct pages were extracted. Chunking matters too. Tables, footnotes, and repeated legal boilerplate can produce weak retrieval even when every vector is structurally valid. Monitor answer grounding and retrieval relevance separately from ingestion health. The dimension check prevents malformed writes; it cannot certify semantic quality.
Shape is only one signal.
The operational rule is concise: reject shape drift locally, version intentional changes, and observe the pipeline by contract cohort. Fix the producer or choose a new index deliberately. Never edit vector arrays until an error message disappears.
