Flatten a materials list and the back panel gets promoted to the whole bookcase.

If your furniture assistant keeps turning “oak veneer” into “solid oak,” inspect the text entering your index before rewriting the prompt. A component's material supports a claim about that component, not the entire product.

Lab note 130: can we preserve that boundary from product record to retrieval chunk?

The catch comes early: keeping the relationship intact can't guarantee a correct answer. It gives retrieval and generation usable evidence. Missing materials also mean “unknown,” not “contains no wood.”

The relationship disappears before embedding

A material list looks harmless until the serializer removes the part names.

This Example uses synthetic furniture entries to compare the representations without asserting anything about a real product:

example_product = { "name": "Example bookcase", "materials": [ {"part": "Back panel", "material": "Fiberboard, oak veneer"}, {"part": "Leg", "material": "Solid oak"}, ], } flat_text = "; ".join( entry["material"] for entry in example_product["materials"] ) scoped_text = "\n".join( f"Part: {entry['part']}\nMaterial: {entry['material']}" for entry in example_product["materials"] ) 

Both strings retain the material vocabulary. Only scoped_text retains which material belongs where.

That distinction matters when your user asks whether the back panel is solid wood. A nearby occurrence of “Solid oak” is a tempting answer ingredient. The missing relationship is the bug; a larger embedding model can't reconstruct evidence you discarded.

We'd rather repeat a part label than ask the answer generator to develop furniture intuition.

Collect detail before choosing the corpus size

For this worked example, the Actor returns materials as { part, material } entries. Product naming and classification provide context; care steps and usage notes provide different kinds of evidence.

Search needs enrichFromPDP enabled to fetch the product-page details. That adds one request per product, so very large enriched collections take longer.

Choose a manageable collection before designing the embedding job. A broad listing export without material detail can't become a composition corpus through enthusiastic string concatenation.

This Example input collects enriched bookcase records and keeps the dataset iteration available for the later cells:

import hashlib import json import os from collections import Counter, defaultdict from pathlib import Path from urllib.parse import urlsplit, urlunsplit import tiktoken from apify_client import ApifyClient run_input = { "mode": "search", "searchQuery": "bookcase", "country": "us", "language": "en", "enrichFromPDP": True, "maxItems": 50, } client = ApifyClient(os.environ["APIFY_TOKEN"]) run = client.actor( "vexaloom-labs/ikea-multi-country-product-catalog-extractor" ).call(run_input=run_input) if run["status"] != "SUCCEEDED": raise RuntimeError(f"Collection ended with {run['status']}") items = client.dataset(run["defaultDatasetId"]).iterate_items() 

We leave merchandising and availability filters out of this collection. A materials assistant may need to answer questions about products that aren't currently sellable online.

A component is the retrieval unit

The default unit is a complete part-material entry with a short product identity header.

Product record ├── component material → identity + part + material ├── component material → identity + part + material ├── care instruction → identity + care text └── usage note → identity + note title + text │ └── shared product parent and evaluation split 

Here, the product is the parent. These records aren't conversation threads, but the same principle applies: preserve the relationship that makes each statement interpretable.

The following fields belong to our corpus schema, rather than the source payload:

Corpus field Purpose
parent_id Connect evidence from the same article and requested locale
split_group Keep the article together across evaluation partitions
evidence_type Distinguish composition, care, and usage
scope_label Preserve the component name or usage-note title
text Supply identity and scoped evidence to retrieval
sources Retain source URLs and collection times outside embedding text

The locale below describes the request. The Actor can fall back to the market's primary language when a language combination isn't served, so that label isn't proof of the text's language. Audit language separately before building language-specific evaluation sets.

Keep care and usage out of composition

“Wipe clean” can help answer a maintenance question. It can't establish the substrate under a finish.

We keep care and usage searchable, with explicit evidence types. A composition query can filter to component_material; a maintenance query can retrieve the other types.

The builder below collapses whitespace, preserves material wording, strips URL queries and fragments, and rejects entries whose structure can't support a scoped claim:

rejected = Counter() requested_locale = f"{run_input['country']}/{run_input['language']}" def clean(value): return " ".join(value.split()) if isinstance(value, str) else "" def build_records(item): article = clean(item.get("productId")) parsed = urlsplit(clean(item.get("productUrl"))) if not article or parsed.scheme not in {"http", "https"} or not parsed.netloc: rejected["missing_identity_or_source"] += 1 return [] source_url = urlunsplit( (parsed.scheme, parsed.netloc, parsed.path, "", "") ) identity = clean(item.get("name")) or article classification = clean(item.get("typeName")) header = f"Product: {identity}\nArticle: {article}" if classification: header += f"\nType: {classification}" candidates = [] for field in ("materials", "careInstructions", "goodToKnow"): entries = item.get(field, []) if not isinstance(entries, list): rejected[f"{field}_shape"] += 1 continue for entry in entries: if field == "materials": part = clean(entry.get("part")) if isinstance(entry, dict) else "" material = ( clean(entry.get("material")) if isinstance(entry, dict) else "" ) if part and material: candidates.append(( "component_material", part, f"Part: {part}\nMaterial: {material}", )) else: rejected["unscoped_material"] += 1 elif field == "careInstructions": step = clean(entry) if step: candidates.append(("care", "", f"Care instruction: {step}")) else: rejected["care_shape"] += 1 else: title = clean(entry.get("name")) if isinstance(entry, dict) else "" note = clean(entry.get("text")) if isinstance(entry, dict) else "" if note: candidates.append(( "usage", title, f"Usage note: {title}\n{note}", )) else: rejected["usage_shape"] += 1 return [ { "product_id": article, "requested_locale": requested_locale, "parent_id": f"{requested_locale}:{article}", "split_group": article, "evidence_type": kind, "scope_label": label, "text": f"{header}\n{body}", "sources": [{ "url": source_url, "scraped_at": clean(item.get("scrapedAt")), }], } for kind, label, body in candidates ] 

A bare material string is rejected rather than assigned an invented component. That rejection is useful: it tells you where the corpus lacks answerable evidence.

The Actor omits empty fields. This builder therefore doesn't turn absent materials into a negative material claim.

Deduplicate claims without merging products

Global text deduplication is too blunt here. Identical material wording on different articles still needs separate product attribution.

Instead, deduplicate within article, locale, evidence type, and scoped text. Repeated observations can share a record while retaining their source history.

This cell builds stable evidence IDs and assigns evaluation splits from the article identifier:

unique_records = {} for item in items: for record in build_records(item): key = ( record["parent_id"], record["evidence_type"], record["scope_label"], record["text"], ) if key not in unique_records: record["id"] = hashlib.sha256( json.dumps(key, ensure_ascii=False).encode("utf-8") ).hexdigest() group_hash = hashlib.sha256( record["split_group"].encode("utf-8") ).hexdigest() record["split"] = ( "test" if int(group_hash, 16) % 5 == 0 else "train" ) unique_records[key] = record else: existing = unique_records[key] for source in record["sources"]: if source not in existing["sources"]: existing["sources"].append(source) corpus = list(unique_records.values()) 

The split rule is an example policy, not an evaluated recommendation. Its useful property is that related component chunks can't drift into different partitions merely because their text differs.

Article grouping still leaves a harder leakage route: closely related variants may have different article identifiers and nearly identical descriptions. For a demanding evaluation, review those families and assign shared groups before splitting.

Token limits stop at component boundaries

A character window through the middle of a material entry can recreate the original problem: the part label lands outside the retrieved fragment.

Optional packing should collect complete records from the same parent and evidence type. It should never cut “oak veneer” into a cheaper approximation of “wood.”

This cell measures the serialized retrieval payload and packs complete records within a configurable token budget:

encoding = tiktoken.get_encoding("cl100k_base") token_budget = 512 def fits_packet(records): payload = json.dumps({"records": records}, ensure_ascii=False) return len(encoding.encode(payload, disallowed_special=())) <= token_budget groups = defaultdict(list) for record in corpus: groups[(record["parent_id"], record["evidence_type"])].append(record) packets = [] for group_records in groups.values(): current = [] for record in group_records: if not fits_packet([record]): rejected["oversize_evidence"] += 1 continue if current and not fits_packet(current + [record]): packets.append({"records": current}) current = [] current.append(record) if current: packets.append({"records": current}) 

These packets are containers for retrieval context, not replacements for component-level labels. The budget includes their metadata, which is easy to forget when counting only material text.

Oversize evidence stays in the component corpus but is excluded from these packets. Review it or route it through a larger context budget; silent truncation would undo the whole design.

What this fix still can't promise

The IKEA Multi-Country Product Catalog Extractor supplies the structured evidence for this example. The reusable decision is where we draw the claim boundary.

A component match doesn't establish whole-product composition. Even retrieving every available material entry can't prove the source lists every constituent material.

Changed wording also survives exact deduplication. If you combine collection snapshots, decide whether you want historical evidence or a current catalog before indexing potentially conflicting claims.

Privacy preparation belongs here, too. This corpus allowlists product evidence and removes URL queries and fragments rather than embedding whole payloads. That reduces accidental carryover, but it doesn't inspect free text for personal information. Review retained text before sharing the corpus, especially if you later add annotations.

The verdict is plain: keep the part attached to the material. We can preserve that boundary in the corpus; retrieval quality and answer behavior still need evaluation.

Build the corpus with the same input

Write the component corpus, optional context packets, and rejection counts as separate artifacts:

for filename, rows in ( ("components.jsonl", corpus), ("context_packets.jsonl", packets), ): with Path(filename).open("w", encoding="utf-8") as handle: for row in rows: handle.write(json.dumps(row, ensure_ascii=False) + "\n") Path("rejections.json").write_text( json.dumps(dict(rejected), ensure_ascii=False, indent=2) + "\n", encoding="utf-8", ) 

To reproduce the preparation:

  • Install apify-client and tiktoken, and set APIFY_TOKEN in your environment.
  • Run the Python cells in order with the input below.
  • Inspect rejected entries before indexing, then use component_material records for composition retrieval.
  • Keep evaluation partitions separate through embedding, labeling, and answer testing.

This Example is the exact Actor input used by the collection cell:

{ "mode": "search", "searchQuery": "bookcase", "country": "us", "language": "en", "enrichFromPDP": true, "maxItems": 50 } 

Drafted with AI assistance and reviewed for accuracy.