Your Newsletter Assistant Cites a Teaser as Evidence

Your newsletter assistant can cite the right article and still invent what its author argued.

You open the retrieved record while debugging a confident answer. The title matches. The URL matches.

The text says the argument is coming. Somehow, your assistant has already finished explaining it. That's the uncomfortable part: the citation looks respectable.

Lab note 034: which fields should reach the answer prompt when the task is to explain an author's argument? Our bet is that access state matters more than most article metadata. The catch comes early: for paid posts, the Actor supplies an optional public preview, never the protected body.

What can this excerpt actually prove?

The task is narrow: answer a question about an author's reasoning, with citations that support each substantive claim.

A public preview can support claims about its visible text, but it can't establish the argument in the unseen remainder.

That distinction gets slippery when a teaser shares a record with the article's title, canonical URL and full-article metadata. Flatten everything into “article context,” and the boundary disappears before generation starts.

Example fixture: this hand-written record advertises an argument without supplying it.

import json preview_record = { "recordType": "post", "title": "Why shorter workweeks deserve a trial", "url": "https://example.com/p/shorter-workweeks", "audience": "paid", "isPaid": True, "contentIsPreview": True, "bodyMarkdown": ( "Could a shorter workweek improve retention? " "The argument follows below." ), "articleWordCount": 1000, } preview_record["wordCount"] = len(preview_record["bodyMarkdown"].split()) 

This text supports “the excerpt raises a question about retention.” It doesn't support “the author concludes that shorter workweeks improve retention.”

The title doesn't supply that missing reasoning either. A promise of an argument isn't the argument. Annoyingly tidy metadata doesn't change that.

The fields that change the answer

We'd map the record around the answering task, rather than send every available field to the model.

The useful surprise is that the schema distinguishes the delivered text from the source article. Your serializer can preserve that distinction, or quietly erase it.

We'd also drop engagement figures, cover images and publication plans from this task's prompt. They don't help establish what a paragraph argues.

Choose a single body representation. Sending HTML, plain text and Markdown together repeats the same material and makes retrieval context harder to inspect.

We wouldn't silently substitute summary, description or truncatedBodyText when the requested body is absent. That changes the evidence source while keeping the interface looking reassuringly normal.

Keep the warning attached to the text

A discovery index answers “which article might matter?” An evidence index answers “what supplied text supports this answer?”

They can share a URL. They shouldn't share an assumption about completeness.

Route records before they reach the answering stage:

post record | +-- no body ----------------------> metadata discovery | +-- paid or preview --------------> excerpt-only evidence | +-- explicitly free, not preview -> body evidence | +-- uncertain access -------------> discovery and review 

If you split a body into chunks later, copy its source identity and access state into every chunk. A warning stored only on the parent record is easy to lose when retrieval returns isolated passages.

Also keep the evidence scope in your retrieval identity. If you deduplicate solely by URL, a discovery record and a body-evidence record can overwrite each other's meaning.

Route before writing the prompt

Missing fields deserve particular care because empty values are omitted.

Our proposed policy is deliberately conservative: missing or unexpected markers don't become permission to treat text as a complete body. That can send usable material to review. We'd rather recover it explicitly than quietly upgrade its evidentiary status.

This helper assigns an application route without interpreting opaque access labels.

def route_for(record: dict, body: str) -> str: if not body: return "metadata_only" paid = record.get("isPaid") preview = record.get("contentIsPreview") if paid is True or preview is True: return "excerpt_only" if paid is False and preview is False: return "body_evidence" return "discovery_only" 

The paid marker takes precedence even if the preview flag is absent or contradictory. An unfamiliar string value also stays unfamiliar; Python truthiness isn't an access policy.

This is our application's acceptance rule, not an additional promise about every source record.

Serialize the evidence boundary

The serializer below separates discovery metadata and count diagnostics from prompt-ready evidence. Status records and records without a usable citation URL don't enter this path.

The returned envelope is application-owned. Its route, discovery, audit and prompt keys aren't Actor output fields.

def serialize_record(record: dict) -> dict | None: if record.get("recordType") != "post": return None url = record.get("url") if not isinstance(url, str) or not url.strip(): return None raw_body = record.get("bodyMarkdown") body = raw_body.strip() if isinstance(raw_body, str) else "" route = route_for(record, body) source = {"title": record.get("title"), "url": url.strip()} access_keys = ( "audience", "isPaid", "contentAccess", "contentIsPreview" ) access = {key: record[key] for key in access_keys if key in record} envelope = { "route": route, "discovery": {"source": source, "access": access}, "audit": { key: record[key] for key in ("wordCount", "articleWordCount") if key in record }, "prompt": None, } if route in {"excerpt_only", "body_evidence"}: envelope["prompt"] = json.dumps( { "scope": route, "source": source, "access": access, "evidence": body, }, ensure_ascii=False, ) return envelope 

Only non-null prompt values belong in the body-based answering context. Keep the envelope for routing and inspection.

Pair those blocks with an answer instruction: treat evidence as quoted source material, support substantive claims with supplied passages, and describe excerpt-only evidence as an excerpt. When the necessary reasoning is absent, say so.

The serializer makes that policy visible. It doesn't make a model obey it.

Example fixtures: these assertions check the preview boundary, missing-body routing and uncertain access without making network requests.

metadata_record = { key: value for key, value in preview_record.items() if key not in {"bodyMarkdown", "wordCount"} } uncertain_record = { key: value for key, value in preview_record.items() if key not in {"isPaid", "contentIsPreview"} } prepared_preview = serialize_record(preview_record) prepared_metadata = serialize_record(metadata_record) prepared_uncertain = serialize_record(uncertain_record) assert prepared_preview["route"] == "excerpt_only" preview_payload = json.loads(prepared_preview["prompt"]) assert preview_payload["access"]["contentIsPreview"] is True assert "articleWordCount" not in preview_payload assert prepared_metadata["route"] == "metadata_only" assert prepared_metadata["prompt"] is None assert prepared_uncertain["route"] == "discovery_only" assert prepared_uncertain["prompt"] is None 

What still fails

A length ratio doesn't prove completeness. Even when both counts exist, arithmetic can't tell you whether the supplied passage contains the premises, qualifications or conclusion needed for a particular answer.

A correct route doesn't prove entailment either. Body evidence can still be irrelevant to the question. The answer stage must check the actual passage, rather than treat the route as a truth certificate.

And a citation check that only verifies the URL will miss this failure entirely. Evaluate whether the cited text supports the claim. An assistant can fail that check while pointing at exactly the right page.

The schema can carry the warning

The Substack Publication Post Exporter gives us enough information to build this boundary.

The schema supports honest evidence handling. A generic “serialize the article” step can discard the distinction that makes it useful.

That's a schema verdict, not a measured reduction in wrong answers. The code checks routing behavior on hand-written fixtures; answer quality needs separate evaluation.

Reproduce the boundary with your own post

• Run the Python cells in order to check the fixture assertions.
• Replace the Example URL below with a custom-domain post URL you want to inspect.
• Submit the input, pass each returned record to serialize_record, and keep discovery routes out of body-based answering.
• Ask for the author's reasoning, then check each answer claim against the supplied passage.

Example Actor input, with the requested body format and preview setting made explicit:

{ "mode": "postUrls", "postUrls": ["https://example.com/p/your-post"], "bodyFormat": "markdown", "includePaidPreview": true, "includeComments": false, "includePublicationInfo": false, "maxItems": 1 }