Your city chart can name a filter that never took effect

Your regional seller chart can carry a city label for a filter that never took effect.

You're preparing the comparison, and DKI Jakarta looks perfectly respectable above a bar. The awkward question is whether that label came from the submitted dropdown or from the location selector that actually governed the query.

cityId overrides city in the Actor's explicit location inputs. If you preserve the dropdown as your reporting label while submitting an ID override, the chart can misstate its sampling frame.

That leaves a useful debugging decision: did you query the wrong geography, or did you label the query incorrectly? There's another possibility too: the labels describe overlapping areas, so the comparison was never a partition.

Which location did you actually request?

For a Tokopedia seller comparison, start with a narrower question than “How big is each regional market?”

“How many distinct sellers appear among the returned listings for this keyword and location selection?” is something these records can support. It names the retrieval mechanism instead of quietly promoting search results into a seller census.

Example inputs below keep the keyword, ordering and product cap fixed while changing the dropdown selection.

import os from itertools import combinations import pandas as pd import duckdb from apify_client import ApifyClient run_input = { "mode": "search", "searchQueries": ["smartphone"], "searchUrls": [], "sortBy": "relevance", "city": "DKI Jakarta", "maxItems": 100, } plans = { "metro": {**run_input, "city": "Jabodetabek"}, "province": run_input, "district": {**run_input, "city": "Jakarta Selatan"}, } 

Those query identifiers are labels we add locally. They aren't fields returned by the Actor.

The keyword excludes sellers without matching listings. The product cap can leave matching listings outside the pull, and relevance ordering makes inclusion dependent on ranking rather than random selection.

A seller with many matching listings has more opportunities to appear than a seller with a narrow matching assortment. Distinct-seller counting removes repeated seller IDs after retrieval; it doesn't undo that selection mechanism.

A dropdown isn't a geographic partition

The location menu presents broad areas and their constituents at the same visual level. That makes choosing an option easy and adding the resulting counts dangerous.

These relationships explain why the labels can't be assumed to form mutually exclusive buckets. They don't establish the source's exact filter boundaries.

The distinction is easier to see as a containment sketch:

Jabodetabek └── DKI Jakarta └── Jakarta Selatan Geographic containment ≠ guaranteed retrieval containment 

That last line matters. Separately capped and ranked queries need not return nested sets, even when their geographic labels describe nested areas.

A product missing from the broader pull may simply sit outside its retrieved window. Absence from that pull doesn't redraw the map.

Keep the selector decision beside the pull

A useful query manifest preserves what you submitted and how you interpreted it. Renaming a spreadsheet tab after retrieval is a surprisingly effective way to lose both.

The following cell parses comma-separated IDs without converting them to numbers. It records the dropdown separately from the selector that takes precedence.

manifest_rows = [] for query_id, actor_input in plans.items(): submitted_city = actor_input.get("city", "any") submitted_ids = actor_input.get("cityId", "") parsed_ids = tuple( token.strip() for token in submitted_ids.split(",") if token.strip() ) submitted_urls = actor_input.get("searchUrls", []) explicit_label = ( "cityId=" + ",".join(parsed_ids) if parsed_ids else "city=" + submitted_city ) explicit_location = bool(parsed_ids) or submitted_city != "any" manifest_rows.append({ "query_id": query_id, "city": submitted_city, "cityId": submitted_ids, "parsed_city_ids": parsed_ids, "searchUrls": submitted_urls, "explicit_location_label": explicit_label, "mixed_location_review": bool(submitted_urls) and explicit_location, "effective_input_label": ( "unresolved URL/input interaction" if submitted_urls else explicit_label ), }) query_manifest = pd.DataFrame(manifest_rows) 

Search URLs need their own review because filters and sorting carried in the URL are applied. Their precedence relative to explicit location inputs is unspecified.

The flag is deliberately conservative: a supplied URL might contain no location filter at all. Inspect the retained URL before deciding. Guessing a query parameter's meaning would just move the labelling problem into Python.

The manifest also separates an omitted cityId from an inferred location. It records the input decision; it doesn't certify the source's geographic interpretation.

Attach each pull to its query identifier and retain its dataset identifier before counting anything.

client = ApifyClient(os.environ["APIFY_TOKEN"]) pull_rows = [] pull_log = [] for query_id, actor_input in plans.items(): run = client.actor( "dataspryva/tokopedia-search-category-and-price-drop-scraper" ).call(run_input=actor_input) if run["status"] != "SUCCEEDED": raise RuntimeError(f"{query_id}: {run['status']}") pull_log.append({ "query_id": query_id, "dataset_id": run["defaultDatasetId"], }) pull_rows.extend( {**item, "query_id": query_id} for item in client.dataset( run["defaultDatasetId"] ).iterate_items() ) pull_manifest = query_manifest.merge( pd.DataFrame(pull_log), on="query_id", validate="one_to_one", ) 

Streaming through the dataset avoids making the comparison depend on a listing response's page size. Rejecting an unsuccessful run also prevents partial retrieval from quietly becoming a regional result.

Count sellers without manufacturing identities

Empty fields are omitted from records. For this question, missing shopId is especially consequential: the listing exists in the pull, but it can't contribute a known seller identity.

Don't replace that missing ID with a shared placeholder and count the placeholder as a seller. Don't substitute shopLocation either; a place label isn't an identity.

This cell preserves missing values and converts identifiers to text before pandas can infer a numeric column.

def clean_text(value): if value is None: return pd.NA cleaned = str(value).strip() return cleaned if cleaned else pd.NA identity_fields = ("productId", "shopId", "shopLocation") records = pd.DataFrame( [ { "query_id": row["query_id"], **{ field: clean_text(row.get(field)) for field in identity_fields }, } for row in pull_rows ], columns=["query_id", *identity_fields], ).astype("string") 

Keep returned rows, identifiable listings and identifiable sellers separate. Each answers a different question.

DuckDB can calculate those counts alongside the omissions that limit their interpretation.

counts = duckdb.sql(""" SELECT query_id, count(*) AS returned_rows, count(DISTINCT "productId") AS distinct_listings, count(DISTINCT "shopId") AS distinct_sellers, count(*) FILTER ( WHERE "productId" IS NULL ) AS rows_without_product_id, count(*) FILTER ( WHERE "shopId" IS NULL ) AS rows_without_shop_id, count(*) FILTER ( WHERE "shopLocation" IS NULL ) AS rows_without_location FROM records GROUP BY query_id """).df() regional_counts = ( counts.set_index("query_id") .reindex(list(plans), fill_value=0) .rename_axis("query_id") .reset_index() ) 

Keeping planned queries with empty results prevents them from disappearing from the report. An empty retrieved set still doesn't establish that no matching sellers exist in the underlying market.

Measure overlap before adding regions

Product overlap and seller overlap reveal different problems.

Shared productId values show duplicated listing membership across pulls. Shared shopId values can also reveal a seller appearing through different listings, which product deduplication alone would miss.

This cell calculates both intersections, retaining each side's denominator.

overlap_rows = [] for identifier in ("productId", "shopId"): memberships = { query_id: set( records.loc[ records["query_id"].eq(query_id), identifier ].dropna() ) for query_id in plans } for left_query, right_query in combinations(plans, 2): left_ids = memberships[left_query] right_ids = memberships[right_query] shared_ids = left_ids & right_ids overlap_rows.append({ "identifier": identifier, "left_query": left_query, "right_query": right_query, "left_ids": len(left_ids), "right_ids": len(right_ids), "shared_ids": len(shared_ids), "left_shared_fraction": ( len(shared_ids) / len(left_ids) if left_ids else pd.NA ), "right_shared_fraction": ( len(shared_ids) / len(right_ids) if right_ids else pd.NA ), }) overlap = pd.DataFrame(overlap_rows) 

Missing identifiers can't participate in these intersections. Report their omission counts beside the overlap table.

Observed overlap demonstrates shared retrieved membership. An empty intersection doesn't establish disjoint geography: ranking, truncation, query timing and missing IDs can all limit what the comparison observes.

Build comparisons you can defend

For this worked example, Tokopedia Search, Category & Price Drop Scraper supplies listing and seller identifiers; the manifest supplies the analytical meaning of each pull.

Use a single location-control route per comparison. With dropdown inputs, omit cityId and URL-carried filters. With exact IDs, retain their source mapping and label the comparison from that mapping. With filtered search URLs, preserve the URLs and inspect their location selections before assigning chart labels.

Keep the other selection conditions fixed. Changing keyword, sorting or eligibility filters alongside geography makes a regional difference harder to interpret.

Equal caps don't create equal coverage. A pull that reaches maxItems is a reason to examine truncation; a pull below it isn't proof of exhaustive coverage.

For overlapping areas, report separate query counts. If you also calculate a union, deduplicate by the identity relevant to the metric and call it the union of retrieved listings or sellers. Adding the bars gives repeated identities extra votes.

A defensible chart caption names the keyword, location-control route, ordering, cap, retrieval window and missing-ID treatment. “Distinct sellers observed in returned keyword results” is less glamorous than “regional market size”. It's also a claim the dataset can carry.

The geography stops at the seller

shopLocation describes the seller geography exposed by the source. It doesn't establish buyer demand, delivery coverage or warehouse inventory.

These pulls also can't establish the source's full geographic boundary definitions. Shared identifiers help diagnose overlap; they don't turn a search interface into an administrative boundary dataset.

Your city label is a claim about who could enter the count. Make it earn its place above the bar.