Web Scrape API Empty Content: 3 Steps for JavaScript Pages

Treat an empty response from a JavaScript-rendered product page as a data-path problem first: inspect the browser's network activity, call the JSON endpoint that supplies the listing, and assert that the expected fields exist. Add browser rendering only when no usable data endpoint exists. For a competitor price watch, that order matters because rendering every page multiplies the expensive part of the index pipeline without making the resulting records more trustworthy.

Short answer: prefer the page's data endpoint, fail closed on empty content, and keep rendering as an explicit fallback tier. A successful HTTP status is not evidence that a listing was captured.

What Should You Do When a Web Scrape API Returns Empty Content?

The scrape API may be returning the server's HTML correctly. Many storefronts deliver a small document and let JavaScript request the title, offer, availability, and seller data after load. A non-rendering fetch sees the shell, so selectors aimed at browser-created nodes find nothing. Retrying the same fetch changes no part of that sequence.

Confirm the diagnosis in a browser before changing infrastructure. Open the network panel, reload one representative product page, filter for fetch or xhr, and inspect JSON responses. The useful request is the one whose response contains the listing fields you need. Record its method, URL, required parameters, and authentication behavior; do not copy browser cookies into a long-lived collector unless you have permission and a rotation plan.

Don't promote the change yet.

One trap is especially costly: treating a blank title or price as a valid deletion. In a price-watch index, that converts an extraction failure into false market movement. Empty is an error state until a separate signal proves that the listing was removed.

Choose the least expensive trustworthy path

The options are not interchangeable. Evaluate them against a fixed sample of real listing types, including pagination and sold-out products, then measure the number of valid records produced per run. Do not compare on HTTP success alone. A source that needs rendering may be perfectly suitable at low volume yet become the dominant cost as the catalog grows; conversely, a brittle private JSON call can cost more operator time than a browser tier if its parameters change every week. The right unit is a validated listing, not a request.

That last boundary is important. Infrai offers one REST API and one key across its capability surface, so the collector's contract can stay put while the provider behind it changes. The trade-off is control: it does not replace Playwright when the runbook requires precise browser interaction, and no unified contract makes server HTML contain DOM nodes that only a browser creates.

Probe the upstream contract before indexing

The following program calls the verified scrape route, rejects non-2xx responses, limits the response size, verifies that the body is JSON, and asserts a target-specific marker. The exact scrape request schema is obtained from the public discovery surface, so the program accepts that validated JSON as SCRAPE_REQUEST_JSON instead of guessing fields that may not exist. This is intentional: generate the request from the discovery path and schema, then place only the secret key in INFRAI_API_KEY. A retry after 429 honors Retry-After when it is a valid number of seconds and otherwise uses bounded exponential backoff.

package main import ( "context" "encoding/json" "errors" "fmt" "io" "net/http" "os" "strconv" "strings" "time" ) const maxBody = 4 << 20 func main() { baseURL := os.Getenv("INFRAI_BASE_URL") key := os.Getenv("INFRAI_API_KEY") requestJSON := os.Getenv("SCRAPE_REQUEST_JSON") marker := os.Getenv("EXPECTED_MARKER") if baseURL == "" || key == "" || requestJSON == "" || marker == "" { panic("INFRAI_BASE_URL, INFRAI_API_KEY, SCRAPE_REQUEST_JSON, and EXPECTED_MARKER are required") } if !json.Valid([]byte(requestJSON)) { panic("SCRAPE_REQUEST_JSON must be valid JSON") } body, err := scrape(baseURL, key, requestJSON) if err != nil { panic(err) } if !json.Valid(body) { panic("scrape response is not JSON") } if !strings.Contains(string(body), marker) { panic(errors.New("expected listing content is empty")) } fmt.Printf("validated marker in %d-byte response\n", len(body)) } func scrape(baseURL, key, requestJSON string) ([]byte, error) { ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second) defer cancel() for attempt := 0; attempt < 4; attempt++ { req, err := http.NewRequestWithContext(ctx, http.MethodPost, strings.TrimRight(baseURL, "/")+"/web/scrape", strings.NewReader(requestJSON)) if err != nil { return nil, err } req.Header.Set("Authorization", "Bearer "+key) req.Header.Set("Content-Type", "application/json") resp, err := http.DefaultClient.Do(req) if err != nil { return nil, err } body, readErr := io.ReadAll(io.LimitReader(resp.Body, maxBody+1)) resp.Body.Close() if readErr != nil { return nil, readErr } if len(body) > maxBody { return nil, errors.New("response exceeded 4 MiB") } if resp.StatusCode >= 200 && resp.StatusCode < 300 { return body, nil } if resp.StatusCode != http.StatusTooManyRequests || attempt == 3 { return nil, fmt.Errorf("scrape returned %s: %s", resp.Status, body) } delay := time.Second << attempt if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 { delay = time.Duration(seconds) * time.Second } select { case <-time.After(delay): case <-ctx.Done(): return nil, ctx.Err() } } return nil, errors.New("retry limit reached") } 

Keep this assertion immediately before the index write as well. Validation only at discovery time misses later contract drift. Use a stable listing identifier as the idempotency key for the write so a timeout and retry cannot create duplicate documents; a content hash can decide whether the existing document actually needs an update.

Do not retry every failure identically. A timeout or 429 can be retried with bounded exponential backoff while honoring Retry-After; an empty required field should stop that record and raise a contract alert. Fast failure protects the index.

A 200 proves transport, not content.

Verify the fix without trusting the happy path

Start with a small fixture set: one normal listing, one variant-heavy listing, one sold-out listing, and one known missing page. Store the raw response alongside the parser version for a limited diagnostic window, subject to your data policy. The acceptance check should distinguish four outcomes: valid record, intentional absence, upstream rejection, and extraction failure.

Run the direct-data collector and any browser fallback against the same fixtures. Compare required-field completeness and stable identifiers, not markup. Then canary the change on a narrow source slice before allowing it to update the main index. Watch the empty-record count, fallback rate, duplicate-write count, and age of the newest successful listing.

Zero is suspicious.

Index choice is a separate cost decision. Pinecone is a managed vector database, Weaviate and Qdrant offer dedicated vector-search systems, while pgvector keeps vectors beside relational product data in Postgres. For a small price-watch corpus already living in Postgres, pgvector can reduce operational surface; a dedicated system may fit when retrieval scale or vector-specific operations justify another service. None repairs an empty scrape. Chroma and Milvus are additional real options, but changing the destination before fixing extraction only indexes the same absence somewhere else.

The fallback rate is also a budget signal. If it rises abruptly, the data endpoint may have changed or begun rejecting requests; automatically sending the entire workload to browsers can turn a target change into an unbounded capacity event. Cap browser concurrency and define a maximum fallback share that pauses ingestion rather than silently increasing spend.

Roll back without corrupting the price history

Rollback should switch collection paths, not erase indexed state. Keep the last validated document until a replacement passes the same schema and content checks. If the direct endpoint changes, disable that adapter for the affected source and route only its bounded canary set to rendering while the contract is re-inspected.

Make the rollback unit the source adapter and parser version. That gives operators a reversible change with an audit trail, while stable listing IDs make replay safe. Never turn off validation to restore throughput; stale, identified data is easier to reason about than fresh-looking empty records.

The decision rule is straightforward: use the JSON call when it is available and permitted, use a browser when rendering is genuinely required, and index nothing that has not passed an explicit content assertion. The vendor decision comes after that architecture, based on who should own browser operations and how much contract portability the system needs.

References

• Chrome DevTools: Inspect network activity
• Playwright documentation
• Browserless documentation
• Apify Actors documentation
• MDN: HTTP response status codes
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks