How can I prevent duplicate processing of SEMrush Scraper output?
To prevent duplicate output from the SEMrush Free Website Stats Scraper, your data pipeline needs a robust mechanism for tracking what has already been processed. Maintain a list of all domains for which you have successfully retrieved and processed SEMrush stats, storing this information in persistent storage. Before initiating a new run, filter your input list of domains against this stored state.
While Apify offers named Key-Value Stores as a suitable solution for this, similar results can be achieved using other persistent storage solutions like Redis, a relational database, or cloud-based object storage. Named Key-Value Stores, unlike unnamed storages which expire, are persistent and provide a reliable place to store your processed domain identifiers and their scrapedAt timestamps. This approach, leveraging persistent storages, is crucial for long-running or scheduled pipelines, as free-tier accounts only retain the 10 most recent run storages for four months.
Here's how a Python application might manage this state before feeding a list of domains to the semrush-scraper Actor:
from apify_client import ApifyClient import os # Initialize ApifyClient with your API token # For local testing, ensure APIFY_TOKEN is set in environment variables apify_client = ApifyClient(os.environ["APIFY_TOKEN"]) # Define the name for your state storage PROCESSED_DOMAINS_KV_STORE = "semrush-processed-domains" def load_processed_domains(client: ApifyClient) -> set[str]: """Loads the set of domains previously processed from a named Key-Value Store.""" try: kv_store = client.get_or_create_key_value_store(store_name=PROCESSED_DOMAINS_KV_STORE) processed_data = kv_store.get_record(key="processed_domains") if processed_data and processed_data["value"]: return set(processed_data["value"]) return set() except Exception as e: print(f"Error loading processed domains: {e}") return set() def save_processed_domains(client: ApifyClient, domains: set[str]): """Saves the current set of processed domains to a named Key-Value Store.""" try: kv_store = client.get_or_create_key_value_store(store_name=PROCESSED_DOMAINS_KV_STORE) kv_store.set_record(key="processed_domains", value=list(domains), content_type="application/json") except Exception as e: print(f"Error saving processed domains: {e}") # Example usage: # all_domains_to_check = ["wikipedia.org", "github.com", "example.com", "apify.com"] # # processed = load_processed_domains(apify_client) # print(f"Previously processed domains: {processed}") # # domains_for_new_run = [d for d in all_domains_to_check if d not in processed] # print(f"Domains for current run: {domains_for_new_run}") # # # In a real scenario, you'd now pass domains_for_new_run to the Actor. # # After the Actor runs and data is consumed, you'd update 'processed' and save it. After a scraper run completes, fetch its dataset items, process them, and then persist the identifiers of successfully handled items into a named Apify Key-Value Store or an external database to prevent reprocessing in future runs. This post-run update of your state ensures that only newly acquired or modified data is considered for downstream actions. Once the SEMrush Free Website Stats Scraper Actor run finishes, you need to retrieve its results from the default dataset and then update your state.
Here's how you might fetch the dataset items and update your set of processed domains, including handling domains for which SEMrush reported no data (notFound):
from apify_client import ApifyClient import os apify_client = ApifyClient(os.environ["APIFY_TOKEN"]) PROCESSED_DOMAINS_KV_STORE = "semrush-processed-domains" # load_processed_domains and save_processed_domains functions are defined as above # ... def process_and_update_state(client: ApifyClient, run_id: str, current_processed_domains: set[str]): """ Fetches dataset items from a run, processes them, and updates the set of processed domains. """ run = client.get_run(run_id=run_id) dataset = client.get_dataset(dataset_id=run["defaultDatasetId"]) newly_processed_domains = set() for item in dataset.iterate_items(): domain = item.get("domain") if domain: # Here you would typically send 'item' to your downstream system # For demonstration, we just mark it as processed print(f"Processing data for domain: {domain}. Not found: {item.get('notFound', False)}") newly_processed_domains.add(domain) else: print(f"Warning: Dataset item missing 'domain' field: {item}") updated_processed_domains = current_processed_domains.union(newly_processed_domains) save_processed_domains(client, updated_processed_domains) print(f"State updated. Total processed domains: {len(updated_processed_domains)}") # Example of how you would integrate this: # # # 1. Load existing state # existing_processed = load_processed_domains(apify_client) # all_input_domains = ["apify.com", "example.com", "nodata.org", "wikipedia.org"] # Your full list # # # 2. Filter input domains # domains_for_run = [d for d in all_input_domains if d not in existing_processed] # # if domains_for_run: # print(f"Starting run for {len(domains_for_run)} new domains.") # run_input = { # "domains": domains_for_run, # "mode": "full", # "proxyConfiguration": {"useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US"} # } # actor_run = apify_client.actor("crawlerbros/semrush-scraper").call(run_input=run_input) # print(f"Actor run started with ID: {actor_run['id']}") # # # 3. After the run completes (or after fetching its status), process output and update state # # In a real pipeline, you'd wait for run completion or use a webhook. # # For this example, let's assume 'actor_run' is already a completed run object. # # process_and_update_state(apify_client, actor_run["id"], existing_processed) # else: # print("No new domains to process.") By storing the set of domain values for every item successfully pulled from the dataset, you create a robust processing history. Even if a run fails midway or a downstream system experiences an outage, subsequent runs can always pick up where the last successful processing left off, preventing redundant work and ensuring data consistency. The notFound status, when present in an item, indicates that SEMrush has no data for that domain. This is still a processed state; you've checked the domain, and the Actor has reported its status. Therefore, such domains should also be added to your processed_domains set to avoid repeatedly checking them in future runs.
What unique identifier works best for SEMrush Website Stats?
The domain field from the scraper's output, once normalized, serves as the most effective unique identifier for tracking individual records and preventing duplicates across pipeline runs. Each record represents the stats for a single domain, making domain a natural candidate for an ID.
The semrush-scraper output for each domain includes a domain field that has already been normalized (e.g., https://www.custify.com/ becomes custify.com). This ensures consistency and makes it suitable for use as a primary key in a database or a unique identifier in your state management logic.
Consider the typical structure of a single output item from the semrush-scraper:
{ "type": "semrush_website_stats", "domain": "wikipedia.org", "authorityScore": 92, "visits": 1100000000, "visitsText": "1.1B", "organicSearchTraffic": 485000000, "organicSearchTrafficText": "485M", "referringDomains": 3900000, "referringDomainsText": "3.9M", "referringDomainsChange": 1.2, "backlinks": 2960000000, "backlinksText": "2.96B", "backlinksChange": 0.5, "asOf": "July 2026", "sourceUrl": "https://www.semrush.com/website/wikipedia.org", "scrapedAt": "2026-09-28T10:30:00.000Z" } The domain field is consistently present and represents the specific website whose statistics were scraped. For tracking historical data, domain can be combined with asOf (the data month) or scrapedAt (the UTC ISO timestamp of the scrape) to create a composite key, allowing you to store multiple data points for the same domain over time without collisions. When a domain is not found by SEMrush, the output includes notFound: true and a reason, but crucially, the domain field is still present, allowing you to record that specific domain was checked and no data was available.
How do I orchestrate runs beyond the 300-second synchronous limit?
For semrush-scraper runs that might exceed five minutes due to enableBrowserFallback for many domains, use the asynchronous run API endpoint (POST /v2/acts/<actor>/runs) and implement a webhook or polling mechanism to receive results. This avoids the hard-capped synchronous endpoint which returns HTTP 408 past 300 seconds.
The semrush-scraper README explicitly states that some domains, particularly those where SEMrush doesn't have a public overview page, trigger a "browser fallback" which involves launching a real Chrome instance, warming Google cookies, navigating to SEMrush, and rotating through residential proxy IPs. This process can easily take "60+ seconds for some domains." If your input domains list contains many such cases, a batch run is very likely to exceed the 300-second (5-minute) synchronous API timeout.
To reliably run the scraper for larger batches or domains requiring browser fallback, you must initiate the Actor asynchronously and then either poll its status or configure a webhook to be notified upon completion. This kind of orchestration could also be handled by dedicated workflow management systems like Airflow, Prefect, or custom-built solutions using cloud functions and message queues.
from apify_client import ApifyClient import os apify_client = ApifyClient(os.environ["APIFY_TOKEN"]) ACTOR_ID = "crawlerbros/semrush-scraper" def run_semrush_scraper_async(domains_list: list[str], webhook_url: str = None): """ Initiates an asynchronous run of the SEMrush scraper and optionally sets up a webhook. """ run_input = { "domains": domains_list, "mode": "full", "enableBrowserFallback": True, "proxyConfiguration": {"useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], "apifyProxyCountry": "US"} } # Define the webhook if a URL is provided webhooks = [] if webhook_url: webhooks.append({ "eventTypes": ["ACTOR.RUN.SUCCEEDED", "ACTOR.RUN.FAILED"], "requestUrl": webhook_url, "payloadTemplate": "{\"runId\": {{run.id}}, \"datasetId\": {{run.defaultDatasetId}}, \"status\": \"{{run.status}}\", \"actorId\": \"{{run.actorId}}\"}" }) # Call the Actor asynchronously print(f"Initiating asynchronous run for {len(domains_list)} domains...") actor_run = apify_client.actor(ACTOR_ID).call( run_input=run_input, webhooks=webhooks, timeout_secs=0, # Crucial: setting timeout_secs to 0 ensures asynchronous execution wait_for_finish=0 # Also ensures async, does not wait for finish ) print(f"Actor run initiated. Run ID: {actor_run['id']}") print(f"View run details at: https://console.apify.com/actors/{ACTOR_ID}/runs/{actor_run['id']}") return actor_run['id'] # Example usage: # domains_to_scrape = ["github.com", "youtube.com", "example.com", "nodata.org", "wikipedia.org", "apify.com"] # my_webhook_endpoint = "https://your-n8n-or-service-webhook-url.com/semrush-callback" # # if domains_to_scrape: # run_id = run_semrush_scraper_async(domains_to_scrape, my_webhook_endpoint) # print(f"Scraper run {run_id} started asynchronously. Webhook configured to {my_webhook_endpoint}.") # else: # print("No domains to scrape.") When you call an Actor and set timeout_secs=0 (or omit it and rely on the default behavior for POST requests), the API immediately returns the runId, and the Actor execution continues in the background. You can then use this runId to poll the run status or, more efficiently, rely on the webhook to trigger your downstream processing once the run completes, fails, or is aborted. This asynchronous pattern is fundamental for building robust pipelines that aren't constrained by short API timeouts.
How does the mode input parameter affect the SEMrush Scraper's output?
The mode input parameter controls the scope of data returned by the semrush-scraper, allowing you to retrieve either a full set of metrics or narrow the output to authority_only, backlinks_only, or traffic_only. This enables efficient data retrieval by only fetching the specific information relevant to your use case.
The semrush-scraper offers four distinct mode options, which directly influence which fields are populated in the output for each domain. By default, the mode is set to full, providing a comprehensive set of metrics. However, if your application only requires a specific subset of data, selecting a more granular mode can potentially optimize the scraping process by reducing the amount of data that needs to be extracted and processed from the raw SEMrush page. While the Actor's pricing is per result item, fetching less data might reduce downstream processing load and storage.
Here's how each mode impacts the output fields:
-
full(default): This mode returns all available metrics for a domain. This includesauthorityScore,visits(andvisitsText),organicSearchTraffic(andorganicSearchTrafficText),referringDomains(andreferringDomainsText,referringDomainsChange), andbacklinks(andbacklinksText,backlinksChange). It also always includestype,domain,asOf,sourceUrl, andscrapedAt.
{ "domains": ["wikipedia.org"], "mode": "full" } ``` {% endraw %} The resulting output item would contain all fields, such as: {% raw %} ```json { "type": "semrush_website_stats", "domain": "wikipedia.org", "authorityScore": 92, "visits": 1100000000, "visitsText": "1.1B", "organicSearchTraffic": 485000000, "organicSearchTrafficText": "485M", "referringDomains": 3900000, "referringDomainsText": "3.9M", "referringDomainsChange": 1.2, "backlinks": 2960000000, "backlinksText": "2.96B", "backlinksChange": 0.5, "asOf": "July 2026", "sourceUrl": "https://www.semrush.com/website/wikipedia.org", "scrapedAt": "2026-09-28T10:30:00.000Z" } ``` {% endraw %} * **{% raw %}`authority_only`{% endraw %}**: This mode focuses solely on retrieving the {% raw %}`authorityScore`{% endraw %}. Other metric-related fields will be omitted from the output. {% raw %} ```json { "domains": ["google.com"], "mode": "authority_only" } ``` {% endraw %} An example output for {% raw %}`authority_only`{% endraw %}: {% raw %} ```json { "type": "semrush_website_stats", "domain": "google.com", "authorityScore": 100, "asOf": "July 2026", "sourceUrl": "https://www.semrush.com/website/google.com", "scrapedAt": "2026-09-28T10:30:00.000Z" } ``` {% endraw %} * **{% raw %}`backlinks_only`{% endraw %}**: This mode retrieves {% raw %}`backlinks`{% endraw %}, {% raw %}`backlinksText`{% endraw %}, {% raw %}`backlinksChange`{% endraw %}, {% raw %}`referringDomains`{% endraw %}, {% raw %}`referringDomainsText`{% endraw %}, and {% raw %}`referringDomainsChange`{% endraw %}. Traffic and authority score data are excluded. {% raw %} ```json { "domains": ["apify.com"], "mode": "backlinks_only" } ``` {% endraw %} An example output for {% raw %}`backlinks_only`{% endraw %}: {% raw %} ```json { "type": "semrush_website_stats", "domain": "apify.com", "referringDomains": 10000, "referringDomainsText": "10K", "referringDomainsChange": 0.8, "backlinks": 500000, "backlinksText": "500K", "backlinksChange": 1.1, "asOf": "July 2026", "sourceUrl": "https://www.semrush.com/website/apify.com", "scrapedAt": "2026-09-28T10:30:00.000Z" } ``` {% endraw %} * **{% raw %}`traffic_only`{% endraw %}**: This mode exclusively fetches {% raw %}`visits`{% endraw %}, {% raw %}`visitsText`{% endraw %}, {% raw %}`organicSearchTraffic`Source：https://dev.to/crawlerbros/why-preventing-duplicate-semrush-scraper-output-needs-named-storage-fhfRelated posts
Nibnib says 4bpp. The file still has a header.
I counted words in the same 100 MiB file with ten programs: straightforward and tuned versions in Rust, Go, Node and Python, plus two-thread Rust and Go. Read the full article : https://medium.com/git
Redis 8.12 RC2's SEXPIRE command cuts a set membership check from two round trips to one
Does AbortSignal Cancel a Node.js Database Query? 4 Drivers Tested
I read all 308 open jobs at xAI: 54% are in its data center team, and most of them are in Memphis
