Tired of configuring residential proxies, managing captcha solvers, or getting stuck with costly $50/month scraper subscriptions?

Here is a lightweight Python pattern to extract clean Markdown from protected URLs on demand. It uses the x402 protocol on Base L2, settling per-request micropayments (0.05 USDC) on-chain with zero API keys or pre-shared secrets.

Prerequisites

pip install requests web3

Quickstart

import json
import requests
from web3 import Web3

GATEWAY_URL = "https://agentfi-gatewaye.onrender.com/v1/extract"
BASE_RPC = "https://mainnet.base.org"
USDC_CONTRACT = "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913"
TRANSFER_ABI = json.loads('[{"constant":false,"inputs":[{"name":"_to","type":"address"},{"name":"_value","type":"uint256"}],"name":"transfer","outputs":[{"name":"","type":"bool"}],"type":"function"}]')

def extract(url, private_key=None, tx_hash=None):
# 1. Probe payment requirement
r = requests.post(GATEWAY_URL, json={"url": url})
if r.status_code == 200:
return r.json()

pay_info = r.json() 
 2. Settle 0.05 USDC on Base L2 if private key provided 
 
final_tx = tx_hash
 if not final_tx and private_key:
 w3 = Web3(Web3.HTTPProvider(BASE_RPC))
 acc = w3.eth.account.from_key(private_key)
 contract = w3.eth.contract(address=Web3.to_checksum_address(USDC_CONTRACT), abi=TRANSFER_ABI)
 
 
tx = contract.functions.transfer( Web3.to_checksum_address(pay_info["payTo"]), int(float(pay_info["price"]) * 1e6) ).build_transaction({ "from": acc.address, "nonce": w3.eth.get_transaction_count(acc.address), "gas": 80000, "maxFeePerGas": w3.eth.gas_price, "maxPriorityFeePerGas": w3.eth.gas_price, "chainId": 8453 }) signed = w3.eth.account.sign_transaction(tx, private_key) final_tx = w3.to_hex(w3.eth.send_raw_transaction(signed.raw_transaction)) 
 
 
if not final_tx:
 return {"error": "Payment required", "details": pay_info}
 
 3. Retrieve clean markdown with on-chain receipt header 
 
headers = {"x-402-signature": final_tx}
 return requests.post(GATEWAY_URL, json={"url": url}, headers=headers).json()




Example usage:

result = extract("https://example.com", private_key="YOUR_PRIVATE_KEY")

print(result)

Why x402?

  • No Monthly Commitments: Pay strictly for what you extract (0.05 USDC / call).
  • Keyless Architecture: Transactions act as cryptographic bearer tokens.
  • Agent-Native: Autonomous AI agents can fund their own wallets and pay for data autonomously without human intervention.