Fifteen days, twenty-five releases. PageSift went public as 0.20 on October 20; today it ships 1.0.
PageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including /s/), an MCP server, and Python / Node.js / Rust SDKs.
1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is feature completion.
What 1.0 adds
OCR, built from scratch. Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer. Models are open weights, downloaded on first use, off by default. On the olmOCR-bench scanned set it scores 0.543 vs Docling (GPU) at 0.410.
Selector extraction with memory. Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports relocated, ambiguous, or missing instead of silently returning wrong data.
Change monitoring. A /v2/monitor endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.
Browser actions, sessions, and an agent endpoint across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + [[rules]] system, and per-host secret injection.
The architecture in three ideas
- Measured instead of fixed. The hundreds of hand-tuned timeouts, thresholds and concurrency limits are gone. A statistical kernel derives them from runtime observations — survival analysis (Kaplan–Meier) for timeouts, a Universal Scalability Law fit for global concurrency, and "posterior expected value − shadow price × cost" for render decisions. One global meta-parameter remains: δ = 0.05. With no observations, behaviour matches the old fixed values; an estimate takes over only when the difference is statistically significant.
- Scheduling that follows the hardware. One resident CPU worker per core (no blocking thread pool), pinned to NUMA node sets rather than individual cores — because measurements said so: hard per-core pinning cost −33.9% throughput in the 0.70 experiments, node-set pinning gained +8.63%. The code comments cite the measurement behind each such decision.
- Failures are reported, never dressed up as success. 0.91 was a release about exactly this: jobs whose results couldn't be saved, replicated, or delivered by webhook previously ended as
completed. They now fail loudly, with an errors endpoint. 1.0 extends the rule to the API surface: options the engine cannot honor are rejected (unknown MCP arguments error; unimplementable Jina headers returnUNSUPPORTED_OPTION), and the release notes carry a Known limits section that lists what isn't tested and what doesn't work — including security caveats.
Performance (re-measured 2026-11-04)
| Workload | PageSift 1.0 | Fastest rival | Gap |
|---|---|---|---|
| PDF → Markdown, olmOCR-bench 1,269 pages, single thread | 2.4 ms/page | pypdf 50.0 ms/page | ×20.8 |
| HTML → Markdown, 600 documents | 0.24 ms/page | Turndown 1.9 ms/page | ×8.1 |
| Main content, Firecrawl 597 pages, single thread p50 | 1.0 ms/page | crw 7.4 ms/page | ×7.3 |
| Office / e-books, 81 files | 93.6 files/s | Kreuzberg 19.5 | ×4.8 |
| Wide crawl, 200 sites × 100 pages, unthrottled | 3,500–4,400 pages/s, 280–316 MB peak | Spider 2,000–2,700 pages/s, 5.5 GB | — |
Honest footnotes, because they matter: competitor numbers are the recorded late-October runs, not re-run for this release; and the workloads where PageSift loses are listed in the same document ("Largest gaps") — e.g. a single-host replay looks slow mostly because of the default 1-second per-host politeness interval, which is policy, not engine speed. Full data and reproduction scripts live in the repo (docs/benchmarks.md, benchmarks/).
Battle-tested once, properly
In late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it. A user found it; the project fixed it and made it impossible to reintroduce silently. That's the workflow 1.0 is built around.
Who it's for / not for
For: teams feeding RAG or agent pipelines who want a self-hosted Firecrawl/Jina alternative without per-page pricing, and anyone who needs web pages, PDFs and Office files in one pipeline.
Not for: people who need a hosted cloud API (PageSift is self-hosted; a hosted version is not part of 1.0), or bypassing hard anti-bot targets (robots.txt and politeness are respected; captchas are reported, not solved).
PageSift is a solo project. It exists because the crawler feeding my other project was too slow and not clean enough, so I wrote my own. 1.0 closes the feature list. What I want next is more production workloads breaking it in interesting ways.
Repo: https://github.com/YOLOW92/pagesift — AGPL-3.0, commercial licensing available. Issues and head-to-head benchmarks welcome, good or bad.
