Most "search APIs" are a thin layer over Google or Bing results. NestDaddy is not. It has its own crawler, its own inverted index, its own ranking and its own news collection, designed, built and operated end to end by our team.
This post walks through how the platform fits together and the engineering principles we follow. If you are thinking about building search yourself, or you want to know what sits behind the API you are calling, this is for you.
The shape of the platform
There are four parts, and each one is isolated so that no part can take another down:
- Crawler. Fetches pages, obeys robots.txt, extracts the readable text and stores it in PostgreSQL.
- Indexer. Turns new pages into posting lists in a RocksDB index that is built next to the live one.
- Swap. Every day, the new index replaces the live index in one atomic step, with an automatic rollback.
- Search. A BM25 ranker reads the live index, read-only, behind the web site, the REST API and an MCP server for AI agents.
The rule that ties them together: the search path never waits for the crawler or the indexer, and neither of them can compete with it for resources.
The crawler: polite, bounded, isolated
A good crawler is a polite one, so ours is built around clear limits.
- robots.txt first. One fetch per site, cached for 24 hours. A 401 or 403 on robots.txt means "disallow everything".
- Spread across hosts. Each batch claims URLs from the queue with
FOR UPDATE SKIP LOCKEDand keeps at most 3 URLs per host, so a batch touches hundreds of different sites instead of hammering one. The same claim lets many crawler workers share one queue safely. - Bounded downloads. Every response is streamed and cut at 5 MB or 20 seconds, so no single site can slow the crawl down or flood it with data.
- Isolation. The crawler runs in its own resource slice with dedicated CPU and memory limits and a low priority. Search traffic always comes first.
The index: RocksDB and posting lists
The index is an inverted index stored in RocksDB: for every term, a posting list of document IDs, plus per-document statistics for BM25.
We picked RocksDB over a search server for three reasons. It is a library, so there is no extra service to run. It is built for fast point lookups and multi-gets, which is exactly what a query does. And it can be opened read-only by many processes while another process builds the next version.
Text goes through a tokenizer (lowercasing, stop words, Porter stemming with a memoised stemmer) before it reaches the index, and the same tokenizer runs on every query.
Replacing the index safely, every day
The daily swap is deliberately boring:
- Take the writer lock so the indexer pauses.
- Hard-link the immutable SST files into a staging directory and copy the small metadata files (MANIFEST, CURRENT, OPTIONS, WAL). This takes about a second and uses no extra disk.
- Release the lock and verify the staging copy read-only: the document counter must not go down, sample terms must resolve, the newest indexed pages must be readable.
- Rename the live directory to a "previous" directory and the staging directory to live.
- Reload the search workers and run a live search check. If anything fails, roll back automatically.
Previous generations are kept, so going back is a rename, not a restore.
Ranking: BM25, measured
The ranker is BM25 with practical additions: a two-stage pre-score (rare terms first, then the top 3,000 candidates re-ranked by term frequency with one RocksDB multi-get), a domain match signal for navigational queries, and image documents kept out of web results.
We do not change ranking by feel. An evaluation suite of real queries, each with a judgment ("the right answer is this URL pattern" or "the homepage of this site"), runs against the live index after every change, and a change ships only when precision goes up. The November release moved every metric we track:
| Metric | Before | After |
|---|---|---|
| Right answer at #1 (P@1) | 45% | 70% |
| Right answer in the top 3 | 60% | 77.5% |
| Right answer in the top 10 | 70% | 87.5% |
| Mean reciprocal rank | 0.536 | 0.754 |
The changes behind it were classic search engineering: true document frequencies for IDF, an average document length measured from real documents, deterministic sampling, and keeping images out of web search.
Latency engineering
Fast search is mostly about removing surprises. Three practices matter most for us:
Linear-time text processing. Every regular expression on the query path is checked for backtracking. Patterns like \S+@\S+ are replaced by linear-time equivalents such as (?<!\S)(?=\S+@\S)\S+, which give the same output and stay fast on any input.
Keep the hot index in memory. A dedicated service checks which parts of the index file are resident (mincore) and reads back only what is missing, and it runs with a cgroup memory guarantee so the kernel keeps the index cached. One detail worth knowing: cgroup v2 memory protection is hierarchical, so the parent slice needs the same guarantee as the service.
Audit every scheduled job. Generic "free up memory" scripts such as echo 1 > /proc/sys/vm/drop_caches ignore cgroup protection and quietly undo all of the above. We review every scheduled job on the search path.
The result is sub-second search on fresh queries, measured on the same evaluation suite after every release.
What is next on the roadmap
Owning the whole stack means we can improve every layer ourselves. These are the upgrades in progress:
- Deeper answers. The next index version adds full body text on a tiered storage design, so answers that live deep in a page rank where they should.
- Numbers and more scripts. A new tokenizer that keeps short numbers ("windows 11", "iphone 17") and splits Chinese, Japanese and Thai text ships with the next index build.
- Freshness at scale. A targeted recrawl of the most searched results and the most linked sites is running now, and the crawler is moving to a continuous multi-process pipeline.
Why build it at all
Because an index you own is an index nobody can switch off. Microsoft retired the Bing Search APIs in August 2025 and many apps lost their search overnight. Products that need raw results, news by country, or readable page text for AI agents need a source that answers in plain JSON and does not depend on someone else's roadmap.
That is what NestDaddy is for. You can try the search, news and page extraction APIs with a free 7-day key at nestdaddy.com/apis, or connect it to Claude, Cursor or any MCP client through nestdaddy.com/mcp.
If you are building something similar and want to compare notes, we would like to hear from you.
Originally published on the NestDaddy blog. NestDaddy is an independent search engine with its own index: nestdaddy.com.
