portage-cli is a Rust reimplementation of Gentoo's Portage tools; Like Portage, it reads the package repository from the file system: ~132k files plus some more if you are using git as backend:

What Files Share of files Size Avg per file
Ebuilds (cat/pkg/*.ebuild) 33,788 25.5% 56.5 MiB 1.7 KiB
md5-cache (metadata/md5-cache/) 33,788 25.5% 64.7 MiB 2.0 KiB
Package metadata.xml 19,460 14.7% 9.5 MiB 0.5 KiB
files/ (patches, configs, init scripts) 18,375 13.9% 26.0 MiB 1.5 KiB
Manifest 18,220 13.8% 28.1 MiB 1.6 KiB
metadata/glsa/ (security advisories) 3,855 2.9% 9.6 MiB 2.6 KiB
profiles/ 3,414 2.6% 2.9 MiB 0.9 KiB
licenses/ 645 0.5% 4.9 MiB 7.7 KiB
metadata/ other (news, DTDs, QA checks, …) 279 0.2% 2.0 MiB 7.3 KiB
eclass/ 255 0.2% 2.3 MiB 9.4 KiB
Category metadata.xml 174 0.1% 0.2 MiB 1.2 KiB
Top-level files, .github/, scripts/ 8 <0.1% <0.1 MiB —
Total 132,261 100% 207 MiB 1.6 KiB

That is a lot of small files to sync, store and stat.

Since on file systems like ext4 it does have a large impact, I tried to see how different would be if the repository is a single redb database instead, filled straight from git, with no checkout at all.

TL;DR

It works:

  • Sync is much faster: one day behind, 0.54 s against 0.83–0.97 s for git, and 8.28 s for emerge-webrsync. There is no worktree to update.
  • About a quarter of the space on a plain filesystem: 174 MiB against 593 MiB of checkout files plus a 96 MiB .git. NOTE: btrfs with zstd provides similar gains if tuned properly.
  • Reads are faster, but not dramatically: em -p @world is about 10% faster cold and the same warm.

The idea

A git repository is already a database of objects. A sync only needs the new commit's objects and a diff of two trees. A checkout is something you write out afterwards for tools that want files.

So the database holds:

  • the git objects (blobs and trees, zstd-compressed when that saves space), keyed by object id;
  • a paths index from repository path to object;
  • derived tables: the md5 of every ebuild and eclass, the package list, and what symlinks point to.

A sync speaks git protocol v2 directly, thanks to gix:

  1. ask for a thin pack against the stored commit;
  2. complete it with base objects already in the database, all in memory;
  3. check the tip's signature;
  4. diff the old and new root trees;
  5. apply everything in one write transaction.

There is no .git directory, no worktree and no temporary files. The tree is always exactly one signed commit: the old one or the new one, never a mix.

As long we do not need the actual files materialized we can at least spare space.

The main work

All the codebase had to avoid doing direct file system access and that required a bit of refactor here and there, zero cost abstraction is a Rust promise and it held well:

  • TreeSource reads bytes by repository-relative path: read, list, kind. FsTree is the file system checkout and RedbTree is the database. The parsers for profiles, layout.conf, categories, Manifests and USE descriptions are shared and no longer care where the bytes come from.
  • EntrySource produces valid metadata entries, and each backend answers it in its own way:
    • A fs checkout needs mtimes, a gap index and md5 re-digests, because files may change under you.
    • A redb tree is always a commit, so its answer is a merge of two ordered tables.
  • A build gets one real directory: FILESDIR, EBUILD and the Manifest need a package directory on disk. For redb that is a copy of just that package directory, kept per tree. Ebuild text, eclasses and profiles are read through the tree.

Masters, overlays and alias repositories sit above storage, so they mix freely. An overlay on disk can use a redb ::gentoo as its master, and inherit reads eclasses through the master's tree without copying anything.

This split was worth doing even if redb experiment goes nowhere. On the default path it costs nothing measurable:

em -p @world, warm, 15 runs master branch (no redb)
851 ± 10 ms 855 ± 9 ms

Results

All runs were on one machine (Ampere-1a, 128 cores), pinned to one NUMA node, with the checkout and the database at the same commit. That last detail matters, as you'll see later.

Sync: the clear win

Behind git fetch + reset --hard redb sync Wire pack
1 day 0.83–0.97 s 0.54 s 106 KB
7 days 1.46–1.48 s 1.25–1.26 s 1.81 MB
12 days 1.96–1.98 s 1.41 s 3.83 MB

Over GitHub, 12 days behind, it was 2.7–2.9 s against 3.5–5.3 s for git. For comparison, emerge-webrsync takes 8.28 s for the 1-day case and downloads 50 MB.

About 40% of the 1-day redb sync is compact(). It keeps the file near its import size; without it, 14 daily syncs grow the file from 143 to 178 MiB.

Size: it depends on your filesystem

checkout files .git (depth 1) redb tree + md5-cache
Plain filesystem (ext4, 4 KiB blocks) 593 MiB 96 MiB 143 + 31 MiB
btrfs with compress=zstd:3 112 MiB not measured 173 MiB

On an uncompressed filesystem, every small file costs at least a block, and redb is about a quarter of the size. On btrfs with zstd, most of the tree's files are stored inline in metadata. Then the checkout's files are smaller than the database, which is already zstd-compressed, and the saving is only the .git directory.

Reading metadata: parallelism matters more than the container

Reading the whole md5-cache:

Threads directory redb
1 318–350 ms 335–336 ms
8 61–76 ms 66–74 ms
12 46–57 ms 38–40 ms
32 33 ms 20–21 ms

Most of the gain is parallelism, and a directory read gets that too. In em itself:

checkout redb
Valid entries for ::gentoo 70 ms 55 ms
em -p @world, warm 934 ± 41 ms 920 ± 40 ms
em -p @world, cold page cache 1.32–1.42 s, 124 MiB read 1.19–1.25 s, 119 MiB read
em search -S, warm 62 ms 76 ms (every entry is decompressed)

The two backends give the same @world plan and the same search output. Cold, redb is about 10% faster with about the same bytes read; warm, they are equal.

A slow path lingering

Running with no metadata cache at all exposed something unrelated to storage. Ebuilds without a valid cache entry were checked and sourced one at a time, on every backend. Moving that onto the worker pool em regen already uses made the process snappier:

Case before after
Overlay without md5-cache, 3,909 ebuilds, cold 12.75 s 1.50 s
redb, no cache database, 33,977 ebuilds, cold 73.1 s 7.64 s

So far

Sync is where a database pays. One thin pack and one transaction beat git fetch plus reset --hard at every delta, and leave no worktree to repair. Since the tree is always one verified commit, its metadata needs no freshness checks.

The tree and the metadata cache live different lives. They ended up as two files. The cache can lag behind the tree, run ahead of it, or be missing. Each backend says which entries it lacks, and shared code fills them.

A build needs one real directory, not a checkout. My first attempt copied eclass/ and the repository's identity files and opened the copy as the repository. It was a shortcut, and it hid profiles/thirdpartymirrors. Tracing a real build showed that only the package directory has to exist on disk.

Measure on matching inputs. One early cold-cache figure (99 against 124 MiB read) came from a build that still read profiles and eclasses from the checkout. A redb tree from a different commit than the checkout made an overlay's cache look stale on every run. Re-measured on one commit, it is 119 against 124 MiB. Several wrong turns were beliefs that a trace, a profile or the reference tool then corrected.

redb's readers and writer exclude each other. In its default mode both lock the file when they open it, so a running em keeps a sync from starting, and the other way round. Now the second one fails at once with an error naming the first.

Independent reviews find what the author stops seeing. Reviews of the branch found:

  • pruning old package copies could delete one another em was still building from;
  • a locked or corrupt database silently fell back to a stale checkout next to it;
  • a remote that hadn't moved skipped the signature check against the current keyring;
  • symlinks read as their target's name instead of its contents;
  • em clean treated a database-stored repository's distfiles as unreferenced.

Each of these became a regression test.

Try it (experimental)

The redb backend lives on the tree-source branch, behind cargo features, and the database format may still change.

cargo install --git https://github.com/lu-zero/portage-cli --branch tree-source portage-cli --features redb,sync-gix

Then set the sync type in repos.conf:

[gentoo] location = /var/db/repos/gentoo sync-type = git-redb sync-uri = https://github.com/gentoo-mirror/gentoo.git

em sync imports the checkout's current commit from its git objects, or fetches the tip if there is no checkout. Later syncs fetch thin packs into the database.

What's still open

This first experiment is far from being a complete redb backend:

  • the db layout is naive
  • the compression can be tweaked
  • the db vacuum/housekeeping strategy can be better
  • the whole sync db-updates can be batched/streamed so the resident set can be reduced.

The design notes and the full measurements are in the repository: tree-storage.md and tree-storage-experiment.md.