Opening a 50 GB text file without reading it

Most text editors open a file by reading all of it. That is fine for a source file and hopeless
for a 50 GB log: the work done before the first screen grows with the file, and no processor
saves you from that. BigText, a free and open-source editor for Windows, Linux and macOS, is
built on the opposite rule:

The time and memory of every operation depend on the text on screen, not on the size of the
file.

This article is about the five decisions that make that rule hold, what they measure at on a
50 GB file, and what I learned building the editor with an AI model.

BigText opens a 50 GB log at once, jumps to its end and its middle, searches it with a regular expression and follows a growing log.

A fast machine does not help

All the numbers below come from one machine:

On a machine like this everything should be instant, and for years opening a big log, CSV export
or SQL dump was not. The editor would think for a long time, the window would freeze, sometimes
it would refuse the file.

The reason is simple. Notepad++ allocates a buffer for the whole file and warns above 2 GB that
opening it may take minutes
(source).
VS Code keeps roughly the size of the file in memory
(VS Code blog).
Reading a file from start to end and splitting it into lines is one sequential pass, usually on
one thread: one of the 24 cores works and the rest wait. The GPU has nothing to do with it.

That pass is easy to measure, because BigText makes it too, in the background, to build its line
index. For the 50 GB file it takes 17–18 seconds on this machine, with the file already in the
operating system's cache, that is, in RAM. An editor that makes this pass before it shows the
first line keeps you waiting at least that long in the best case. If the file does not fit in
memory, it does not open at all.

The problem is not the hardware but the algorithm. If the work before the first screen is
proportional to the file, no processor helps.

The idea: read what the screen needs

A screen holds fifty or sixty lines, a few kilobytes of text. Showing the first screen of a 50 GB
file does not require reading 50 GB. Five decisions follow from that.

1. The file is mapped, not loaded, and in windows

Memory mapping (mmap, MapViewOfFile on Windows) ties a file to the program's address space
without reading anything. The actual read happens when the program touches a byte, and only for
that page. Opening a file this way costs the same whatever its size.

There was a surprise here. On Windows, mapping a 50 GB file as one view costs almost 100 MB of
memory by itself: the system keeps 8 bytes of bookkeeping for every 4 KB page. Memory use
depended on the file size again; with a 50 GB file open the process took 205 MB.

So the single view became a cache of windows: 288 MB each, starting every 256 MB so that a line
near a boundary always fits in one of them, with at most six mapped at a time. Memory dropped from
205 MB to 107 MB and stopped depending on the file size.

2. Opening reads the first megabyte

When a file opens, a worker thread reads its first megabyte. From that sample it decides the
encoding (UTF-8, UTF-16, Windows-1251 and others), the line endings and whether the file is
binary. That takes 1.1–1.4 ms. Then the lines that fit on the screen are drawn, and the file is
open.

3. The screen is anchored to a byte offset, not a line number

An ordinary editor knows its position as "I am at line one million", and to know that it has to
count every line before it. BigText knows "I am at byte 26,843,581,344". The scrollbar is
proportional to bytes too. Jumping to the middle of the file counts nothing: it reads a few
kilobytes there, finds the nearest line start and draws.

The line number is shown at once anyway. At first it is an estimate with a ~ in front of it;
when the index reaches that part of the file, the exact number replaces it.

4. A sparse line index: one number per 64 KB

Storing where every line starts is expensive. My test file has 453,565,956 lines; a full table of
offsets would take 3.4 GB. BigText splits the file into 64 KB blocks and stores one number per
block: how many lines came before it.

file: [ 64 KB ][ 64 KB ][ 64 KB ][ 64 KB ] ... 819 thousand blocks index: 0 554 1,107 1,661 ... lines before each block (example) 

For the 50 GB file the index is 8.12 MB, 0.016 % of the file. "Go to line N" finds the block by
binary search and then scans at most 64 KB of text, counting newlines with SIMD.

The index is built in the background from the start of the file, at about 2.9 GB/s: 17–18 seconds
for 50 GB. The editor works normally meanwhile: scrolling, jumping and searching all work.

5. The UI thread never waits

The UI thread reads only the bytes on screen. Indexing, search, replace and save run on worker
threads, each with a progress bar and a cancel button. The goal is that no frame takes more than
16 ms. While the 50 GB file was being indexed, a page-down frame took 1.2 ms on average.

There is one more edge case. A file can be a single enormous line, such as a 2 GB minified JSON
document. Such lines are split softly every 64 KB, so working out any line never reads more than
128 KB.

Measurements

The numbers come from a release build of BigText 0.4.1 and its own benchmark
(bigtext --bench-ui). Each file was measured three times, with the files in the OS cache.

The important row is the second one: the file grew fifty times and the time to open did not
change. Six more runs on the 50 GB file gave 35 ms at worst.

Memory hardly changed either. Most of the 100 MB is the graphical interface itself: the core
without a window holds the 50 GB file and its index in 12.1 MB.

What the 30 ms means needs saying precisely. Opening the file and detecting the encoding take
1.2 ms and laying out the first screen of text 0.03 ms; the rest is waiting for the window's next
frames. The benchmark stops the clock when the editor has painted the text, which does not
include the frame reaching the monitor, and it opens the file in the program's first frame.

In a window that is already running there are two cases. With another file open, the 50 GB file
opened in a new tab in 10 ms. The first file opened into an empty window took 0.1–0.2 s to show:
the editor's own work is again a few milliseconds, and the rest is the GPU driver or the
compositor presenting the first frame with a document in it. The cause of that one slow frame is
not known yet; a second file in the same window paints in about 10 ms.

Other numbers on the 50 GB file:

• Go to line before the index was ready: no frame above 2.1 ms.
• Plain-text search: 4.0–4.3 GB/s, the whole file in 12 seconds.
• Regular-expression search: 3.6–4.3 GB/s.

A regular expression found 830 matches in the 50 GB file in 15.5 seconds. The results list is under the text; a click jumps to a match.

Editing and saving

Viewing is half of it; big files also need editing. BigText uses a piece table, an old and proven
structure. The original file is never touched. Typed text goes into a separate buffer, and the
document is a list of pieces: "this part of the original, then this part of the buffer". A
thousand edits in a 50 GB file take a few kilobytes.

Saving streams the result into a temporary file in the same folder, flushes it to disk and only
then replaces the original atomically (ReplaceFileW on Windows, which keeps attributes and
permissions). If something fails on the way or the power goes off, the original stays intact.

The 25th gigabyte of the same file in hex view. Jumping there is instant too.

I built it with Claude

I built BigText with Anthropic's Claude model, through Claude Code. Claude wrote the code. My
part was to say precisely what was needed, check the result and choose the next step. Every one
of the 174 commits in the repository names Claude as co-author.

It started with one document: the goal of the program, eight architecture rules that must not be
broken, measurable speed targets and a list of edge cases that must not break it. It is in the
repository as prompt.md. Four of its requirements:

• Never read the whole file into memory.
• Never block the UI thread; no task on it may take more than 16 ms.
• Do not use ready-made editor components: they all keep the whole document in memory.
• Measure the speed targets and report real numbers.

Claude first wrote a plan (PLAN.md), then worked through it in stages: opening and viewing,
search, replace, editing and saving, encodings. After each stage it measured on the test files
and compared the result with the target.

• 24 October, 09:46: the first commit.
• 24 October, 14:38: version 0.1.0, five hours later.
• 29 October: version 0.4.1, the one measured above.
• 10–12 November: versions 0.5.0 to 0.7.3.

Today the project has 151 Rust files, about 70,000 lines of code including tests, 335 tests and an
in-app GUI self-test that drives the real window with synthetic mouse and keyboard input through
229 checks.

Four lessons:

1. Make the goal measurable. Not "it should be fast" but "the first screen in under 200 ms" and "under 200 MB of memory with a 50 GB file open". A goal like that can be checked.
2. Write the architecture rules down first. The decisions that mattered most were made before any code was written.
3. Measurement beats guessing. Memory dropping from 205 MB to 107 MB was not in the plan; it came out of a measurement.
4. Give the AI tests it can run itself. When the GUI self-test was added, it found four bugs, which were then fixed.

What else it does

Besides opening big files, BigText:

• searches with regular expressions, filters lines like grep and replaces across the whole file;
• follows growing logs like tail -F, jumps to a time, filters by level and opens logs on servers over SSH;
• opens CSV and TSV as tables with sorting, statistics and pivots, and JSON with a structure view;
• compares two huge files;
• reads files inside .gz, .zst, .zip and .tar without unpacking them by hand;
• highlights code in about two dozen languages and formats, with a folder side bar and go-to-file, for the files that are not huge.

Two logs compared: four differences in 425 thousand lines, found in 0.04 seconds.

Try it, or measure it yourself

BigText runs on Windows, Linux and macOS. It is free and open source under the MIT license, and
its interface is in eight languages. The test-file generator and the benchmark come with the
portable downloads:

gen-testfiles --only random-lines-10g.txt bigtext --bench-ui testdata/random-lines-10g.txt --report report.txt 

The 50 GB file is the 10 GB one joined five times.

• Website: https://erkinbek.github.io/bigtext/
• Downloads: https://github.com/Erkinbek/bigtext/releases/latest
• Source: https://github.com/Erkinbek/bigtext

If you have a file your editor cannot open, try it in BigText and tell me how it went.