Reading a multi-gigabyte MBOX in a browser tab

What broke at 512 MB, and the one-pass scan that replaced it · 20 September 2026 · measurements added 21 September 2026

This is a write-up of one bug in Mailward and the rewrite that fixed it. The numbers come from its tests and from its failure telemetry, which records a category like “mbox, over 1 GB, parse failed” and never anything from the file.

The failure

The first MBOX reader did the obvious thing: read the file, decode it to a string, split it into messages. It worked on the mailboxes we tested with. Then the telemetry showed someone on an Android phone opening an mbox in the “over 1 GB” bucket, failing, trying again, and failing again.

The cause is not memory in the usual sense. V8 caps the length of a single string at 2²⁹ − 24 characters, a little over 512 MiB. Decode a bigger file into one string and you do not get a slow page — and, as it turns out, you do not reliably get an error either. We first wrote here that you get an exception; measuring it in Chrome 153 showed something worse. Building a string one character past the cap does throw (RangeError: Invalid string length), but TextDecoder.decode() — what the old reader called — returns a string of length 535,822,336 for 511 MiB of input and a string of length 0 for 513 MiB: an empty string, no exception, with or without fatal: true. The reader then split nothing into no messages. Blob.text() on the same 600 MiB fails with a NotReadableError whose message blames file permissions. A Google Takeout export of a Gmail account that has been in use for ten years is routinely several gigabytes, and it arrives as one file. The files people most want to open were exactly the ones that could not be opened.

Why you cannot just split on “From ”

An mbox is every message concatenated into one text file, each preceded by a line that starts with From (with the space), an envelope sender and a date. That is the whole format. The well-known catch is that a message body may also contain a line starting with “From ”. Writers are supposed to escape it as >From (the mboxrd convention escapes already-escaped lines too, so it is reversible), but plenty of real files are written by tools that never escape anything.

If the reader splits on the bare prefix, as many importers do, every such body line cuts a message in two. So the separator we match is a whole envelope line: From , anything, then a date with seconds in asctime form (Thu Jan 1 00:00:00 1970). Takeout puts a UTC offset before the year and some Unix spools put a zone name there, so the pattern allows both. A sentence in a body that happens to start with “From ” almost never ends in a timestamp with seconds and a four-digit year.

The scan

The rewrite never holds the file as a string. In a Web Worker, FileReaderSync reads the user’s File through file.slice(start, end) in fixed 4 MB steps, and one pass over those chunks finds the separators and records a byte range per message. Three details did most of the work.

Decode each chunk as Latin-1. The scan needs offsets into the file, but a regular expression gives offsets into a string, and with UTF-8 those differ as soon as there is one non-ASCII character. Latin-1 maps every byte to exactly one character, so a string index is a byte offset. (In browsers the latin1 label actually selects windows-1252, which has the same one-byte-one-character property, and that property is all the scan needs.) The separator is pure ASCII, so nothing is lost by scanning this way; the real decoding happens later, per message, from the original bytes.

const latin1 = new TextDecoder("latin1"); // one byte -> one char
for (let offset = 0; offset < size; offset += chunkSize) {
  const bytes = loader.loadPage(offset, Math.min(chunkSize, size - offset));
  const base = carry.length > 0 ? carryBase : offset;
  const text = carry + latin1.decode(bytes);
  // run the separator regex over `text`; base + match.index is a file offset
}

Carry the unfinished line across the boundary. A chunk boundary can land in the middle of a From_ line. After each chunk the scan keeps the tail from the last newline onward and prepends it to the next chunk, so the line is seen whole. Two details are easy to miss: if the mailbox uses CRLF, the carry has to start at the \r, not the \n, or a separator split between those two bytes is missed; and the carry is capped at 4,096 bytes, because a “line” longer than that is a base64 blob or a minified HTML body, not an envelope line, and carrying megabytes of it forward would quietly bring the memory problem back.

Never take the same separator twice. Because the carry replays the end of the previous chunk, a separator found there can match again. The scan remembers the offset of the last newline it accepted and ignores anything at or before it.

What survives the pass is small: for each message a start offset, an end offset, and the summary the list needs — subject, sender, date, whether it has attachments. Getting that summary right means parsing each message once during the scan (postal-mime, on the message’s own bytes), but the parsed result is dropped as soon as the summary is taken. Memory grows with the number of messages, not with the size of the file.

Opening a message

Clicking a row re-reads just that byte range, drops the envelope line, undoes the escaping — one leading > comes off every line matching ^>+From — and parses it properly, with its real charset. A cache of the last four parsed messages exists for one reason: reading a message and then saving one of its attachments should not parse the same ten megabytes twice.

Takeout’s labels

Gmail has labels, mbox has no folders, and Takeout resolves this by flattening everything into one file and writing each message’s labels into an X-Gmail-Labels header. Since the scan already parses headers for the summary, it collects those values and rebuilds them as virtual folders next to the flat “all messages” view. A message with three labels is listed under all three, which is what Gmail showed. The sidebar keeps the hundred largest labels; past that it stops being navigation.

The test that matters

The property worth pinning is that the chunk size is invisible. The test builds a mailbox — LF and CRLF variants, escaped and unescaped body “From ” lines, a message much larger than a chunk, a UTF-8 byte-order mark at the start — and opens it with a series of chunk sizes chosen to cut envelope lines and CRLF pairs in every possible place. Every size must produce the identical archive. A single fixed chunk size in the tests would hide exactly the boundary bugs described above.

Measured

Numbers, so the claim can be checked rather than believed. Machine: Apple M3, 24 GB RAM (model identifier Mac15,12), internal SSD; macOS 27.0 (26A428). Browser: Google Chrome 153.0.8010.52, headless, driven by Playwright; a fresh profile per run. Target: the production build at https://pst.aivismonitor.com/open-mbox-file-online, free tier. Files: synthetic Google-Takeout-shaped mailboxes written by scripts/bench/gen-mbox.mjs: plain-text messages of 6 to 35 short paragraphs, every eighth message carrying a 200 KB base64 attachment (about 39 KB per message on average), X-Gmail-Labels on every message, LF line endings. Each row is a single run on 2026-09-21; “open” is from handing the file to the page until the folder list and the first page of messages are on screen, memory is the peak resident size of the tab’s renderer process (the worker lives in it) sampled twice a second while opening, and “full body pass” is a search for a string that occurs nowhere, which cannot stop early and therefore re-reads every message.

Three things to read off it. Opening is linear in file size — about 55 MB/s on this machine, reading and parsing every message once. Memory follows the number of messages, not the bytes: a file 102 times larger cost 408 MB more, roughly 1.5 KB per message for offsets and the listing summary. And reading a message costs the same in a 10.7 GB file as in a small one, because it is one slice of the file either way.

What these numbers are not: they are one desktop with a fast SSD. A phone reading from slower storage will take longer roughly in proportion to its read speed, and we have no measured phone figures to offer. 10.7 GB is the largest file we have opened, not a limit we found. The generator and the measuring script are in the repository (scripts/bench/); the small public mailbox that checks correctness rather than capacity is on the sample files page.

What it does not fix

• Body search has to re-read the file, because the bodies are not kept. It reports subject and sender hits first and streams the rest with progress.
• A download that was cut short is still cut short. The last message simply ends early.
• A phone will open a 2 GB mailbox this way, but the first pass still has to read 2 GB from storage. It takes as long as it takes; the progress bar is honest about it.

After this shipped, the same “over 1 GB” bucket started showing successful opens from phones. The whole file is listed and searchable for free at any size; reading messages past the first 500 MB of a very large mailbox is part of the paid licence.

If you have a Takeout export sitting in your downloads folder, the reader is here: open an MBOX file in the browser — nothing is uploaded, and you can disconnect from the network once the page has loaded. The format details for MBOX and its neighbours are in the email archive formats reference, and converting the other way is covered under PST to MBOX.