Working with large MBOX files: open, split and back up

Ten years of email can easily produce an MBOX file of several gigabytes. Big files are perfectly valid, but they push against the limits of memory, file systems and some tools. This article covers what to do about each.

Why large files are slow or fail to open in MBox Viewer

Many tools, including browser-based viewers and some mail programs, read a whole archive into memory before they can list the messages. The amount of free RAM then decides how big a file they can handle.

MBox Viewer for Windows takes a different approach. It supports archives larger than 4 GB and, on the first opening, reads the file once to build an index with the .mboxview extension. That first pass can take several minutes for a big archive; later openings use the index and are much faster.

If a file will not load or takes too long:

• Close other programs to free memory and let the first indexing run finish.
• Use the 64-bit program, mboxview64.exe, on 64-bit Windows.
• Keep the archive and the program's data folder on a drive with plenty of free space.
• Split the file into smaller parts.

How to split an MBOX file safely before MBox Viewer

Do not use a generic file splitter. It cuts at a fixed size and will break a message in half. Split at message boundaries instead.

Python, which is preinstalled on most Linux systems and easy to add on Windows and macOS, can do this with a few lines. This example writes files of 5,000 messages each:

import mailbox
src = mailbox.mbox("big.mbox")
part, out = 0, None
for i, msg in enumerate(src):
  if i % 5000 == 0:
    if out: out.close()
    part += 1; out = mailbox.mbox(f"part-{part:03}.mbox")
  out.add(msg)
out.close()

Each part is a normal MBOX file that opens on its own. The original stays unchanged.

Check that a copy is complete before opening in MBox Viewer

When you copy a large archive to another drive, make sure the copy is identical. A checksum gives each file a fingerprint; if the fingerprints match, the files match.

Save the result in a small text file next to the archive. Years later you can run the command again and confirm that nothing has changed.

You can also compare message counts. Run grep -c "^From " archive.mbox on both copies, or open each in MBox Viewer and compare the mail count in the status bar.

Storing archives for MBox Viewer in the long term

• Follow the 3-2-1 idea: three copies, on two different types of storage, with one kept in a different place.
• Avoid FAT32. It cannot store files larger than 4 GB. Use NTFS, APFS, ext4 or exFAT.
• Compress for transfer, not for storage. A ZIP saves space when sending, but a plain MBOX file is easier to check and open later.
• Encrypt if the mail is sensitive. Use an encrypted drive or an encrypted archive, and store the password safely.
• Test once a year. Open the archive and read a few messages. A backup that has never been tested is only a hope.

MBox Viewer large file summary

Large MBOX files are normal. When one will not open, free memory or split it at message boundaries. When you store one, keep several copies, record a checksum and check now and then that it still opens.