A local conversation archive: importing, indexing and reading 1.35 million messages
A client held years of exported conversations spread across thousands of folders in inconsistent formats, effectively unreadable and unsearchable. Paymentery built a desktop platform that imports the entire archive, indexes it for instant search, and presents it in Arabic and English. The client is not named at their request. The scope described here was completed and handed over. Every screen on this page comes from a demonstration build filled with invented data; no real message, name or number is shown.
The requirement
The archive was not a database. It was a decade of exports: over ten thousand conversation folders, spreadsheets whose text had to survive byte for byte, attachments referenced by names that no longer matched the files on disk, and five exports containing characters that crash ordinary spreadsheet readers outright.
Two constraints shaped every decision. Nothing could leave the machine: no cloud, no upload, no third-party service, not even a font request. And nothing could be altered: message text and timestamps had to be preserved exactly as exported, with no timezone inferred and no silent normalisation.
What was built
The importer
A memory-bounded, resumable importer that reads each export in a single transaction, so a failure can never leave a half-imported conversation behind. Re-importing the same source, version and row ordinal produces no duplicates, while genuinely repeated messages with identical text are preserved as the distinct records they are. A conservative XML reader handles the malformed exports that off-the-shelf libraries reject.
Search
A local SQLite FTS5 index using trigram tokenisation narrows candidates, then every hit is verified against the original text literally before it is shown, so the index can never invent a match. Search runs archive-wide or inside one conversation, and jumps to the exact message in its surrounding context rather than returning a detached snippet.
The interface
A React workspace with virtualised lists, so a ten-thousand-row conversation list never builds ten thousand DOM nodes, and paginated message loading over a local API, so opening a forty-thousand-message conversation does not load forty thousand messages. Arabic and English, light and dark, desktop and mobile layouts.
Engineering decisions
One database per conversation
Every conversation keeps its own SQLite database and stays independent. Nothing is merged on the platform's initiative: identities are not fused, contacts are not consolidated across conversations, and a conversation type is never inferred from sender count alone. Where the evidence is inconclusive, the record says unclassified rather than guessing.
Two searches, deliberately separate
The field above the conversation list returns matching conversations and matching messages together, the behaviour people already expect from a messaging app. Archive-wide search is a separate surface with its own result pages. Collapsing them into one box was tried and rejected: the two have different result shapes and different intents.
Attachments with evidence, not guesses
Attachment references are resolved into three explicit states: confirmed links backed by matching metadata, unapproved candidates where the evidence is only partial, and unresolved references where the file was not found. An unresolved reference is never presented as a deleted file, and a changed file extension is never accepted as proof on its own.
Original text, untouched
Stored text is the exported text. The search index holds a separate derived copy, versioned independently, so improving search can never rewrite the archive. Dates are kept as the strings they were exported as, and the interface states plainly that the timezone is unspecified rather than quietly assuming one.
Nothing leaves the machine
The service binds to the loopback interface only. Fonts, styles and scripts are served locally with no external requests at runtime, path access is bounded and symlink-checked, and no endpoint can be induced to read outside the approved archive. Verification during development was done with anonymous statistics — counts, timings and checksums — so that even the build process never handled message content.
Arabic and English
Arabic is a first-class locale. The layout direction, the icon order, the scroll behaviour and the date rendering all change with it, and both languages ship the same features. Typography uses locally bundled Arabic and Latin variable fonts so that switching language never triggers a network request or a visible reflow.
Working with the archive
Beyond reading, the platform supports the work of actually using an archive: bookmarking individual messages, tagging and archiving conversations, reviewing proposed identity links with an undo for every decision, and browsing attachments by conversation or across the archive with the confirmed and candidate scopes kept apart.
Quality
- The full archive was imported and verified end to end: every stored record matched the search index in both text and ordering, and the message API was exercised for all 10,685 conversation routes.
- Interface behaviour was tested with Playwright across Chrome and WebKit, desktop and phone viewports, in Arabic and English, against synthetic fixtures only.
- Automated accessibility checks with axe-core covered contrast, labelling, dialogs and keyboard navigation on every screen.
- Path-traversal, symlink and origin-restriction tests confirmed that no request can reach a file outside the approved scope.
- Import was verified as resumable and idempotent: interrupted runs, repeated runs and byte-identical duplicate messages were each tested for the correct outcome.
Result
The complete archive is imported and readable: 10,685 conversations and 1,352,442 records — 1,233,291 messages and 119,151 system notifications — with zero failed conversations. 41,865 attachment links are confirmed, and 4,105 contact representations are available without any identity being merged.
Archive-wide search returns in well under a second once the index is warm. The platform runs entirely offline on the client's own machine, and the source exports were never modified.
Delivered and handed over: the importer, the local service and index, the interface, the verification tooling that reports anonymous statistics only, and documentation of every decision and known limitation. Cross-conversation contact linking and a standalone mobile package are documented as later phases rather than implied as present.
Have an archive nobody can search?
Describe the exports you hold and the privacy constraints around them. Format, volume and confidentiality are assessed before any scope is defined.