Project memory from audio

My name is Daniel. I’m the developer of Nucleate, a local-first project history tool that’s built around voice capture. It’s released as a donationware-style product, in that it’s a serious tool but is free and without ads.

To be clear, Nucleate is not an automatic speech recognition (ASR) product. It does ASR-shaped work, but only to ingest data. Its real job is to be a project memory tool and historian.

I say this because there are literally dozens of ASR tools already out there already, and nearly all share the same backbone. Some tools are free, others are distributed as polished SaaS and charge a premium for cloud-based processing. In nearly all cases, they can correctly capture “What did I say?” but few actually extract meaningful project history.

Existing pipelines create individual, disconnected documents without real chronology. The ball is moving towards Agents + Karpathy-style post-processing, but currently cross-examination is largely weak or absent in the ASR space.

I don’t think this insight is all too surprising. When people talk out loud, thoughts are free-flowing and messy. The classic rule of thumb is that it always sounds better in your head.

After transcription, the obvious next step is to hand the transcript to an LLM. I did this too, but there’s a core problem with the handoff: every recording starts from zero. To me, the core issue is a lack of coherency across the lifetime of a project.

Nucleate is built around the idea of combining project history with automatic speech recognition. It uses layered time periods to track project barriers, progress, and direction.

It’s a pretty straightforward model. Audio becomes a transcript. Transcript becomes a summary. Multiple summaries become weekly reviews. Multiple weeklies become monthly rollups.

The “progressive compression” approach means that the system gets multiple shots to identify recurring topics and priorities. You don’t have to extract everything perfectly for each individual file.

A lot of value comes from structure and repetition, and so you can cheat a bit with smaller models like Qwen or Mistral. Because you provide the context and probably talk about like-kind topics, it’s easier to collect project history and meaningful status inside higher-level captures.

Nucleate is my attempt at what an ASR project memory system should look like: free, private, and flexible. I’ve tried hard to shape Nucleate into that kind of tool, and—while it’s not perfect—it’s a fair shake at a missing link in the space.

For those interested, there’s a ton of “how it works” info on my website. The one-sentence version is that it’s free, offline/local-friendly, connects to Obsidian/Notion, and can be heavily customized at both the model/AI-level and at the user-level to suit just about any kind of project.

Nucleate literally became a ~1000h journey. I use it daily and think it was worth it, if nothing else as a passion project that got me excited about something new.

I’ve made the decision to share it publicly as a free product. I think it punches well above its weight and is worth something, but I also know it to be strangely shaped and won’t necessarily suit most workflows. You kind of have to “buy in” to the idea of daily audio-to-notes pipelines, and I think we’re a rare breed.

I am probably going to close the book on Nucleate for a bit and migrate back to game dev, but I’d genuinely appreciate feedback from folks who have experimented with ASR, local AI, PKM systems or project tracking.