Disclosure: I built and maintain the open-source tool discussed in this article. It is MIT-licensed and free.

My coding agent can quote any file in my repo. It can search the web, hit APIs, and refactor a module while I get coffee. Ask it "what was I working on before lunch?" and it has nothing. Total blank.

That blank spot has a name in the research: episodic memory for AI agents, a record of experience situated in time. A 2025 position paper called it the missing piece for long-term LLM agents, and I kept nodding while reading it, because I'd hit the wall myself. My agent knew my code better than I did and knew my day worse than a stranger.

So I spent a few weeks building the layer that fixes it, then measured everything on 55 days of my own screen capture. The short version: you can compile raw screen recordings into agent-readable memory with plain code, no LLM anywhere in the pipeline, and the result is 88x smaller than the raw data. Here's how, with the numbers.

AI Agent Memory Remembers the Wrong Thing

Look at what "memory" means in the current agent stack. MemGPT pages conversation history in and out of the context window. Mem0 extracts salient facts from your chats. Zep builds a temporal knowledge graph over dialogue and business data.

All of it is conversation memory. What you told the model.

But think about what actually defines your working day. Not the chat box. The pull request you spent 90 minutes on, the forty tab switches after lunch, the email you drafted and abandoned at 4:40 pm. None of that flows through the agent, so none of it gets remembered.

The raw material exists, though. Screen recorders have been a commodity for years. ActivityWatch has logged window focus for a decade. Microsoft ships Recall on Copilot+ PCs. OpenAI quietly added Chronicle to Codex in April 2026, capturing screen content to build memories for its own assistant. Capture is solved.

Consumption is not. A single day of event-driven capture on my machine produces about 2,000 database rows, and each row says only this: at 22:53:05, Chrome showed linkedin.com/in/someone. Two thousand timestamped instants. An agent can't reason over that any more than you can read a movie from its film strip.

Two Ways to Feed Screen Data to an Agent (Both Bad)

The obvious approaches both fall apart in practice. I tried both before building the third.

The first option is raw search: give the agent a search tool over the capture database and let it figure things out at query time. This is what most recorder APIs expose. I measured what it costs: serializing one day of snapshot rows as JSON, the way a search API returns them, comes to 126,812 tokens with the cl100k tokenizer. For one day. The agent then has to do sessionization, deduplication, and duration math in its head, every single query. You're paying the model to do janitorial work.

The second option is LLM summarization. Pipe the rows through a model, store the summary. This compresses well, but now your memory inherits every weakness of its summarizer. It costs money per run. It's non-deterministic, so you can't cache it, diff it, or test it in CI. Long-context models use the middle of long inputs poorly. And a summarizer can hallucinate a meeting that never happened, which is a strange property for something you're calling memory.

Neither gives you what memory actually needs to be: cheap, reproducible, and trustworthy. There's a third option, and it's deliberately boring.

Deterministic Compilation: The Boring Middle Layer

Compile the snapshots into episodes with plain code. No model in the loop.

A compiler doesn't have opinions about your code. Same input, same output, every time. Apply that idea to screen capture and each day compiles into bounded activity frames: an app and a site, a start and an end, active time, the pages you actually looked at, and how much you typed. One real frame from my data (entities anonymized):

- app: Google Chrome
  site: linkedin.com
  start: "20:24:04"
  end: "20:42:11"
  duration_min: 18.0
  pages:
    - {kind: people_search, entity: "cto paris", count: 2}
    - {kind: profile, entity: jane-doe}
    - {kind: company, entity: acme}
  input: {keys: 214, clicks: 31}
  evidence: {frame_ids: "99871..100147"}

Three rules do most of the work, and all three are just arithmetic.

Dwell. Capture is event-driven: a frame gets stored when the screen changes. Each frame earns min(gap_to_next_frame, 90s) of active time. The cap matters because a static screen (or an absent human) shouldn't accumulate credit forever, while normal reading, which produces no screen changes, still counts.

Session gaps. A silence longer than 300 seconds closes the current frame and gets reported as a gap. Not smoothed over. Reported.

Flicker merge. You check Slack for 10 seconds mid-task, constantly. The sequence A to B back to A, where B lasts under 20 seconds, folds into one A frame, and B gets recorded as an interruption with its measured seconds. So the Slack detour still shows up. It just stops shredding your focus blocks into noise.

URLs get typed the same way: pure string parsing turns linkedin.com/in/jane-doe into profile: jane-doe and github.com/acme/api/pull/412 into pull_request: acme/api#412. Twenty-odd site parsers, each a pure function, with a generic fallback so no URL is ever lost.

Measured vs Inferred: The Contract That Keeps Memory Honest

Here's the design decision I'd defend in a bar fight. The schema has two tiers, and the wall between them is the whole point.

The measured tier contains only fields that a computer can derive from capture data. Sessions, durations, typed entities, and input counts. There are no intent labels, because code can't observe intent. The compiler will tell your agent, "two profile views and a people search in 18 minutes." It will never say "prospecting." Your agent is an LLM. Drawing that conclusion is its job, and it can do it at query time with full context.

Anything interpreted lives in a second tier, where it must be namespaced, tagged with a confidence level, and linked to the measured evidence supporting it. Strip the inferred tier, and you always have pure fact left.

Why so strict? Because agent memory is turning into an attack surface. There's active research on poisoning stored memories to steer agent behavior, and the defense starts with provenance. Every compiled frame carries pointers back to the raw rows it came from, so an auditor can recompute it mechanically. A planted or hallucinated label can't masquerade as an observation when observations are, by construction, recomputable.

What 55 Days of My Own Screen Showed

I ran the compiler against my own corpus: 55 calendar days, 100,561 snapshot rows, 188,317 input events, 52 applications. Some numbers survived contact with reality better than others.

Metric                           Result                      
One day as raw rows              126,812 tokens              
Same day, compiled document      32,216 tokens (3.9x smaller)
Same day, compact context block  1,441 tokens (88x smaller)  
Full-day compile time            68ms median (M3 MacBook)    
Reproducibility                  byte-identical across runs  
Tokens spent compiling           zero                        

The 88x is the headline, but 68ms is the number that changed how I think about it. At that speed, you stop maintaining memory and just rebuild it on every query. Staleness stops being a concept.

The surprise was fragmentation. Across 40 active days, the median activity frame lasted 0.3 minutes, and 74% of all frames ran under one minute. I expected tidy work blocks. Strict segmentation showed me confetti. That's a finding about how I actually work. An LLM summarizer would have smoothed it into fiction like "spent the afternoon focused on development," and I would have believed it.

Honest limitation: this is one user, one machine, my corpus. Cadence and fragmentation will differ for yours. But the mechanism (deterministic rules over event-driven capture) doesn't care whose data it runs on.

Wiring Episodic Memory Into an Agent Over MCP

The consumption side is where this stops being an experiment. The compiler exposes an MCP server, so any MCP-capable agent picks it up in one line:

pip install activity-frames
claude mcp add activity-frames -- aframes mcp

After that, "what was I doing before lunch?" resolves through a get_context tool call against local data. For agents outside MCP, it's two lines of Python:

from activity_frames import ActivityLog
context = ActivityLog().context(hours=4)  # paste-ready, ~1.4k tokens

I open-sourced the whole thing as activity-frames (the activity-frames repo on GitHub, MIT) because the format matters more than the code. Recall and Chronicle keep their memory inside walled gardens. The open-source capture engine I run underneath exposes raw search. The layer that turns capture into honest, typed episodes should be a standard, not a moat.

FAQ

Doesn't this mean recording everything I do? Yes, locally. Capture, storage, and compilation never leave the machine; the compiler opens the database read-only, and typed text stays out of every output unless you explicitly opt in. You get keystroke counts by default. The actual text never leaves the database unless you flip a flag. The database at rest is sensitive, same as any recorder, so treat it like your browser history.

Why not just RAG over the raw rows? Retrieval solves finding, not understanding. Pull 50 raw snapshot rows into context, and the agent still has to infer sessions, durations, and what mattered. Compilation does that once, deterministically, for everything.

Does it need a specific recorder? The schema is engine-agnostic: anything that writes frames with timestamps, apps, and URLs into SQLite can feed it. The reference implementation ships with an open-source capture engine it can provision for you, and reads its SQLite format out of the box.

Where This Goes

Three takeaways if you're building agents.

First, conversation memory isn't enough. Your agent's biggest blind spot is the 8 hours of context your user generates outside the chat.

Second, put deterministic code between capture and model. Anything an if-statement can compute, an LLM shouldn't guess. Save the model for interpretation, where it's actually needed.

Third, keep facts and inferences in separate tiers with an auditable boundary. Memory you can't recompute is memory you can't trust, and agents are about to make that distinction expensive.

The bigger shift is economic. When episodic memory costs 68 milliseconds and zero tokens per day, it stops being a feature and becomes infrastructure, like logging. Agents that know what their humans actually did will feel unreasonably competent next to those that don't. The data's already on your disk. It's just waiting for a compiler.

Disclosure: I built and maintain activity-frames, the open-source tool discussed above. MIT-licensed, free, no commercial tier.