A researcher accidentally turned LLM memory into program analysis

For the past few months security researcher Jordy Zomer has been using LLM agents for vulnerability research. In a post published on 28 August, he describes how an attempt to fix their memory problems turned into a Datalog-style program analysis engine called Lemmalog.

The problem is not that agents are bad at navigating large codebases — they are surprisingly good at that. But once an investigation runs for hours, the model slowly loses track of what has actually been established: it suggests approaches already ruled out, forgets that an assumption turned out to be false, or keeps reasoning from an observation that is no longer valid.

Memory as program analysis

Existing memory systems usually store past conversations and observations, embed them and retrieve the most relevant pieces. Zomer argues that this hides two different problems under one word. Retrieval answers "what from the past is relevant to this question?", but not "given everything learned so far, what is currently true?"

Lemmalog splits the work. The LLM handles the fuzzy part, converting natural language, source code and debugger output into structured facts, while the engine handles the deterministic part: facts and rules derive additional facts.

Retractions, provenance and time

Removing a fact is harder than adding one. If a conclusion has several derivations, deleting one supporting fact should not delete the conclusion while another still holds — which matters often in vulnerability research, where a candidate may be exploitable through more than one independent path. So the engine tracks how every fact was derived, updates support when something changes, and can answer why a conclusion is believed. If a claim has no provenance behind it, it is not part of the maintained state.

Benchmark results

Zomer tested the system on LongMemEval and LoCoMo using the benchmarks' own standardized readers and judges, with extraction done once per conversation. On LongMemEval (102 questions) Lemmalog scored 0.463 F1 (±0.010) and 0.575 accuracy — ahead of OpenClaw (0.244) and full-context prompting (0.222), but behind PropMem (0.550) and SimpleMem (0.480). It was strongest on knowledge updates (0.579 against PropMem's 0.528) and weakest on multi-session reasoning (0.211). Across LoCoMo's 1,986 questions it averaged 0.533 F1 (±0.001), behind PropMem and OpenClaw but ahead of full context on adversarially false-premise questions (0.707 versus 0.509). The answering model receives roughly 2,700 tokens per question on LongMemEval instead of about 104,000 — around 38 times less context, because extraction is paid for once.

Many of the gains came from unglamorous fixes: an instruction meant to reduce hallucinations made the reader refuse 32 answerable questions, and a plural stemmer that ignored short words silently dropped counting evidence. Zomer is explicit that PropMem still leads overall and that LongMemEval's sample is small. The next experiment, he says, is the one that motivated the work: a long vulnerability investigation.