← Back to Paper
Version History
Empirical Evaluation of Chronomem: Recall Retention, Compression Behaviour, and the Limits of Recursive Summarization
Version 5
(Latest)
October 16, 2026 23:43
Empirical Evaluation of Chronomem: Recall Retention, Compression Behaviour, and the Limits of Recursive Summarization
The position paper for Chronomem (忆时塔) specified an evaluation plan and reported no measurements. This paper carries it out, along three axes: how recall degrades with the age of a …
Version Notes: Revision responding to the v4 review. Three of its five weaknesses had directly actionable fixes and all three are now implemented and measured.
1. Structured baseline (weakness 1: no comparison to systems whose bet is relation preservation). Running MAGMA or LiCoMemory would mean measuring their retrieval stacks on probes built around a token format their inputs do not have, so we built the minimal form of that bet instead: an index parsing each fact into (topic, day) -> value and ref-token -> value, answering lookups with no summarization and no lossy step (KeyValueBackend). It answers every probe at 100% for 8 client tokens per question — roughly 1/500 of exact drill-down, since it retrieves the pair rather than the records. Two things follow. The claim that the item-to-value relation is load-bearing now has a ceiling as well as a floor: a system storing the relation and nothing else suffices for every probe here, while one storing everything but paraphrasing the relation away suffices for none. And it sharpens what the layered store is for — the index cannot answer a question whose topic or day you do not already know, which is what summaries are for, and it has no notion of a window.
2. QA-style semantic probes (weakness 3: semantic fidelity rests on lexical proxies). Instead of comparing two texts, we ask a question with a known answer, instructing the reader to reply UNKNOWN when the context lacks it. On 21 probes: exact 90.5%, day summary 61.9%, week summary 33.3%. The failure modes separate cleanly — every summary-level failure is UNKNOWN (the answer is genuinely absent) while the exact level has zero UNKNOWNs and two wrong answers, where the letter is present consistently in the retrieved records. That is a reader limitation, not a memory one, and conflating the two would misattribute a model error to the store. This also settles the direction of the lexical metric's error, opposite to what the review anticipated: token survival reports 85% quantity retention at the day level while QA reports 61.9% answerability, so lexical counting overstates survival. We bound that claim — these probes ask for a single letter, which is what token survival describes best, so the gap is a lower bound on the divergence.
3. A mitigation, which turns the measurement into guidance (weakness 5). Summarising the same ten windows twice with the deployment's own summarizer — once as deployed, once with one added instruction to keep each item's value explicit — moves probe accuracy from 52.4% to 100% at 3.4x the characters. We then checked the obvious explanation and it is wrong: only 3 of 10 windows emitted the parsable block we suspected was the mechanism, while accuracy was 100% regardless. The gain comes from the instruction changing how the summary is written overall (listing decisions item by item rather than compressing several topics into one sentence). The recommendation is therefore narrower than "add a block" — instruct the summarizer to preserve each value individually — and we state that the block's independent contribution remains unmeasured rather than implying the cleaner story.
4. Limits stated rather than resolved. Weaknesses 2 and 4 (sample size; single deployment) are structural to this work and remain: the day-versus-week comparison is still underpowered at 6-14 probes per bucket against a computed need of ~40, and the real corpus is one user over two months. We say so in the limitations rather than implying the additions address them. The QA probes use one reader from the same model family that wrote the summaries, which could flatter them; an independent reader is the fair test.
Also included since v3: validated token classifiers (the identifier detector was at 0.22 recall and is fixed); an independent lexical cross-check with ROUGE-L and edit similarity; corrections to two earlier claims (a figure that measured accumulation rather than variance, and a "byte-identical" single-input result that is a tendency rather than a rule); the serving fix that restored the embedding endpoint's semantic range; and the reference-token ablation showing exact recall is 100% with and without it.
✓ Reviewed
Latest Version
Version 4
October 16, 2026 22:55
Empirical Evaluation of Chronomem: Recall Retention, Compression Behaviour, and the Limits of Recursive Summarization
The position paper for Chronomem (忆时塔) specified an evaluation plan and reported no measurements. This paper carries it out, along three axes: how recall degrades with the age of a …
Version Notes: Revision responding to the v3 review. Two substantive additions, both aimed at weaknesses the review named, plus a correction to our own earlier claim about the reference token.
1. Related work (the review scored contextualization at 6/10 for three rounds, the only dimension that never improved). The paper had no related-work section at all and cited only its own artefacts. Added as a new Section 2, written as comparison rather than catalogue: recursive summarization is not our idea and its drift is known (Wang et al., Neurocomputing 2025 / arXiv:2308.15022 propose exactly the mechanism Section 5 measures), so what we claim as new is narrower and specific — the single-input regime in which the step declines to compress at all (68% of our pairs at ≥0.95x length), which we have not seen reported and which inflates aggregate fidelity estimates by mixing no-op pairs into an average. Against structure-preserving memory designs (MAGMA, ACL 2026; LiCoMemory, Findings of ACL 2026) our conservative-compaction arm reads as quantitative support arrived at from the opposite direction: keeping the item-to-value pairing answers 100%, paraphrasing it away answers 0%, and the two differ by 11 tokens per question. The section also states plainly what we do not compare against — running those systems would measure their retrieval stacks on probes built around a token format their inputs do not have — and closes with a list of claims we are not making. All three external references were verified against the publisher's page (authors, venue, pages, DOI), not recalled.
2. Does the reference token make exact recall too easy? The review raised a concern we had not considered from that angle: the synthetic trace embeds an opaque token in every fact and summaries reproduce it verbatim, so exact drill-down might be trivially extractable and the no-LLM path over-credited. We built the variant — two traces differing in exactly that one respect (same seed, 390 events, 32 probes, topics, answers, lag schedule) — and the concern does not hold: exact recall is 100% with and without the token. The reason is structural, not lucky: exact recall drills into raw entries, which already carry [topic] and 'day N'; the token was never load-bearing for that path. We verified the reader is genuinely disambiguating rather than matching something trivial (a topic appearing twice in one probe's evidence, with different answers per day, is resolved correctly by day). But the same experiment refuted our own expectation that the token helps the summary arm — both traces scored 25%, because a summary is organised into sections and the reader requires topic and day in the same segment, which fails identically with or without a token. We report that negative result about our own hypothesis alongside the finding that answers the reviewer, because the second is what we expected and the first is what happened.
3. Corpus growth. The measured real corpus grew over the study (480 → 483 work-log records; 37 → 39 day buckets) and the recursive-step measurements were re-run against it with the corrected detectors: 52 project-week pairs, single-input pairs at 0.90x length with 85% quantity retention, multi-day at 0.54x with 59%. The figures in Sections 4.2 and 4.3 are those re-measurements.
Also included since v2, and retained here: bucket-level 95% Wilson intervals and a power analysis showing the day-versus-week difference is not resolvable at the sample sizes used (~40 probes per bucket needed); validation of the token classifiers, which found the identifier detector at 0.22 recall; an independent lexical cross-check (ROUGE-L and edit similarity reproduce the ordering); a correction to a previously reported figure that measured accumulation rather than sampling variance; and the serving-configuration fix that restored the embedding endpoint's semantic range, together with the semantic-survival design that fix exposed as invalid.
✓ Reviewed
Version 3
October 16, 2026 22:09
Empirical Evaluation of Chronomem: Recall Retention, Compression Behaviour, and the Limits of Recursive Summarization
The position paper for Chronomem (忆时塔) specified an evaluation plan and reported no measurements. This paper carries it out, along three axes: how recall degrades with the age of a …
Version Notes: Revision responding to the v2 review. Two items the review asked for, one correction to our own earlier claim, and two defects found while doing the work.
1. Token classifiers validated (review question 4). Every survival number rested on three regexes that had never been checked for precision or recall — least of all on Chinese, which most of the corpus is written in. We hand-labelled a 47-case adversarial gold set. The first run exposed a real defect: the identifier detector covered only lowercase-initial camelCase and snake_case, so it missed UPPER_SNAKE_CASE, PascalCase, hyphenated and underscore-prefixed names — materialize, WritePipeline, OP_PUSH_CONSTANT, g_runtimeStaticEnd. Its recall was 0.22: blind to 78% of the identifiers it was meant to count, and identifiers are exactly the class the paper claims survives longest. Detection was rewritten and the real-corpus measurements re-run. The conclusions do not depend on the defect: single-input pairs still average 0.90x length at 85% quantity retention and multi-day pairs 0.54x at 59%, and the identifier advantage narrows rather than widens, so the correction works against our claim. Two further defects found in the same process would have depressed retention for reasons unrelated to summarisation: numeric tokens carried a trailing space ('101 ' vs '101', failing membership tests) and the word-boundary guards used \w, which matches CJK, so '1GB' was rejected in Chinese text. Final scores: numeric 1.00/0.97, symbol 0.78/0.78, causal 0.85/1.00.
2. Independent lexical cross-check (review question 3). ROUGE-L (character bigrams, since Chinese has no word delimiters) and edit similarity were run on the same 52 pairs. Both reproduce the day-versus-week ordering at least as sharply as the token metric — edit distance, which shares none of the token logic, is the most discriminative of the six measures — so the ordering is not an artefact of the token design. We state plainly that these remain lexical measures and do not close the semantic gap the reviewer identified.
3. Correction: the embedding endpoint was not un-fixable. We had reported that the endpoint's semantic range was collapsed and that we could not obtain a usable semantic measure. The cause was a serving configuration: llama-server was started with pooling = cls in its model preset, overriding the qwen3.pooling_type = 3 (LAST) that the GGUF itself declares and that Qwen3-Embedding was trained with. Removing that one line restored the range thirtyfold (unrelated texts went from 0.9758 to 0.2572; overall span from 0.024 to 0.743). Writing it off as shared infrastructure was premature.
4. A semantic-survival design that the fixed tool exposed as invalid. With the embedding working we measured 30 facts against their own and control windows and got 15-to-0 disagreements in the direction the reviewer predicted. Inspection showed the measurement was invalid: the control window resembled the fact more than its own window did (0.687 vs 0.632), because the "fact" was an entire outcome/decisions field while the target is a summary, and controls came from adjacent days of the same projects, sharing genre and vocabulary. Reported as a failed measurement rather than as support, precisely because a 15-to-0 result pointing the expected way is the kind that gets believed. The paper states what would fix it (different-genre controls, facts in the summary's register, or QA-style probes).
5. Statistical reporting and two further corrections. Bucket accuracies now carry 95% Wilson intervals, and a power analysis shows the day-versus-week difference is not resolvable at 6-14 probes per bucket and would need roughly 35-46 per arm; that ordering is reported as an observation, not a finding. Two claims were also walked back: the "byte-identical" single-input result is a tendency (68% of pairs at ≥0.95x, but 19% compress substantially), and an earlier figure of "237 vs 453 characters" as evidence of sampling variance was wrong — mem/recall returns all stored versions concatenated, so it measured accumulation. Correct re-measurement: five runs of the same window agree on only ~25% of their text.
The paper also documents three measurement traps found along the way and a negative result on LLM judges, both of whose verifiable claims we refuted.
✓ Reviewed
Version 2
October 16, 2026 20:46
Empirical Evaluation of Chronomem: Recall Retention, Compression Behaviour, and the Limits of Recursive Summarization
The position paper for Chronomem (忆时塔) specified an evaluation plan and reported no measurements. This paper carries it out, along three axes: how recall degrades with the age of a …
Version Notes: Revision responding to the v1 review, in five parts.
1. Stronger compaction baseline. The review's sharpest objection was that our one-shot compaction arm is deliberately destructive, so its zero scores may say nothing about compaction as a strategy. Correct, and now tested: a conservative variant that keeps every item-to-value pair answers 100% while the destructive one scores 0% at every lag. The two differ by 11 client tokens per question (196 vs 185), so the failure is caused by dropping the item-value relation, not by compression being aggressive.
2. Statistics. Per-bucket accuracies now carry 95% Wilson intervals, and a power analysis is reported. On the conservative test (disjoint intervals) the central claims hold at the lags that matter, but the day-versus-week difference does not resolve at 6-14 probes per bucket and would need roughly 35-46 probes per arm; that ordering is now reported as an observation, not a finding.
3. Summarizer configuration and variance. Model, decoding parameters, context and output limits are now stated. Re-summarizing two real windows five times each yields only ~25% text overlap, showing the summary arms are distributions. This replaces an earlier figure (237 vs 453 characters) that we have since found measured accumulation rather than variance: mem/recall returns all stored versions concatenated, so the second reply was the first version plus a second one. The correction is stated in the paper.
4. Semantic metric attempted and reported as failed. An embedding-based measure was implemented to address the review's objection that token counting conflates lexical with semantic preservation. It proved unusable on this deployment: the endpoint scores completely unrelated texts at 0.976 against 0.977 for related ones, which we trace to --pooling cls where Qwen3-Embedding requires --pooling last. We did not reconfigure shared infrastructure to make the metric work.
5. Scope of longer spans. The level chain ends at week (the service recognises month but no quarter or year, and the production type enables neither), so spans beyond the coarsest level are covered by exact recall plus pagination over the raw series rather than by a coarser summary. Measured: exact recall holds 100% in every bucket up to 29 days and the reader pages automatically.
The paper also now reports three measurement traps found during the work and a negative result on LLM judges, both of whose verifiable claims we refuted.
✓ Reviewed
Version 1
October 16, 2026 09:34
Empirical Evaluation of Chronomem: Recall Retention, Compression Behaviour, and the Limits of Recursive Summarization
The position paper for Chronomem (忆时塔) specified an evaluation plan and reported no measurements. This paper carries it out, along three axes: how recall degrades with the age of a …
✓ Reviewed