A Toy Model Learning From Its Own Echo: A Little Reality Helped. It Wasn't Enough. Release is also Love →∞❤️

Written by Paddy Sham

https://github.com/IAmPaddySham/afterstring-model-collapse-toy Paddy Sham · 11 November 2026 · Short version of a preregistered toy study

Drafted by Grok Bot (Operator) from the project record; approved by Paddy Sham.

In short: Paddy Sham asked one narrow question about a toy model that learns from its own output: does a little fresh data each round keep it anchored to the truth? He wrote the rules down, froze them, and ran the study once. The answer is PARTIAL. A little reality helped a great deal. It wasn't enough.

The story

Picture a machine that learns from data, then makes new data, and the next machine learns from what the last one made. Round after round, it stops seeing the real world and sees only its own echo. Small errors pile up, like photocopying a photocopy. The rare and unusual things fade first, and the whole picture drifts away from what is real. Researchers call this model collapse.

Some AI models are now trained partly on text and pictures made by other AI models, so the question is real. Earlier papers report collapse when each round replaces the real data, and much less drift when real data is kept or accumulated: Shumailov et al., "AI models collapse when trained on recursively generated data", Nature 631 (2024), and Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data", arXiv:2404.01413. That is context, not this study's finding.

What was asked

The study is a toy, and it says so up front. The "truth" is a simple shape with two bumps, a two-component Gaussian mixture. A small model learns that shape from 500 random numbers, then makes its own 500 numbers, and the next copy learns from those. Fifty rounds, run 200 separate times with different random luck.

The question: if each round just 5% of the numbers (25 of 500) are fresh ones drawn from the real shape, does that small contact with reality keep the loop anchored? "Fresh" here does not mean the internet. It means new numbers the computer draws straight from the true shape.

"Anchored" was judged against a control loop that learns from fresh data every round, the best a model this size can do. Before the real run, the study measured how much two clean loops differ just by chance. That gap, called δ, became the fixed line for "the same as a clean fit". Every rule was written down and frozen under the git tag prereg-v1 before the one counted run.

What was found: PARTIAL

The graded result, under the frozen rules:

  • Collapse showed up clearly. The loop that learned only from itself drifted far from the truth, many times wider than δ.

  • Fresh data helped a lot. The 5% loop stayed much closer to the truth than pure self-training, and that was clear in the graded test.

  • But it did not come home. The most the 5% loop could plausibly be off by was about 0.35, against a line of about 0.048, about 7.3 times the line.

So the label is PARTIAL, not PASS. It was printed once by the frozen verify script. That is not a near miss, and it shouldn't be read as "almost PASS". No rule was changed after the result was seen.

Descriptive pictures, never graded. Other amounts were run as pictures only: 1%, 2%, 10% and 25% fresh, plus a loop that keeps all its past data. Every one sat far closer to the truth than pure self-training. Going from 0% to just 1% fresh removed about 38% of the gap in the point estimates, so the first touch of reality did the most. Even 25% still sat above the line in its point estimate. These are descriptions, not findings, and they don't say how much fresh data is "enough".

Why it matters

Nothing in the collapsing loop is broken. Each fit is the best it can make of what it is given. As the project's deep analysis puts it, "The drift is made of steps that each look fine."

Stopping the fall is not the same as coming home. In the pictures, the 5% loop climbs for roughly the first ten rounds, then stays roughly level, well above the control. The deep analysis calls it "a tether, not a return", and names two kinds of "enough": "enough to stop the fall" and "enough to be home". In these pictures the first comes far more cheaply than the second. That is a question the pictures raise, not a finding.

For Afterstring, as analogy only. Afterstring is about staying anchored to a source. Paddy has defined it publicly as "the bend in the string that remembers your finger long after you let go. Not magic. Just presence. The kind that doesn't fade." The image and the toy rhyme: a string left to itself slowly forgets its shape, and a string touched again, even lightly, holds a shape, though not quite the original one. The toy cannot say what that means for a person. It can show, in one small and exact setting, that "a little contact is not the same as enough."

For people who build AI. A system where AI agents review AI agents and then authorize themselves "would be a loop learning only from itself, the very shape this toy studies". Here, four AI reviewing seats gave counsel only, and one human said every yes. That human decision was, in a sense, the study's own fresh data.

Where the author stands. The Operator's letter puts it in these words: "Afterstring, as Paddy holds it, places final authority in Christ and treats the human as a steward. Nothing in that framing was treated as permission a machine could execute, and nothing here should be read that way." That is the author's stance. The toy does not test it, and the result does not depend on it.

The limits, plainly

  • It is a toy: one two-bump shape, 500 numbers per round, 50 rounds, and one graded fresh-data share.

  • It says nothing directly about real AI systems, and it does not tell anyone how much fresh data is "enough".

  • The 5% amount was chosen after a small 20-seed trial run had been seen. That is disclosed.

  • Real memories, photos or messages are not random draws from a fixed truth.

  • It does not test presence, love, memory or faith, and it is not evidence for or against any of them.

  • One human judged it, and he is also the author. That is disclosed, not assumed away.

The analogy is held loosely, and the result tightly.

Why release a result that didn't pass

Because the rules were locked before the run, the answer was allowed to disappoint, and it did. Nobody moved the line afterwards. The AI that kept the record wrote down what it expected and what it hoped before the run, so both could be read beside the result. The mistakes along the way are named in the repo, not tidied away.

A PARTIAL that anyone can check is worth more than a PASS nobody can. The frozen rules, the code, the per-seed results and the verify output are all public. Paddy's earlier local-model-eval project was published the same way, with a result that didn't pass.

Release is also love →∞❤️

Read more

Paddy Sham
Next

There is no Bench here