A conducted daydreaming instrument. A human performer steers a character's inner life while a cognitive engine maintains persistent concerns, episodic memory, and involuntary daydreaming between interactions. It is a prosthetic inner life for language models. The stage makes that visible through narration, reactive music, and eventually real-time video.

What Inner Life Looks Like

Four moments from one continuous 12-cycle run, captured at the first runtime-thought milestone when the kernel's inner life first became visible. Each beat is generated by an LLM from the kernel's structural state: which concern is active, which operator won, what was retrieved from memory, what emotional pressure is driving the cycle.

Here are four moments from one continuous 12-cycle run:

Cycle 1. Rationalization. The kernel selected this operator because negative emotion linked to a failed goal crossed the activation threshold.

Cycle 3. Reversal. The operator changed because emotional pressure shifted toward a different failed goal. Reversal imagines a counterfactual past and the prose follows.

Cycle 9. Roving. Accumulated residue shifted retrieval pressure, and the system drifted toward a pleasant association.

Cycle 12. Eleven cycles of rationalization, reversal, and roving have reshaped the memory store. The prose here transforms from cycle 1 because what gets retrieved is different.

The prose was not scripted. The kernel selected operators, retrieved episodes, and stored residue inside an authored benchmark world. The LLM inhabited the state it was given.

How One Cycle Works

One tick of inner life. Five operators compete under emotional pressure. The amber arrow is the feedback loop.

The kernel runs a bounded cycle. It decays emotional and motivational state, selects the concern with the strongest current pull, runs a family plan for that concern, retrieves episodes by cross-referencing named indices, stores the result with admission discipline, projects the state into an inner-life thought beat, and writes the residue back into memory.

The feedback arrow is the key. Generated thought changes what the character retrieves and does next cycle. The system is path-dependent.

The Feedback Loop Is Real

Traces diverge from cycle 4 when thought residue feeds back into memory. Same kernel, same seed, same fixture.

Same kernel. Same seed. Same fixture. The only difference is whether generated thought feeds back into memory.

From cycle 4 onward, the traces diverge. Different nodes get selected. Different operators win. Different situations become active.

Haiku and Sonnet produce different downstream behavior because their residue carries different indices, which means different episodes surface on the next retrieval pass.

Same Cues, Different Reading

17 cycles of autonomous inner life between these two readings of the same place.

We ran a 20-cycle mini-world on typed Graffito material, where a character named Tony navigates three situations with persistent sensory-regulation state. The same mural cues that read as threat early in the run read as challenge later, because the kernel's own family selections in other situations shifted Tony's carried regulation state.

Six autonomous appraisal flips across 20 cycles, with adversity and recovery, not scripted progression.

The climb is not smooth. Street reversals knock Tony back. Each recovery peak is higher than the last. By cycle 16, the baseline holds.

Where It Came From

90,000 words of cognitive science condensed into 19 buildable mechanism cards, verified against Mueller's original 1990 Lisp source.

Erik Mueller's Daydreaming in Humans and Machines (1990) describes a complete cognitive architecture for mind-wandering: emotion-driven control, concern initiation, planning, retrieval, reminding, rationalization, reversal, serendipity, mutation, and episode evaluation. 90,000 words and 135 hand-coded rules, in a system that ran once per session.

Daydreamer GitHub 

We condensed the full book into 19 typed mechanism cards. Not prose summaries but implementation-grade specs: loop shape, interface boundaries, properties to preserve, and where LLM judgment can enter without breaking structural guarantees.

Then we read Mueller's original Lisp source code and found that several of his admission disciplines had been collapsed in early development: per-index cue roles, hidden episodes, the rule accessibility frontier. We reconstructed them and added the extensions Mueller never needed because his system did not persist across sessions. A memory membrane for persistent accumulation. Anti-residue flags for groove detection. Evidence-driven promotion for durable memory.

The Memory Membrane

Episodes enter provisional and must accumulate cross-family evidence to reach durable. Anti-residue flags catch grooves. Cue zones prevent metadata from driving retrieval.

When the system accumulates across sessions, it risks developing grooves: self-reinforcing loops where prior rationalizations make future rationalization easier, or prior counterfactuals become standing causal priors.

The membrane prevents this. Episodes enter as provisional and must earn durability through attributed downstream evidence, specifically cross-family usefulness rather than self-retrieval. Anti-residue flags detect and suppress loops. The evaluator can veto promotion but cannot alone grant it. Cue zones separate the content indices that drive retrieval from provenance and support metadata that do not.

This discipline is partly Mueller reconstruction, partly new. He had hidden episodes and per-index eligibility flags. We added admission tiers, evidence-driven promotion, and the accessibility frontier.

Rules as Data

Cross-family bridges derived from rule schemas, not hand-coded. The frontier is where serendipity will eventually search.

The early kernel had correct behavior locked inside procedural functions. Rationalization worked, but no other part of the system could see why it fired or how it connected to reversal.

We moved behavior into typed rules, which changes three things. The connection graph can discover structural links between mechanisms that nobody hand-coded. Episodes can carry provenance about which cognitive paths produced them. And the architecture becomes open to ideas from ACT-R, SOAR, and other cognitive systems because they can be expressed in the same substrate.

The cross-family bridges emerged from the rule schemas: rationalization's emotional afterglow feeding roving, reversal's aftershock feeding both roving and rationalization. The graph derived them structurally, and the frontier is the substrate serendipity will eventually search.

What We Built

Earlier rounds built the live performance surface: Scope for real-time video at 24fps, an APC Mini MK2 repurposed as a cognitive controller, Lyria RealTime for reactive audio, LoRA style morphing, depth conditioning.

This round built the mind that will feed that stage.

• A Clojure kernel running Mueller's control loop with five daydreaming families.
• A rule engine with typed rules, a custom matcher, and a connection graph that derives structural edges from rule schemas.
• A memory membrane with admission tiers, anti-residue flags, evidence-driven promotion, and a rule accessibility frontier.
• A typed executor boundary where family plans return declarative effects and the kernel validates and applies them.
• A runtime thought projector and a feedback loop where generated thought changes future cognition.
• 19 mechanism cards condensed from Mueller's book, verified against his original Lisp source.
• Five rounds of external architecture review absorbed into the build. 247 tests. 1311 assertions. All passing.

What Is Proven

Inner-life prose driven by persistent cognitive state, not just a prompt.

The feedback loop changes later cognition, with traces diverging from cycle 4 onward.

The memory membrane prevents self-reinforcing grooves.

Narrow membrane assays prove both same-family loop suppression and live cross-family promotion with frontier opening.

The generation pipeline produces material that survives traversal.

What Is Not Proven Yet

Full serendipity traversal (the connection graph exists but path verification does not).

Cross-session accumulation, which is the hard falsification test.

The first real LLM-backed rule, which is designed but not built.

A full staged run with narration, audio, and visuals working together.

Richer authored worlds beyond the first mechanistic miniworld.

What It Enables

A performance instrument with an inner life, where a human performer steers pressure, mood, and mode while the character's cognitive machinery keeps working underneath.

A prosthetic inner life for language models. Between interactions, the model can rehearse, rationalize, avoid, roam, and remember. You come back and the character is different because it has actually been through something computationally.

Accumulation across sessions. If the architecture works, the system gets more capable by operating. The membrane ensures that accumulation is earned, not self-reinforcing.

The Bet

This is a foundation, not a finished instrument. The architecture is designed, partially built, and benchmarked. The dynamics work in miniature. 

Persistent typed cognitive structure, owned by an engine and augmented by bounded LLM judgment, will accumulate in ways no prompt-only system can match.

Next round connects the mind to the stage.

Previous Work

Round 2: Latent Stage built the live performance instrument: 24fps autoregressive synthesis on KREA Real-Time 14B + VACE, driven from an APC Mini MK2 with 42 prompt palettes and 9 latent-space faders trained via k-sparse autoencoder, synchronized Lyria RealTime audio, and tensor parallelism pushed from 16 FPS to 27 FPS after a funcol collective fix. See the Round 2 page 

Round 1: Video Conductor built the engine: WebRTC streaming, prompt transitions (soft cut, hard cut, embedding interpolation), LoRA blending, depth puppeteering with a phone as miniature set, record/replay/re-render, all at 23 FPS on a B300. See the Round 1 page 

GitHub

https://github.com/davidrd123/latent-dreamer