Adventures of Lolo Agent
Twenty-one runs to one heart.
An agent that gets pixels, a controller, and save states. No rules, no object names, no solutions, no demonstrations. This page is one case study from the larger project: two days of experiments on Room 3, where the agent learned how to make a heart reachable and then repeatedly failed to collect it.
The honest ledger
The best run collected three of five hearts and never opened the chest. Six runs reached the target heart, but only one intervention produced the intended final step, and a simpler standing rule reproduced it.
The one passing result has since been re-confirmed on the fixed code path, in a preregistered pair of runs that both completed: the treatment collected the heart at decision 17 as predicted, the control never did, and neither lost a life. Everything below is the assisted track, meaning the agent uses pixel detectors for the player and hearts as development scaffolding. The strict interaction-only claim, which is the actual research goal, has no object-centric evidence yet. That gap is the single most important caveat on this page.
First, the game
Adventures of Lolo is one of those core memory games for me. I would argue it is the best puzzle game on the NES after Tetris. So this started from affection rather than from a benchmark. I wanted to know whether something could work these rooms out the way I once did, without being handed any of what I already know.
It is a 1989 NES puzzle game by HAL Laboratory, the studio that would later make Kirby. You are Lolo, a blue ball with a face, working through fifty single-screen rooms to rescue Lala from the Great Devil. Nothing scrolls. Nothing is timed. There are no reflexes involved.
Collect every heart in the room to open the treasure chest. Take the treasure, and the door out unlocks. What stands between Lolo and the hearts is a fixed arrangement of blocks, water, and monsters, plus a very small toolkit: Lolo can push certain blocks one square at a time and fire magic shots that turn some enemies into eggs. Those eggs can be pushed, used as rafts, and eventually turn back into enemies.
The monsters are part of the puzzle rather than an action challenge. Some sit inert until you cross their line of sight. Some patrol. Some shoot along a row or column the moment you step into it. Which ones are dangerous depends entirely on where everything else in the room is standing.
Why a puzzle game from 1989 is a hard target
The game is deterministic and discrete, which makes it tractable to study. The difficulty is that it is a game about preparation with irreversible consequences. Pushing a block one square too far can make a room permanently unsolvable, and the game will not tell you. It keeps letting you play a room you can no longer win.
So the thing a player has to learn is not a reaction or a route. It is which rearrangements of the room open which futures, several moves before any of it pays off, with mistakes that are silent and permanent. That is the structure this project is trying to get an agent to discover on its own, without being told that a block is pushable, that an egg is a raft, or that a heart is worth having.
The claim, and why it is hard
The agent may use four things: rendered RGB frames, controller actions and durations, opaque save/load capabilities, and the pixel-observed consequences of its own actions. It may not receive game rules, object names, level solutions, ROM memory, demonstrations, or solution videos.
That restriction is enforced by the shape of the code, not by good intentions. An object here is just a patch of pixels the agent noticed moving together and decided to keep watching. It has no name, because nothing ever told it there is such a thing as a block or a monster. It gets a serial number instead, so the seventeenth thing it started following is track-17 and nothing more. A data contract is the fixed set of fields one part of the program is allowed to hand another, rather like the blanks on a form. Those forms have no blank labelled push, enemy, or egg, so even a component that somehow worked out what an egg was would have nowhere to write it down.
The target is stronger than finishing fifty rooms: solve held-out rooms with the learned parameters frozen, then attempt the sequel with them still frozen. Frozen means the numbers inside the model stop changing. During training those numbers are adjusted after every mistake; freezing them locks the model as it stands, so it faces the new rooms with only what it already understood. This matters because a model allowed to keep learning on the test can simply memorise its way through, and you learn nothing about whether it grasped anything general. A room it has never seen, attempted without further tuning, is the honest version of the question.
The preparation structure described above is what makes this hard to reward. It is invisible to every cheap objective: frame-level prediction, distance-to-goal, and raw novelty all report progress while the agent goes nowhere. Each of those was tried here, and each is in the failure list below.
Practical goal
Beat all fifty rooms. This measures whether the system can turn its experience, search and memory into progress through the actual game.
Research goal
Learn reusable mechanics from interaction, solve withheld rooms with persistent parameters frozen, then attempt Adventures of Lolo 2 without further training.
Before Room 3
This page does not cover the whole project. Before the campaign documented below, the system had already learned through interaction to move through the title and story screens and enter gameplay. In Room 2, a strict pixel-only run collected all four hearts. A separate assisted positive-control run, using pixel detectors as development scaffolding, collected the treasure and cleared the room.
Those are different claims and should stay different. The strict run showed genuine progress under the intended input rules, but did not establish a room clear. The assisted run proved the emulator, controller, telemetry and search loop could complete the task, but it does not satisfy the final research standard. Room 3 is where the assisted development track became stuck long enough to expose the representation and planning failures described here.
Room 3: the case study
One room, seen by the agent as pixels. The map below is derived from the frames themselves, using a per-cell modal colour over 80 recorded frames, not from any supplied level data.
The six images that follow are editorial illustrations assembled from recorded frames. Their labels and annotations explain the evidence to a person; they were never shown to the agent or added to its inputs.
The campaign in five turns
- 01The agent established that changing one tracked object altered which parts of the room it could reach. Removing it expanded certified access from 7 cells to 24; a later audit showed 67 cells were actually reachable.
- 02Five planning interventions tried to preserve and exploit that prepared configuration. Each failed for a different measurable reason, including mechanisms that never fired and a restore that spent its win standing still.
- 03Twenty-one completed runs from the same checkpoint reached toward one target heart without ever collecting it. That is the origin of this page's subtitle, not a count of all project runs.
- 04One intended intervention finally produced the last step onto the heart. A simpler standing rule reproduced it, and a preregistered treatment/control pair re-confirmed the result after a search-cache bug was fixed.
- 05The room still was not solved. The best run collected 3 of 5 hearts; no run in this campaign opened the treasure chest.
What is in the room
Everything the agent has to work with, and the one thing it never got.
Room 3 on the 16×15 coarse grid. Hearts in signal red, the manipulable object at its home cell in ochre, the chest in blue, and the one genuinely static obstruction.
Three sets of cells
The whole campaign turns on the difference between these.
The 7-cell and 24-cell sets are certified measurements: what a depth-12 search verifiably retained. The 67-cell set is what is actually reachable. Mistaking the first for the third cost this project two days.
Why save states change the problem
A save state is a complete snapshot of the console's memory. You can restore one and the machine continues exactly as it was, every time. That gives this setting something almost no learning environment has: cheap, exact counterfactuals. A counterfactual is the answer to "what would have happened if I had done nothing instead?" Normally that question is unanswerable, because you only get to live one history. Here you can snapshot, press a button, snapshot again from the same instant, wait the same length of time doing nothing, and compare the two futures pixel by pixel.
That second run is the matched control: same starting point, same duration, only the action differs. It is the same logic as a control group in a drug trial, except it costs milliseconds and is exact rather than statistical. Every causal claim in this project rests on one, which is why the honest failures below are failures about what mattered rather than about what happened.
Two more consequences follow. The emulator is the oracle, the thing that settles questions: a learned model may propose an action, but only running it decides. And because branching is cheap, the agent explores by snapshotting promising positions into an archive and jumping back to them later rather than replaying from the start, the same family of trick behind Go-Explore.
How an experiment was judged
Four pieces of method vocabulary appear throughout, and they are the reason the negative results here are worth anything.
- Preregistration. Before a run starts, the exact success conditions are written down and committed to the repository. Afterwards you compare the result to what you already wrote, so a failure cannot quietly become a success by redefining what you were looking for. This is borrowed from clinical and psychological research, where it exists to prevent exactly that (background).
- Control arm and treatment arm. Two runs identical in every setting except the one thing being tested. If the control also does the thing, the mechanism gets no credit for it.
- Ablation. Remove a component and re-run. If nothing changes, that component was not doing the work, whatever the story said. One mechanism here was removed on exactly this basis after its own counterfactual instrument showed it had never changed a single decision.
- Discriminator. The specific observable that separates "the agent can do this" from "the agent got lucky". Here it was one heart that twenty-one runs had never collected. A discriminator has to be checked for health: at one point the control was quietly closing in on it, which would have made a win meaningless.
How this was actually run
All of it on one laptop. An Apple Silicon MacBook Pro, no cloud, no cluster, nothing rented.
The game runs in a headless emulator, meaning an emulator with no window and no speaker: it advances the console's state and hands back the raw picture as an array of pixels, as fast as the machine will go. The emulator core is Nestopia, driven through libretro, the same emulator plumbing RetroArch uses. It runs inside a small separate host process, which matters for the research rules rather than for speed: the ROM contents and the save-state bytes stay inside that process, and the agent's side of the boundary only ever receives RGB frames and opaque handles it cannot decode. The constraint is enforced by the architecture rather than by good intentions.
Speed is what makes the method possible. The host verifies roughly 586 branches per second and emulates around 9,400 frames per second, so a search that tries 25,000 different action sequences from one position costs about 43 seconds. A full experiment, 24 decisions with a wide search at each one, takes 30 to 60 minutes and produces about 85,000 telemetry events and 12,000 verified branches.
Every run writes a self-contained directory: an append-only event log, deduplicated frames, a manifest recording the digest of every input (ROM, emulator core, host binary, model checkpoints), and a decision-by-decision table. That manifest is why claims here can be checked rather than believed. It is also how the runs are known to be deterministic: give the system identical inputs and it produces a byte-for-byte identical run. Six separate control runs across four experiments produced the same 85,594 events and the same 24 decisions, which is what makes any difference in a treatment run attributable to the treatment.
One negative result about infrastructure, since it saved money: renting a cloud GPU was benchmarked and rejected. The bottleneck is a sequential, CPU-bound emulator, and the GPU did not accelerate it. A real-data training benchmark came back at 0.196 times local speed. The laptop won.
Replay the decision traces
The player contains 27 late-campaign Room 3 runs, versions 322 through 348, replayed as committed trajectories through the room. Playback advances through all 27 automatically; pause it to choose a particular run or inspect a decision. The markers are moments the telemetry flagged: an archive restore, a deposit, or a heart collection.
What the telemetry stores is the cell the agent occupied at each committed decision, not every step in between. The connecting line is therefore reconstructed: a shortest walkable route between consecutive recorded positions, so it stays on floor the agent could actually stand on. Trust the marked positions, and read the line between them as plausible rather than observed.
What did not work
Here are the same 27 selected late-campaign runs at once. Each panel is one run, and the line is how far the agent was from a particular heart as the run went on. These are not every run conducted during the project, or even every Room 3 run.
The interesting part is not that things failed but that each failed for a different, nameable reason. Five separate mechanisms were built to make the planner prefer a prepared configuration.
Objectives that resembled progress
Raw novelty and screen change. Useful for exploration, unreliable as progress. The agent spent most of its budget in title animation while its coverage metrics improved.
Straight-line distance to a visible goal. Accelerated easy routes, then stalled at every obstacle requiring preparation. Later hardened into a general rule: any consistent preference for target proximity fails, not merely distance rewards, because the excursions away from a target are what deposit the archive ladder that later progress climbs. Steering toward the goal destroyed reaching it.
Temporal precedence as causal credit. A button press was credited for a scene change 51 decisions later that a timer, not the agent, caused. Every causal claim since requires a duration-matched control from the same root.
Bigger search instead of better representation. One run rejected 1,756 changed-layout branches and accepted zero as reliable manipulation evidence, through 3,202 verified branches. The bottleneck was detection, not coverage.
Gates that could not fail informatively
A promotion gate asked whether a learned mask reproduced the incumbent's masks byte-for-byte. It passed on bits that turned out to be mask-irrelevant, then failed on a criterion no learned mask could satisfy, the incumbent's own defects included. Replacing it with a functional gate, refereed by detector-free ground truth rather than by the incumbent, revealed that the incumbent itself misses 6–12% of real manipulations.
Four times the project proved itself wrong
These are the entries worth reading, because each was a conclusion already written down, already acted on, and then falsified by the project's own measurement.
The ladder that was never entered
After an experiment failed, the record said the failure had moved inside the planner's commit ladder, which had no tier referencing a target cell. Designing the fix falsified it: at the decisive decisions the planner returned early and never built a candidate set at all: branches_examined: 0. The ladder also already had such a tier. A new tier would have had zero opportunities across ten runs.
The closing mechanism won its contest and spent the win standing still.
The restore that was supposed to hold a valuable position selected the state the agent was already standing in, consumed the decision, and then removed that position from the archive.
A budget mistaken for a wall
The roadmap said the two remaining hearts lay outside the reachable region and therefore required a second manipulation. Three independent methods then put both hearts inside a 67-cell region reachable from the collected heart. An older run already held an emulator-verified 24-action branch that collects one of them with zero life loss. And the failing run's own search had climbed to five steps from the other heart at branch depth exactly 12, the configured ceiling, before a restore pulled it back.
Outside the certified envelope, therefore unreachable. A certified record is a lower bound
produced by one budget from one root. Depth 12 had been silently load-bearing across the entire campaign.
A correctness bug under the whole methodology
A six-event discrepancy between two runs was traced rather than dismissed as noise. The option search memoised matched no-op controls on (id(parent), duration), which is a memory address. Entry lifetime outlives key lifetime, so a reused address serves another node's control. It was reachable and firing, roughly one parent per search at depth four and beyond, and the corrupted value reaches the beam score. A wall collision was observed being scored as a measured local effect.
Impact is provably empty below depth four and unbounded at or above it, which is where every experiment on this page ran.
A healthy run called dead
A monitoring check reported a run had died. Both of its signals were broken: the process pattern could not match the process, and the event count it called stalled was exactly on profile. The general rule that came out of it, that two readings sharing an author are one signal and not two, later caught the mirror-image error, a monitor matching its own search pattern and so calling a finished run alive.
What shows promise
Accessibility as a measured quantity
The strongest positive result. Removing an object from the room was shown, under a certified configuration-hold predicate, to change reachable space from 7 cells to 24, including a cell bearing a heart. The comparison is honest in both directions: an earlier manipulation the project had celebrated and preserved for four generations was measured as strategically neutral. Value is a property of resulting configurations, not of manipulations.
Perception without detectors
A learned controllable-region tracker trained purely on counterfactual structure, with no player detector anywhere in the label path, now exceeds the assisted incumbent on stability, preservation of adjacent objects, and detection of in-place changes. It took six gate iterations, each isolating one mechanism, and it is still only promoted to shadow: it has never run in the planner.
The measurement discipline itself
This is the part that actually compounds. Preregistering the scoring rule before the run caught a discriminator that was quietly dying, a unit error that would have "fixed" a non-problem, a false-positive bound falsified before it cost a run, a lever credited with an effect it never had, and the correctness bug above. Most of the failures on this page were caught by measurement, not by luck.
And an uncomfortable one
Plain search at depth 24–36 did things the entire planning apparatus could not do at depth 12.
Where this goes
The immediate queue is short and ordered by what could invalidate what. The first item is now done: the one passing result has been re-confirmed on the fixed code path, so it no longer rests on a corrupted scoring input. Next is to re-measure reachability at greater depth against a depth-12 control. That is one cheap run that could retire a large amount of planned work.
Then a strategic question that this campaign forced into the open. The roadmap optimises a research claim: strict inputs, frozen parameters, transfer to unseen rooms and a sequel. The practical goal is beating the game. Those diverged, and nobody noticed for two days, because every experiment was scored against a gate rather than against room progress. Under the research claim the entire campaign scores zero, because all of it is assisted-track. Under the practical goal, the evidence now points at search budget rather than at planning.
There is a second-order irony worth recording. The rule "never raise depth after a failure" was earned honestly: it was written after searches got wider instead of smarter. Over-generalised past its precondition, it is exactly what stopped anyone re-measuring the budget once the representation finally existed.
Was Gate 4 measuring a planning capability, or measuring depth 12?
Questions I would like other people to think about
If you have worked on object-centric learning, model-based planning, or puzzle-solving agents, these are the questions where another approach would be most useful.
- How should an agent discover persistent objects and transformations from pixels? A skull, an egg and an empty cell may be three appearances in one causal chain. The model needs to recognize that chain when it encounters the same mechanic again, without being given the labels.
- How should it represent irreversible states? A pushed block can create a safe path or silently make a room impossible. The consequence may not become visible until much later.
- How do you tell preparation from luck, without a rule book? The behaviour worth rewarding here is doing something now whose only payoff arrives several moves later. Every cheap objective tried, prediction error, distance to the goal, raw novelty, scored the preparation the same as an accident that happened to land in the same place.
- When a lever changes nothing, how do you find out quickly? Five separate mechanisms were built here and each failed for its own reason, but the expensive part was never the building, it was the days between shipping one and learning it had never fired. Ship every lever with an instrument that records what the system would have done without it.
- How much of what looks like a reasoning failure is a budget? Twice on this page a conclusion about the agent's competence turned out to be a statement about how deep it was allowed to search. The uncomfortable version: if raising a limit dissolves your result, was there a result.
- What is the honest unit of progress? Scoring against internal gates rather than against the game let the project look busy for two days while the room stayed unsolved. Whatever your equivalent of the room is, it is worth checking that your metric still moves when it does.
- Can a hand-built detector ever be removed later? Every result on this page uses pixel detectors as scaffolding, added to make progress and intended to come out. They have not come out. I would like to know whether anyone has managed that transition cleanly, or whether scaffolding of this kind is load-bearing the moment you lean on it.
This experiment has been relatively costly, expending all my max-plan tokens on multiple services. There is a real opportunity cost that trades off against exploring other interests. There is a path forward here, and I may revisit it at a later time after thinking more about the research direction.
Terms, and where to read more
A short glossary of everything technical on this page, in the order it tends to matter.
| Term | What it means here |
|---|---|
| headless emulator | An emulator with no window or sound, advancing the console and returning raw pixels as fast as the CPU allows. |
| save state | A snapshot of the console's entire memory. Restoring one resumes play exactly, which is what makes exact counterfactuals possible. |
| branch | Restore a save state, take an action, see the result. The unit of work in every experiment here. |
| matched control | A second branch from the same instant that does nothing for the same duration. The comparison that turns "this happened" into "this action caused it". |
| archive | A store of promising positions the agent can jump back to instead of replaying from the beginning. |
| beam / depth | How many candidate action sequences are kept at each step, and how many steps ahead the search looks. Depth 12 turned out to be silently load-bearing. |
| telemetry | The event log a run writes. Everything asserted here was read back out of it rather than remembered. |
| determinism | Identical inputs produce a byte-identical run, so any difference between two runs is attributable rather than noise. |
| ground truth | An independent answer used to score a method, here derived from counterfactual structure rather than from the thing being tested. |
| falsification | Stating in advance the result that would prove you wrong, then looking for it. Four conclusions on this page died this way. |
Sources worth your time
- toddsherman/lolo-agent. The project itself: the agent, the plan of record with every amendment, and docs/learnings.md, the negative-results log that every numbered reference on this page points into. Read the reproducibility note in the README first.
- Adventures of Lolo. The game itself: HAL Laboratory, 1989, fifty rooms across ten floors.
- libretro / RetroArch documentation. The emulator interface used to drive the game headlessly.
- NESdev Wiki. The reference for how the console actually works, if you want to understand what a frame or a save state contains.
- Go-Explore (Ecoffet, Huizinga, Lehman, Stanley, Clune). The archive-and-return idea that makes save-state exploration powerful, and the closest published relative of the search used here.
- World Models (Ha, Schmidhuber). The learned predictive model this project deliberately does not trust as an oracle, and useful for seeing the alternative design.
- Preregistration (Center for Open Science). Why writing the success condition down first is the load-bearing habit on this page.
The code
Python · PyTorch · libretro / Nestopia · pixel-only agent boundary
The repository contains the agent, the headless emulator boundary, the learned world model, save-state branching and planning code, telemetry and replay tools, experiment configurations, the roadmap, and the running log of what did not work. The commercial game ROM is not included.