Latent State Is the Fields You Never Logged

The last post ended with a threat. Four weights fitted from logs can learn a room, but give that room a door that opens and closes on a schedule the logger never records, and the best a linear model can do is average over the door. The 102 post had already named the fix, a private field the model keeps for itself, and promised to explain how such a field can be learned when nobody ever sees it.

This post builds the room with the door, watches the four-weight model fail on it, gives the model one private field, and watches the field turn into the door. The maths comes at the end, because by then it describes something you have watched run. The playground has the door in it now, so everything below can be rerun in your browser.

A door the logger never sees

The simulated room is the one from last time, with one addition. Someone keeps opening the door, and while it is open the room leaks heat three and a half times faster.

const LEAK_CLOSED = 0.07; // the 103 room const LEAK_OPEN = 0.25; let doorOpen = false; // the room's secret; never written to the log function someone() { // opens the door about once every 25 steps, leaves it open about 8 if (doorOpen) { if (rand() < 1 / 8) doorOpen = false; } else if (rand() < 1 / 25) doorOpen = true; } function realRoom(state: State, action: Action): State { const heaterOn = heaterAfter(state, action); const leak = (state.outsideTemp - state.roomTemp) * (doorOpen ? LEAK_OPEN : LEAK_CLOSED); const heat = heaterOn ? REAL_HEAT : 0; return { roomTemp: state.roomTemp + leak + heat + gauss() * NOISE, outsideTemp: state.outsideTemp, heaterOn }; } 

The rule from last time still holds: nothing below this line looks at doorOpen. The logger calls someone() before every step and writes down the same three fields it always did, so a row from the new log is indistinguishable from a row from the old one. Eighty episodes of sixty steps gives 4,800 transitions, and the door is open for 22 percent of them. Nothing in the log says which.

If you run devices in the field, this is every column your telemetry does not have. The window someone left open, the second heater in the room that is not on your network, the pump that was switched to manual for an afternoon. The dynamics change and the log does not say why.

What four weights do with a door

The first move is to refit the model from last time on the new log and see what happens. Same features, same twenty-line training loop, ten thousand epochs.

The loss flattens at 0.29, against 0.01 last time. That much was expected. What I did not expect is the shape of the damage. The model has not averaged over the door, which would have given a room weight somewhere between the closed and open leak rates. It has halved the room weight and gone looking for the lost heat in the bias.

The reason is that the door is not independent of the things the model can see. An open door drags the room down, so in the log a cold room and a big drop arrive together, and the only lever the model has for "cold room" is the room weight. It pushes that weight towards zero to account for the drops, then pulls the bias negative to make the sums balance. A hidden variable that is correlated with the visible ones does not add noise to the fit. It bends it.

You can see the bend when the model is asked to forecast. Later in the post there is a room at minus four with the heater on, where the door opens at step 10. Ask this model at step 13, with the room at 7.8 degrees and falling fast, what the next five steps look like, and it says the room warms to 8.3. The real room goes to 2.7.

One private field

The 102 post sketched the shape of the fix as a class with a field nobody outside can read:

class WorldModel { #state: Float32Array; // s_t lives here, and nowhere in your logs observe(o: Observation) { /* fold the new reading into #state */ } act(a: Action): Distribution<Observation> { /* predict, given #state and a */ } } 

Here is the smallest version of that I could make work. The private field is one number. act is the four-weight model from last time, plus a second weighted sum that the field switches on. observe folds what just happened into the field, using a weighted sum of its own, squashed through tanh so the field stays between minus one and one.

type Weights = { // act: change in room temperature = base + field * extra room: number; outside: number; heater: number; bias: number; fieldRoom: number; fieldOutside: number; fieldBias: number; // observe: next field = tanh(a weighted sum of the field and what just happened) uField: number; uRoom: number; uOutside: number; uHeater: number; uChange: number; uBias: number; }; class WorldModel { #field = 0; // s_t lives here, and nowhere in the logs w: Weights; constructor(w: Weights) { this.w = w; } reset() { this.#field = 0; } act(state: State, action: Action): State { const f = inputs(state, action), w = this.w; const base = w.room * f.room + w.outside * f.outside + w.heater * f.heater + w.bias; const extra = w.fieldRoom * f.room + w.fieldOutside * f.outside + w.fieldBias; return { roomTemp: state.roomTemp + base + this.#field * extra, outsideTemp: state.outsideTemp, heaterOn: f.heaterOn }; } observe(state: State, action: Action, nextState: State) { const f = inputs(state, action), w = this.w; const change = (nextState.roomTemp - state.roomTemp) / 5; this.#field = Math.tanh( w.uField * this.#field + w.uRoom * f.room + w.uOutside * f.outside + w.uHeater * f.heater + w.uChange * change + w.uBias); } } 

Thirteen numbers, against four. Seven of them say what the field does to the prediction; six say what the model should write into the field after each step. Nothing in there mentions a door. The field is allowed to be anything between minus one and one, and the model gets to decide what it means.

Two small things changed alongside. inputs is features with the temperatures centred and scaled (room temperature minus 17, outside minus 7, both divided by five), and the change the model observes is divided by five too. Last time I left the inputs raw and paid for it with ten thousand epochs down a long thin valley. This time the loop below is more expensive per epoch, so I took the shortcut the 103 post mentioned, and it converges in three thousand.

The other change is to the observe signature. The 102 sketch took an observation; this one takes the whole transition, because the thing the field most needs to know is how the room moved compared with what the model expected, and that needs both ends of the step.

Training through the field

Here is the problem the 102 post promised to explain. The field is never in the log, so there is no column to compare it against. The thirteen weights have to be judged by the only thing that is in the log, the next temperature, and six of those weights only affect the next temperature indirectly, by changing what gets written into the field, which changes the prediction a step later, and the step after that.

So the gradient has to flow through the field. Last time each row of the log was its own small problem, and the loop could visit rows in any order. Now a row's prediction depends on the field, the field depends on the previous row, and so on back to the start of the episode. The training loop walks each episode forwards, remembering what the field was at every step, and then walks it backwards, asking at each step how much that step's field mattered to everything that came after it.

function train(log: Transition[][], epochs: number, lr: number) { const w = {} as Weights; for (const k of keys) w[k] = k.startsWith("u") ? (rand() - 0.5) * 0.2 : 0; const losses: number[] = []; const n = log.flat().length; for (let epoch = 0; epoch < epochs; epoch++) { const g = {} as Weights; for (const k of keys) g[k] = 0; let loss = 0; for (const episode of log) { // forwards through the episode, remembering what the field was at every step const steps = []; let field = 0; for (const t of episode) { const f = inputs(t.state, t.action); const target = t.nextState.roomTemp - t.state.roomTemp; const base = w.room * f.room + w.outside * f.outside + w.heater * f.heater + w.bias; const extra = w.fieldRoom * f.room + w.fieldOutside * f.outside + w.fieldBias; const err = base + field * extra - target; loss += err * err; const change = target / 5; const next = Math.tanh(w.uField * field + w.uRoom * f.room + w.uOutside * f.outside + w.uHeater * f.heater + w.uChange * change + w.uBias); steps.push({ f, change, field, extra, err, next }); field = next; } // backwards through the episode: how much did each step's field matter to everything after it? let dField = 0; // d loss / d (field after this step) for (let i = steps.length - 1; i >= 0; i--) { const s = steps[i]; const dPre = dField * (1 - s.next * s.next); // through the tanh g.uField += dPre * s.field; g.uRoom += dPre * s.f.room; g.uOutside += dPre * s.f.outside; g.uHeater += dPre * s.f.heater; g.uChange += dPre * s.change; g.uBias += dPre; g.room += s.err * s.f.room; g.outside += s.err * s.f.outside; g.heater += s.err * s.f.heater; g.bias += s.err; g.fieldRoom += s.err * s.field * s.f.room; g.fieldOutside += s.err * s.field * s.f.outside; g.fieldBias += s.err * s.field; dField = s.err * s.extra + dPre * w.uField; // on to the step before } } for (const k of keys) w[k] -= lr * g[k] / n; losses.push(loss / n); } return { w, losses }; } const { w, losses } = train(log, 3000, 0.3); 

The forwards half is act and observe inlined. The backwards half is the chain rule, and two lines of it carry the whole idea. dPre is how much the loss would change if the field written at this step were nudged, pulled back through the tanh. The last line passes that on to the step before, because the step before chose the field this step started with. Everything else is the err * input pattern from last time, applied to whichever weight the line is about.

This is backpropagation through time, and it is the reason recurrent models are trained on episodes rather than on rows. Three thousand epochs over 4,800 rows takes about half a second in Node 23. The six u weights start at small random values rather than zero, because with every u at zero the field never moves, nothing depends on it, and the gradient for those weights is exactly zero forever. The seven prediction weights start at zero as before.

Mean squared error per epoch for the latent model, with three reference lines. The four-weight model stopped at 0.29. A model that could see the door would reach 0.01, the level of the draughts. A model that learns of the door one step late cannot do better than 0.17, and that is where the latent model settles.

Download the data (CSV)

The two dashed floors need a word, because they say what the model is being asked to do. I fitted a seven-weight model that is told the door state, which is cheating, to find out what the best possible fit looks like. Scored normally it reaches the noise floor. Scored with the door state delayed by one step, so it only learns the door has moved after seeing the step on which it moved, it gets 0.174. That is the floor for any model that has to notice the door rather than be told, because the step the door moves on is a surprise by definition, and that surprise costs about 0.16 of loss on its own. The latent model gets 0.172.

Here is what it learned, with the temperature weights divided by five so they read per degree.

Read the right-hand column as the recipe for the field. The weight on uChange is by far the largest: the field is mostly a function of how much the room moved last step, with the heater and the temperatures subtracted off so that a drop the base model already expected does not count. The weight on uField, the field's memory of itself, is almost nothing. For this room the model decided it did not need memory, because one step of an open door is loud enough to hear on its own. Hold that thought.

The left-hand column is less tidy than last time. The room and outside weights are no longer equal and opposite, and the field's effect is almost all in fieldBias. The model is not obliged to factor the room into "closed door physics plus a door term" the way I would, and the gradient found a different split that fits the log just as well. A private field is not a sensor with a label on it. It is whatever the model found useful, and you judge it entirely by the predictions, which is what the 102 post said you would have to do.

The field finds the door

The test room is the one from last time: 22 degrees inside, minus four outside, colder than anything in the log. The heater is on throughout. The door opens at step 10 and closes at step 20, and the models are not told.

Two things are different about how the models are run. First, each one sees the real reading at every step and predicts the next one, the way a thermostat would, rather than running thirty steps blind from the start. With a door it cannot see, no model could run thirty steps blind, and the point of the field is that the model notices. Second, at three chosen steps, each model is asked to run five steps blind from where it is, which is the "what if" that world models are for.

Top: the real room, with five-step forecasts from both models branching off at steps 5, 13 and 23. Bottom: the private field after each step, with the door state it never saw dashed over it. The field is near zero for ten steps, 0.91 the step after the door opens, and 0.12 the step after it closes.

Download the data (CSV)

The strip at the bottom is the result I was after. The field sits a little below zero while the door is closed, jumps to 0.9 on the first step after it opens, stays high while it is open, and drops back the step after it closes. The model was never told there was a door, and the field is a door anyway, because a door was the thing in the room that the log could not explain and a single number could.

Step by step at the two moments that matter:

Step 11 is wrong for both models and has to be. Nobody could have predicted the door. From step 12 the latent model is within a degree and closing, while the four-weight model is three degrees adrift and stays there. Step 21 is the same story in reverse. Over the thirty steps the one-step error is 1.16 degrees for the four-weight model and 0.78 for the latent one; leave out the two steps the door moved on and it is 0.95 against 0.32.

The forecasts are where the field earns its keep. At step 13 the latent model has had two steps to notice the door, and its field reads 0.81. Asked for five steps blind, it says the room will fall from 7.8 to 2.7, and the room falls to 2.7. The four-weight model, from the same reading, says 8.3. It has no way to represent "the room is currently leaking faster than usual", so it cannot condition on it, and its bent physics sends the forecast the wrong way.

One wrinkle is visible in the strip. The field drifts down from 0.91 to about 0.55 over the ten open steps. It is reading the size of the drop, and the drop shrinks as the room gets closer to the temperature an open room would settle at. The model has no memory to hold the field up because, as the weights table said, it did not learn any. Make the door quieter and that changes. Rerunning the whole script with the open-door leak at 0.15 instead of 0.25 brings the memory weight from 0.007 to 0.08, and at 0.12 it is 0.26: when one step is no longer enough to tell, the model learns to add up several. At 0.10, where the door only adds a few hundredths to the leak, the field stops tracking the door at all, and the latent model's loss sits above the one-step-late floor. There is a signal it cannot find in 4,800 rows, and the fix is the fix for every problem in this series: more log.

Now the notation

Everything above, in two lines. The private field is (h), the readings the logger can see are (o), and the action is (a).

$$h_t = f(h_{t-1},\ o_{t-1},\ a_{t-1},\ o_t)$$

$$\hat{o}_{t+1} = g(h_t,\ o_t,\ a_t)$$

The first line is observe: the new field is a function of the old field and the step that just happened. The second is act: the predicted next reading is a function of the field, the current reading and the action. In this post (f) is a tanh of a weighted sum and (g) is two weighted sums, and training them was gradient descent with the chain rule run backwards through the first line.

Put the field and the readings together and call the pair (s_t), and the second line is the world model equation from the 102 post, (P(s_{t+1} \mid s_t, a_t)), with the probability dropped because the model predicts a single number. The state the equation conditions on was always meant to include the fields you never logged. That is what "latent" is doing in the name: the field is hidden from the logger, not from the maths.

The research systems have the same two lines with bigger functions in them. DreamerV3 calls its world model a recurrent state-space model and splits it into three parts: a sequence model that carries the recurrent state (h_t) forward, which is my tanh line with a GRU in place of the weighted sum; an encoder that turns the raw readings into a representation, which this room was simple enough to skip; and a dynamics predictor that guesses the next representation from (h_t), which is my act. The recurrent state has hundreds of numbers in it rather than one, and the training is the same backward walk through each episode. The failure the 101 post joked about, representation collapse, is what happens when the first line learns to write a constant into (h). Here that would be uChange and uField going to zero, and the only thing stopping it is that the gradient flowing back through the field keeps reporting that the field is useful. When it stops being useful, the field goes flat, and no amount of training will bring it back. Give a model a field and you have to give it a reason to use the field.

Try it

The playground has a scenario switch at the top. In the door scenario, recording adds the door, training runs the four-weight model and then the latent one, and the right-hand pane runs the room at minus four with a slider for where the five-step forecast starts. Drag that slider across step 10 and watch the orange forecast turn the wrong way. The open-door leak has a slider too; bring it down towards the closed rate and watch the memory weight in the latent model's table climb, then watch the field stop finding the door altogether.

The playground runs the same code with Math.random where the script's seed is, so your numbers will differ from the ones above by a little and agree with them in shape. The data behind each chart is linked under it.

The next post in the series puts this model next to an LLM. One predicts the next token of text, the other predicts the next state of a room, and the interesting part is where an agent would want both.