Mon February 03, 2025

A new paper released on Friday is making waves in the AI community, not because of the model
it describes, but because it shows how close we are to some very large breakthroughs in AI. The model
is just below state of the art, but it can run on my laptop. More important, it sheds light on how all
this stuff works, and it’s not complicated.

OpenAI were the first to claim the inference-time scaling laws. Basically, an LLM can get higher performance
if it can “think” longer before answering. But, like, how do you do it? How do you make it think longer?

OpenAI and R1 had cool graphs showing performance scaling with average thinking time (this from
the s1 paper):

But how do they control the length of an LLM response? Everyone skipped over that part, but s1
shows us details, and it is fun.

Context: When an LLM “thinks” at inference time, it puts it’s thoughts inside and
XML tags. Once it gets past the end tag the model is taught to change voice into a confident
and authoritative tone for the final answer.

In s1, when the LLM tries to stop thinking with "", they force it to keep going
by replacing it with "Wait". It’ll then begin to second guess and double check it’s answer. They
do this to trim or extend thinking time (trimming is just abruptly inserting "").

It’s really dumb, I love it. It feels like the kind of hack I would try.

So for o3-mini-low versus o3-mini-high, that’s likely how they do it. They probably trained 3
models, and with each with a different average thinking time (as measured during training). Eventually the
training process begins to encode that behavior into the model weights.

The Entropix Tie In

The trick is so dumb you can do it at inference time too. I’m kicking myself for not understanding
this earlier, because it’s what entropix is all about, and I wrote a lot about entropix.

In entropix, they look at the entropy & varentropy of the logits (and attention) to change how the
tokens are selected. In fact, they used tokens like “Wait” to f