Post
19
During World War II, the British psychologist Norman Mackworth studied how long radar operators could stay alert: the answer would decide how long a shift should be. In his experiments, after just half an hour on watch, people missed signals noticeably more often, even though they kept watching the screen as before.
A language model has a watch of its own. The longer the text it works with, and the further it gets into its response, the more often it misses one rule or another. And it never lets on: toward the end of a long response it sounds just as confident as in its first lines.
How long does a model's watch last? The context window won't tell you. It says how much text fits into the model, not how much the model can handle.
The model passport built by the IEC instrument has a measure for this: fatigue capacity. It is the length of a text at whose last word the model gets a replacement right only half the time, even when there is just one rule.
I measured it for seven models, and the spread is huge. Nemotron 3.5 Lightning tires after just two hundred words, while Nemotron 3 Ultra holds out for about forty thousand. The models also tire in different ways. Gemini 3.1 Flash Lite doesn't care how long the input is and gives way only as its response goes on, whereas Ultra ignores length for a long time and then gives way sharply on texts longer than eight thousand words.
The new article has the plots, a table of fatigue capacities, and the laws they are derived from. The passports of all seven models are open.
Article: https://huggingface.co/blog/kutomanovhennadii/how-a-model-tires-toward-the-end-of-its-response
Passports: kutomanovhennadii/iec-passports-6ae3ad9fe2c798814eb8a3dc
Originally written in Russian. English translation assisted by AI.
A language model has a watch of its own. The longer the text it works with, and the further it gets into its response, the more often it misses one rule or another. And it never lets on: toward the end of a long response it sounds just as confident as in its first lines.
How long does a model's watch last? The context window won't tell you. It says how much text fits into the model, not how much the model can handle.
The model passport built by the IEC instrument has a measure for this: fatigue capacity. It is the length of a text at whose last word the model gets a replacement right only half the time, even when there is just one rule.
I measured it for seven models, and the spread is huge. Nemotron 3.5 Lightning tires after just two hundred words, while Nemotron 3 Ultra holds out for about forty thousand. The models also tire in different ways. Gemini 3.1 Flash Lite doesn't care how long the input is and gives way only as its response goes on, whereas Ultra ignores length for a long time and then gives way sharply on texts longer than eight thousand words.
The new article has the plots, a table of fatigue capacities, and the laws they are derived from. The passports of all seven models are open.
Article: https://huggingface.co/blog/kutomanovhennadii/how-a-model-tires-toward-the-end-of-its-response
Passports: kutomanovhennadii/iec-passports-6ae3ad9fe2c798814eb8a3dc
Originally written in Russian. English translation assisted by AI.