The Jacobian lens (Gurnee et al. 2026) reads a hidden activation \(h_\ell\) at layer \(\ell\) by transporting it into the final-layer basis with the corpus-averaged Jacobian \(J_\ell\) and decoding with the model’s own unembedding: \(\mathrm{lens}_\ell(h) = \mathrm{unembed}(J_\ell\, h)\). The tokens it surfaces are what the model is verbalizably holding at that depth, often intermediate concepts that are neither the input nor the next token.
The logit lens toggle skips the transport and applies \(\mathrm{unembed}(h)\) directly. That is the classic baseline, and comparing the two is the point: the J-lens recovers interpretable intermediates many layers before the logit lens does.
Row shading marks the three regions of the network. The early sensory layers are greyed out; their readouts are noisy and largely uninterpretable. The middle workspace band (layers 12–28 on this model) is where the interpretable action is. In the final motor layers the readout collapses onto the actual next-token prediction. The top row is the model’s real output distribution.
Pinning a token colors every (layer × position) cell by that token’s rank in the readout (log scale: 1 · 10 · 100 · 1k+) and draws its rank trajectory in the charts, so you can watch a concept surface in the workspace and dissolve into the motor layers.
Chat mode wraps your text in Qwen’s chat template (thinking disabled, per the paper’s verbal-report protocol), with an optional system prompt and assistant prefill. Generation lets the model continue greedily and extends the lens over its own output tokens, so you can watch the workspace while the model writes. The J-Space panel aggregates the whole grid: every token in any top-10 readout, ranked by how often it appears, with its per-layer distribution.
Interventions (paper §2.5) edit the residual stream through the lens itself. Swap exchanges two tokens’ lens coordinates across the workspace band (a reflection across their bisector); the model’s “thought” is replaced and its output often follows. Steer injects a concept by adding a token’s lens direction at every position, and the model becomes preoccupied with it. Both show the default and modified generations side by side. Effects at 4B scale are real but weaker than the paper’s frontier-model numbers, and that contrast is itself the finding.