Does a looping brain need a looping model?
A recurrent transformer reproduces the cortex's depth→time hierarchy for language
Uri Dar, Dr. Ariel Goldstein
The Hebrew University of Jerusalem · Department of Cognitive and Brain Sciences
The Hebrew University of Jerusalem Department of Cognitive and Brain Sciences

1Introduction & Background

  • In a feed-forward LLM (GPT-2 XL), successive layers map onto successive time lags of neural activity in language cortex — model depth tracks the brain's processing time (a "depth→time" hierarchy; Goldstein et al. 2025).
  • But the cortex is not feed-forward: it computes by recurrence, reusing the same circuits over time.
  • Looped/recurrent transformers (e.g. Ouro-2.6B) reuse the same weights across several passes — architecturally closer to cortical recurrence than a deep feed-forward stack.
RESEARCH QUESTION Does a looped transformer reproduce the depth→time hierarchy along its recurrent depth — and does the hierarchy come from the recurrence rather than from raw network depth?

2Methods

  • Data: public "Podcast" ECoG dataset (Zada et al. 2025; OpenNeuro ds005574) — 9 epilepsy patients, intracranial high-gamma (70–200 Hz), listening to a ~30-min spoken story (~5,000 words).
  • Regions (Destrieux atlas): higher-order language cortex — IFG, aSTG; auditory control — mSTG (posterior STG; not a perfectly clean control — holds some semantically-responsive electrodes).
  • Models: GPT-2 XL (49 layers, feed-forward) vs Ouro-2.6B (looped; 4 recurrent passes → "effective recurrent depth").
  • Encoding: per electrode × model representation → predictive word-alignment → PCA-50 → ridge regression → encoding correlation r across a 25 ms lag grid. Peak encoding lag = grid lag where r is maximal; plot peak lag vs depth, quantify with Pearson r.
HypothesisLanguage-ROI peak lag rises with model depth in both models, and in Ouro it tracks effective recurrent depth.

3Results

GPT-2 XL: peak encoding lag rises with layer in IFG and aSTG; flat in mSTG control.
Fig 1 · Replication — GPT-2 XL (feed-forward)

GPT-2 XL reproduces the depth→time hierarchy. Peak lag rises with layer in language cortex, while the auditory control stays flat.

IFG r = +0.82 · aSTG r = +0.80 · control mSTG r = −0.26 (flat).
n electrodes: IFG 54 · aSTG 44 · mSTG 10.
Ouro-2.6B: peak encoding lag rises along effective recurrent depth in IFG and aSTG.
Fig 2 · Looped model — Ouro-2.6B (recurrent)

Ouro-2.6B shows the same rising hierarchy along its effective recurrent depth — a recurrent architecture recreates the cortical timing hierarchy.

IFG r = +0.66 · aSTG r = +0.60 · control mSTG r = +0.84 (also rises — Ouro's drift is more global; see Discussion).
IFG encoding peaks shift to later lags with model depth, in GPT-2's layers and Ouro's recurrent passes.
Fig 3 · The depth→time hierarchy, layer by layer (IFG)

In IFG, the encoding peak shifts to later lags as model depth increases — in GPT-2’s feed-forward layers and in Ouro’s recurrent passes alike. Each stripe is one layer / recurrent step (colour = depth, early→late); its dot marks the peak lag. The looped model traces the same cortical timing hierarchy as the feed-forward model.

Peak lag vs depth (IFG): GPT-2 r = +0.82 · Ouro r = +0.66.

4Discussion & Conclusion

  • Replicating Goldstein's effect validates our pipeline and confirms the depth→time hierarchy is robust.
  • The new, exciting result: a looped recurrent model recreates the cortical timing hierarchy along its iterations — consistent with language cortex computing by recurrence.
  • Recurrence is a viable substrate: Ouro maps onto neural processing time along its recurrent passes, tracing the same depth→time hierarchy that GPT-2 traces across feed-forward layers (Fig 3).
  • Caveats (stated honestly): the mSTG control is not perfectly clean, and Ouro's drift is somewhat global rather than strictly language-specific.
  • Future directions: explore other recurrence schemes — recur a single transformer block; grouped recurrency (e.g. loop the first three blocks 3×, then another three blocks 3×); vary the number of passes; and untie the looped weights — to test which form of recurrence best tracks the brain’s timing hierarchy.
Take-home A recurrent transformer reproduces the cortex's depth→time hierarchy for language — its recurrent passes track neural time just as a feed-forward model's layers do.

5References

  1. Goldstein, A. et al. (2025). Alignment of LLM layers to a neural-time hierarchy in language cortex. Nature Communications.
  2. Ouro-2.6B — looped / recurrent (weight-tied) transformer. ByteDance, 2025.
  3. Zada, Z. et al. (2025). The "Podcast" intracranial (ECoG) speech dataset (OpenNeuro ds005574). Scientific Data.