Introduction

Neural networks (NNs) were originally inspired by biological brains, yet they remain significantly distinct from their biological counterparts. Brains demonstrate complex neural dynamics that evolve over time, but modern NNs intentionally abstract away such temporal dynamics in order to facilitate large-scale deep learning. For instance, the activation functions of standard NNs can be seen as an intentional abstraction of a neuron’s firing rate, replacing the temporal dynamics of biological processes with a single, static value. Such simplifications, though enabling significant advancements in large-scale machine learning , have resulted in a departure from the fundamental principles that govern biological neural computation.

Over hundreds of millions of years, evolution has endowed biological brains with rich neural dynamics, including spike-timing-dependent plasticity (STDP) and neuronal oscillations. Emulating these mechanisms, particularly the temporal coding inherent in spike timing and synchrony, presents a significant challenge. Consequently, modern neural networks do not rely on temporal dynamics to perform compute, but rather prioritize simplicity and computational efficiency. This abstraction, while boosting performance on specific tasks, contributes to a recognized gap between the flexible, general nature of human cognition and current AI capabilities, suggesting fundamental components, potentially related to temporal processing, are missing from our current models .

For these reasons, we argue that time should be a central component of artificial intelligence in order for it to eventually achieve levels of competency that rival or surpass human brains . Therefore, in this work, we address the strong limitation imposed by overlooking neural activity as a central aspect of intelligence. We introduce the Continuous Thought Machine (CTM), a novel neural network architecture designed to explicitly incorporate neural timing as a foundational element. Our contributions are as follows:

• We introduce a decoupled internal dimension, a novel approach to modeling the temporal evolution of neural activity. We view this dimension as that over which thought can unfold in an artificial neural system, hence the choice of nomenclature.
• We provide a mid-level abstraction for neurons, which we call neuron-level models (NLMs), where every neuron has its own internal weights that process a history of incoming signals (i.e., pre-activations) to activate (as opposed to a static ReLU, for example).
• We use neural synchronization directly as the latent representation with which the CTM observes (e.g., through an attention query) and predicts (e.g., via a projection to logits). This biologically-inspired design choice puts forward neural activity as the crucial element for any manifestation of intelligence the CTM might demonstrate.

Reasoning models and recurrence

The frontier of artificial intelligence faces a critical juncture: moving beyond simple input-output mappings towards genuine reasoning capabilities. While scaling existing models has yielded remarkable advancements, the associated computational cost and data demands are unsustainable and raise questions about the long-term viability of this approach. For sequential data, longstanding recurrent architectures have largely been superseded by transformer-based approaches . Nevertheless, recurrence is re-emerging as a natural avenue for extending model complexity. Recurrence is promising because it enables iterative processing and the accumulation of information over time. Modern text generation models (sometimes referred to as ‘reasoning models’) use intermediate generations as a form of recurrence that enables additional compute during test-time. Recently, other works have demonstrated the benefits of the recurrent application of latent layers . While such methods bring us closer to the recurrent structure of biological brains, a fundamental gap nevertheless remains. We posit that recurrence, while essential, is merely one piece of the puzzle. The temporal dynamics unlocked by recurrence — the precise timing and interplay of neural activity — are equally crucial. The CTM differs from existing approaches in three ways: (1) the decoupled internal dimension enables sequential thought on any conceivable data modality; (2) private neuron-level models enables the consideration of precise neural timing; and (3) neural synchronization used directly as a representation for solving tasks.

---

Method

The Continuous Thought Machine (CTM) is a neural network architecture that enables a novel approach to thinking about data. It departs from conventional feed-forward models by explicitly incorporating the concept of Neural Dynamics as the central component to its functionality. The video above gives a pictorial overview of the internal workings of the CTM. We give all technical details, including additional figures and verbose explanations in our Technical Report. A GitHub repository is also available. We will provide links to relevant parts of the repository as we explain the model below.

The CTM consists of three main ideas:

1. The use of internal recurrence, enabling a dimension over which a concept analogous to thought can occur. The entire process visualised in the video above is a single tick; the interactive maze demo at the top of the page uses 75 ticks. This recurrence is completely decoupled from any data dimensions.
2. Neuron-level models, that compute post-activations by applying private (i.e., on a per-neuron basis) MLP models to a history of incoming pre-activations.
3. Synchronization as a representation, where the neural activity over time is tracked and used to compute how pairs of neurons synchronize with one another over time. This measure of synchronization is the representation with which the CTM takes action and makes predictions. Listing 3 in the Technical Report shows the logic for this, and Appendix K details how we use a recursive computation for efficiency.

Internal ticks: the ‘thought’ dimension

We start by introducing the continuous internal dimension: $t in { 1, ldots ,T }$t∈{1,…,T}. Unlike conventional sequential models — such as RNNs or Transformers — that process inputs step-by-step according to the sequence inherent in the data (e.g., words in a sentence or frames in a video), the CTM operates along a self-generated timeline of internal thought steps. This internal unfolding allows the model to iteratively build and refine its representations, even when processing static or non-sequential data such as images or mazes. To conform with existing nomenclature used in related works , we refer to these thought steps as ‘internal ticks’ from here on.

Recurrent weights: synapses

A recurrent multi-layer perceptron (MLP structured in a U-NET fashion ) acts as a synapse model for the CTM. At any internal tick $t$t, the synapse model produces what we consider pre-activations:

$bold{a}^t = f_{theta_{text{syn}}}(text{concat}(bold{z}^t, bold{o}^t)) in~mathbb{R}^D,$a​t​​=f​θ​syn​​​​(concat(z​t​​,o​t​​))∈ R​D​​,

where $bold{o}^t$o​t​​ is from input data. The $M$M most recent pre-activations are then collected into a pre-activation ‘history’:

$bold{A}^t = begin{bmatrix} bold{a}^{t-M+1} & bold{a}^{t-M+2} & cdots & bold{a}^t end{bmatrix} in~mathbb{R}^{D times M}.$A​t​​=[​a​t−M+1​​​​​a​t−M+2​​​​​⋯​​​a​t​​​​]∈ R​D×M​​.

Neuron-level models

$M$M effectively defines the length of the history of pre-activations that each neuron level model works with. Each neuron, ${1, ldots, D}${1,…,D}, is then given its own privately parameterized MLP that produces what we consider post-activations:

$bold{z}_d^{t+1} = g_{theta_d}(bold{A}_d^t),$z​d​t+1​​=g​θ​d​​​​(A​d​t​​),

where $theta_d$θ​d​​ are the unique parameters for neuron $d$d, and $bold{z}_d^{t+1}$z​d​t+1​​ is a single unit in the vector that contains all post-activations. $bold{A}_d^t$A​d​t​​ is a $M$M-dimensional vector (time series). The full set of neuron post-activations are then concatenated with attention output and fed recurrently into $f_{theta_{text{syn}}}$f​θ​syn​​​​ to produce pre-activations for next step, $t+1$t+1, in the unfolding thought process.

Synchronization as a representation: modulating data

How should the CTM interact with the outside world? Specifically, how should the CTM consume inputs and produce outputs? We introduced a timing dimension over which something akin to thought can unfold. We also want the CTM’s relationship with data (its interaction, so to speak) to depend not on a snapshot of the state of neurons (at some $t$t), but rather on the ongoing temporal dynamics of neuron activities. By way of solution, we turn again to natural brains for inspiration and find the concept of neural synchronization both fitting and powerful. For synchronization we start by collecting the post-activations into a post-activation ‘history’:

$bold{Z}^t = begin{bmatrix} bold{z}^{1} & bold{z}^{2} & cdots & bold{z}^t end{bmatrix} in mathbb{R}^{D times t}.$Z​t​​=[​z​1​​​​​z​2​​​​​⋯​​​z​t​​​​]∈R​D×t​​.

The length of $bold{Z}^t$Z​t​​ is equal to the current internal tick, meaning that this dimension is not fixed and can be arbitrarily large. We define neural synchronization as the matrix yielded by the inner dot product between post-activation histories:

$bold{S}^t = bold{Z}^t cdot (bold{Z}^t)^intercal in~mathbb{R}^{Dtimes D}.$S​t​​=Z​t​​⋅(Z​t​​)​⊺​​∈ R​D×D​​.

Since this matrix scales in $O(D^2)$O(D​2​​) it makes practical sense to subsample $(i,j)$(i,j) row-column pairs, which capture the synchronization between neurons $i$i and $j$j. To do so we randomly select $D_text{out}$D​out​​ and $D_text{action}$D​action​​ $(i,j)$(i,j) pairs from $bold{S}$S, thus collecting two synchronization representations, $bold{S}^t_text{out} in~mathbb{R}^{D_text{out}}$S​out​t​​∈ R​D​out​​​​ and $bold{S}^t_text{action} in~mathbb{R}^{D_text{action}}$S​action​t​​∈ R​D​action​​​​. $bold{S}^t_text{out}$S​out​t​​ can then be projected to an output space as:

$bold{y}^t = bold{W}_{text{out}} cdot bold{S}^t_text{out}.$y​t​​=W​out​​⋅S​out​t​​.

Modulating input data

$bold{S}^t_text{action}$S​action​t​​ can be used to take actions in the world (e.g., via attention as is in our setup):

$bold{q}^t = bold{W}_{text{in}} cdot bold{S}^t_text{action}$q​t​​=W​in​​⋅S​action​t​​

where $bold{W}_{text{out}}$W​out​​ and $bold{W}_{text{in}}$W​in​​ are learned weight matrices that project synchronization into vectors for observation (e.g., attention queries, $bold{q}^t$q​t​​) or outputs (e.g., logits, $bold{y}^t$y​t​​). Even though there are $(D times (D+1))/2$(D×(D+1))/2 unique pairings in $bold{S}^t$S​t​​, $D_text{out}$D​out​​ and $D_text{action}$D​action​​ can be orders of magnitude smaller than this. That said, the full synchronization matrix is a large representation that has high future potential.

In most of our experiments we used standard cross attention :

$bold{o}^t = text{Attention}(Q=bold{q}^t, KV=text{FeatureExtractor}(text{data}))$o​t​​=Attention(Q=q​t​​,KV=FeatureExtractor(data))

where a ‘FeatureExtractor’ model, e.g., a ResNet , is first used to build useful local features for the keys and values. $bold{o}^{t}$o​t​​ is concatenated with $bold{z}^{t+1}$z​t+1​​ for the next cycle of recurrence.

Loss function: optimizing across internal ticks

The CTM produces outputs at each internal tick, $t$t. A key question arises: how do we optimize the model across this internal temporal dimension? Let $bold{y}^t in mathbb{R}^{C}$y​t​​∈R​C​​ be the prediction vector (e.g., probabilities of classes) at internal tick $t$t, where $C$C is the number of classes. Let $y_{true}$y​true​​ be the ground truth target. We can compute a loss at each internal tick using a standard loss function, such as cross-entropy:

$mathcal{L}^t = text{CrossEntropy}(bold{y}^t, y_{true}),$L​t​​=CrossEntropy(y​t​​,y​true​​),

and a corresponding certainty measure, $mathcal{C}^t$C​t​​. We compute certainty simply as 1 – normalised entropy. We compute $mathcal{L}^t$L​t​​ and $mathcal{C}^t$C​t​​ for all $t in {1, ldots, T}$t∈{1,…,T}, yielding losses and certainties per internal tick, $mathcal{L} in mathbb{R}^{T}$L∈R​T​​ and $mathcal{C} in mathbb{R}^{T}$C∈R​T​​.

A natural question arises: how should we reduce $mathcal{L}$L into a scalar loss for learning? Our loss function is designed to optimize CTM performance across the internal thought dimension. Instead of relying on a single step (e.g., the last step), which can incentivize the model to only output at that specific step, we dynamically aggregate information from two internal ticks: the point of minimum loss and the point of maximum certainty:

• the point of minimum loss: $t_1=text{argmin}(mathcal{L})$t​1​​=argmin(L); and
• the point of maximum certainty: $t_2=text{argmax}({mathcal{C}})$t​2​​=argmax(C).

This approach is advantageous because it means that the CTM can perform meaningful computations across multiple internal ticks, naturally facilitates a curriculum effect, and enables the CTM to tailor computation based on problem difficulty. The final loss is computed as:

$L = frac{mathcal{L}^{t_1} + mathcal{L}^{t_2}}{2}.$L=​2​​L​t​1​​​​+L​t​2​​​​​​.

---

Experiment: ImageNet

Experiment: Solving 2D Mazes – doing it the hard way