Why Foundation Model Plasticity Is Hard: About Linear Fast Weights, Neural Memory and What's next?

1. Why Is Foundation Model Plasticity So Difficult?

Updating neural-network weights has well-established methods.

Given a supervised learning task, we collect training examples, define a loss function, and update model parameters through gradient descent. Even online learning from one example at a time is a well-established problem.

Continual learning in a foundation model must preserve and extend the model’s capabilities acquired through pretraining.

A foundation model’s parameters already encode a broad collection of knowledge, representations, and capabilities acquired through extensive pretraining.

Ideally, when the model encounters a new experience, it should learn from it immediately. The update should also preserve what the model already knows, generalize beyond the current exemplar instance, and potentially improve its ability to learn future tasks.

Then, what makes foundation model plasticity much harder than ordinary online optimization?

In summary:

So far we’ve seen much progress in online weight adaptation (mostly through linear weight association; some add non-linearity; a more detailed review of those methods is described in the following sections), a little bit of progress in learning what to consolidate and what to adapt (like the inner and outer loops in TTT).

But as a continuous data stream comes in, and as the downstream tasks are NOT predetermined, what knowledge should be consolidated into the model weights and what knowledge should be actively explored and acquired by the deployed agents remains largely unknown to us.

1.1 Choosing What to Update and What to Learn

Consider a pretrained foundation model with parameters $\theta_0$.

After observing a new experience $e_t$, we can always construct a loss function and perform an update:

$$\theta_{t+1} = \theta_t-\eta\nabla_\theta\mathcal L(e_t;\theta_t).$$

The update itself is straightforward.

The difficult questions are:

1.2 The stability–plasticity problem is amplified in foundation models

A foundation model must balance two competing objectives:

• Plasticity: incorporate new information and acquire new capabilities.
• Stability: retain previously learned knowledge and broadly useful representations.

Suppose an update $\Delta\theta$ improves the current task. Its effect on a previously learned task can be approximated by:

$$\Delta\mathcal L_{\mathrm{old}} \approx \nabla_\theta\mathcal L_{\mathrm{old}}^\top\Delta\theta + \frac12\Delta\theta^\top H_{\mathrm{old}}\Delta\theta.$$

Here, $\Delta\mathcal L_{\mathrm{old}}$ is the change in the old task’s loss, and the gradient and Hessian $H_{\mathrm{old}}$ are evaluated at the pre-update parameters.

Even when the gradient of the old loss is approximately zero, an update can increase the old loss through its second-order effect.

Evaluating every parameter update against the enormous range of capabilities acquired during pretraining would be prohibitively expensive.

Alongside benchmark performance, the model must retain the underlying representations that support generalization to unevaluated tasks.

Making all parameters trainable leaves these questions of retention and generalization unresolved.

1.3 Plasticity Capacity vs. Functional Plasticity

Ideally, we would like to find parameter updates that improve performance across future tasks while preserving existing capabilities.

Conceptually, the objective is:

$$\boxed{ \Delta\theta_t^* = \arg\max_{\Delta\theta} \mathbb E_{\mathcal T\sim p_{\mathrm{future}}} \left[ U_{\mathcal T}(\theta_t+\Delta\theta) - U_{\mathcal T}(\theta_t) \right] }$$

subject to constraints on forgetting and update costs.

Here, $U_{\mathcal T}$ measures the model's utility on a future task $\mathcal T$.

The difficulty is immediate: the future task distribution is unknown, and a new experience provides very limited evidence about it.

Many parameter updates may explain the latest experience equally well while having completely different consequences for future learning. Choosing among them requires an objective as well as an optimizer.

A general solution requires useful inductive biases about which aspects of experience are likely to transfer.

That brings us to the central question of this review:

How much can a foundation model change through online learning, and how can it determine which changes are worth making?

Two dimensions matter:

Recent fast-weight architectures have made substantial progress on the first question. The second remains much less understood.

2. Two Types of Online Plasticity

Recent fast-weight methods can be divided into two broad categories based on the function class that is updated online.

This classification describes the online plasticity of the memory component. The entire foundation model can have a much broader function class.

A nonlinear foundation model can combine powerful nonlinear computation with online updates confined to a linear association matrix.

With an MLP memory, online updates can change the memory’s neural-network weights while leaving the pretrained backbone fixed.

These methods are better understood as progressively more expressive forms of online parametric memory embedded in a larger model.

3. Type I: Linear Associative Memory

Linear associative memory: DeltaNet, Falcon, and MemT

Model	What changes online	update mechanism
Model DeltaNet	What changes online A linear key–value memory matrix.	update mechanism Prediction-error correction writes the residual between the target value and the current memory prediction.
Model Falcon	What changes online A linear fast-weight memory matrix.	update mechanism Falcon-1 combines prediction-error correction, input-normalized step size, and weight decay. Falcon-2 adds channel-wise updates; Falcon-3 uses a context window.
Model MemT-1.0	What changes online Persistent matrices across memory channels and, with MoS, experts.	update mechanism Stable GDN-2 uses gated erase, write, and forgetting. Token-level storage runs at turn boundaries; generation reads the incoming memory snapshot.

3.1 Linear Attention: Writing Associations into a Matrix

Consider a stream of key-value pairs $(k_t,v_t)$ and a fast-weight memory matrix:

Associative-memory symbols

Symbol	Meaning	Dimensions / structure
Symbol $k_t$	Meaning Key vector	Dimensions / structure $k_t\in\mathbb R^{d_k}$
Symbol $v_t$	Meaning Value vector	Dimensions / structure $v_t\in\mathbb R^{d_v}$
Symbol $q_t$	Meaning Retrieval query	Dimensions / structure $q_t\in\mathbb R^{d_k}$
Symbol $W_t$	Meaning Fast-weight memory matrix	Dimensions / structure $W_t\in\mathbb R^{d_v\times d_k}$

• The matrix implements a linear mapping:

$$\hat v_t=W_{t-1}k_t.$$

• At inference time, a query retrieves information through:

$$o_t=W_tq_t.$$

• A basic Hebbian update is:

$$W_t=W_{t-1}+v_tk_t^\top.$$

• Each new experience directly writes a key-value association into the memory through a rank-one outer product.

• The update is computed directly from the key and value.

• The update ignores whether the current memory already predicts the value correctly.

• Repeated writing can cause interference and inefficient use of memory capacity.

3.2 DeltaNet: Updating Based on Prediction Error

DeltaNet improves the learning rule by introducing error correction.

• A simplified Delta Rule is:

$$\boxed{ W_t=W_{t-1} +\beta_t(v_t-W_{t-1}k_t)k_t^\top }$$

• where:

$$e_t=v_t-W_{t-1}k_t$$

is the prediction error. The scalar $\beta_t$ is an input-dependent writing strength, constrained to $(0,1)$ in DeltaNet.

• This is equivalent to a gradient descent step with step size $\beta_t$ on the local squared-error objective:

$$\mathcal L_t(W) = \frac12\|Wk_t-v_t\|_2^2.$$

• Both Hebbian and Delta writes have rank at most one.

• DeltaNet’s distinctive mechanism is prediction-error correction.

• The error-correcting update becomes small when the memory already predicts the target accurately.

• Gated DeltaNet adds input-dependent forgetting, allowing the model to control how much of the previous memory is retained.

These mechanisms improve associative memory management without requiring a trainable nonlinear memory network.

3.3 Falcon: Normalized Online Regression

Fast Weight Attention for Continual Learning (Falcon) investigates fast-weight updating as online regression.

Falcon-1 updates

Purpose	Formula	Explanation
Purpose Weight update	Formula $$W_t=(1-\eta_t\lambda_t)W_{t-1} +\eta_t(v_t-W_{t-1}x_t)x_t^\top,$$	Explanation The residual $v_t-W_{t-1}x_t$ drives prediction-error correction. The factor $1-\eta_t\lambda_t$ comes from an $L_2$ penalty on the memory weights and controls regularization-induced decay.
Purpose Normalized step size	Formula $$\eta_t= \frac{\beta_t} {\|x_t\|_2^2+\lambda_t+\epsilon}.$$	Explanation The normalization makes the update less sensitive to input magnitude.
Purpose Temporal alignment	Formula $$x_t=\phi(k_{t-1}),\qquad y_t=v_t.$$	Explanation For its next-latent prediction objective, Falcon uses the previous key representation to predict the current value.

• For $x_t\in\mathbb R^{d_x}$, the memory has shape $W_t\in\mathbb R^{d_v\times d_x}$, the transpose of the state convention used in the paper. Here, $\beta_t$ controls the update gain, $\lambda_t\geq0$ is the regularization coefficient, and $\epsilon>0$ stabilizes the denominator.

• Falcon-2 introduces channel-specific adaptation, while Falcon-3 extends the update to a sliding window of recent samples.

• The resulting memory retains a linear mapping:

$$\boxed{f_W(k)=Wk.}$$

3.4 MemT-1.0: Persistent Associative Memory

MemT-1.0 separates dynamic local memory from trainable hyper memory.

• Local memory contains persistent matrices; hyper memory outputs projections and gates learned during training.

• During online interaction, the backbone and hyper-memory parameters remain fixed. MemT therefore belongs to Type I: Linear Associative Memory.

• Nonlinear computation elsewhere in the architecture does not change which memory parameters adapt online.

Memory update

• Its Stable Gated DeltaNet-2 update can be written in our value-by-key convention as:

$$\begin{aligned} u_i&=\sqrt{\beta_i}\odot k_i,\\ W_i&=W_{i-1}D_i(I-u_i u_i^\top) +(\omega_i\odot v_i)u_i^\top. \end{aligned}$$

• Here, $W_i\in\mathbb R^{d_v\times d_k}$, and $k_i$ is an $L_2$-normalized key column.

• The erase vector $\beta_i\in(0,1)^{d_k}$ and write vector $\omega_i\in(0,1)^{d_v}$ control storage; $D_i=\operatorname{Diag}(\exp(\lambda_i))\in\mathbb R^{d_k\times d_k}$ is the diagonal forget gate, with each component of $\lambda_i$ negative.

• Index $i$ runs over tokens within one turn; layer, channel, and turn indices are omitted. Decay precedes the symmetric erase operation in the transposed convention.

• Finally, a query reads the matrix through $Wq/\sqrt{d_k}$.

• Generation reads the incoming memory snapshot. After the turn completes, storage processes its input and response token by token for use in the next turn.

• MoS: Mixture-of-Storage densely routes reads and writes across multiple matrix experts to expand capacity. This is something I find particularly interesting!

• These mechanisms are described in Sections 2.2–2.3 and 4.3 and Appendix C; the official model card also documents saving and restoring session memory states.

3.5 What Limits Linear Associative Memory?

The good part is that, in linear associative memory updates, a sequence of rank-one updates can accumulate into a high-rank matrix.

However, the central constraint is the feature space: online learning takes place within a feature space defined by the pretrained model.

• Consider:

$$f_t(x)=W_t\phi(x),$$

where $\phi$ is a frozen representation network.

• This update mechanism modifies $W_t$ while keeping the input representation $\phi$ fixed.

• If two states require different predictions but are mapped to the same feature representation, no linear update to $W_t$ can distinguish them.

• This becomes important when an agent’s pretrained representation conflates states in an unfamiliar environment.

Then why do these methods work well in some tasks?

• Well, sufficiently powerful pretrained features can support many complex associations; the online learner’s reach depends on that representation space.

• Linear Associative Memory is therefore an efficient way to update memory, but a restricted mechanism for acquiring genuinely new representations.

4. Type II: Nonlinear Neural Memory

TTT, Titans, and ATLAS explore a more expressive direction.

• They use a neural network as the online memory:

$$\boxed{f_{\theta_t}(k).}$$

• The parameters $\theta_t$ can include multiple layers, allowing online experience to change nonlinear transformations inside the memory network.

• This expands the class of functions that can be learned during inference.

Nonlinear neural memory: TTT-MLP, Titans, and ATLAS

Model	What changes online	update mechanism
Model TTT-MLP	What changes online The fast weights of a two-layer nonlinear memory network.	update mechanism Key/value views define a local learning task; outer-loop next-token loss shapes the projections and update settings. TTT-Linear remains Type I.
Model Titans	What changes online Neural-memory parameters and their momentum state.	update mechanism Associative-loss gradients update memory with momentum and input-dependent forgetting.
Model ATLAS	What changes online Neural-memory parameters optimized over a context window.	update mechanism Omega defines a weighted multi-sample local objective. Separately, Muon processes gradient momentum with Newton–Schulz iterations to form the update direction.

4.1 TTT: A Trainable Neural Network as the Hidden State

Learning to (Learn at Test Time): RNNs with Expressive Hidden States proposes that the hidden state of a recurrent model can itself be a machine learning model.

Two variants are studied:

• TTT-Linear: a linear model serves as the hidden state.
• TTT-MLP: a two-layer MLP serves as the hidden state.

Nonlinear memory

The nonlinear core of a simplified TTT-MLP takes the form:

$$f_\theta(k)=W_2\sigma(W_1k).$$

• For hidden width $h$, $W_1\in\mathbb R^{h\times d_k}$ and $W_2\in\mathbb R^{d_v\times h}$. This schematic expression omits the residual connection and layer normalization used in the paper’s implementation.

• At test time, its weights are updated by minimizing a local self-supervised loss:

$$\boxed{ \theta_t=\theta_{t-1} -\eta_t\left.\nabla_\theta\mathcal L_{\mathrm{local}}(\theta;x_t)\right|_{\theta=\theta_{t-1}}. }$$

• Here, $\theta$ denotes the memory network’s fast parameters. The online update can modify both $W_1$ and $W_2$.

• This is substantially more expressive than modifying only a linear key-value matrix.

• Updating the MLP requires local backpropagation, with computational and memory costs each time the network adapts during inference.

• Online learning occurs within the TTT layer: its memory weights adapt while the surrounding pretrained parameters remain fixed during inference.

4.2 Titans: Momentum, Surprise, and Forgetting

Titans: Learning to Memorize at Test Time extends neural memory by incorporating gradient-based surprise, momentum, and forgetting.

Titans memory updates

Purpose	Formula	Explanation
Purpose Momentum update	Formula $$S_t= \mu_tS_{t-1} -\eta_t\nabla_\theta \mathcal L_t(\theta_{t-1}),$$	Explanation Momentum preserves information about recent update directions.
Purpose Memory update	Formula $$\theta_t= (1-\alpha_t)\theta_{t-1}+S_t.$$	Explanation Forgetting allows the model to control how much of its previous memory is retained.

Titans symbols

Symbol	Meaning	Dimensions / structure
Symbol $\theta_t$	Meaning Neural-memory parameters	Dimensions / structure Memory-network parameter structure
Symbol $S_t$	Meaning Momentum state	Dimensions / structure Same parameter structure as $\theta_t$
Symbol $\mu_t$	Meaning Momentum retention	Dimensions / structure Input-dependent coefficient
Symbol $\eta_t$	Meaning Gradient step size	Dimensions / structure Input-dependent coefficient
Symbol $\alpha_t$	Meaning Parameter decay	Dimensions / structure Input-dependent; $\alpha_t\in[0,1]$

• The associative loss is $\mathcal L_t(\theta)=\|f_\theta(k_t)-v_t\|_2^2$, with gradients taken with respect to the memory parameters.

• This gives the neural memory richer learning dynamics than simple online gradient descent.

• Most importantly for me, the memory parameters adapt in the inner loop; key/value projections and other backbone parameters are learned in the outer loop.

• The surprise signal is the gradient of the local associative loss with respect to the memory parameters.

• A large prediction error indicates a mismatch with the current memory; its value for lasting changes in behavior depends on how the experience transfers.

• Titans improves the mechanisms for writing and managing neural memory.

• However, identifying transferable knowledge remains an open problem.

4.3 ATLAS: Contextual Optimization of Neural Memory

ATLAS: Learning to Optimally Memorize the Context at Test Time extends neural-memory learning in two directions.

• First, ATLAS introduces the Omega Rule, which optimizes memory over multiple recent samples.

A contextual loss can be written as:

$$\mathcal L_t^\Omega(\theta) = \sum_{i=t-c+1}^{t} \gamma_{t,i} \left\| f_\theta(\phi(k_i))-v_i \right\|_2^2.$$

• Here, $c$ denotes the context window length. The input-dependent gate $\gamma_{t,i}\in[0,1]$, written as $\gamma_i^{(t)}$ in the paper, controls the contribution of sample $i$ to the update at time $t$.

• Values near zero prune that sample from the local objective; values near one retain its full contribution.

• At the beginning of a sequence, the window includes only the available samples.

This allows recent experiences to jointly constrain the update.

• Second, ATLAS explores more sophisticated optimization dynamics, including Muon-style matrix updates using Newton–Schulz iterations.

• Omega changes the samples and weights in the local learning objective. Muon processes the accumulated gradient momentum into an approximately semi-orthogonal matrix update direction before applying the learning rate and memory decay.

The contribution combines a more expressive memory architecture with contextual optimization.

The optimization targets reconstruction or memorization of the local context.

However, selecting the information that will help on unseen tasks remains outside this objective.

4.4 What Have These Methods Actually Achieved?

TTT, Titans, and ATLAS demonstrate that online memory can be more expressive than a linear associative matrix.

They also show that neural-memory updates can be integrated into large pretrained foundation models and trained efficiently enough to be practical in some tasks, but not generally good and not generally with impressive performance.

In these architectures, the broader model is largely governed by pretrained slow parameters, while selected memory parameters change rapidly during inference.

The fast-memory weights may encode substantial contextual information, but many implementations initialize or reset them at sequential task streams.

Retaining memory states across sessions becomes necessary, but it raises further questions about long-term interference and how to consolidate transferable knowledge. Thus, for foundation model plasticity, the unresolved question is whether these online changes can go beyond memorizing recent context and become durable improvements in general capabilities.

5. Functional Plasticity: What Should a Foundation Model Learn?

Functional Plasticity refers to the ability of a foundation model to acquire useful, transferable capabilities through experience-driven parameter updates, rather than merely fitting new data or storing new associations.

The previous sections focus on how much a model can change.

Allowing more parameters to change makes the choice of update direction more consequential:

How should those parameters change?

Suppose a neural-memory network is trained online using:

$$\theta_{t+1} = \theta_t-\eta\nabla_\theta \mathcal L_{\mathrm{local}}(\theta_t).$$

The objective might be to predict a value from a key, reconstruct a representation, or minimize an error over recent context.

The model is ultimately evaluated on downstream tasks.

If we denote the future task loss by $\mathcal J_{\mathrm{future}}$, a first-order approximation gives:

$$\mathcal J_{\mathrm{future}}(\theta_{t+1}) \approx \mathcal J_{\mathrm{future}}(\theta_t) -\eta \left\langle \nabla_\theta\mathcal J_{\mathrm{future}}(\theta_t), \nabla_\theta\mathcal L_{\mathrm{local}}(\theta_t) \right\rangle.$$

Both gradients are evaluated at the pre-update parameters $\theta_t$. For sufficiently small steps, a positive inner product predicts lower future-task loss, while a negative inner product predicts interference. A zero inner product leaves the effect to higher-order terms.

This exposes a common limitation of both types of plasticity.

Linear fast-weight methods and neural-memory methods are largely built around local learning signals.

Alignment with downstream utility requires more than an expressive update network.

Greater plasticity can amplify the effects of a misaligned objective by allowing it to change more parameters.

5.1 TTT’s Inner and Outer Loops

TTT's inner-loop / outer-loop formulation offers a way to address this problem.

The global training objective can shape the local self-supervised learning objective.

TTT symbols

Symbol	Meaning	Dimensions / structure
Symbol $W_t$	Meaning Fast parameters at step $t$	Dimensions / structure Parameter structure of the chosen memory model
Symbol $\phi$	Meaning Outer-loop parameters	Dimensions / structure Key, value and query projections, fast-weight initialization, learning-rate parameters, and other network parameters
Symbol $\theta_K,\theta_V,\theta_Q$	Meaning Key, value and query projections	Dimensions / structure Learned linear maps from layer inputs to memory views
Symbol $x_t$	Meaning Input representation at a TTT layer	Dimensions / structure Layer-input vector
Symbol $s_t$	Meaning Token in the training sequence	Dimensions / structure Discrete token

Inner Loop: Learning from the current input

TTT constructs learnable key and value views from the layer input $x_t$:

$$k_t=\theta_Kx_t,\qquad v_t=\theta_Vx_t.$$

TTT inner-loop operations

Purpose	Formula	Explanation
Purpose Local objective	Formula $$\boxed{ \ell_t(W;\phi) = \|f_W(\theta_Kx_t)-\theta_Vx_t\|_2^2. }$$	Explanation The dependence on the outer-loop parameters $\phi$ is explicit.
Purpose Fast-weight update	Formula $$W_t= W_{t-1} -\eta_t\nabla_W\ell_t(W_{t-1};\phi).$$	Explanation Update the fast weights using the local objective.
Purpose Query readout	Formula $$z_t=f_{W_t}(\theta_Qx_t).$$	Explanation Query the updated memory using another learned projection.

• These equations describe the per-token online update; the paper also develops mini-batch TTT for efficient execution.

Outer Loop: Learning which local learning process improves the final objective

The outer loop optimizes the model's next-token prediction loss.

• For a training token sequence $\mathcal S=(s_1,\ldots,s_T)$, the objective can be written conceptually as:

$$\boxed{ \phi^*= \arg\min_\phi \mathbb E_{\mathcal S} \left[ \sum_{t=1}^{T-1}-\log p_\phi\!\left(s_{t+1}\mid s_{\leq t};W_{1:t}(\phi)\right) \right]. }$$

The trajectory $W_{1:t}(\phi)$ is generated by the inner-loop updates starting from $W_0(\phi)$.

• During outer-loop training, gradients pass through this trajectory, including the inner gradients, to shape the projections and update settings.

• The outer optimization therefore learns representations and update settings that make the inner-loop learning process useful for the final prediction objective.

• Therefore, TTT attempts to learn what should be learned locally, based on what helps the larger model perform better.

5.2 Open Problems in TTT’s Learning Framework

Three limitations remain unsolved.

5.3 The Hardest Problem: Discovering Transferable Meta-Skills

We take two levels of thinking on this problem.

• The first level is about learning what helps the current task.

For an interactive agent, this may require connecting experience with delayed rewards, long-horizon credit assignment, goal inference, or planning outcomes. This is difficult, but the current task can eventually provide an evaluation signal.

• The second level is about transferable learning mechanisms. It’s much harder than the first one.

It’s about what the model should learn now to improve its ability to learn future tasks that have not yet been encountered.

Consider an agent exploring a new environment.

It might learn a particular map, memorize a sequence of actions, or discover a rule that applies only to that environment.

The experience could also help the agent acquire a more general capability: a better exploration strategy, a method for identifying causal relationships, or a way to discover reusable subtasks.

These can provide a family of transferable meta-skills.

A foundation model should ideally accumulate both task-specific knowledge and more general learning capabilities, while distinguishing which updates deserve long-term knowledge consolidation.

One possible research objective is:

$$\boxed{ \mathcal U(\Delta\theta_A) = \mathbb E_{\mathcal T_B\sim p_{\mathrm{future}}} \left[ \operatorname{Regret}_B(\theta_0) - \operatorname{Regret}_B(\theta_0+\Delta\theta_A) \right]. }$$

Here, $\Delta\theta_A$ is the parameter update obtained from experience in task A.

$\operatorname{Regret}_B(\theta)$ denotes the expected cumulative regret incurred while learning and acting in task B, starting from parameters $\theta$. The goal is to identify parameter changes whose benefits extend from the current experience to lower expected regret on future tasks.

The objective exposes the fundamental difficulty: the future task distribution is not fully known, and the utility of a particular meta-skill may only become apparent after encountering many different environments.

A general learner therefore needs inductive biases toward reusable structure and mechanisms for testing which learned abstractions actually transfer.

And this relates back to Prof. Rich Sutton’s feature construction in the “Alberta Plan” paper.

TTT takes an important step by allowing the global training objective to shape local learning.

Extending this approach to foundation model plasticity would require optimizing learning mechanisms across tasks, with the possibility of improving those mechanisms through post-deployment experience.

6. Commercialization values: A New Foundation Model or an Agent Companion Model (Agent-CM)?

Beyond the technical challenges, native-memory models (like MemT and others) face a difficult commercial question: Where should memory sit in the AI agent stack?

Consider two competing approaches.

Memory-native foundation models: an embarrassing approach

The first is to build a new memory-native foundation model, where memory is an intrinsic part of the model. MemT-1.0 is a recent example. It attempts to incorporate persistent, dynamically updated memory directly into its model, reducing the need to repeatedly process long interaction histories. However, these memory-native foundation models usually do not work with the agent’s base model, which is usually a GPT Astra or Claude Fable model and is more capable on many tasks.

Then, providing a better memory-native model seems not to add any value here. What customers care about the most is task performance, and such memory-native foundation models have not so far exhibited better task performance compared to the commonly used proprietary models such as GPT Astra or Claude Fable.

Just having better memory management performance is not adding any commercialization value for the final agent’s task performance. This is embarrassing for these models.

External memory systems: the moat is weak!

The second approach is to manage memory outside the agent’s base model, using context compaction, retrieval, structured memory stores, and agent-level memory management. Products such as Mem0, Zep, Letta, and LangMem follow different versions of this approach.

From a commercial perspective, the second approach has an important advantage over the first one: it can improve existing agents without replacing their foundation models.

A developer can continue using the strongest available reasoning or coding model (GPT Astra or Claude Fable), while adding an independent memory system. When a better foundation model becomes available, the memory infrastructure can remain largely unchanged.

In summary, a standalone memory-native model faces a much higher bar. It is not enough to demonstrate better memory retention, lower KV-cache storage, or longer effective context. It must compete against the best available foundation model combined with a strong external memory system.

If a native-memory model remembers more but performs worse at reasoning, coding, planning, or tool use, why would developers replace their existing foundation model?

Memory improvements alone may not justify sacrificing the broader capabilities of a frontier model.

Agent Companion Models (Agent-CMs)

This suggests a third possibility that I find particularly interesting: an Agent Companion Model (Agent-CM).

Instead of building another general-purpose foundation model, we could build a smaller, specialized neural learning system that operates alongside an existing foundation model.

The foundation model would remain responsible for general reasoning, language understanding, and tool use. The companion model would continuously learn from the agent's interactions, updating its own parametric memory and extracting reusable knowledge from experience.

Each agent could maintain an independent, persistent learning state without requiring a separate copy of the entire foundation model. The companion would communicate what it has learned through a defined interface, supplying relevant knowledge, task constraints, or reusable skills to the main agent.

This architecture would combine two advantages:

• The general capabilities of the strongest available foundation models.
• The online plasticity of neural-memory architectures such as TTT, Titans, or ATLAS.

It would also avoid requiring the memory model to outperform frontier foundation models across every capability.

Of course, an Agent Companion Model (Agent-CM) would still have to demonstrate something that context-based memory systems cannot achieve as effectively. If it merely retrieves facts or summarizes previous conversations, the additional complexity of neural online learning may not be justified.

Check this article for a detailed explanation on Agent-CMs: Beyond the Agent Harness: Why do we need continual learning Agent Companion Models?

7. Conclusion

There is a critical distinction between updating a parameterized memory and acquiring continual learning improvements of a foundation model. For that, we need to answer three questions:

1. Plasticity Capacity: What can a foundation model change through online parameter updates?
2. Functional Plasticity: What new, useful, and transferable capabilities do these changes produce?
3. Meta-Skill Discovery: Which experiences should produce lasting changes that make future learning more efficient?

Recent fast-weight methods have made meaningful progress on the first question.

TTT's bi-level optimization offers a useful starting point for the second.

The third remains largely open.

---

References