Continual Learning

Standard neural network training assumes a fixed dataset, shuffled and sampled independently and identically throughout training. Real deployments rarely look like that: new product categories appear, a support team starts fielding tickets in a new language, a robot encounters a room it’s never seen. Train a neural network on a new task by naively continuing to fine-tune it, and something specific and severe usually happens: accuracy on everything the model previously knew collapses, often to near-random, even though nothing about the new task actively resembles the old one. This is catastrophic forgetting, and continual learning (also called lifelong learning or incremental learning) is the field devoted to preventing it, letting a single model absorb a sequence of tasks over time while retaining what came before.

Why Forgetting Happens

Catastrophic forgetting isn’t a bug in any one architecture; it’s a structural consequence of how gradient descent works when the same parameters are shared across tasks. A weight important for Task A’s predictions is, from Task B’s optimizer’s perspective, just another free parameter to push wherever minimizes Task B’s loss. Nothing in the vanilla training objective represents “don’t move this weight, Task A needs it here.” The result is that the very parameter sharing that lets neural networks generalize is also what lets a new task silently overwrite an old one.

flowchart LR
    classDef default fill:#ffffff,stroke:#4338CA,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef data    fill:#EEF0F7,stroke:#0D9488,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef process fill:#F7F8FC,stroke:#6366F1,stroke-width:2px,color:#0F172A,rx:8px,ry:8px;
    classDef output  fill:#4338CA,stroke:#4338CA,stroke-width:2px,color:#ffffff,rx:8px,ry:8px;

    T0[Model trained\non Task A]:::data
    T1[Fine-tune on\nTask B, naive]:::process
    T2["Task A accuracy: high\nTask B accuracy: high"]:::output
    T3["Task A accuracy: collapsed\nTask B accuracy: high"]:::output

    T0 --> T1
    T1 -->|with continual\nlearning method| T2
    T1 -->|naive fine-tune,\nno safeguard| T3

This is exactly the same underlying tension that model merging’s loss-barrier widget illustrates from a different angle: whether a path through weight space stays in a low-loss region for both tasks, or crosses a barrier that’s low for the new task and high for the old one.

The Stability-Plasticity Dilemma

Every continual learning method is navigating the same trade-off, inherited from a term used in the computational neuroscience literature well before deep learning: the stability-plasticity dilemma. A model that is maximally stable (resistant to any weight change) never forgets, but also never learns anything new. A model that is maximally plastic (free to change any weight) learns new tasks fully but forgets old ones completely. Continual learning methods are, almost without exception, mechanisms for controlling where a model sits on this spectrum, either globally via a single hyperparameter or locally, protecting different parameters by different amounts based on how important they were to prior tasks.

The three families of methods below tackle this dilemma with three different levers: penalizing changes to important weights (regularization), replaying old data alongside new (rehearsal), or physically separating which parameters each task gets to use (architecture).

Regularization-Based Methods

Elastic Weight Consolidation (EWC), introduced by Kirkpatrick et al. at DeepMind, is the best-known regularization approach. After training on Task A, EWC estimates the Fisher information for each parameter, a measure of how much the Task A loss would increase if that parameter were perturbed. Parameters with high Fisher information were doing important work for Task A; parameters with near-zero Fisher information were barely used and are safe to repurpose. When training on Task B, EWC adds a quadratic penalty term to the loss:

L(θ) = L_B(θ) + Σᵢ (λ/2) · Fᵢ · (θᵢ − θ*_A,ᵢ)²

where θ*_A are the parameter values at the end of Task A training, Fᵢ is the Fisher information for parameter i, and λ is a scalar controlling how strongly old-task parameters are protected overall. This is, in effect, a spring anchoring each parameter back to its Task-A value, with spring stiffness proportional to how much Task A needed that parameter, and an overall strength dial (λ) the practitioner sets. The interactive widget below lets you drag that dial.

Synaptic Intelligence (SI), from Zenke, Poole, and Ganguli, computes a similar per-parameter importance score but does it online, accumulating how much each parameter contributed to reducing the loss over the entire course of Task A training (via the path integral of the parameter’s gradient times its update), rather than requiring a separate Fisher-estimation pass after training finishes. This makes SI cheaper to run continuously as new tasks arrive.

Learning without Forgetting (LwF), from Li and Hoiem, takes a different tack that doesn’t touch the loss landscape’s geometry at all: before fine-tuning on Task B, record the current model’s outputs on Task B’s inputs (no Task A data needed), then during Task B training add a knowledge-distillation-style loss pulling the model’s Task-B-input outputs back toward those recorded predictions. This is architecturally identical to the response-based knowledge distillation mechanism, just applied to the same model at two points in time instead of to a separate teacher and student.

# Learning without Forgetting: distill the pre-fine-tuning model's own
# predictions on new-task inputs, instead of penalizing weight movement.
import torch.nn.functional as F

old_model.eval()
for batch in task_b_loader:
    inputs, labels = batch
    with torch.no_grad():
        old_logits = old_model(inputs)          # frozen snapshot, pre-Task-B

    new_logits = model(inputs)
    task_b_loss = F.cross_entropy(new_logits, labels)
    # T softens both distributions the same way distillation does
    T = 2.0
    distill_loss = F.kl_div(
        F.log_softmax(new_logits / T, dim=-1),
        F.softmax(old_logits / T, dim=-1),
        reduction="batchmean",
    ) * (T ** 2)

    loss = task_b_loss + 1.0 * distill_loss
    loss.backward()

Replay-Based Methods

Regularization methods never look at Task A data again after estimating importance scores. Replay-based methods take the more direct route: keep some Task A data (or a proxy for it) around, and mix it back in while training on Task B.

Rehearsal is the simplest version: retain a small buffer of real examples from each previous task and interleave them with new-task batches during training, so the optimizer sees a gradient signal for both tasks on every step, approximating the i.i.d. assumption standard training relies on.

iCaRL (Rebuffi et al.) formalizes this for class-incremental image classification: rather than storing random old examples, it keeps a fixed-size “exemplar set” per class chosen to be the examples whose feature-space representations are closest to that class’s mean representation, and it combines this rehearsal with a distillation loss (in the LwF style above) to further stabilize the learned representation as new classes are added.

Generative (or pseudo-) replay avoids storing any real data at all: a generative model trained alongside the classifier learns to produce synthetic samples resembling old-task data, which are replayed instead of, or alongside, real stored exemplars. This matters in settings where retaining real data isn’t allowed, for privacy or licensing reasons, but a decent generative model of the old task’s distribution can be kept instead.

# Rehearsal: a fixed-size ring buffer of old-task examples, replayed
# alongside every new-task batch during training.
import random

class ReplayBuffer:
    def __init__(self, capacity: int = 2000):
        self.capacity = capacity
        self.data = []

    def add(self, examples):
        for ex in examples:
            if len(self.data) < self.capacity:
                self.data.append(ex)
            else:
                self.data[random.randrange(self.capacity)] = ex  # reservoir sampling

    def sample(self, batch_size: int):
        return random.sample(self.data, min(batch_size, len(self.data)))

replay_buffer = ReplayBuffer(capacity=2000)
replay_buffer.add(task_a_examples)  # populated once Task A training finishes

for new_batch in task_b_loader:
    old_batch = replay_buffer.sample(len(new_batch))
    combined_batch = new_batch + old_batch   # interleave old and new
    loss = train_step(model, combined_batch)

Architecture-Based (Parameter Isolation) Methods

Rather than penalizing or replaying, a third family of methods sidesteps interference by giving each task its own dedicated slice of the network.

Progressive Neural Networks (Rusu et al.) freeze the columns of parameters trained on each previous task and, for every new task, instantiate an entirely new column of parameters with lateral connections into the frozen old columns, letting the new task reuse old features without ever being able to overwrite them. This gives strong forgetting resistance (old columns are literally frozen) at the cost of the network growing with every new task.

PackNet (Mallya and Lazebnik) takes the opposite resource strategy: instead of growing the network, it iteratively prunes a single fixed-size network after each task, freeing up the least-important weights for that task, then hands those freed weights to the next task while keeping the previous tasks’ weights fixed. A binary mask per task tracks which weights belong to which task at inference time.

Per-task adapters, the pattern most directly relevant to modern LLMs, sidestep the whole problem for the base model’s weights entirely: keep the large pretrained backbone completely frozen, and train a small LoRA adapter per task. Since each adapter is a separate, tiny set of low-rank matrices, adding Task B never touches Task A’s adapter at all, only which adapter gets loaded at inference time changes. The trade-off is that this only prevents forgetting for tasks the backbone doesn’t itself need to change to handle, not for cases requiring the shared backbone representation itself to shift.

Interactive: The EWC Lambda Trade-Off

The λ (lambda) term in the EWC loss above is the clearest example of the stability-plasticity dial in action: it’s a single scalar that directly trades retained accuracy on the old task against how much new-task accuracy the model can still reach. Drag it below to see both curves move:

At λ near 0, EWC reduces to plain fine-tuning: new-task accuracy is high, old-task retention is low. As λ grows, the penalty increasingly locks important Task-A weights in place, retention climbs, and new-task accuracy falls because the optimizer has less room to move where Task A needed stability. There is no single correct λ, only a point on this curve that matches how much old-task accuracy a given deployment can afford to trade away.

Continual Learning in Large Language Models

The catastrophic forgetting problem doesn’t disappear at LLM scale, it just shows up in different guises. A comprehensive 2024 survey on continual learning of LLMs documents forgetting across several distinct settings: continual pretraining, where a base model is updated on newer web data or a new domain corpus and can lose calibration on the original pretraining distribution; continual instruction tuning, where fine-tuning sequentially on new instruction datasets degrades performance on earlier instruction-following behaviors; and forgetting introduced during alignment stages like RLHF, where optimizing a reward model on new preference data can measurably erode capabilities the base model had before alignment even started. The survey also notes an architectural finding worth knowing in practice: decoder-only models tend to forget less than encoder-decoder models under the same continual fine-tuning regime, and general instruction tuning itself tends to make a model somewhat more resistant to forgetting during subsequent task-specific fine-tuning, likely because broad instruction tuning already spreads task-relevant behavior across more of the network rather than concentrating it.

In production, purpose-built continual learning algorithms like EWC are used less often for frontier LLMs than the simpler alternatives covered elsewhere in this glossary: per-task LoRA adapters swapped at inference time avoid touching the shared backbone at all, and periodic model merging of sequential fine-tuning checkpoints has become a popular lightweight substitute for a formal regularization term, trading a training-time penalty for a post-hoc weight-space average.

Continual Learning vs. Related Techniques

What’s New (2025-2026)

Nested Learning. Google Research’s Nested Learning paper (Behrouz and Mirrokni, presented at NeurIPS 2025) reframes a neural network itself as a nested system of optimization problems running at different update frequencies, rather than treating “architecture” and “learning algorithm” as separate concerns. Its HOPE architecture, built on this idea, includes a continuum memory system designed to keep updating at multiple timescales without the kind of abrupt overwrite that causes catastrophic forgetting in a standard single-timescale network, reporting lower perplexity and better continual-learning benchmarks than comparable recurrent and transformer baselines.

Merging as the pragmatic default. Reflecting the point above, more 2025 work explicitly frames iterative model merging (folding each new fine-tuning checkpoint back toward the previous one) as a practical continual-learning strategy for LLMs specifically, rather than treating classical regularization or replay methods as the default. This is a notable shift from the vision- and RL-dominated continual learning literature of the 2016-2020 period, where EWC, SI, and PackNet were developed.

Agentic memory as a training-free alternative. A parallel trend sidesteps continual learning’s weight-update problem altogether: instead of updating model parameters to retain new information, agent systems maintain external, retrievable memory stores that get written to and read from at inference time, with the underlying model’s weights left untouched. This trades a hard training problem (avoid catastrophic forgetting) for an engineering problem (retrieve the right memory at the right time), and has become the default approach for giving deployed LLM agents persistent, updatable knowledge without retraining.

Applications

On-device personalization. A keyboard app or voice assistant that adapts to an individual user’s vocabulary and corrections over weeks of use needs to retain general-purpose language capability while incorporating user-specific patterns, a canonical continual learning setting.

Robotics and embodied agents. A robot deployed across multiple environments needs to learn new object types or new manipulation skills as it encounters them, without losing competence at tasks it already mastered in earlier deployments.

Enterprise LLMs with rolling document updates. A retrieval-free internal assistant that gets periodically fine-tuned on newly added company documents needs safeguards against overwriting its general reasoning and instruction-following ability with each update round.

Recommendation systems under distribution shift. User preferences and item catalogs change continuously; a recommender retrained naively on only the most recent window of interactions forgets longer-term patterns that still matter for less-active users.

Class-incremental vision systems. Security and inspection systems that need to recognize new object or defect categories as they’re identified in the field, without full retraining on the original labeled dataset (often no longer available), are the classic setting iCaRL and PackNet were built for.

import torch
import torch.nn.functional as F

# Scenario: a model already trained on Task A (e.g. English support
# tickets) is about to be fine-tuned on Task B (e.g. Spanish tickets).
# EWC adds a penalty that discourages changing the parameters Task A
# relied on most, so Task B training doesn't erase Task A performance.

def compute_fisher(model, task_a_loader, device) -> dict:
    """Estimate the diagonal Fisher information for each parameter:
    how sensitive Task A's loss is to changes in that parameter."""
    fisher = {n: torch.zeros_like(p) for n, p in model.named_parameters()}
    model.eval()
    for batch in task_a_loader:
        inputs, labels = batch[0].to(device), batch[1].to(device)
        model.zero_grad()
        log_probs = F.log_softmax(model(inputs), dim=-1)
        # Sample from the model's own predicted distribution, per the
        # standard EWC Fisher estimator (Kirkpatrick et al., 2017).
        sampled = torch.multinomial(log_probs.exp(), 1).squeeze()
        loss = F.nll_loss(log_probs, sampled)
        loss.backward()
        for n, p in model.named_parameters():
            if p.grad is not None:
                fisher[n] += p.grad.detach() ** 2 / len(task_a_loader)
    return fisher

def ewc_loss(model, old_params: dict, fisher: dict, ewc_lambda: float = 400.0):
    """Quadratic penalty anchoring important Task-A parameters near
    their old values. ewc_lambda controls stability vs. plasticity:
    higher = more Task-A retention, lower = more Task-B flexibility."""
    penalty = 0.0
    for n, p in model.named_parameters():
        penalty += (fisher[n] * (p - old_params[n]) ** 2).sum()
    return ewc_lambda * penalty

# Training loop on Task B
fisher = compute_fisher(model, task_a_loader, device)
old_params = {n: p.clone().detach() for n, p in model.named_parameters()}

for batch in task_b_loader:
    inputs, labels = batch[0].to(device), batch[1].to(device)
    optimizer.zero_grad()
    task_b_loss = F.cross_entropy(model(inputs), labels)
    loss = task_b_loss + ewc_loss(model, old_params, fisher, ewc_lambda=400.0)
    loss.backward()
    optimizer.step()