Bård StandalAI and Power
Menu

What is AI?

Accuracy on unseen equations rose long after the model had memorised the training set.

Alethea Power later told the story of an experiment that a colleague left running over a holiday. Before the break, the model had memorised its training equations but still failed on held-back ones. After training continued, its accuracy on the held-back equations rose towards 100 per cent.1

Original experiment · division modulo 97

Training accuracy was nearly 100 per cent while accuracy on unseen equations remained near chance.

Training and validation accuracy during grokkingTraining accuracy approaches 100 percent before one thousand updates. Accuracy on unseen equations remains near chance until much later, then rises sharply towards 100 percent near one million updates.
Shown equations
nearly 100%
Unseen equations
near chance

The network reproduces the training equations but still fails on equations it was not shown.

Redrawn from Figure 1 in Power et al. The horizontal axis runs from 100 to one million updates on a logarithmic scale. The curves are approximate.

The published experiments used arithmetic tables built from 97 abstract symbols. In the starkest reported run, training accuracy for division modulo 97 became nearly perfect before 1,000 updates. Researchers called it grokking when accuracy on unseen equations rose only after hundreds of thousands of further updates.2

Training continued on the same examples and with the same architecture, so the later improvement came from further changes to the saved weights.

The task

Modulo 97 returns to zero after 96.

Ordinary addition gives 48 + 75 = 123. Subtracting 97 leaves 26, so the same calculation is written 48 + 75 = 26 (mod 97). The network saw symbols and examples, not this rule in words.

48 + 7526 after wrapping at 97

Inside a model

A model is a calculation with adjustable numbers.

Training changes parameters such as weights and biases. A forward pass uses those parameters to turn an input into a prediction.

One complete forward pass
  1. 01 · Input2, 1

    Two numbers enter the first layer.

  2. 02 · Weighted sum
    2 × 0.51 × 0.50.5 bias2

    The weights control how much each input contributes. The bias is another learned number.

  3. 03 · Activation
    max(0, 2)2

    This nonlinear operation keeps positive values and replaces negative values with zero.

  4. 04 · Output layer
    2 × 1.53

    The last layer returns the prediction.

Saved parameters
weights 0.5, 0.5 and 1.5; bias 0.5
Activations
2 inside the hidden layer, then 3 at the output

Larger networks repeat these operations across many layers and recalculate the activations for every new input.

Training

Training adjusts the weights to reduce the loss.

The loss function measures the difference between the model's prediction and the target.

One weight · one exact update

This example uses one weight. The input is 2 and the target is 4; the starting weight is 0.5.

Current values

With weight 0.5, the prediction is 1 and the loss is 4.5.

Weight
0.5
Prediction
1
Loss
4.5
  1. 01

    Prediction

    The forward pass multiplies the input by the current weight.

    prediction = 2 × 0.51
  2. 02

    Loss

    The target is 4, so the prediction is 3 too low. Squaring makes either direction count as an error.

    ½ × (1 − 4)²4.5
  3. 03

    Backpropagation

    A small change in the weight changes the prediction and therefore the loss. Backpropagation uses the chain rule to calculate that combined rate.

    Loss responds to prediction1 − 4 = −3Raise the prediction by 0.01 and the loss falls by about 0.03.Prediction responds to weightx = 2Raise the weight from 0.50 to 0.51 and the prediction rises from 1.00 to 1.02.Loss responds to weight−6A 0.01 weight rise makes the loss fall by about 0.06.

    The first rate is −3 because this loss is half the square of the miss. The half cancels the doubling caused by the square, leaving prediction minus target: 1 − 4. The second rate is 2 because the input multiplies every change in the weight. Multiplying the local rates, −3 × 2, assigns −6 to the weight. This multiplication is the chain rule.

  4. 04

    Gradient descent

    A learning rate of 0.1 means taking one tenth of the −6 gradient. Subtracting that negative step raises the weight by 0.6.

    0.5 − 0.1 × (−6)1.1
  5. 05

    Repeat

    The next forward pass uses the changed weight, moving the prediction closer to 4 and reducing the loss.

    new prediction · 2 × 1.12.2new loss · ½ × (2.2 − 4)²1.62

The loss curve

For this example, loss reaches zero when the weight reaches 2.

Loss as the weight moves toward twoThe loss falls from 4.5 at weight 0.5 to 1.62 at weight 1.1. Further gradient steps approach weight 2, where the loss is zero.w 0.5 · loss 4.5w 1.1 · loss 1.62w 2 · loss 0weightloss
  1. Startw 0.50loss 4.50
  2. Update 1w 1.10loss 1.62
  3. Update 2w 1.46loss 0.58
  4. Update 3w 1.68loss 0.21
  5. Limitw 2.00loss 0

A deep network has many links between its prediction and its weights. Backpropagation starts at the loss and carries the same local-rate calculation backwards through layer after layer. Each weight receives a gradient: how a tiny change in that weight would change the loss at that moment.8

Backpropagation is not used during an ordinary response, and the optimiser does not run. The saved weights therefore remain fixed.9

Grokking

A separate modulo-113 experiment

The original modulus-97 experiment reported circular structure in weights learned for addition. The later modulo-113 study identified generalising and memorising components behind the accuracy curve.3

Separate experiment · addition modulo 113
generalising circuit formsheld-out accuracy rises

Sine and cosine components combined positions around a circle.

Language models

In a Transformer, each position combines information from itself and earlier positions.

Most current large language models use stacked Transformer blocks.

  1. Artificial intelligence

    Artificial intelligence is the wider field; language models are one kind of AI system.

  2. Machine learning

    Machine-learning methods adjust parameters from examples instead of receiving every rule in advance.

  3. Neural networks

    Neural networks are machine-learning models made from layers of weighted calculations.

  4. Transformer

    A Transformer is a neural-network architecture that uses attention to process sequences.

  5. Large language model

    “Large” has no agreed numerical threshold.

Before the Transformer · illustrative identifiers

A tokenizer assigns numbers to pieces of text. The model replaces each number with a vector.

A tokenizer may keep a word intact or split it into smaller pieces. Punctuation can be its own token.4

Thetoken 1 · id 464 modeltoken 2 · id 2746 learnedtoken 3 · id 7274 thetoken 4 · id 262 ruletoken 5 · id 3278 becausetoken 6 · id 780 trainingtoken 7 · id 3047 continuedtoken 8 · id 6407
Token id 3278learned embedding[ 0.2, −0.7, 0.1, 0.8, … ]
Position 5position information[ −0.1, 0.3, 0.6, −0.2, … ]
Input to the first blockstarting vector for “rule”[ 0.1, −0.4, 0.7, 0.6, … ]

A vector is learned from the contexts in which a token appears. Later layers update it for the current input.16

Masked self-attention

The attention layer calculates how much information to take from the current position and each earlier position.

Selected positionrule
Query
the vector used to score the available positions
Keys and values
keys determine the scores; values supply the information that is combined
The4%
model14%
learned42%
the6%
rule34%
becausemasked
trainingmasked
continuedmasked
Calculate scores

The query-key scores are normalised into shares. Later positions are masked.

Combine values

For “rule”, the largest shares go to the values at “learned” and “rule”.

The percentages are recalculated for each input and are not saved weights. Real models use several heads, and attention weights do not reliably explain an answer.6

One decoder block

A decoder block combines attention with a separate feed-forward calculation.

Residual paths carry the representation around the attention and feed-forward calculations, while normalisation keeps the numerical scale manageable.5

  1. 01Embedding + position

    One vector arrives for every token position.

  2. 02Masked attention

    Each position mixes information from itself and earlier positions.

  3. 03Add + normalise

    The attention result joins the residual path.

  4. 04Feed-forward network

    Another learned nonlinear transformation runs at each position.

  5. 05Add + normalise

    The block returns one changed vector per position.

token vectorsblock 1block 2…final blockvocabulary scores
Inference

A language model extends the context one token at a time.

Current context

The model found the

The context grows while the weights stay fixed.
rule47%
pattern31%
training12%
.10%
selected tokenrule

The model appends “rule” and runs the fixed weights again on the longer context.

The model found therulebecausetrainingcontinued.

The displayed probabilities are constructed. A decoding rule may select the highest value or sample from the distribution.17, 7

1986–2022

Backpropagation remained in use as networks grew and their architectures changed.

  1. 1986
    weightshidden layeroutputloss← gradient travels back

    Backpropagation

    Rumelhart and colleagues described how error information could be carried backwards through a network. An optimiser could then adjust each weight according to its contribution to the error.9

  2. 1989
    one learned filter · reused across the image

    Shared convolutional filters

    At Bell Labs, Yann LeCun used backpropagation to train filters that were reused across different parts of a postal-digit image. The network did not need a separate detector for every position.10, 11

  3. 2012
    1.2m imagesGPUdeep network

    AlexNet

    Alex Krizhevsky and colleagues trained a deep convolutional network on ImageNet using graphics processors. Its top-five test error in the 2012 competition was 15.3 per cent, compared with 26.2 per cent for the second-best entry.12, 13

  4. 2017
    attentionfeed-forward
    attentionfeed-forward
    attentionfeed-forward

    The Transformer

    Vaswani and colleagues replaced recurrent sequence processing with stacked attention and feed-forward blocks. More of the work inside a sequence could therefore run in parallel during training.5

  5. 2018–20
    text examplesnext-token trainingsaved checkpoint

    Generative pretraining

    GPT trained a Transformer on text before adapting it to particular tasks. GPT-3 increased the scale and could follow tasks described through examples in the prompt without changing its saved weights for each request.15, 17

  6. 2022
    pretrained modelpost-trainingdialogue

    Post-training

    InstructGPT used human demonstrations and ranked outputs after pretraining. ChatGPT placed a related model behind a dialogue interface that supplied the growing conversation as its current context.18, 19

During an answer

During inference, each selected token changes the running state while the learned weights normally remain fixed.

Saved model stateLearned weights

The same saved numbers are used at every step of an ordinary response.

fixed during inference
  1. 01
    Current context

    The prompt and all selected tokens so far enter the next pass.

    grows
  2. 02
    Attention cache

    Keys and values from earlier positions can be retained for the rest of the response.

    grows
  3. 03
    New activations

    Each layer calculates values for the newest position from the weights and the running state.

    recalculated
  4. 04
    Token scores

    The final values become a probability distribution over possible next tokens.

    recalculated
  5. 05
    Selected token

    One token is appended to the context, and the model runs another pass.

    returns to 01
Training

Weights are updated

  1. Example with a known target
  2. Loss measures the prediction error
  3. Backpropagation calculates gradients
  4. The optimiser updates the weights
Inference

Weights remain fixed during the response

  1. The current context and cache enter
  2. The newest position produces activations
  3. One token is selected
  4. The context and cache grow for the next pass

The physical machine

The calculations run on processors and memory.

Accelerator memory holds the learned weights and cached intermediate values while processors run the matrix operations. The equipment needs electricity and cooling before another token can be returned.

Continue to The system
Sectional drawing of the physical equipment used to run model inference.
Saved weights remain in accelerator memoryTemporary activations change as the context grows

Sources

Show the list
  1. Girl Geek X OpenAI Lightning Talks. Girl Geek X. Power recounts the training run that preceded the published grokking experiments.
  2. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv. Grokking reports delayed generalisation on small operation tables, including arithmetic modulo 97.
  3. Progress measures for grokking via mechanistic interpretability. International Conference on Learning Representations. The authors trace the gradual formation of a generalising circuit in a separate modulus-113 model.
  4. SentencePiece: A simple and language independent subword tokenizer and detokenizer. arXiv. SentencePiece explains subword tokenisation and why a token need not be a whole word.
  5. Attention Is All You Need. NeurIPS. Vaswani and colleagues introduced the Transformer architecture here.
  6. Attention is not Explanation. NAACL. Jain and Wallace find that attention weights do not reliably explain a model's answer.
  7. Scaling laws for neural language models. arXiv. Kaplan and colleagues measure how loss changes as training scale increases.
  8. Deep Learning. MIT Press. The cited chapters cover feed-forward networks and gradient-based optimisation.
  9. Learning representations by back-propagating errors. Nature. Rumelhart and colleagues describe backpropagation in multilayer networks.
  10. Backpropagation applied to handwritten zip code recognition. Neural Computation. LeCun and colleagues describe the Bell Labs network used to read handwritten postal digits.
  11. Yann LeCun faculty profile. New York University. NYU lists LeCun's biography and research fields.
  12. ImageNet: A large-scale hierarchical image database. IEEE Conference on Computer Vision and Pattern Recognition. ImageNet introduced the labelled dataset used for large-scale object recognition.
  13. ImageNet classification with deep convolutional neural networks. NeurIPS. Krizhevsky and colleagues report the GPU-trained AlexNet architecture and its ImageNet result.
  14. Neural machine translation by jointly learning to align and translate. arXiv. Bahdanau and colleagues present an early attention mechanism for machine translation.
  15. Improving language understanding by generative pre-training. OpenAI. Radford and colleagues describe generative Transformer pretraining followed by task adaptation.
  16. Language models are unsupervised multitask learners. OpenAI. The GPT-2 report describes a decoder-only model and byte-level byte-pair tokenisation.
  17. Language models are few-shot learners. NeurIPS. Brown and colleagues report GPT-3's scale and its use of examples supplied in the prompt.
  18. Training language models to follow instructions with human feedback. NeurIPS. Ouyang and colleagues describe instruction tuning and post-training based on ranked model outputs.
  19. Introducing ChatGPT. OpenAI. OpenAI's release note gives the November 2022 launch date and relates ChatGPT to InstructGPT.