tinyGPT Gym

See how a language model learns from a training document: text becomes tokens, tokens become prediction examples, and each training step nudges the model toward better next-token guesses.

Back to gym index

Loading tinyGPT artifacts...

You start with raw AI-book text; this tab turns it into the vocabulary and token IDs tinyGPT can learn from.

Act 1 — The problem → Act 2 — BPE builds the vocabulary → Act 3 — Text becomes IDs

Training document

Training Document: AIClub Introduction to Artificial Intelligence

This fixed AI textbook corpus is the only text tinyGPT learns from in this gym.

Act 1 — The problem: words or characters, both break

Two approaches to tokenizing the same sentence — and why each falls short.

Approach A — whole words

One token per word

Easy to read, but the vocabulary explodes. Every name, typo, and domain term needs its own entry. Unseen words fail at test time.

7 tokens

but vocab can reach 500k+ words

▾

subwords
balance both

Approach B — characters only

One token per character

Tiny vocabulary, no unknowns — but every sentence explodes into dozens of pieces. The context window fills fast.

47 tokens

for one sentence alone

Act 2 — BPE builds the vocabulary

Byte-level BPE repeatedly merges common adjacent pieces, then uses that learned vocabulary to encode the selected snippet.

Follow along in the notebook — Section 2 is the tokenizer. ↗ Open in Colab

Vocabulary is learned from repeated text patterns.

The tokenizer starts from bytes, finds frequent neighboring pieces, creates merged tokens, and eventually represents the AI book as a shorter stream of token IDs.

1 Count neighbors Scan the training text and count adjacent token pieces.

2 Add one piece The most repeated pair becomes a new vocabulary token.

3 Replace everywhere Every occurrence is re-written using the new shorter piece.

Current merge candidate

Where this appears in the document

step 1 120 merges

As BPE adds vocabulary pieces, the same document is represented with fewer token pieces.

Act 3 — Text becomes a stream of IDs

The learned vocabulary encodes the selected text as colored token blocks with numeric IDs underneath.

Token IDs

So what? Notice common AI-book phrases can become compact tokens, while rarer wording may split into smaller pieces. BPE reflects what is frequent in this specific corpus.

At a Glance

Small corpus and model stats for orientation.

Corpus

AI book

Tokenizer Setup

Matches the notebook configuration.

Pre-tokenizerByteLevel(add_prefix_space=True)

Vocabulary target8000 tokens, min_frequency = 2

Special tokens<pad>, <bos>, <eos>, <unk>

Template<bos> $A <eos>

Vocabulary Trace

Early merges the tokenizer discovers first.

Step	Merge	Frequency	Tokens Left

You now have a stream of token IDs; this tab turns that stream into fixed-size X and y batches so the model has inputs and next-token answers.

Act 1 — One window: x and y → Act 2 — Stack into a batch → Act 3 — Train / val split

How the token stream becomes training examples

A training step samples 32 windows from the encoded document. Each window is 128 tokens wide and becomes one row in the batch.

Follow along in the notebook — Section 3 covers batching. ↗ Open in Colab

Act 1 — One window: input x and target y

One window of 5 tokens from the stream gives one training example — automatically, no human labeling needed.

stream

668 664 143 106 1053

input x

668 664 143 106 —

target y

— 664 143 106 1053

y is x shifted one position to the right — 4 free prediction tasks from 4 tokens.

1. Encode documentThe AI book becomes one long token stream.

2. Pick windowsChoose random start positions in the train split.

3. Stack rowsEach 128-token window becomes one row of X.

4. Provide labelsy is the next-token answer sheet, shifted one step ahead.

Act 2 — Stack many windows into a batch

One batch is a 32 x 128 grid of token IDs.

The model sees 32 independent samples at once. Each sample is a fixed-length row of 128 real tokens. The target row y is also 128 tokens: one next-token answer for each X position.

Act 3 — Train / val split

Train

Val

Shapes entering the model (preview of Tab 3)

X and y are full tensors. The causal mask (explained in Tab 3) determines which X positions each column can attend to.

X and y are both full 128-token tensors — the batch format is simple. X and y are both full tensors. The blue/gray ladder shows which parts of a 128-token X row each position can attend to after the causal mask is applied.

X shape [32, 128] y shape [32, 128] same 128 positions

Input X

[32, 128] 32 samples, each with 128 context tokens.

Target y

[32, 128] 32 samples, each with 128 next-token labels.

Block Size

128

Batch Size

Seed

Validation Snippet

Held-out text is used for evaluation, not optimizer updates.

You have a 32 x 128 batch of X and y tensors; this tab shows what happens when those tensors enter GPT.forward() and how 3,000 training steps produce a working model.

Act 1 — Loss: measuring how wrong → Act 2 — The training loop → Act 3 — Validation and perplexity

One Training Iteration

Forward pass, loss, backward pass, gradient clipping, and AdamW update.

Training loop code is in Section 4 of the notebook. ↗ Open in Colab

Act 1 — Cross-entropy loss

correct next token "▁Intelligence" · id 664

Model is confident It gives the correct token a high probability, so the penalty is small.

p = 0.72loss ≈ 0.33

Model is unsure It gives the correct token a tiny probability, so the penalty is large.

p = 0.003loss ≈ 5.81

loss = −log(probability assigned to correct token)

Act 2 — The training loop

checkpoint step0

train loss0.00

val perplexity0.00

Act 3 — Validation and perplexity

Gradient clipping: prevents a single bad batch from making a wildly large parameter update.

AdamW: an optimizer that adjusts each weight by a small amount in the direction that reduces loss, with weight decay to prevent overfitting.

perplexity = e^loss how many tokens the model was effectively choosing between

PPL ≈ 100confused between 100 options — early training

PPL ≈ 10narrowed to ~10 plausible next tokens

PPL = 1perfect — always certain about the right answer

Inside the forward pass (Tab 2 → Tab 3 connection)

Click any step to see the Python code and tensor shapes. Embedding dimension is 256 — every token becomes a 256-value vector.

idx / X [32, 128] Input token IDs. The model receives this tensor. feeds GPT.forward

targets / y [32, 128] Not an input to the forward pass; used only when cross-entropy computes loss. loss only

You now have trained weights; this tab uses the tokenizer and forward pass to turn a prompt into last-position logits, sample one token, and feed it back as generated text.

Generate With The Trained tinyGPT Checkpoint

Runs the converted PyTorch model in the browser. No backend, no server state, no browser training.

New Tokens

hint

Temperature

low = safe · high = creative

Top-k

sample from top k candidates

Ready. The ONNX model loads on the first generation.

Generated Text

Each new token comes from the last-position logits over the 6,090-token vocabulary.

Why last-position logits? Because of the causal mask you saw in Tab 3, position 127 has seen all 128 input tokens, so its logits are the only ones used to predict the next token.

Real Inference Path

This is the honest path to a prompt box that behaves like the Colab notebook.

Inference code is in the last section of the notebook. ↗ Open in Colab

ids = tokenizer.encode(prompt)

idx = ids[-128:] # block_size context

logits = tinygpt.onnx(idx)

next_id = sample_top_k(logits[-1], temperature, k)

ids.append(next_id)

Next-token Candidates

The latest model call exposes the highest-probability next tokens before sampling.

Browser Model

Converted from the supplied PyTorch checkpoint.