Step
0
Byte-Pair Encoding (BPE) Demo
Starts with character tokens, then repeatedly merges the most frequent adjacent token pair.
Vocabulary Size
0
Context Size
0
Last Merge
-
Token Tables
Created tokens and likely next merges for the current step.
Created Tokens (Latest First)
Shows base character tokens and merged tokens with their current frequency.
| Token ID | Token | Current Frequency |
|---|
Top Next Merge Candidates
Most frequent adjacent token pairs that BPE can merge next.
| Pair | Merged Token | Frequency |
|---|
Vocabulary Size vs Context Size
X-axis: vocabulary size. Y-axis: context size (number of tokens used to represent the original text).