☰
Transformer models
Attention
GPT-2
GPT-3
Llama-8B
Llama-70B
Llama-405B
Input token length (n)
Vocab size
Embedding dim (d)
Heads (h)
Head dim (derived d/h)
Layers
Parameters
OFF
Input
70,000
tokens
Embedding
Positional
Encoding
40 TIMES
Transformer
Block
(Attention)
Mixing & FFN
Unembedding
Prediction
→
→
→
→
→
1347
-
-
Params (B)
FLOPs (TFLOPs)
-
-
-
-
-
-
-
-
-
-
-
-
Details