← All Models | Allen AI OLMo - Fully open language models with training data/code:

allenai/OLMo-2-1124-13B

📊 Model Parameters

Total Parameters 13,716,198,400
Context Length 4,096
Hidden Size 5120
Layers 40
Attention Heads 40
KV Heads 40

💾 Memory Requirements

FP32 (Full) 51.10 GB
FP16 (Half) 25.55 GB
INT8 (Quantized) 12.77 GB
INT4 (Quantized) 6.39 GB

🔑 KV Cache (Inference)

Per Token (FP16) 819.20 KB
Max Context FP32 6.25 GB
Max Context FP16 3.12 GB
Max Context INT8 1.56 GB

⚙️ Model Configuration

Core Architecture

Vocabulary Size100,352
Hidden Size5,120
FFN Intermediate Size13,824
Number of Layers40
Attention Heads40
KV Heads40

Context & Position

Max Context Length4,096

Attention Configuration

Attention BiasNo
Attention Dropout0%
Tied EmbeddingsNo

Activation & Normalization

Activation Functionsilu
RMSNorm Epsilon1e-06

Special Tokens

Pad Token ID100,277
BOS Token IDNot set
EOS Token ID100257

Data Type

Model Dtypefloat32
Layer Types:
Attention
MLP/FFN
Normalization
Embedding