← All Models | Allen AI OLMo - Fully open language models with training data/code:

allenai/OLMo-2-1124-13B

📊 Model Parameters

Total Parameters 13,716,198,400
Context Length 4,096
Hidden Size 5120
Layers 40
Attention Heads 40
KV Heads 40

💾 Memory Requirements

FP32 (Full) 51.10 GB
FP16 (Half) 25.55 GB
INT8 (Quantized) 12.77 GB
INT4 (Quantized) 6.39 GB

🔑 KV Cache (Inference)

Per Token (FP16) 819.20 KB
Max Context FP32 6.25 GB
Max Context FP16 3.12 GB
Max Context INT8 1.56 GB

⚙️ Model Configuration

Core Architecture

Vocabulary Size100,352
Hidden Size5,120
FFN Intermediate Size13,824
Number of Layers40
Attention Heads40
KV Heads40

Context & Position

Max Context Length4,096
RoPE Base Frequency500,000
RoPE ScalingNot set

Attention Configuration

Tied EmbeddingsNo
Attention BiasNo
Attention Dropout0%

Activation & Normalization

Activation Functionsilu
RMSNorm Epsilon1e-06

Special Tokens

BOS Token IDNot set
Pad Token ID100,277
EOS Token ID100257

Data Type

Model Dtypefloat32
Layer Types:
Attention
MLP/FFN
Normalization
Embedding