← All Models | Allen AI OLMo - Fully open language models with training data/code:

allenai/Olmo-3-1025-7B

📊 Model Parameters

Total Parameters 7,298,011,136
Context Length 65,536
Hidden Size 4096
Layers 32
Attention Heads 32
KV Heads 32

💾 Memory Requirements

FP32 (Full) 27.19 GB
FP16 (Half) 13.59 GB
INT8 (Quantized) 6.80 GB
INT4 (Quantized) 3.40 GB

🔑 KV Cache (Inference)

Per Token (FP16) 524.29 KB
Max Context FP32 64.00 GB
Max Context FP16 32.00 GB
Max Context INT8 16.00 GB

⚙️ Model Configuration

Core Architecture

Vocabulary Size100,278
Hidden Size4,096
FFN Intermediate Size11,008
Number of Layers32
Attention Heads32
KV Heads32

Context & Position

Max Context Length65,536
RoPE Base Frequency500,000
RoPE Scalingyarn (factor: 8.0)
Sliding Window Size4,096
Layer Attention Types[32 items]

Attention Configuration

Tied EmbeddingsNo
Attention BiasNo
Attention Dropout0%

Activation & Normalization

Activation Functionsilu
RMSNorm Epsilon1e-06

Special Tokens

BOS Token IDNot set
Pad Token ID100,277
EOS Token ID100257

Data Type

Model Dtypebfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding