← All Models
|
Allen AI OLMo - Fully open language models with training data/code:
OLMo-2-1124-7B
OLMo-2-1124-13B
Olmo-3-1025-7B
Olmo-3-1125-32B
allenai/OLMo-2-1124-7B
📊 Model Parameters
Total Parameters
7,298,617,344
Context Length
4,096
Hidden Size
4096
Layers
32
Attention Heads
32
KV Heads
32
💾 Memory Requirements
FP32 (Full)
27.19 GB
FP16 (Half)
13.59 GB
INT8 (Quantized)
6.80 GB
INT4 (Quantized)
3.40 GB
🔑 KV Cache (Inference)
Per Token (FP16)
524.29 KB
Max Context FP32
4.00 GB
Max Context FP16
2.00 GB
Max Context INT8
1.00 GB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
100,352
Hidden Size
4,096
FFN Intermediate Size
11,008
Number of Layers
32
Attention Heads
32
KV Heads
32
Context & Position
Max Context Length
4,096
RoPE Base Frequency
500,000
RoPE Scaling
Not set
Attention Configuration
Tied Embeddings
No
Attention Bias
No
Attention Dropout
0%
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-06
Special Tokens
BOS Token ID
Not set
Pad Token ID
100,277
EOS Token ID
100257
Data Type
Model Dtype
float32
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All