← All Models
|
EleutherAI - Non-profit AI research lab focused on open-source LLMs:
gpt-j-6b
gpt-neo-125m
gpt-neo-1.3B
gpt-neo-2.7B
gpt-neox-20b
comma-v0.1-2t
common-pile/comma-v0.1-2t
📊 Model Parameters
Total Parameters
7,002,656,768
Context Length
16,384
Hidden Size
4096
Layers
32
Attention Heads
32
KV Heads
32
💾 Memory Requirements
FP32 (Full)
26.09 GB
FP16 (Half)
13.04 GB
INT8 (Quantized)
6.52 GB
INT4 (Quantized)
3.26 GB
🔑 KV Cache (Inference)
Per Token (FP16)
524.29 KB
Max Context FP32
16.00 GB
Max Context FP16
8.00 GB
Max Context INT8
4.00 GB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
64,256
Hidden Size
4,096
FFN Intermediate Size
11,008
Number of Layers
32
Attention Heads
32
KV Heads
32
Head Dimension
128
Context & Position
Max Context Length
16,384
RoPE Base Frequency
100000.0
RoPE Scaling
Not set
Attention Configuration
Attention Bias
No
Attention Dropout
0%
MLP Bias
No
Tied Embeddings
No
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-05
Special Tokens
BOS Token ID
1
Pad Token ID
Not set
EOS Token ID
2
Data Type
Model Dtype
bfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All