← All Models
|
EleutherAI - Non-profit AI research lab focused on open-source LLMs:
gpt-j-6b
gpt-neo-125m
gpt-neo-1.3B
gpt-neo-2.7B
gpt-neox-20b
comma-v0.1-2t
EleutherAI/gpt-neo-125m
📊 Model Parameters
Total Parameters
163,795,968
Context Length
2,048
Hidden Size
768
Layers
12
Attention Heads
12
KV Heads
12
💾 Memory Requirements
FP32 (Full)
624.8 MB
FP16 (Half)
312.4 MB
INT8 (Quantized)
156.2 MB
INT4 (Quantized)
78.1 MB
🔑 KV Cache (Inference)
Per Token (FP16)
36.86 KB
Max Context FP32
144.0 MB
Max Context FP16
72.0 MB
Max Context INT8
36.0 MB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
50,257
Hidden Size
768
Number of Layers
12
Attention Heads
12
FFN Intermediate Size
Not set
Context & Position
Max Context Length
2,048
Sliding Window Size
256
Attention Configuration
Attention Dropout
0%
Tied Embeddings
Yes
Activation & Normalization
Activation Function
gelu_new
RMSNorm Epsilon
1e-05
Dropout (Training)
Residual Dropout
0%
Embedding Dropout
0%
Special Tokens
BOS Token ID
50,256
EOS Token ID
50256
Pad Token ID
Not set
Data Type
Model Dtype
Not set
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All