← All Models
|
Deepseek - Chinese AI lab known for efficient MoE architectures:
DeepSeek-V2
DeepSeek-R1
deepseek-ai/DeepSeek-V2
📊 Model Parameters
Total Parameters
235,741,434,880
Context Length
163,840
Hidden Size
5120
Layers
60
Attention Heads
128
KV Heads
128
💾 Memory Requirements
FP32 (Full)
878.21 GB
FP16 (Half)
439.10 GB
INT8 (Quantized)
219.55 GB
INT4 (Quantized)
109.78 GB
🔑 KV Cache (Inference)
Per Token (FP16)
1.23 MB
Max Context FP32
375.00 GB
Max Context FP16
187.50 GB
Max Context INT8
93.75 GB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
102,400
Hidden Size
5,120
FFN Intermediate Size
12,288
Number of Layers
60
Attention Heads
128
KV Heads
128
Context & Position
Max Context Length
163,840
RoPE Base Frequency
10,000
RoPE Scaling
{...} (7 fields)
Attention Configuration
Attention Bias
No
Attention Dropout
0%
Tied Embeddings
No
Multi-Head Latent Attention
KV LoRA Rank
512
Query LoRA Rank
1,536
QK RoPE Head Dimension
64
Value Head Dimension
128
QK Non-RoPE Head Dimension
128
Mixture of Experts
Expert FFN Size
1,536
Shared Experts
2
Number of Experts
160
Routing Scale Factor
16.0
TopK Method
group_limited_greedy
Expert Groups
8
Groups per Token
3
Experts per Token
6
MoE Layer Frequency
1
Dense Initial Layers
1
Normalize TopK Probabilities
No
Router Scoring Function
softmax
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-06
Special Tokens
BOS Token ID
100,000
Pad Token ID
Not set
EOS Token ID
100001
Data Type
Model Dtype
bfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All