← All Models
|
Deepseek - Chinese AI lab known for efficient MoE architectures:
DeepSeek-V2
DeepSeek-R1
deepseek-ai/DeepSeek-R1
📊 Model Parameters
Total Parameters
671,026,419,200
Context Length
163,840
Hidden Size
7168
Layers
61
Attention Heads
128
KV Heads
128
💾 Memory Requirements
FP32 (Full)
2499.77 GB
FP16 (Half)
1249.88 GB
INT8 (Quantized)
624.94 GB
INT4 (Quantized)
312.47 GB
🔑 KV Cache (Inference)
Per Token (FP16)
1.75 MB
Max Context FP32
533.75 GB
Max Context FP16
266.88 GB
Max Context INT8
133.44 GB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
129,280
Hidden Size
7,168
FFN Intermediate Size
18,432
Number of Layers
61
Attention Heads
128
KV Heads
128
Context & Position
Max Context Length
163,840
RoPE Base Frequency
10,000
RoPE Scaling
{...} (7 fields)
Attention Configuration
Attention Bias
No
Attention Dropout
0%
Tied Embeddings
No
Multi-Head Latent Attention
KV LoRA Rank
512
Query LoRA Rank
1,536
QK RoPE Head Dimension
64
Value Head Dimension
128
QK Non-RoPE Head Dimension
128
Mixture of Experts
Expert FFN Size
2,048
Shared Experts
1
Number of Experts
256
Routing Scale Factor
2.5
TopK Method
noaux_tc
Expert Groups
8
Groups per Token
4
Experts per Token
8
MoE Layer Frequency
1
Dense Initial Layers
3
Normalize TopK Probabilities
Yes
Router Scoring Function
sigmoid
Speculative Decoding
Next-N Prediction Layers
1
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-06
Special Tokens
BOS Token ID
0
Pad Token ID
Not set
EOS Token ID
1
Data Type
Model Dtype
bfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All