← All Models
|
Microsoft Phi - Small language models with strong reasoning:
phi-1
phi-1_5
phi-2
Phi-3-mini-4k-instruct
Phi-4
microsoft/phi-2
📊 Model Parameters
Total Parameters
2,779,683,840
Context Length
2,048
Hidden Size
2560
Layers
32
Attention Heads
32
KV Heads
32
💾 Memory Requirements
FP32 (Full)
10.36 GB
FP16 (Half)
5.18 GB
INT8 (Quantized)
2.59 GB
INT4 (Quantized)
1.29 GB
🔑 KV Cache (Inference)
Per Token (FP16)
327.68 KB
Max Context FP32
1.25 GB
Max Context FP16
640.0 MB
Max Context INT8
320.0 MB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
51,200
Hidden Size
2,560
FFN Intermediate Size
10,240
Number of Layers
32
Attention Heads
32
KV Heads
32
Context & Position
Max Context Length
2,048
RoPE Base Frequency
10000.0
RoPE Scaling
Not set
Attention Configuration
Attention Dropout
0%
Tied Embeddings
No
Activation & Normalization
Activation Function
gelu_new
RMSNorm Epsilon
1e-05
Dropout (Training)
Residual Dropout
10.0%
Embedding Dropout
0%
Special Tokens
BOS Token ID
50,256
Pad Token ID
Not set
EOS Token ID
50256
Data Type
Model Dtype
float16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All