← All Models
|
Qwen - Alibaba Cloud's multilingual LLM series:
Qwen1.5-0.5B
Qwen1.5-1.8B
Qwen1.5-4B
Qwen1.5-7B
Qwen1.5-14B
Qwen1.5-32B
Qwen1.5-72B
Qwen1.5-110B
Qwen1.5-MoE-A2.7B
Qwen2-0.5B
Qwen2-1.5B
Qwen2-7B
Qwen2-57B-A14B
Qwen2-72B
Qwen2.5-0.5B
Qwen2.5-1.5B
Qwen2.5-3B
Qwen2.5-7B
Qwen2.5-14B
Qwen2.5-32B
Qwen2.5-72B
Qwen3-0.6B
Qwen3-1.7B
Qwen3-4B
Qwen3-8B
Qwen3-14B
Qwen3-32B
Qwen3-30B-A3B
Qwen3-235B-A22B
Qwen3-235B-A22B-Thinking-2507
Qwen3-Next-80B-A3B-Thinking
Qwen/Qwen3-Next-80B-A3B-Thinking
📊 Model Parameters
Total Parameters
79,674,391,296
Context Length
262,144
Hidden Size
2048
Layers
48
Attention Heads
16
KV Heads
2
💾 Memory Requirements
FP32 (Full)
296.81 GB
FP16 (Half)
148.41 GB
INT8 (Quantized)
74.20 GB
INT4 (Quantized)
37.10 GB
🔑 KV Cache (Inference)
Per Token (FP16)
98.30 KB
Max Context FP32
48.00 GB
Max Context FP16
24.00 GB
Max Context INT8
12.00 GB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
151,936
Hidden Size
2,048
FFN Intermediate Size
5,120
Number of Layers
48
Attention Heads
16
KV Heads
2
Head Dimension
256
Context & Position
Max Context Length
262,144
Layer Attention Types
[48 items]
Uses Sliding Window
No
Attention Configuration
Tied Embeddings
No
Attention Bias
No
Attention Dropout
0%
Mixture of Experts
MoE Layer Frequency
1
Expert FFN Size
512
Shared Expert FFN Size
512
Experts per Token
10
Number of Experts
512
Normalize TopK Probabilities
Yes
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-06
Special Tokens
Pad Token ID
Not set
BOS Token ID
151,643
EOS Token ID
151645
Data Type
Model Dtype
bfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All