← All Models | Other Chinese/Asian AI Labs:

XiaomiMiMo/MiMo-7B-Base

📊 Model Parameters

Total Parameters 7,833,409,536
Context Length 32,768
Hidden Size 4096
Layers 36
Attention Heads 32
KV Heads 8

💾 Memory Requirements

FP32 (Full) 29.18 GB
FP16 (Half) 14.59 GB
INT8 (Quantized) 7.30 GB
INT4 (Quantized) 3.65 GB

🔑 KV Cache (Inference)

Per Token (FP16) 147.46 KB
Max Context FP32 9.00 GB
Max Context FP16 4.50 GB
Max Context INT8 2.25 GB

⚙️ Model Configuration

Core Architecture

Vocabulary Size151,680
Hidden Size4,096
FFN Intermediate Size11,008
Number of Layers36
Attention Heads32
KV Heads8
Head Dimension128

Context & Position

Max Context Length32,768
Uses Sliding WindowNo
Sliding Window SizeNot set
Window Attention Layers32
Layer Attention Types[36 items]

Attention Configuration

Attention Dropout0%
Tied EmbeddingsNo
Attention BiasYes

Speculative Decoding

Next-N Prediction Layers1

Activation & Normalization

Activation Functionsilu
RMSNorm Epsilon1e-05

Special Tokens

Pad Token IDNot set
BOS Token IDNot set
EOS Token IDNot set

Data Type

Model Dtypebfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding