Model Architecture: moonshotai/Kimi-K2-Thinking

📊 Model Parameters

Total Parameters 39,064,524,800

Context Length 262,144

Hidden Size 7168

Layers 61

Attention Heads 64

KV Heads 64

💾 Memory Requirements

FP32 (Full) 145.53 GB

FP16 (Half) 72.76 GB

INT8 (Quantized) 36.38 GB

INT4 (Quantized) 18.19 GB

🔑 KV Cache (Inference)

Per Token (FP16) 1.75 MB

Max Context FP32 854.00 GB

Max Context FP16 427.00 GB

Max Context INT8 213.50 GB

⚙️ Model Configuration

Core Architecture

Vocabulary Size163,840

Hidden Size7,168

FFN Intermediate Size18,432

Number of Layers61

Attention Heads64

KV Heads64

Context & Position

Max Context Length262,144

RoPE Base Frequency50000.0

Attention Configuration

Attention BiasNo

Attention Dropout0%

Tied EmbeddingsNo

Multi-Head Latent Attention

KV LoRA Rank512

Query LoRA Rank1,536

QK RoPE Head Dimension64

Value Head Dimension128

QK Non-RoPE Head Dimension128

Mixture of Experts

Expert FFN Size2,048

Shared Experts1

Number of Experts384

Routing Scale Factor2.827

TopK Methodnoaux_tc

Expert Groups1

Groups per Token1

Experts per Token8

MoE Layer Frequency1

Dense Initial Layers1

Normalize TopK ProbabilitiesYes

Router Scoring Functionsigmoid

Speculative Decoding

Next-N Prediction Layers0

Activation & Normalization

Activation Functionsilu

RMSNorm Epsilon1e-05

Special Tokens

Pad Token ID163,839

BOS Token ID163,584

EOS Token ID163586

Data Type

Model Dtypebfloat16

Layer Types:

Attention

MLP/FFN

Normalization

Embedding