← All Models | GPT - OpenAI's Generative Pre-trained Transformers:

openai-community/gpt2-medium

📊 Model Parameters

Total Parameters 406,286,336
Context Length 1,024
Hidden Size 1024
Layers 24
Attention Heads 16
KV Heads 16

💾 Memory Requirements

FP32 (Full) 1.51 GB
FP16 (Half) 774.9 MB
INT8 (Quantized) 387.5 MB
INT4 (Quantized) 193.7 MB

🔑 KV Cache (Inference)

Per Token (FP16) 98.30 KB
Max Context FP32 192.0 MB
Max Context FP16 96.0 MB
Max Context INT8 48.0 MB

⚙️ Model Configuration

Core Architecture

Vocabulary Size50,257
Number of Layers24
Attention Heads16
FFN Intermediate SizeNot set

Context & Position

Max Context Length1,024

Attention Configuration

Attention Dropout10.0%
Tied EmbeddingsYes

Activation & Normalization

Activation Functiongelu_new
RMSNorm Epsilon1e-05

Dropout (Training)

Residual Dropout10.0%
Embedding Dropout10.0%

Special Tokens

BOS Token ID50,256
EOS Token ID50256
Pad Token IDNot set

Data Type

Model DtypeNot set
Layer Types:
Attention
MLP/FFN
Normalization
Embedding