← All Models | GPT - OpenAI's Generative Pre-trained Transformers:

openai-community/openai-gpt

📊 Model Parameters

Total Parameters 147,621,888
Context Length 512
Hidden Size 768
Layers 12
Attention Heads 12
KV Heads 12

💾 Memory Requirements

FP32 (Full) 563.1 MB
FP16 (Half) 281.6 MB
INT8 (Quantized) 140.8 MB
INT4 (Quantized) 70.4 MB

🔑 KV Cache (Inference)

Per Token (FP16) 36.86 KB
Max Context FP32 36.0 MB
Max Context FP16 18.0 MB
Max Context INT8 9.0 MB

⚙️ Model Configuration

Core Architecture

Vocabulary Size40,478
Number of Layers12
Attention Heads12

Context & Position

Max Context Length512

Attention Configuration

Attention Dropout10.0%
Tied EmbeddingsYes

Activation & Normalization

Activation Functiongelu
RMSNorm Epsilon1e-05

Dropout (Training)

Residual Dropout10.0%
Embedding Dropout10.0%

Special Tokens

BOS Token IDNot set
Pad Token IDNot set
EOS Token IDNot set

Data Type

Model DtypeNot set
Layer Types:
Attention
MLP/FFN
Normalization
Embedding