← All Models | EleutherAI - Non-profit AI research lab focused on open-source LLMs:

EleutherAI/gpt-neo-1.3B

📊 Model Parameters

Total Parameters 1,418,502,144
Context Length 2,048
Hidden Size 2048
Layers 24
Attention Heads 16
KV Heads 16

💾 Memory Requirements

FP32 (Full) 5.28 GB
FP16 (Half) 2.64 GB
INT8 (Quantized) 1.32 GB
INT4 (Quantized) 676.4 MB

🔑 KV Cache (Inference)

Per Token (FP16) 196.61 KB
Max Context FP32 768.0 MB
Max Context FP16 384.0 MB
Max Context INT8 192.0 MB

⚙️ Model Configuration

Core Architecture

Vocabulary Size50,257
Hidden Size2,048
Number of Layers24
Attention Heads16
FFN Intermediate SizeNot set

Context & Position

Max Context Length2,048
Sliding Window Size256

Attention Configuration

Attention Dropout0%
Tied EmbeddingsYes

Activation & Normalization

Activation Functiongelu_new
RMSNorm Epsilon1e-05

Dropout (Training)

Residual Dropout0%
Embedding Dropout0%

Special Tokens

BOS Token ID50,256
EOS Token ID50256
Pad Token IDNot set

Data Type

Model DtypeNot set
Layer Types:
Attention
MLP/FFN
Normalization
Embedding