← All Models
|
Llama - Meta's Large Language Model Meta AI:
llama-65b
llama-30b
llama-13b
llama-7b
Llama-2-7b-hf
Llama-2-13b-hf
Llama-2-70b-hf
Meta-Llama-3-8B
Meta-Llama-3-70B
Llama-3.1-405B
huggyllama/llama-13b
📊 Model Parameters
Total Parameters
13,015,864,320
Context Length
2,048
Hidden Size
5120
Layers
40
Attention Heads
40
KV Heads
40
💾 Memory Requirements
FP32 (Full)
48.49 GB
FP16 (Half)
24.24 GB
INT8 (Quantized)
12.12 GB
INT4 (Quantized)
6.06 GB
🔑 KV Cache (Inference)
Per Token (FP16)
819.20 KB
Max Context FP32
3.12 GB
Max Context FP16
1.56 GB
Max Context INT8
800.0 MB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
32,000
Hidden Size
5,120
FFN Intermediate Size
13,824
Number of Layers
40
Attention Heads
40
KV Heads
40
Head Dimension
128
Context & Position
Max Context Length
2,048
RoPE Base Frequency
10000.0
RoPE Scaling
Not set
Attention Configuration
Attention Bias
No
Attention Dropout
0%
MLP Bias
No
Tied Embeddings
No
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-06
Special Tokens
BOS Token ID
1
Pad Token ID
0
EOS Token ID
2
Data Type
Model Dtype
float16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All