← All Models
|
architecture:
Trinity-Nano-Base
Trinity-Mini
Trinity-Large-TrueBase
arcee-ai/Trinity-Nano-Base
📊 Model Parameters
Total Parameters
6,120,003,328
Context Length
131,072
Hidden Size
1024
Layers
56
Attention Heads
8
KV Heads
2
💾 Memory Requirements
FP32 (Full)
22.80 GB
FP16 (Half)
11.40 GB
INT8 (Quantized)
5.70 GB
INT4 (Quantized)
2.85 GB
🔑 KV Cache (Inference)
Per Token (FP16)
57.34 KB
Max Context FP32
14.00 GB
Max Context FP16
7.00 GB
Max Context INT8
3.50 GB
⚙️ Model Configuration
Core Architecture
Vocabulary Size
200,192
Hidden Size
1,024
FFN Intermediate Size
3,072
Number of Layers
56
Attention Heads
8
Head Dimension
128
KV Heads
2
Context & Position
Max Context Length
131,072
RoPE Base Frequency
10,000
Sliding Window Size
2,048
Layer Attention Types
[56 items]
Attention Configuration
Attention Bias
No
Attention Dropout
0%
Tied Embeddings
No
Mixture of Experts
Expert FFN Size
256
Experts per Token
8
Number of Experts
128
Expert Groups
1
Groups per Token
1
Activation & Normalization
Activation Function
silu
RMSNorm Epsilon
1e-05
Special Tokens
EOS Token ID
Not set
Pad Token ID
Not set
BOS Token ID
Not set
Data Type
Model Dtype
bfloat16
Layer Types:
Attention
MLP/FFN
Normalization
Embedding
Attention
MLP
Norm
Embedding
Clear
Expand All
Collapse All