Dense
Google · 2026-04
Gemma 4 (E2B)
Dense decoder architecture with MQA + QK-Norm + SWA attention mechanism.
Gemma 4 (E2B) decoder block architecture: Attention: MQA + QK-Norm + SWA with QK-Norm with Sliding Window Attention. Normalization: RMSNorm. FFN: SwiGLU. Position encoding: RoPE. Scale: 5.1B, 128K context, 24 layers. Decoder type: Dense.
MQA + QK-Norm + SWA·SwiGLU
5.1B|128K context|MQA + QK-Norm + SWA|Dense
Architecture Specifications
Parameters5.1B
Context Window128K
Decoder TypeDense
AttentionMQA + QK-Norm + SWA
Vocabulary Size262K
Release Date2026-04
CategoryEfficient & Small
OrganizationGoogle
Key Features
Effective 2.3B parametersMQA efficiencyOn-device
Enterprise AI platform
Compare, evaluate, and deploy LLM architectures at scale
Colaberry AI provides architecture specifications, benchmark comparisons, and deployment guidance for enterprise AI teams.