Zen: 1T+ Parameter MoE Architecture

Zen is Hanzo's flagship language model family. The largest model, zen4-ultra, is a 1.04 trillion parameter Mixture of Experts (MoE) architecture that activates 140B parameters per forward pass. It is the culmination of three years of model development, from zen-nano (600M) to the current zen4 family.

This post describes the architecture, the training methodology, and the on-chain verification system that makes Zen models auditable.

Architecture

Zen4-ultra uses a sparse Mixture of Experts architecture. The core design:

• Total parameters: 1.04T
• Active parameters per token: ~140B (13.5% activation ratio)
• Experts: 128 experts per MoE layer, top-8 routing
• Layers: 96 transformer layers, every other layer is MoE (48 MoE layers, 48 dense)
• Context window: 256K tokens (YaRN positional interpolation)
• Vocabulary: 152K tokens (multilingual BPE)

The MoE routing uses a learned gating network with auxiliary load-balancing loss to prevent expert collapse. Each expert is a standard FFN (feed-forward network) with SwiGLU activation. The router selects 8 of 128 experts per token — enough diversity to capture complex reasoning patterns while keeping compute tractable.

Why MoE

Dense models scale linearly: 2x parameters = 2x compute per token. MoE breaks this relationship. Zen4-ultra has 7.4x the total parameters of a 140B dense model but the same inference cost per token. The extra parameters provide capacity — the model stores more knowledge and handles more diverse tasks — without proportional compute increase.

The tradeoff is memory. All 1.04T parameters must be loaded, even though only 140B are active. This requires distributed inference across multiple nodes, which is where the ACI network becomes essential.

The Zen4 Family

Model	Total Params	Active Params	Architecture
zen4-mini	4B	4B	Dense
zen4-base	8B	8B	Dense
zen4-pro	14B	14B	Dense
zen4-max	30B	8B	MoE (32 experts, top-2)
zen4-coder	80B	20B	MoE (64 experts, top-4)
zen4-ultra	1.04T	140B	MoE (128 experts, top-8)

Every model in the family shares the same tokenizer, attention implementation, and training data pipeline. The dense models (mini through pro) are distilled from ultra, not trained independently. This ensures behavioral consistency across the family.

Model Fusion

Zen models are not trained from scratch on random initialization. They are fused from top-performing open-weight models using a methodology we call structured knowledge transfer.

The base architecture is Qwen3. Every Zen model starts from a Qwen3 checkpoint — this is a hard governance constraint under Zoo Foundation policy (Qwen3+ only). The fusion process:

1. Architecture alignment: Map source model architectures to the Zen target architecture. For MoE models, this includes expert initialization from dense model layers using the DROP upcycling technique (described in our DROP post).

2. Knowledge distillation: Train the Zen model on a mixture of (a) raw pretraining data and (b) teacher outputs from source models. The teacher ensemble includes Qwen3, DeepSeek, and Mistral checkpoints selected by benchmark performance on target capabilities.

3. Expert specialization: After initial fusion, individual experts are fine-tuned on domain-specific data — mathematics, code, scientific reasoning, multilingual text — to encourage expert specialization. The router learns to route domain-specific tokens to domain-specialized experts.

4. Behavioral alignment: DSO (Decentralized Semantic Optimization) experience entries from the Zoo Foundation ledger are used for final behavioral tuning. This is the community governance layer — the behaviors Zen exhibits are determined by the entries the community has voted into the ledger.

Distributed Training

Zen4-ultra training runs on Hanzo Network compute clusters — bare-metal nodes with 8x H100 GPUs each, connected via InfiniBand. The training infrastructure:

• Framework: PyTorch with FSDP (Fully Sharded Data Parallel) for dense layers, custom expert-parallel sharding for MoE layers
• Nodes: 512 nodes (4096 H100 GPUs) for the full ultra training run
• Training time: 14 days for the final fusion + specialization phase
• Data: 15T tokens, deduplicated, quality-filtered, multilingual (97 languages)
• Checkpointing: Every 1000 steps, uploaded to Hanzo Storage with content-addressed hashing

The MoE expert-parallel strategy places each expert on a single GPU and uses all-to-all communication for token routing. This is more communication-intensive than data parallelism but avoids the memory overhead of replicating all experts on every GPU.

On-Chain Model Attestation

Every Zen model checkpoint has an on-chain attestation on the Hanzo Network. The attestation includes:

• Model hash: SHA-256 of the complete weight tensor, computed deterministically from the canonical safetensors serialization
• Training config hash: SHA-256 of the full training configuration (hyperparameters, data mixture, compute allocation)
• Benchmark results: Standardized benchmark scores (MMLU, HumanEval, MATH, etc.) computed by independent evaluators on the Zoo Foundation evaluation committee
• TEE attestation: Intel TDX or AMD SEV attestation from the training nodes, proving the training ran on the claimed hardware with the claimed code

The attestation contract is deployed on the Hanzo Network at a well-known address. Anyone can verify that a given set of model weights matches the attested hash, and that the benchmark results were produced by the evaluation committee rather than self-reported.

This does not prevent all forms of gaming — benchmark contamination in training data remains hard to detect — but it provides a verifiable chain from training hardware to published weights to claimed performance. If the hash does not match, the weights have been modified. If the TEE attestation is invalid, the training environment was different from what was claimed.

Inference on ACI

Zen4-ultra requires approximately 2TB of memory to load all parameters in FP16. No single node has this much GPU memory. Inference is distributed across ACI compute nodes using tensor parallelism:

• 8-way tensor parallel across H100 GPUs on a single node for the dense layers
• Expert parallel across nodes for MoE layers — each node holds a subset of experts
• Speculative decoding with zen4-mini as the draft model, achieving 3.2x throughput improvement on long-form generation

End-to-end latency for a 256-token generation: 1.8 seconds time-to-first-token, 45 tokens/second thereafter. Competitive with dense models at the same active parameter count, despite the distributed overhead.

Open Weights

All Zen models are released under Apache 2.0 on HuggingFace and the hanzoai/zen repository. Weights, tokenizer, training configs, and evaluation scripts are included. The open weights mandate is a structural requirement of Zoo Foundation governance — it is written into the organizational charter, not a discretionary policy.

Links

• Models: huggingface.co/zenlm
• Source: github.com/hanzoai/zen
• Attestations: explore.hanzo.network — on-chain model attestation records
• Benchmarks: zenlm.org/benchmarks
• DROP upcycling: hanzo.ai/blog/drop-upcycling-moe

---

Hanzo AI is Techstars '17. Zen models: zenlm.org. Hanzo: hanzo.ai.