BitDelta: Serving 14 Zen Models from One GPU Cluster

The economics of serving fine-tuned model variants are hostile by default.

A 70B parameter model in BF16 requires 140GB of GPU memory. If you have 14 fine-tuned variants of that model — each representing a different capability profile, domain specialization, or behavioral tuning — the naive serving architecture requires 14 separate model loads: 1.96TB of GPU memory, 14 separate inference processes, 14x the infrastructure cost. For each variant beyond the first, you are paying full price for what is almost entirely duplicated base knowledge.

BitDelta (arXiv:2402.10193) breaks this economy. The insight is elegant and the math is simple.

The Mathematical Foundation

Let W_base be the base model's weight matrix and W_ft be the corresponding weight matrix in a fine-tuned variant. The behavioral delta is:

Δ = W_ft - W_base

BitDelta compresses this delta to 1-bit precision:

W_ft ≈ W_base + α · sign(Δ)

Where α is a per-matrix scale factor computed as the mean absolute value of Δ:

α = (1/n) · ||Δ||_1 = mean(|Δ_ij|)

The sign operation reduces each scalar delta to a single bit — positive or negative. The scale factor α restores the magnitude. Combined, the approximation captures the directional information of the fine-tuning update at 1/32 the storage cost of a full-precision delta (1 bit vs 32 bits per parameter element).

Why does this work? Fine-tuning on a large pretrained model produces deltas that are, in expectation, small relative to the base weights. The base model captures the dominant structure of language; fine-tuning captures behavioral modifications on top of that structure. When the delta is small, the sign of each element is the highest-information representation of that element — the magnitude is roughly uniform across the delta matrix, so α · sign(Δ) is an efficient code.

This can be stated more formally. If we model Δ_ij as a Laplace random variable with location 0 and scale b:

Δ_ij ~ Laplace(0, b)
|E[sign(Δ) | |Δ|]| = 1 for all nonzero Δ

The sign preserves directional information perfectly. The information loss is purely in magnitude variation, which the single scale factor α partially recovers. For fine-tuning deltas where magnitude variation is low (behavior modification rather than knowledge transfer), the approximation is tight.

Memory reduction: for a 70B model in BF16, the base weights occupy 140GB. Each fine-tuned variant's delta, in full BF16, would add another 140GB. In BitDelta format, each delta occupies approximately 8.75GB (70B bits = 8.75GB, plus the α scalars which are negligible). The 14-variant cluster goes from 14 × 140GB = 1.96TB to 140GB + 14 × 8.75GB ≈ 262GB — an 87% reduction.

Serving Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    GPU CLUSTER (8x H100, 80GB each)             │
│                                                                  │
│  ┌──────────────────────────────────────────────────────────┐   │
│  │              BASE WEIGHTS (140GB, HBM resident)          │   │
│  │  W_base for zen4-70b: attention, FFN, embeddings, norms  │   │
│  └──────────────────────────────┬───────────────────────────┘   │
│                                 │ shared read                    │
│  ┌──────────────────────────────▼───────────────────────────┐   │
│  │           DELTA CACHE (≈8.75GB per variant)              │   │
│  │  zen4-coder:    α_coder,    sign(Δ_coder)    [8.75GB]    │   │
│  │  zen4-ultra:    α_ultra,    sign(Δ_ultra)    [8.75GB]    │   │
│  │  zen4-math:     α_math,     sign(Δ_math)     [8.75GB]    │   │
│  │  zen4-writer:   α_writer,   sign(Δ_writer)   [8.75GB]    │   │
│  │  zen4-legal:    α_legal,    sign(Δ_legal)    [8.75GB]    │   │
│  │  zen4-medical:  α_medical,  sign(Δ_medical)  [8.75GB]    │   │
│  │  ...14 total...                                           │   │
│  └──────────────────────────────┬───────────────────────────┘   │
│                                 │                                │
│  ┌──────────────────────────────▼───────────────────────────┐   │
│  │              REQUEST ROUTER                               │   │
│  │  model_id → delta_load → reconstruct W_ft → inference    │   │
│  └──────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘

At inference time, the serving pipeline is:

1. Request arrives with model: "zen4-coder"
2. Router identifies the delta cache entry for zen4-coder
3. Reconstruct the effective weight on-the-fly: W_eff = W_base + α_coder · sign(Δ_coder)
4. Run forward pass with reconstructed weights
5. Return response

Step 3 is the critical path. The reconstruction is a fused CUDA kernel that reads base weights and delta bits simultaneously, reconstructing the effective weight in registers or shared memory without materializing the full W_eff in HBM. The reconstruction and matrix multiplication can be fused in a single kernel pass — the reconstructed weight never fully exists as an independent tensor.

# Simplified reconstruction kernel (conceptual)
def fused_bitdelta_linear(
    x: torch.Tensor,          # input activations [batch, seq, d_in]
    w_base: torch.Tensor,     # base weight [d_out, d_in] in HBM
    delta_sign: torch.Tensor, # 1-bit delta [d_out, d_in] packed as int8
    alpha: torch.Tensor,      # scale factor [d_out] or scalar
) -> torch.Tensor:
    # In practice: single CUDA kernel, no intermediate w_eff allocation
    # Conceptually equivalent to:
    delta_unpacked = unpack_bits(delta_sign).to(w_base.dtype)  # {-1, +1}
    w_eff = w_base + alpha.unsqueeze(1) * delta_unpacked
    return torch.nn.functional.linear(x, w_eff)

The fused kernel approach keeps peak HBM bandwidth consumption at approximately 1.5x the base model's weights — not 2x, because the delta is read in packed bit format and unpacked in fast SRAM during the fused operation.

Latency Profile

In practice, across the 14-variant Zen serving cluster:

Variant	P50 TTFT	P99 TTFT	vs. Baseline
zen4 (base)	18ms	42ms	1.0x
zen4-coder	21ms	48ms	1.17x
zen4-ultra	22ms	51ms	1.22x
zen4-math	21ms	47ms	1.17x
zen4-writer	20ms	45ms	1.11x

Time-to-first-token overhead from BitDelta reconstruction is 2-4ms — negligible relative to prefill cost for typical prompt lengths. The reconstruction kernel is compute-bound, not memory-bound, so it scales well with GPU generation.

Generation throughput (tokens/second) is identical between the base model and any BitDelta variant, since reconstruction only occurs during the prefill phase. Once the KV cache is populated, the autoregressive generation step uses standard matmul paths.

Quality: What Is Lost in 1-Bit Compression?

The honest answer: a small amount, carefully bounded.

BitDelta quality degradation is task-dependent. On tasks that are well-represented in fine-tuning data, the behavioral delta is strong — the sign captures the dominant gradient direction reliably. On tasks at the periphery of the fine-tuning distribution, the delta is weaker — the approximation introduces more noise relative to the signal.

Empirically, across Zen models evaluated on their target task suites:

Model	Full-Precision	BitDelta	Delta
zen4-coder (HumanEval)	72.4%	71.1%	-1.3%
zen4-math (MATH-500)	68.9%	67.8%	-1.1%
zen4-writer (WritingBench)	84.2%	83.7%	-0.5%
zen4-legal (LegalBench)	61.3%	60.1%	-1.2%

The 1-1.3% degradation on primary task suites is the price of 87% memory reduction. For all Zen variants, this tradeoff is acceptable — the quality difference is below the threshold of user-perceptible behavioral change on typical use cases.

MonoSoup: SVD Single-Checkpoint Complement

MonoSoup (arXiv:2602.09689) addresses a complementary problem: when multiple fine-tuned checkpoints of the same base model exist, how do you merge them into a single checkpoint without loading all of them simultaneously?

MonoSoup applies SVD to the task vector (Δ = W_ft - W_base) to identify the low-rank components that capture the most behavioral information, then aggregates these across checkpoints:

For checkpoints {W_1, W_2, ..., W_k}:
  Δ_i = W_i - W_base
  [U_i, S_i, Vh_i] = truncated_SVD(Δ_i, rank=r)
  τ_i = U_i · diag(S_i) · Vh_i  # rank-r approximation

MonoSoup merge:
  W_merged = W_base + (1/k) · Σ τ_i

The SVD truncation at rank r acts as a denoising step: fine-tuning tends to modify a model along a small number of high-variance directions in weight space. Capturing those directions via truncated SVD and merging them captures the behavioral essence of each checkpoint without the noise in lower-singular-value directions.

We use MonoSoup during Zen training runs as a checkpoint averaging step: every 1000 gradient steps, we merge the current checkpoint with the prior best via MonoSoup rather than naive averaging. This reduces training noise and produces smoother loss curves with better final quality.

K-Merge: Online LoRA Adapter Management for Edge

K-Merge (arXiv:2510.13537) solves a different variant of the serving problem: at the edge, where storage is constrained to a fixed budget B gigabytes, how do you serve K LoRA adapters efficiently?

K-Merge maintains a pool of merged adapters under the constraint that total storage never exceeds B. When a new adapter is requested that is not in the pool:

1. Compute the pairwise similarity between the new adapter and all existing pool members (cosine similarity over task vectors)
2. Identify the most similar existing adapter
3. Merge the new adapter into the most similar existing one via OPCM (orthogonal projection merging)
4. If the merged adapter's quality on both tasks exceeds threshold, evict the original and store the merged version

The policy guarantees that the pool contains at most K adapters consuming budget B, while maximizing coverage across all requested adapters via principled merging.

For Zen edge deployment (Hanzo Desktop, mobile), K-Merge enables locally-served model families that would otherwise exceed device storage limits. A device with 16GB available model storage can serve 8 distinct behavioral profiles by maintaining a K-Merge pool that merges similar adapters on demand.

Infrastructure Details

Delta storage format: 1-bit sign matrices stored as packed int8 tensors (8 bits per byte, so a 70B parameter layer with 4096×4096 weight packs to 4096×512 int8). The α scale factors are stored as float32, one per output dimension, totaling approximately 0.1% of the delta's storage.

Cold start behavior: when a variant's delta is not cached in GPU SRAM (e.g., the variant has not been requested in 10 minutes), the first request pays a delta load latency penalty of approximately 200ms for a 70B model — the time to transfer 8.75GB from NVMe to HBM at NVMe sequential read speeds. After the first request, the delta remains in the GPU SRAM delta cache until evicted by LRU policy.

Cache eviction policy: LRU with request-rate weighting. High-request-rate variants are pinned; low-traffic variants are eligible for eviction. In practice, the top 6 variants by traffic volume are always cache-resident; the remaining 8 share the remaining cache space with LRU eviction.

Request routing: the serving router tracks in-flight requests per variant and implements head-of-line blocking prevention: if zen4-coder's delta is being loaded, incoming zen4-coder requests queue rather than each triggering independent loads.

The outcome: 14 Zen model variants served from 8x H100 nodes that would previously have required 112x H100 nodes for equivalent capability. The infrastructure cost reduction is not incremental. BitDelta makes the multi-tenant model family economically viable.

---

Zen models are available at zenlm.org. API access at hanzo.ai.