Candle: Rust-Native Machine Learning

Python dominates machine learning. But production systems need performance Python cannot deliver. Today we are sharing our investment in Candle, a Rust-native ML framework.

Why Rust for ML

Production ML systems face constraints:

• Latency: Every millisecond matters
• Memory: GPU memory is expensive
• Reliability: No runtime crashes acceptable
• Deployment: Simple binaries preferred

Python excels at research. Rust excels at production.

What is Candle

Candle is a minimalist ML framework in Rust:

use candle_core::{Device, Tensor};
use candle_nn::{Linear, Module};

// Create tensors
let device = Device::Cuda(0);
let x = Tensor::randn(0f32, 1., (32, 768), &device)?;

// Neural network layer
let linear = Linear::new(
    Tensor::randn(0f32, 1., (768, 256), &device)?,
    Some(Tensor::zeros((256,), DType::F32, &device)?)
);

// Forward pass
let output = linear.forward(&x)?;

Core Principles

Minimal Dependencies

Candle depends only on:

• safetensors: Model loading
• cudarc: CUDA bindings
• gemm: CPU matrix operations

No Python. No PyTorch. No heavy runtime.

Explicit Memory Management

// Memory is explicit, not garbage collected
let tensor = Tensor::new(&[1.0f32, 2.0, 3.0], &device)?;

// Clone is explicit
let tensor2 = tensor.clone();

// Drop is automatic when out of scope

Compile-Time Safety

Type errors caught at compile time:

// This won't compile - shape mismatch
let a = Tensor::zeros((32, 768), DType::F32, &device)?;
let b = Tensor::zeros((512, 768), DType::F32, &device)?;
let c = a.matmul(&b)?; // Compile error: incompatible shapes

Model Support

Candle supports major architectures:

Transformers

use candle_transformers::models::llama::{Llama, LlamaConfig};

let config = LlamaConfig::load("config.json")?;
let model = Llama::load("model.safetensors", &config, &device)?;

let output = model.forward(&input_ids)?;

Supported models:

• LLaMA / Mistral
• Falcon
• Phi
• Stable Diffusion
• Whisper

Custom Models

struct MyModel {
    embedding: Embedding,
    layers: Vec<TransformerBlock>,
    output: Linear,
}

impl Module for MyModel {
    fn forward(&self, x: &Tensor) -> Result<Tensor> {
        let mut hidden = self.embedding.forward(x)?;
        for layer in &self.layers {
            hidden = layer.forward(&hidden)?;
        }
        self.output.forward(&hidden)
    }
}

Performance

Benchmarks (LLaMA-7B, single A100):

Framework	Tokens/sec	Memory	Startup
PyTorch	45	14GB	8s
vLLM	120	13GB	12s
Candle	95	11GB	0.5s

Candle advantages:

• 20% less memory
• 16x faster startup
• Single binary deployment

Deployment

Binary Deployment

# Build release binary
cargo build --release

# Deploy single file
scp target/release/inference server:/app/

# Run
./inference --model llama-7b --port 8080

No Python environment. No dependency management. One file.

WebAssembly

// Compile to WASM
// wasm-pack build --target web

#[wasm_bindgen]
pub fn inference(input: &str) -> String {
    let model = load_model();
    model.generate(input)
}

Run ML in the browser with near-native performance.

Embedded

Candle runs on resource-constrained devices:

// Quantized model for edge deployment
let model = Model::load_quantized("model_q4.gguf")?;

// Runs on Raspberry Pi, mobile devices

Hanzo Integration

Candle powers:

• Edge inference: Client-side embeddings
• Real-time scoring: Sub-millisecond predictions
• Batch processing: High-throughput pipelines

Example: Real-time fraud scoring

// 50,000 transactions/second
// < 1ms latency
// Single server

Contributing

Candle is open source. We contribute:

• Quantization kernels
• New model architectures
• Performance optimizations
• Documentation

Repository: github.com/huggingface/candle

What's Next

Our Candle roadmap:

• Custom CUDA kernels for commerce models
• Distributed inference support
• Model compilation and optimization
• iOS/Android deployment targets

The future of production ML is Rust. Candle makes it accessible.

---

Zach Kelling is the founder of Hanzo Industries.