Jin: Unified Multimodal AI Architecture

For years, Hanzo AI has been an ensemble of specialized models. Text understanding here. Image analysis there. Recommendations elsewhere. Today we are introducing Jin, a unified multimodal architecture that replaces the ensemble with a single, coherent intelligence.

The Ensemble Problem

Our previous architecture:

User Query
    ↓
┌─────────────────────────────────┐
│         Router                  │
│  "What kind of query is this?"  │
└─────────┬───────────────────────┘
          ↓
    ┌─────┴─────┬─────────┬─────────┐
    ↓           ↓         ↓         ↓
┌───────┐ ┌─────────┐ ┌───────┐ ┌───────┐
│ Text  │ │ Vision  │ │Search │ │Recomm │
│ Model │ │  Model  │ │ Model │ │ Model │
└───┬───┘ └────┬────┘ └───┬───┘ └───┬───┘
    └──────────┴──────────┴─────────┘
                    ↓
            ┌──────────────┐
            │  Aggregator  │
            └──────────────┘
                    ↓
               Response

Problems:

• Routing errors: Query classification determined model selection
• Information loss: Models could not share intermediate understanding
• Latency: Sequential model calls added latency
• Inconsistency: Different models gave different answers to similar questions

The Jin Architecture

Jin processes all modalities in a unified model:

User Query (text, image, data)
              ↓
    ┌─────────────────────┐
    │                     │
    │        Jin          │
    │                     │
    │  ┌───────────────┐  │
    │  │ Universal     │  │
    │  │ Encoder       │  │
    │  └───────┬───────┘  │
    │          ↓          │
    │  ┌───────────────┐  │
    │  │ Reasoning     │  │
    │  │ Core          │  │
    │  └───────┬───────┘  │
    │          ↓          │
    │  ┌───────────────┐  │
    │  │ Multimodal    │  │
    │  │ Decoder       │  │
    │  └───────────────┘  │
    │                     │
    └─────────────────────┘
              ↓
    Response (any modality)

Architecture Details

Universal Encoder

All inputs convert to a shared representation:

• Text: Subword tokenization → embedding
• Images: Patch embedding → position encoding
• Structured data: Schema-aware encoding
• Time series: Temporal encoding

The encoder learns to align modalities. "Red shoes" and an image of red shoes produce similar representations.

Reasoning Core

A large transformer processes the unified representation:

• Attention across modalities: Text attends to image regions, data attends to text
• Tool integration: Model decides when to query external systems
• Memory: Context from previous interactions informs current reasoning

Multimodal Decoder

Output in any modality:

• Text: Natural language responses
• Structured: JSON, API calls, SQL
• Visual: Image generation (via diffusion head)
• Actions: Agent tool calls

Training

Jin trained on:

• 500B tokens of text
• 2B image-text pairs
• 100M commerce interactions
• 50M structured data examples

Training approach:

1. Pretraining: Large-scale multimodal pretraining
2. Domain adaptation: Commerce-specific fine-tuning
3. Instruction tuning: Following user instructions
4. RLHF: Alignment with human preferences

Capabilities

Unified Understanding

User: "Do you have anything like this but in blue?" [attaches image]

Jin: [Understands image shows a leather jacket]
     [Searches inventory for similar style]
     [Filters by blue color]

Response: "I found 3 leather jackets in blue that match that style..."

No routing. No separate models. One unified understanding.

Contextual Reasoning

User: "Why is this product more expensive than the other one?"

Jin: [Retrieves both product details]
     [Analyzes materials, brand, reviews]
     [Reasons about value differences]

Response: "The first product uses full-grain leather and includes
a lifetime warranty, while the second uses bonded leather..."

Multimodal Generation

User: "Generate a product description for this" [attaches image]

Jin: [Analyzes image for product attributes]
     [Generates appropriate description]
     [Ensures factual accuracy]

Response: "Handcrafted ceramic mug with a speckled blue glaze and
comfortable handle. Holds 12oz. Dishwasher and microwave safe."

Performance

Compared to our previous ensemble:

Metric	Ensemble	Jin	Improvement
Latency	450ms	180ms	60% faster
Accuracy	84%	91%	8% better
Consistency	76%	94%	24% better

API Access

Jin powers all Hanzo AI features. Direct access available:

from hanzo import Jin

jin = Jin(api_key="xxx")

response = jin.query(
    text="Find products similar to this image",
    images=[product_image],
    context={"category": "home_decor", "budget": 100}
)

What's Next

Jin 1.0 is the foundation. Coming:

• Longer context windows
• Real-time streaming responses
• Fine-tuning for specific use cases
• On-premises deployment option

Jin: one model, infinite modalities.

---

Zach Kelling is the founder of Hanzo Industries.