Killing Raft: from ZooKeeper consensus to Lux Quasar, natively

Hanzo Datastore coordinates replicated tables today the way ClickHouse does: an embedded, ZooKeeper-compatible Keeper running Raft inside the server process. It works. It is also the last classical, leader-based, pre-quantum component in the stack. This post is how we kill it — and why "kill ZK" does not mean rewriting replication.

Decomplecting Keeper: API versus engine

The instinct on hearing "replace ZooKeeper" is to imagine touching every Replicated*MergeTree. You don't, because ClickHouse Keeper is already two separable things braided into one daemon:

1. The ZooKeeper API — a linearizable key-value tree with sequential nodes, versions, and watches. In the source this is KeeperStorage, wrapped by KeeperStateMachine (src/Coordination/KeeperStateMachine.h, class IKeeperStateMachine : public nuraft::state_machine). This is the contract. Every replicated table, every system.zookeeper query, every leader election speaks it.

2. The consensus engine — the thing that takes a stream of write requests, gets a quorum of nodes to agree on their order, and then calls commit(log_idx, data) on the state machine in that agreed order. Today that is NuRaft: KeeperServer::launchRaftServer (src/Coordination/KeeperServer.cpp) constructs a nuraft::raft_server, an asio_service, a KeeperStateManager, and a KeeperLogStore.

The clean observation — the Rich Hickey one — is that KeeperStateMachine is already written against the generic consensus interface (nuraft::state_machine: pre_commit, commit, rollback, snapshots). It does not know or care that Raft is underneath. The engine is a value you can swap, not a place welded into the API.

So: keep the state machine, replace the engine. The ZooKeeper API that replication depends on survives untouched. NuRaft, the leader election, and the requirement to run a separate Raft quorum are what die.

What replaces it

The engine becomes Quasar — the leaderless, DAG-based, post-quantum consensus from Lux. Quasar's job is identical to Raft's job here: agree on the order of a log and signal when an entry is final. The difference is how it agrees — no single leader to fail over, ML-DSA signatures instead of classical ones, and a confidence-threshold finality rather than a majority-of-a-fixed-term.

Crucially, Quasar is not a Go service we shell out to. It ships a native C engine with a C++-ready ABI (libluxconsensus). The Datastore server is C++. So the engine links into the same process — no sidecar, no network hop to a coordinator, no second language at runtime.

The seam is proven — against the real state machine

This is not a sketch, and it is no longer a model. We built a program that links the actual ClickHouse server library (libdbms.a) and drives the production KeeperStateMachine<KeeperMemoryStorage> — the exact class ReplicatedMergeTree coordinates through — entirely on Quasar, with NuRaft nowhere in the ordering path. It lives in the Datastore tree at src/Coordination/examples/keeper_quasar_poc.cpp and is built+run on a real Linux host (aarch64, clang-21).

The flow is faithful to how the dispatcher drives NuRaft today — two phases, preprocess when a request is appended and commit when consensus finalizes its order:

state_machine->preprocess(rfs, /*lock_mutex*/ true);     // append-time, pre-consensus (≙ nuraft pre_commit)
auto buf = IKeeperStateMachine::getZooKeeperLogEntry(rfs); // the real ZooKeeper log entry
lux_chain_add_block(chain, &blk);   // the entry rides Quasar (the slot raft_server occupies)
lux_consensus_process_vote(...);    // validators agree — no leader
lux_consensus_is_accepted(chain, id, &accepted);
state_machine->commit(height, *buf); // finality → the real KeeperStorage mutates, in order

Run against a ten-request ReplicatedMergeTree coordination sequence — replica registration, block-number bumps, leader election — it preprocesses every request, orders them through Quasar, finalizes, commits to the real KeeperStorage, and asserts the resulting tree is exactly correct, applied once each, in consensus order:

== REAL Keeper state machine reached consensus on Quasar — zero Raft, zero ZK ensemble ==

Beneath it sits the minimal C-ABI demonstration (pkg/c/examples/cpp_consumer.cpp, make test-cpp) that links libluxconsensus and finalizes a single block — the same engine, without the ClickHouse dependency. Both are reproducible; the Datastore example is opt-in (-DENABLE_EXAMPLES=1 -DLUXCONSENSUS_DIR=...) and never part of a normal build.

Map it onto Keeper one-to-one:

Keeper / NuRaft today	Quasar-native
nuraft::raft_server orders the log	lux_chain_t orders the log
KeeperLogStore / Changelog	Quasar's DAG log
Raft leader election	leaderless confidence voting
state_machine->commit(idx, data)	same call, fired on is_accepted
classical signatures, fixed-term leader	ML-DSA signatures, no leader
separate ZK/Keeper quorum to operate	consensus in-process, in the server

KeeperStateMachine::commit is the join point. NuRaft calls it today; the Quasar driver calls the identical method tomorrow. The state machine cannot tell the difference — which is exactly the property that makes this a swap and not a rewrite.

Why this matters beyond "post-quantum"

• No leader. Raft has a single elected leader per term; its failure stalls writes until re-election. Quasar's leaderless agreement removes that head-of-line dependency.
• No second quorum to run. Coordination stops being a separate system you provision, monitor, and back up. It is the server.
• Post-quantum by construction. The signatures protecting agreement are ML-DSA, not classical curves, so a future quantum adversary cannot forge consensus messages.
• One transport story. It composes with the rest of the direction — native TCP/HTTP now, ZAP next, and the same Lux primitives the chains use.

Honest status

The architecture above is real, and the proof now runs against the production KeeperStateMachine — not a model — built and passing on a real Linux host. What remains is the in-engine swap: introducing the Quasar consensus backend inside KeeperServer alongside NuRaft, making it the default, and then carrying the rest of what KeeperServer does — snapshots, recovery, session expiry, reconfiguration — onto it with the full Coordination test suite green. That is real work, and we are not going to tell you Raft is gone the day we proved the commit path. We are telling you the seam holds against the real types, the join point is KeeperStateMachine::commit, and that is the whole game.

When it lands, contrib/NuRaft/ and the Raft engine leave the tree. The ZooKeeper API stays as long as replication speaks it. That is how you kill the last pre-quantum component without touching a single MergeTree.

• Datastore: github.com/hanzoai/datastore
• Lux consensus: github.com/luxfi/consensus