ZHA uses a hierarchical tree to route attention to a sparse subset of a long context.
The adaptation uses a separate distillation phase; the billion-token figure concerns training budget, not tested context.
What happened?
Zyphra published ZHA, or Zyphra Hierarchical Attention, on November 2 alongside arXiv paper 2610.42112. The method addresses a long-context problem: standard attention compares each query with the full preceding context, while many sparse methods still pay a large cost searching a flat index for relevant tokens. ZHA builds a tree of compressed key summaries and searches it from coarse regions to finer entries, expanding multiple relevant branches rather than scanning every token at the same resolution.
The paper describes ZHA as subquadratic under suitable routing assumptions. It keeps the original key-value cache at the leaves while using learned reference queries to estimate which regions are relevant. Top-p routing makes retrieved information adaptive: concentrated queries can inspect fewer branches, while diffuse queries can expand more. The goal is not a fixed memory that forgets useful information, but a searchable structure whose active work grows more slowly than the full context.
The adaptation claim is important. Zyphra says existing pretrained models can be converted with a separate KL-distillation phase while the backbone remains frozen. The paper and company page describe a training budget on the order of a few billion tokens. That is training/distillation data, not a billion-token context window tested end to end.
What the experiments show—and what they do not
The reported evaluations reach long but much smaller contexts, including 64K, 128K and 256K settings in the research discussion. The authors report performance on retrieval, reasoning, mathematical and agentic coding tasks, and compare ZHA variants with dense attention and other sparse approaches. The results are evidence that the tree can preserve useful retrieval while reducing active work, but remain research evaluations from the authors rather than independent replication.
The headline billion-token numbers come from a performance model. Zyphra projects large speedups over dense attention and MSA at one billion tokens, but the paper does not report an end-to-end model run with a one-billion-token input. A projection can be valuable for deciding whether an architecture is worth building, yet it is not a measurement. Memory movement, tree construction, kernels, batching, quality degradation and pathological retrieval patterns can all change the result outside the tested range.
The method depends on geometry and routing choices. ZHA can expand more than one branch, and its active set is shaped by the top-p threshold. If relevant information is diffuse or summaries rank regions poorly, the method may retrieve much more context. The asymptotic claim is conditional on the learned index and routing behavior, not a guarantee that every long prompt receives the same speedup.
Source: Zyphra ↗ · Source: arXiv ↗ · Source: arXiv ↗ · Source: MarkTechPost ↗
Why does it matter?
Long context is often discussed as if the only question were the maximum number in a model card. ZHA points to a more useful systems question: how much of a huge context must a model actively search for each query, and how can it find that subset without paying the full scan cost? A hierarchy is an intuitive answer because information retrieval moves from broad structure to local detail. If summaries remain faithful, a model can keep access to a large store of keys while doing less work on each query.
The practical value depends on preserving the parts of dense attention users rely on. A system that is fast at one billion tokens but misses a relevant clause is not a successful replacement for a slower system that finds it. Teams evaluating such a method should measure retrieval recall, task accuracy, latency, memory and failure cases at each context length, then compare against a dense or strong sparse baseline. The 256K-class results are a better near-term reference point than the billion-token projection.
The adaptation path is the other meaningful result. Distilling routing components into a frozen pretrained model could be easier than retraining a frontier model from scratch, but it is still training of new attention behavior. Saying “without retraining” would hide the cost and potential quality changes. Zyphra’s evidence supports a narrower claim: hierarchical attention may make very long contexts more tractable, and the authors show a small adaptation budget relative to the base model. Billion-token context remains a future possibility, not a demonstrated capability.
Source: Zyphra ↗ · Source: arXiv ↗ · Source: arXiv ↗ · Source: Zyphra ↗ · Source: MarkTechPost ↗
Read the sources
- ZHA: Zyphra Hierarchical Attention ↗
Zyphra · November 2, 2026 - Subquadratic Attention paper ↗
arXiv · November 2, 2026 - Experimental HTML paper ↗
arXiv · November 2, 2026 - Zyphra announcement thread ↗
Zyphra · November 2, 2026 - Independent ZHA overview ↗
MarkTechPost · November 2, 2026
