An attention-free model offers a research direction—not a speed breakthrough

What happened?

A December 2025 preprint by Zhang Chong drew fresh discussion on Hacker News on October 25. It proposes an attention-free Causal Grassmann architecture that mixes information through local, low-dimensional geometric features rather than explicitly building a matrix of weights for all token pairs. The author presents it as an alternative design direction, not a claim that attention is obsolete.

The paper reports language-model experiments with roughly 13–18 million parameters on Wikitext-2, where its validation perplexity trails size-matched Transformer baselines. Its slightly stronger SNLI result concerns a different classification head on a shared DistilBERT backbone; it does not demonstrate an entirely attention-free replacement for that backbone.

Source: Zhang Chong / arXiv ↗ · Source: Zhang Chong / arXiv ↗ · Source: Hacker News ↗

Linear scaling is not measured speed

The complexity argument assumes a fixed reduced dimension and a fixed number of mixing windows as sequence length grows. The author also reports that the implementation is slower per step than the Transformer baseline for the tested sequence lengths up to 256. The paper describes the work as a proof of concept requiring further implementation improvements, not an optimized engineering solution.

Source: Zhang Chong / arXiv ↗

Why does it matter?

Our analysis: alternative architectures are worth examining precisely because their trade-offs differ. A mathematical scaling advantage can motivate research even before a prototype wins on a GPU, but it should not be advertised as a delivered reduction in the cost of running today’s large models.

The next useful tests would keep quality, training conditions and implementation effort comparable, then extend the comparison to longer contexts and more demanding tasks. Until then, the contribution is narrower and more credible: a geometric mixing rule that supports nontrivial small-scale sequence modeling, with clear gaps between the theoretical motivation, the benchmark results and production readiness.