Inside Ten Open-Weight LLMs: An Architecture Explainer
In the space of about three weeks last January and February, ten notable open-weight language models shipped: from a 3B model that runs on a laptop to two 1-trillion-parameter giants. Sebastian Raschka put all ten side by side in one figure, and it is one of the best snapshots I've seen of how modern LLMs are actually built. This post is a plain-language companion to that figure. It explains each building block once, then walks through what each model does with them, using numbers taken from the published model configs. The surprising takeaway is how few ideas there are. Almost every design is the same transformer with different choices on two dials: how sparse the feed-forward layers are, and how expensive attention is allowed to be.
Raschka's original write-up, A Dream of Spring for Open-Weight LLMs, and his LLM Architecture Gallery are worth reading alongside this. Where his figure and the configs disagree, I've gone with the config.json published with each model on Hugging Face.The ten models at a glance
| Model | Lab | Released | Total / active params | Layers | Attention design | Experts (routed + shared) | Context |
|---|---|---|---|---|---|---|---|
| Kimi K2.5 | Moonshot AI | Jan 27 | 1.03T / 32B | 61 | MLA (latent) | 8 of 384 + 1 | 256K |
| Ling 2.5 | Ant Group (inclusionAI) | Feb 16 | 1T / 63B | 80 | Lightning linear + MLA, 7:1 | 8 of 256 + 1 | 256K (1M with YaRN) |
| GLM-5 | Z.ai | Feb 11 | 744B / 40B | 78 | MLA + DeepSeek Sparse Attention | 8 of 256 + 1 | ~200K |
| Trinity Large | Arcee AI | Jan 27 | 398B / 13B | 60 | GQA, sliding window 3:1 | 4 of 256 + 1 | 256K |
| Qwen3.5 | Alibaba | Feb 16 | 397B / 17B | 60 | Gated DeltaNet + gated attention, 3:1 | 10 of 512 + 1 | 256K |
| MiniMax-M2.5 | MiniMax | Feb 12 | 230B / 10B | 62 | Full GQA | 8 of 256, no shared | 192K |
| Step 3.5 Flash | StepFun | Feb 1 | 199B / 11B | 45 | GQA, sliding window 3:1 | 8 of 288 + 1 | 256K |
| Qwen3-Coder-Next | Alibaba | Feb 3 | 80B / 3B | 48 | Gated DeltaNet + gated attention, 3:1 | 10 of 512 + 1 | 256K |
| Tiny Aya | Cohere Labs | Feb 17 | 3.35B dense | 36 | GQA, sliding window + global | (dense) | 8K |
| Nanbeige 4.1 | Nanbeige | Feb 10 | 3B dense | 32 | Full GQA | (dense) | 256K |
Part 1: The feed-forward half — mixture-of-experts
Dense vs. mixture-of-experts
In a classic "dense" model, every token goes through the same big feed-forward network. Tiny Aya and Nanbeige 4.1 still work this way. The other eight replace that one network with a mixture-of-experts (MoE) layer: hundreds of small feed-forward networks ("experts"), plus a tiny router that picks a handful of them for each token.That gives you two parameter counts. Total parameters decide how much knowledge the model can store and how much memory you need to hold it. Active parameters decide how much compute each token costs. MoE lets those two numbers drift far apart:| Model | Active share of total |
|---|---|
| Qwen3-Coder-Next | 3B of 80B ≈ 3.8% |
| Kimi K2.5 | 32B of 1.03T ≈ 3.1% |
| Trinity Large | 13B of 398B ≈ 3.3% |
| Qwen3.5 | 17B of 397B ≈ 4.3% |
| MiniMax-M2.5 | 10B of 230B ≈ 4.3% |
| GLM-5 | 40B of 744B ≈ 5.4% |
| Step 3.5 Flash | 11B of 199B ≈ 5.5% |
| Ling 2.5 | 63B of 1T ≈ 6.3% |
Many small experts
The trend is toward more, smaller experts. Qwen3.5 and Qwen3-Coder-Next have 512 experts each and route every token to 10 of them. Kimi K2.5 has 384. Each individual expert is small: Qwen3-Coder-Next's experts have a hidden size of just 512. Finer-grained experts let the router assemble a more specific combination for each token, a "team" of specialists instead of a few generalists.The shared expert
Seven of the eight MoE models also run one shared expert that every token goes through regardless of routing. The idea, popularized by DeepSeek, is that some knowledge (basic grammar, common patterns) is useful for every token. Giving it a permanent home frees the routed experts to specialize. MiniMax-M2.5 is the exception: it has 256 experts, picks 8, and has no shared expert at all.Dense layers first
Several models keep their first few blocks dense before switching to MoE: Trinity (6 blocks), Ling 2.5 (4), GLM-5 (3), Step 3.5 Flash (3) and Kimi K2.5 (1). Early layers handle low-level features that every token needs. Routing there tends to be unstable and saves little, so a plain feed-forward layer is simpler and trains more reliably.Part 2: The attention half — where the real variety is
Attention is how each token gathers information from earlier tokens. Standard attention compares every token with every other token, so its cost grows with the square of the context length. It also needs a KV cache, a stored copy of the "keys" and "values" for every past token. At 256K tokens, that cache, not the model weights, is often what fills your GPU. Every attention variant below is a different answer to the question "how much of that cost can we cut without hurting quality?"Grouped-query attention (GQA): the baseline
Attention has multiple "heads" that each look for different patterns. GQA lets several query heads share one set of keys and values, which shrinks the KV cache. MiniMax-M2.5 uses 48 query heads but only 8 key/value heads, a 6× smaller cache than giving every head its own. Nanbeige 4.1 uses 20 query heads and 4 key/value heads. In the figure, GQA is the "default" choice, and MiniMax-M2.5 and Nanbeige 4.1 use it in every layer, with no other tricks. Both are strong for their size, which is a useful reminder that plain designs still compete.Sliding-window attention: look only at recent tokens in most layers
Sliding-window attention (SWA) restricts a layer to the last N tokens rather than the whole context. The cost no longer grows with context length for that layer. Models interleave these "local" layers with occasional "global" layers that see everything, so information can still flow across long documents.- Trinity Large: 3 local layers for every global one (45 local, 15 global), with a 4,096-token window. Its global layers use NoPE (no positional encoding), relying on the local layers to supply position information. This tends to help the model generalize to longer contexts than it was trained on.
- Step 3.5 Flash: also 3:1, but with a very small 512-token window. To compensate, its local layers get more attention heads (96) than its global layers (64).
- Tiny Aya: Cohere's Command-A-style mix of 4,096-token sliding-window layers and global layers.
Multi-head latent attention (MLA): compress the cache
MLA, introduced by DeepSeek, takes a different approach. Instead of storing full keys and values, it stores a small compressed ("latent") vector per token and reconstructs keys and values from it when needed. The KV cache shrinks dramatically while every layer can still see the full context. Kimi K2.5, GLM-5 and Ling 2.5 all use MLA. Kimi K2.5 is essentially DeepSeek V3's architecture scaled up to 1T parameters.Sparse attention: only attend to the tokens that matter
GLM-5 adds DeepSeek Sparse Attention (DSA) on top of MLA. A lightweight "indexer" scans the context and picks the 2,048 most relevant tokens, and full attention runs only over those. At long context lengths, that turns quadratic cost into something close to linear while keeping full-attention quality for the tokens that count.Linear attention: replace most attention layers with a running memory
The most radical option drops standard attention in most layers. Linear attention keeps a fixed-size running state that is updated as each token arrives, much like a recurrent network. Cost per token stays constant no matter how long the context is, and there is no growing KV cache for those layers. The trade-off is that a fixed-size state can't recall everything precisely, so models keep a few full-attention layers to handle exact lookups.- Qwen3.5 and Qwen3-Coder-Next use Gated DeltaNet, a linear-attention layer with a learned "forget gate", in 3 of every 4 layers. The fourth is a gated full-attention layer. Qwen3.5's 60 layers split into 45 DeltaNet and 15 attention layers.
- Ling 2.5 goes further, with a 7:1 ratio of Lightning attention (another linear-attention variant) to MLA. Notably, Ant Group didn't train it that way from scratch: they converted Ling 2.0's standard attention into the hybrid through continued training, and pushed the usable context to 1M tokens.
Spectrum of attention cost
Arranged from most to least expensive at long context:- Full GQA in every layer: MiniMax-M2.5, Nanbeige 4.1
- Full MLA in every layer: Kimi K2.5
- Sliding window + periodic global: Trinity Large, Step 3.5 Flash, Tiny Aya
- MLA + sparse token selection: GLM-5
- Linear attention with periodic full attention: Qwen3.5, Qwen3-Coder-Next (3:1), Ling 2.5 (7:1)
Part 3: The small tweaks that add up
Gated attention
Several models add a learned gate on the attention output, a per-head switch that can dim or silence that head's contribution. Qwen3.5, Qwen3-Coder-Next, Step 3.5 Flash and Trinity Large all use some form of it. It improves training stability and helps avoid "attention sinks", where heads dump attention on the first token when they have nothing useful to attend to.QK-norm and sandwich norms
QK-norm normalizes queries and keys before they are compared, which prevents the attention scores from blowing up during training. MiniMax-M2.5, Step 3.5 Flash and Trinity Large use it. Trinity also uses a "sandwich" of normalization layers before and after each sub-layer. Both are cheap insurance for training very large models without loss spikes.Multi-token prediction (MTP)
Normally a model predicts one token at a time. MTP adds small extra heads that also guess the next few tokens. During training this gives a richer learning signal. At inference it can act as built-in speculative decoding: draft several tokens, verify them in one pass, and keep the ones that are right. GLM-5 and Qwen3.5 have an MTP layer, and MiniMax-M2.5's config ships three. Step 3.5 Flash stands out for using MTP-3 at inference to speed up generation, not just as a training aid.Parallel blocks
Tiny Aya runs attention and feed-forward in parallel within each block rather than one after the other, then adds both results back together. It is a GPT-J / Cohere Command idea that improves hardware utilization at small scale.Part 4: Model by model
Kimi K2.5 (Moonshot AI). DeepSeek V3's recipe at 1T scale: MLA in every layer, 384 experts with 8 routed plus 1 shared, and one dense block at the bottom. The text backbone is unchanged from Kimi K2. The headline change is native multimodality: a vision encoder fused in early and trained on roughly 15T mixed vision and text tokens.Ling 2.5 (Ant Group / inclusionAI). The largest active count of the group at 63B, up from 51B in Ling 2.0, across 80 layers with a hidden size of 8,192. Its 7:1 Lightning/MLA hybrid is the most aggressive bet on linear attention in the set, aimed at million-token contexts and higher throughput.GLM-5 (Z.ai). Z.ai switched to DeepSeek's playbook: MLA plus DeepSeek Sparse Attention, 256 experts, and a multi-token prediction layer. It is shallower than GLM-4.7 (78 vs. 92 layers) but wider, and sits at about 40B active out of 744B.Trinity Large (Arcee AI). A US-built entry with a Gemma-3-style local/global attention pattern, NoPE global layers, gated attention, QK-norm and sandwich norms, on a very sparse MoE: 4 of 256 experts plus a shared one, for about 13B active out of 398B.Qwen3.5 (Alibaba). The Qwen3-Next hybrid recipe (Gated DeltaNet plus gated attention at 3:1) scaled to a 397B flagship with 512 experts, a 248K-token vocabulary that suits its multilingual focus, and native multimodality.MiniMax-M2.5 (MiniMax). Deliberately conventional: full GQA everywhere, QK-norm, partial RoPE, 8 of 256 experts with no shared expert. It competes on cost, at about 10B active, rather than architectural novelty.Step 3.5 Flash (StepFun). Built for speed: a 512-token sliding window in three of every four layers, extra heads in the local layers, a head-wise output gate, and MTP-3 at inference. 288 experts, 8 routed plus 1 shared, at about 11B active.Qwen3-Coder-Next (Alibaba). The same hybrid architecture as Qwen3.5 in miniature: 80B total but only 3B active, with 512 tiny experts. It shows how far a linear-attention coder can go on a fraction of the compute.Tiny Aya (Cohere Labs). A 3.35B dense multilingual model covering about 70 languages, with Command-A-style parallel blocks and sliding-window layers. It ships as several region-tuned variants (Global, Earth, Fire, Water). Note the CC-BY-NC license: it is not for commercial use.Nanbeige 4.1 (Nanbeige). A Llama-3.2-style 3B model with full GQA (20 query and 4 key/value heads), untied embeddings and a 256K context. As Raschka points out, almost all of its gains come from training data, supervised fine-tuning and reinforcement learning, not architecture.What these ten tell us
- The transformer is the constant. None of these is a fundamentally new architecture. They are all GPT-lineage decoders with different choices for the feed-forward and attention halves. Most of the performance difference between them comes from data and post-training, not from the diagram.
- Sparsity is now the default at scale. Every model above about 50B parameters is MoE, typically with 3–6% of weights active, many small experts and one shared expert.
- Attention is where labs differentiate. Cheap-but-complete (GQA, MLA) and cheap-but-approximate (sliding windows, sparse selection, linear attention) all coexist. The hybrids, which mix a cheap layer type with a few full-attention layers, are clearly gaining ground, mainly because long-context agent workloads make the KV cache the bottleneck.
- Serving cost shapes the design. Partial RoPE, MTP at inference, tiny sliding windows and huge expert counts are all optimizations for tokens per dollar, not benchmark scores.
- Plain still works. MiniMax-M2.5 and Nanbeige 4.1 used nothing exotic and stayed competitive. A clever architecture can't replace a good training recipe.
Nine months later
Most of these labs have shipped successors since. In my LLM Landscape leaderboard, Kimi K3.1 Pro is now the strongest model you can download, and the open-weight tier also includes GLM-5.3, Qwen3.8, MiniMax-M3.1-Flash and Ling 3.1 Flash. The vocabulary in this post (active parameters, shared experts, latent, sliding-window, sparse and linear attention, MTP) is still how you read their model cards. When a new open-weight model ships, the two questions worth asking are the same as in February: how much of the model does each token touch, and how much does attention cost at long context. The answers tell you most of what you need to know about how expensive the model will be to run.Sources: Sebastian Raschka, “A Dream of Spring for Open-Weight LLMs: 10 Architectures from Jan–Feb 2026” and the LLM Architecture Gallery; Hugging Face model cards and config.json files for Kimi K2.5, Ling 2.5 1T, GLM-5, Trinity Large, Qwen3.5-397B-A17B, MiniMax-M2.5, Step 3.5 Flash, Qwen3-Coder-Next, Tiny Aya and Nanbeige 4.1; Arcee's Trinity technical report and Cohere's Tiny Aya paper. Parameter counts are rounded. Active-parameter shares are computed from the published totals.
