Every conversation about scaling a decoder past ten billion parameters eventually collapses into one question: do you spend your parameter budget densely, activating every weight on every token, or do you shard that budget across experts and route each token to a handful of them? The Mixture-of-Experts (MoE) literature makes the answer look obvious. You get the representational capacity of a much larger model at the forward-pass FLOP cost of a much smaller one. Switch Transformer, GLaM, Mixtral, DeepSeek-V3 - the trend line is unambiguous, and the loss-per-FLOP curves are real. So why did we build bb-core as a dense 16.4B decoder, trained on roughly 3.1 trillion tokens across more than two million GPU-hours, and hand the weights to the client who owns them?
Because the FLOP curve is not the cost curve. MoE optimizes the quantity that is cheapest to reason about on a slide - training compute per unit loss - and quietly transfers cost into the two quantities that dominate a real deployment: memory footprint and interconnect bandwidth. When you own the hardware and have to serve the model under a latency SLA on a fixed cluster, those transferred costs are the whole ballgame. This is an engineering essay about where that transfer happens, how large it is, and why for a 16B-class model destined for owned infrastructure, dense is not the conservative choice - it is the correct one.
Active parameters, total parameters, and the decoupling everyone quotes
Start with the identity that makes MoE attractive. In a dense transformer, total parameters and active parameters are the same number: every weight participates in every token's forward pass, so the per-token FLOP count scales linearly with parameter count (roughly 2N FLOPs per token for N parameters in the forward pass, 6N including the backward pass). MoE breaks that equality. You replace each dense feed-forward block with E expert FFNs and a router that sends each token to k of them, k typically 1 or 2. Total parameters - the thing sitting in memory - scales with E. Active parameters - the thing that costs FLOPs - scales with k. A model with 8 experts and top-2 routing holds roughly the parameter count of an 8x-wider FFN but pays the compute of a 2x-wider one.
That is the decoupling: capacity is set by total parameters, compute is set by active parameters, and MoE lets you push them apart. The scaling-law reading is that loss is largely a function of total parameters and tokens seen, while training cost is a function of active parameters, so sparsity buys you a better loss-per-FLOP frontier. That claim is true and well replicated. The trap is treating loss-per-training-FLOP as if it were the objective function of a deployed system. It is not even close.
The number that actually sizes your cluster
Inference latency for a single token, in the memory-bound regime that autoregressive decoding lives in, is governed by how many bytes of weights you stream from HBM per token - which tracks active parameters - but the amount of HBM you must provision, and therefore how many accelerators you must buy and keep hot, tracks total parameters. A dense 16.4B model in bf16 is about 33 GB of weights. A sparse model with the same active-parameter compute but 8x the total parameters is about 260 GB before you have loaded a single token of KV cache. You have not made the model cheaper to own. You have made it cheaper to train and dramatically more expensive to hold.
Routing is a training-dynamics liability, not just a lookup
The router is a small learned gate - typically a linear projection from the token's hidden state to E logits, softmaxed, top-k selected. It looks innocent. It is the single most temperamental component in the entire architecture, because it is a discrete, non-differentiable decision that you are trying to train with gradients, and the system has a strong pathological attractor: expert collapse.
Why experts collapse, and the loss you bolt on to stop it
Routing is a rich-get-richer process. Early in training a few experts are marginally better, so the router sends them more tokens, so they receive more gradient signal, so they improve faster, so the router sends them even more. Left alone, a large fraction of your experts die - they receive almost no tokens and contribute nothing, and you have paid the memory cost of capacity you are not using. The standard countermeasure is an auxiliary load-balancing loss: for each batch you compute the fraction of tokens dispatched to each expert and the mean router probability assigned to each expert, and you penalize their dot product, pushing the distribution toward uniform. It works, but it is a second objective fighting your language-modeling objective, with its own coefficient to tune. Set it too low and experts collapse; too high and you are degrading perplexity to satisfy a bookkeeping constraint.
A dense model has no router to collapse, no auxiliary loss to balance, and no expert that can quietly die and take your capacity with it. Every parameter you paid to train is a parameter that fires.
There is more failure surface. Top-k routing needs a capacity factor - a cap on how many tokens any one expert will accept per batch - because you cannot have a ragged, data-dependent tensor shape on the accelerator. Set the capacity factor and tokens beyond it are dropped, passed through unmodified via the residual. Token dropping is training noise you introduced on purpose, and at inference it makes outputs mildly batch-composition-dependent, which is a genuinely unpleasant property to explain to anyone who cares about determinism.
All-to-all: the cost that does not show up in the FLOP count
Here is the part the loss-per-FLOP framing hides completely. At any interesting scale, your E experts do not fit on one accelerator, so you shard them across devices with expert parallelism. But routing is token-wise and data-dependent: a token on device 0 may be routed to an expert living on device 7. So every MoE layer requires two all-to-all collectives - one to dispatch each token to whichever device holds its chosen expert, and one to combine the expert outputs back to the tokens' home devices.
Why all-to-all is the worst collective to depend on
All-to-all moves data between every pair of devices, and its cost is dominated by bisection bandwidth, the exact interconnect property that degrades fastest as you cross NVLink domains onto Ethernet or InfiniBand. Unlike the all-reduce in data-parallel training, it is squarely on the critical path of every forward and backward pass, it cannot be cleanly overlapped with compute because the compute depends on its result, and its latency is sensitive to the load imbalance you were already fighting with the auxiliary loss. Two all-to-all exchanges per MoE layer, across dozens of layers, per token, per step. On a fat-tree cluster with uniform NVLink this is tolerable. On the heterogeneous, owned hardware most enterprises actually run - a few nodes, a real network between them - it is the bottleneck that turns a paper efficiency win into a serving latency regression.
- Dense adds zero collectives beyond the tensor/data-parallel ones you already run; MoE adds two bandwidth-bound all-to-alls per expert layer, on the critical path.
- All-to-all cost is set by bisection bandwidth, which is precisely the metric that collapses when routing crosses node boundaries.
- Load imbalance stretches the collective to the slowest expert, coupling your network latency to your routing dynamics.
- Inference batches with skewed routing distributions get non-uniform latency, undermining tail-latency SLAs.
Predictability is a feature, and dense has it
When you serve a dense model, every token costs the same FLOPs, streams the same weight bytes, and takes a path through the network you can characterize once and trust. Capacity planning is arithmetic: weight bytes plus KV-cache bytes per sequence times concurrency, divided by HBM per device. There is no batch-dependent routing, no token dropping, no expert-imbalance tail, no all-to-all whose latency depends on what your users happened to ask. For a model that will live on a fixed, client-owned cluster under a latency contract, that predictability is worth more than a better training-FLOP frontier you already paid for once and never pay again.
When MoE actually pays off
None of this is anti-MoE dogma. Sparsity genuinely pays when three conditions hold together: you are training and serving at a scale where total parameters run to hundreds of billions and dense is simply infeasible; you own a homogeneous, high-bisection-bandwidth fabric where all-to-all is cheap; and your workload is throughput-oriented and batch-heavy rather than latency-critical, so you can amortize routing overhead across large batches. Frontier labs serving one enormous model to millions of users on purpose-built interconnect are exactly that regime, which is why their flagships are sparse. A 16B-class model going onto a client's own hardware is the opposite regime on all three axes.
How this shaped bb-core
bb-core is a dense 16.4B decoder with a 256K-token context window, trained on approximately 3.1 trillion tokens over more than two million GPU-hours, and the weights belong to the client. Every one of those decisions is downstream of the analysis above. Dense means the served footprint is a known ~33 GB of weights in bf16, KV cache is the only variable, and capacity planning on the client's fixed cluster is deterministic arithmetic with no routing tail. It means no auxiliary balancing loss competing with the language-modeling objective across 3.1T tokens, and no expert able to die and strand capacity we spent GPU-hours to train. It means the long-context work - and 256K tokens is where KV-cache memory and attention cost dominate the budget anyway - is not fighting all-to-all traffic for interconnect bandwidth on every layer.
The sparsity literature is real and we read it closely. But it optimizes the one cost that a client who owns their model pays exactly once. We optimized the costs they pay every single day the model is in production: memory they must provision, latency they must guarantee, and behavior they must be able to predict. At 16B, on owned hardware, dense wins that trade - decisively.
# The trade in one comparison. Same active-parameter compute, very different cost of ownership.
def bf16_gb(params): return params * 2 / 1e9 # 2 bytes/param
# bb-core: dense 16.4B -> active == total
dense_total = dense_active = 16.4e9
# hypothetical MoE matched on ACTIVE compute (what the FLOP curve sees),
# with 8 experts inflating TOTAL params (what HBM must hold)
moe_active = 16.4e9 # same forward-pass FLOPs / token
moe_total = 16.4e9 * 8 # capacity decoupled from compute
print(f"dense weights in HBM : {bf16_gb(dense_total):5.1f} GB") # ~32.8
print(f"MoE weights in HBM : {bf16_gb(moe_total):5.1f} GB") # ~262.4
print(f"same per-token FLOPs : {moe_active == dense_active}") # True
# plus, for MoE: +2 all-to-all collectives / expert layer on the critical path,
# a load-balancing loss to tune, and batch-dependent routing latency.
# None of that appears in the loss-per-training-FLOP frontier.