Most teams don't pretrain. They fine-tune someone else's weights, wrap an API, and call it a model. bb-core 16B is different: a dense 16.4-billion-parameter decoder we pretrained from scratch, with a 256K-token context window, on roughly 3.1 trillion tokens, burning more than 2 million GPU-hours. It runs in production today on hardware the client owns outright.
This is the part of the work that rarely gets written down honestly. Not the benchmark victory lap - the pipeline that curates a trillion tokens, the tokenizer decisions you can never undo, the failure-recovery machinery that keeps a month-long run alive, and the economics that decide whether any of it is worth doing. This is that build log.
Why pretrain at all
Pretraining is expensive, slow, and unforgiving. So the first honest question is why not just fine-tune. The answer is control. A fine-tune inherits every bias, every gap, and every licensing constraint baked into a base model you didn't build and can't inspect. When the base model changes, your product changes underneath you. For a client operating in a domain where the data is proprietary and the deployment has to be sovereign, that dependency is the whole risk.
bb-core 16B exists because the client needed a model whose provenance they could account for end to end - the data it saw, the architecture it runs, the weights it ships. You don't get that by renting. You get it by engineering.
We don't rent AI. We engineer it. bb-core 16B is what that sentence costs when you actually mean it.
The data pipeline at trillion-token scale
A 16.4B dense model trained on ~3.1T tokens sits in a deliberate ratio: enough data per parameter that the model is well-fed, not starved and not drowning. But token count is the easy number. The hard work is what those tokens are.
At trillion-token scale, curation is not a preprocessing step you run once. It's a standing system. Raw text arrives dirty - duplicated across sources, mislabeled by language, riddled with boilerplate and machine-generated noise. Every one of those defects, multiplied across trillions of tokens, becomes capacity the model spends memorizing garbage instead of learning structure.
What curation actually means here
- Deduplication at both document and near-duplicate span level, so the model doesn't overweight text that happens to be copied a thousand times across the corpus.
- Quality filtering that removes low-signal boilerplate and machine noise without silently deleting the long-tail domains that make the model useful.
- Language and domain balancing, because raw web ratios are not the ratios you want the model to learn - the natural distribution over-represents the generic and buries the specialist.
- Contamination control, keeping evaluation and sensitive material out of the training mix so measured performance means something.
- Provenance tracking, so for a client who owns the model, every slice of the corpus can be accounted for rather than hand-waved.
Data work is roughly where the intelligence of the model is decided. Architecture gets the headlines; the corpus gets the results. The ~3.1T tokens that went into bb-core 16B were the survivors of a much larger raw intake - the pipeline's job was to make sure what survived was worth the compute.
Tokenizer choices you can't take back
The tokenizer is the one decision in the whole project you are stuck with. It's fixed before the first training step and it defines the units the model ever gets to reason in. Choose a vocabulary that fragments your domain's terminology into awkward sub-pieces and you've quietly taxed every forward pass for the life of the model.
Two consequences matter most. First, tokenizer efficiency is a compute multiplier: fewer tokens to express the same text means more effective content inside a fixed context and cheaper inference forever. Second, at a 256K context window, tokenization efficiency compounds - every wasted token is a wasted token multiplied by a very long sequence. We chose the vocabulary to fit the corpus the model would actually live in, not a generic benchmark, because the tokenizer is the model's alphabet and you only get to pick it once.
A dense decoder, and the 256K problem
bb-core 16B is a dense decoder - every parameter participates in every token. We didn't route it through a mixture-of-experts. Dense is heavier to run than a sparse model of nominally similar size, but it is predictable: no routing instability, no expert imbalance, uniform behavior across inputs, and a far simpler thing to deploy and reason about on owned hardware. For a model that has to be accountable and operable by the people who own it, predictability is a feature, not a compromise.
Making 256K context real
A 256K-token context window is the part people underestimate. Attention cost scales with sequence length, so a context that large is not a config flag - it's an engineering program. The naive approach makes both memory and compute explode long before you reach 256K, which is why long context is where most from-scratch efforts quietly cap out at a fraction of the headline number.
Getting to a genuine 256K means the attention implementation, the memory layout, and the training schedule all have to cooperate. You extend context deliberately rather than training at full length from step one, because paying the long-sequence cost across the entire run would be wasteful. And a long window is only worth building if the model actually uses distant tokens - a context you can fill but can't attend across is a spec-sheet number, not a capability. The target was a window the model reasons over, end to end.
Distributed training: parallelism, checkpointing, recovery
You cannot fit the training of a 16.4B dense model with 256K sequences onto one device, so the model and the work get split across many. That splitting - parallelism - is the core of large-scale training, and each axis of it buys capacity at the cost of communication.
- Data parallelism replicates the model and splits the batch, then synchronizes gradients - the simplest axis, bounded by how fast devices can exchange updates.
- Model and tensor parallelism split the parameters and the math of individual layers across devices when a single layer is too large to hold alone.
- Pipeline parallelism stages the layers across devices so different micro-batches occupy different stages at once, trading bubble overhead for the ability to hold a bigger model.
- Long-sequence handling for the 256K window, which changes the memory and communication profile of every one of the axes above.
The art is combining these so devices stay busy and the interconnect doesn't become the bottleneck. Push any single axis too far and you spend your GPU-hours moving numbers between chips instead of learning from them.
Assume failure, because it happens
Over 2M+ GPU-hours, hardware fails. Not might - will. A single node dropping out can halt a synchronized job, and across weeks of wall-clock time you will hit many such events. A training run without a recovery story is a bet that thousands of components all stay healthy for a month straight. That bet always loses.
So checkpointing is not a safety afterthought; it is part of the training loop's design. Checkpoint too rarely and a failure costs you days of recomputation. Checkpoint too often and the I/O itself eats your throughput. The recovery machinery has to detect a failure, restore the last good state, and resume without corrupting the run's trajectory - automatically, because a human is not watching at 3 a.m. on day nineteen. Getting this right is the difference between a run that finishes and one that dies at 80%.
Stability and the economics of compute
A month-long run can be destroyed in a single step. Loss spikes, gradient blowups, and silent divergence are the failure modes that don't announce themselves until the curve has already gone wrong. Stability at this scale is watchfulness plus disciplined choices about numerics and schedule - you protect the run from itself, because there is no cheap re-roll.
That's ultimately why the 2M+ GPU-hours figure matters. Compute is the budget the entire project is denominated in, and every inefficiency spends it: a poor tokenizer, an unbalanced parallelism layout, a checkpoint strategy that forces recomputation, an instability that wastes a week. Pretraining economics are the sum of those decisions. The engineering discipline described above isn't perfectionism - it is how you keep a fixed, very large budget from being quietly wasted.
At two million GPU-hours, there is no such thing as a small inefficiency. Every wrong decision is measured in weeks.
Why owned deployment is the point
bb-core 16B runs on infrastructure the client owns. Not a hosted endpoint, not a shared tenancy, not a model that can be deprecated out from under them by a vendor's roadmap. That was the requirement the whole project was built to satisfy, and it shapes every decision above - the dense architecture chosen for predictability, the provenance tracked through the data pipeline, the model handed over as weights the client controls.
Sovereign deployment means the data never leaves, the model never changes without the owner's say-so, and the capability is an asset on the balance sheet rather than a subscription line that renews forever. For teams whose data is their edge and whose compliance surface is real, that is not a nice-to-have. It's the reason to pretrain instead of rent.
bb-core 16B is 16.4 billion dense parameters, a 256K context window, ~3.1 trillion tokens, and 2M+ GPU-hours of disciplined engineering - turned into a model its owner actually owns. That's what pretraining takes. That's what renting never gives you.
# bb-core 16B - the numbers behind the build
params: 16.4B (dense decoder)
context: 256K tokens
train_tokens: ~3.1T
compute: 2,000,000+ GPU-hours
deployment: client-owned infrastructure