Every serious conversation about applied AI eventually arrives at the same fork: take a strong open-weight base model and fine-tune it, or train your own from scratch. The industry consensus - fine-tune, always, scratch is vanity - is mostly correct and occasionally catastrophic. We have taken both roads, and the honest position is that the answer is a function of your constraints, not your ambition.
We build bb-core, a dense 16.4B-parameter decoder with a 256K context window, trained on roughly 3.1 trillion tokens across more than two million GPU-hours. We own the weights, the tokeniser, and the entire pipeline end to end. That did not happen because scratch is fashionable. It happened because for a specific class of problem, fine-tuning does not merely underperform - it fails on axes that no amount of adapter tuning can reach. This post is about telling those cases apart before you spend the money.
What fine-tuning actually is - and isn't
Fine-tuning updates a model that already has a fixed worldview. The base model's tokeniser, its distribution of concepts, its reasoning shortcuts, its safety reflexes, and its silent factual priors were all set during someone else's pretraining run. Fine-tuning nudges the output surface. It does not renegotiate the foundations. This is precisely why it is so efficient: you are paying for the last one percent of adaptation on top of a trillion-token investment you got for free.
For a large majority of production work, that is exactly the right trade. If your task is style, format, tone, tool-calling discipline, or teaching the model a bounded set of behaviours it already half-knows, fine-tuning - or even a good few-shot prompt with retrieval - will beat a bespoke model on every metric that matters, including the one people forget: time to a working system.
Where fine-tuning genuinely suffices
- Adapting register and house style - legal, clinical, brand voice - where the underlying concepts are already in the base model.
- Structured extraction and classification over documents the base model can already read competently.
- Tool and function-calling reliability, where you are shaping behaviour, not adding knowledge.
- Retrieval-augmented tasks where the facts live in a vector store and the model only needs to reason over what it is handed.
- Any project where the base model's licence, latency, and cost are acceptable and the domain is well inside its training distribution.
If you are in this list, do not train from scratch. We will tell you that in the first meeting. Building a foundation model to do sentiment classification is engineering malpractice dressed as sophistication.
Where fine-tuning structurally fails
The failures worth caring about are not quality gaps you can close with more data. They are structural - properties the base model cannot acquire because they were decided before you arrived.
Domain vocabulary and tokenisation
A tokeniser trained on general web text shreds specialist language. Aircraft type codes, genomic sequences, financial instrument identifiers, chemical nomenclature, low-resource languages - these fragment into long, wasteful token streams that the model never saw enough of to represent well. You can fine-tune around the edges, but you cannot change the tokeniser without retraining, and a bad tokeniser is a permanent tax on every inference: more tokens per document, shorter effective context, worse compression of exactly the concepts you care most about. When we control the tokeniser, domain terms become first-class tokens. That is a from-scratch decision, full stop.
Reasoning depth in the domain
Fine-tuning teaches a model what to say. It is remarkably bad at teaching a model how to think in a domain it did not deeply learn during pretraining. If your problem needs the model to hold long chains of domain-specific inference - not retrieve a fact, but reason through a novel case - the reasoning circuitry has to be laid down during pretraining, when the model is still plastic. Bolt-on fine-tuning sits too high in the stack to build that. You get fluent answers that are subtly, confidently wrong in ways your experts catch and your users don't.
Fine-tuning changes what a model says. Pretraining changes what a model is. Confusing the two is the most expensive mistake in applied AI.
IP ownership and data sovereignty
This one is not technical, and it is often decisive. When you fine-tune a third-party base model, your derivative inherits the base licence, its usage restrictions, its indemnity gaps, and its right to change terms under you. Your competitive moat sits on rented land. For clients in regulated sectors, there is a harder constraint still: training data cannot leave a jurisdiction, cannot touch a foreign API, and must be provably contained. When we train from scratch on infrastructure we control, the weights are an asset the client owns outright, the data never leaves the boundary, and there is no upstream licensor who can revoke the ground beneath a shipped product.
Latency and cost at scale
Owning the model means owning the architecture. We size the parameter count, the context strategy, and the serving path to the actual workload rather than paying to run a general-purpose giant. A right-sized 16B that we can quantise, distil, and deploy on our own terms often beats a rented frontier endpoint on cost-per-token and tail latency - while removing the per-call dependency on a vendor's pricing and rate limits entirely. At high volume, the economics invert: the model you own gets cheaper the more you use it; the model you rent does not.
What from-scratch actually costs and buys
Let us be unromantic. Training a foundation model is a serious capital and engineering commitment. bb-core represents over two million GPU-hours and 3.1 trillion tokens, and the compute is the cheap part. The expensive parts are data - curation, deduplication, cleaning, and the domain corpus that makes the whole exercise worthwhile - and the pipeline engineering that keeps a run stable for weeks without silently corrupting itself. Then there is evaluation, safety, and the unglamorous MLOps of serving something you now fully own.
What you buy is a different kind of asset. Not a configuration of someone else's model, but a foundation you can extend, distil, re-tokenise, and re-align on your own schedule. The tokeniser is yours. The reasoning was shaped on your distribution. The weights are a balance-sheet asset with no upstream licensor. And every subsequent fine-tune you do sits on ground you control - which means from-scratch and fine-tuning are not rivals but a sequence: build the foundation once, adapt it cheaply forever.
How we decide
We run every prospective project through the same triage before anyone touches a GPU. The default is fine-tune; scratch has to earn itself against real constraints.
- Is the domain inside the base model's distribution? If yes, fine-tune or retrieve. If the base model can't even tokenise the domain sanely, that is a scratch signal.
- Does the task need retrieved facts or genuine in-domain reasoning? Facts favour RAG. Deep, novel reasoning chains favour pretraining.
- Who must own the weights, and where must the data live? Hard sovereignty or IP-ownership requirements can force scratch on their own, regardless of quality.
- What is the volume and latency envelope? Low volume rents comfortably. High-volume, latency-sensitive workloads reward owning and right-sizing the model.
- Is this a durable capability or a one-off? A moat you will compound on for years justifies foundation investment; a single feature almost never does.
Most projects fail this test and should - and we say so. The minority that pass tend to share a signature: a specialist domain the base models tokenise badly, a real ownership or sovereignty constraint, and enough volume that the unit economics of owning beat the unit economics of renting. That intersection is narrow. When a project lands inside it, fine-tuning is not a cheaper path to the same place; it is a different, lesser destination.
The honest bottom line
Fine-tuning is the correct default and we use it constantly. But it rents another organisation's priors, tokeniser, and licence, and for a specific class of problem those are exactly the things you cannot afford to rent. Training from scratch is expensive, slow, and unnecessary for most work - and irreplaceable for the rest. The engineering discipline is not choosing a side. It is being ruthlessly honest about which problem you actually have, and refusing to solve a rented-priors problem with an owned-model budget, or an owned-model problem with a rented shortcut.
We built bb-core because a class of our clients' problems lived on the wrong side of that line. We don't rent AI. We engineer it - and part of engineering it is telling you, plainly, when you don't need us to.