We don't rent AI. We engineer it. bb-core 16B is a dense 16.4B-parameter decoder-only transformer with a genuine 256K-token context window, trained on roughly 3.1T tokens and delivered client-owned - weights, tokenizer, training recipe and eval harness. This post is the honest engineering account of how the context window was built, not the marketing number. There is a wide gulf between a model that can ingest 256K tokens without crashing and a model that can reason across them, and most of our engineering effort lived in that gulf.
The 256K figure is not a config edit. It is the downstream consequence of three tightly coupled decisions: how positions are encoded so attention still resolves at 16x the training length, how the attention kernel is scheduled so memory stays linear in sequence length, and how the training schedule is staged so the model learns to use the distance rather than merely tolerate it. Get any one wrong and you ship a model that retrieves a needle but cannot compose two facts 180K tokens apart.
Why naive attention caps most from-scratch efforts
Self-attention computes, for each of n query positions, a similarity against all n keys. That is an n-by-n score matrix: O(n^2) in compute and, if you materialize it, O(n^2) in memory. At n=4K the score matrix is 16M entries per head per layer - annoying but survivable. At n=256K it is 68.7 billion entries per head per layer. Multiply by heads and layers and the intermediate activations alone exceed any single accelerator's HBM by orders of magnitude before you have computed a single gradient.
This quadratic wall is why most from-scratch long-context efforts quietly cap out around 8K to 32K. It is not that longer is theoretically impossible; it is that the naive attention primitive makes the memory bill explode faster than the compute bill, and memory is the binding constraint on real hardware. The score matrix is also the wrong thing to be moving: it is written to and read back from high-bandwidth memory, and that IO traffic - not the multiply-accumulates - dominates wall-clock time. Two problems, then: an n^2 memory footprint you cannot afford, and an IO pattern that wastes the compute you can.
RoPE, and why extending it is not free
bb-core encodes position with Rotary Position Embeddings. RoPE rotates each query and key vector by an angle proportional to its absolute position; because the dot product of two rotated vectors depends only on their relative offset, attention becomes relative-position-aware without any additive bias term. Concretely, for a feature pair at dimension index i, the rotation frequency is theta_i = base^(-2i/d), with base typically 10000. Low-index dimensions rotate fast (they encode local, high-frequency position); high-index dimensions rotate slowly (long-range, low-frequency).
The problem: a model trained at 8K has never seen a rotation angle larger than what 8K positions produce. Feed it position 200000 and the slow, high-index frequencies rotate into phase regions that are entirely out of distribution. Attention logits go haywire; perplexity explodes. So extending context is fundamentally a question of how you remap those rotation angles so the model sees in-distribution phases at positions it was never trained on.
Position Interpolation, NTK-aware scaling, YaRN
- Linear Position Interpolation (PI): divide every position by the extension factor s = L_new / L_train, so position 256K at s=32 presents as position 8K. Simple and stable, but it uniformly compresses every frequency - including the fast local ones - which blurs the short-range resolution the model relies on for fluent local reasoning.
- NTK-aware scaling: instead of scaling positions, scale the RoPE base. Increasing base stretches the low-frequency (long-range) dimensions a lot while barely touching the high-frequency (local) dimensions. You extend reach without smearing local structure. The insight is borrowed from neural tangent kernel arguments about how networks resolve high-frequency signals.
- YaRN (Yet another RoPE extensioN): a per-dimension treatment. It leaves high-frequency dims untouched, interpolates low-frequency dims, and blends a narrow band in between, plus a temperature correction on the attention softmax to compensate for the entropy shift long context induces. YaRN reaches target length with the fewest fine-tuning tokens in our ablations, which is why bb-core's extension leans on a YaRN-style schedule.
Position interpolation buys you tokens. It does not buy you reasoning across them. The angle remap is necessary and nowhere near sufficient - it is the price of admission, not the product.
FlashAttention: memory-linear, not compute-linear
FlashAttention is what makes 256K tractable on real hardware, and it is worth being precise about what it does and does not change. It does NOT reduce the compute complexity: attention is still O(n^2) floating-point operations. What it changes is the memory complexity and the IO pattern. By never materializing the full n-by-n score matrix in HBM, it drops attention's memory footprint from O(n^2) to O(n).
The mechanism is tiling with an online softmax. Queries, keys and values are loaded into fast on-chip SRAM in blocks. For each query block, FlashAttention streams over key/value blocks, maintaining a running softmax normalizer and a running weighted sum of values, rescaling the accumulator as each new block shifts the running maximum. The full score matrix is never written to HBM - only the block currently in SRAM exists. Because HBM IO, not arithmetic, is the real bottleneck at long sequence lengths, being IO-aware makes it dramatically faster in wall-clock terms even though the FLOP count is unchanged.
# Online-softmax accumulation over KV tiles (the heart of FlashAttention).
# Full n-by-n scores never touch HBM; memory is O(block), not O(n^2).
import torch
def flash_block(Q, K, V, block=1024):
n, d = Q.shape
O = torch.zeros_like(Q)
scale = d ** -0.5
for i in range(0, n, block): # query tile
q = Q[i:i+block]
m = torch.full((q.shape[0], 1), float('-inf')) # running max
l = torch.zeros((q.shape[0], 1)) # running denom
acc = torch.zeros((q.shape[0], d)) # running sum(V)
for j in range(0, n, block): # stream KV tiles
k, v = K[j:j+block], V[j:j+block]
s = (q @ k.T) * scale
m_new = torch.maximum(m, s.max(-1, keepdim=True).values)
p = torch.exp(s - m_new) # rescale to new max
alpha = torch.exp(m - m_new) # correct old accum
l = alpha * l + p.sum(-1, keepdim=True)
acc = alpha * acc + p @ v
m = m_new
O[i:i+block] = acc / l # normalize once, at the end
return OStaged context-length training: train short, extend late
We do not train bb-core at 256K from step zero, and no serious long-context model does. The reason is economic and pedagogical. Because attention compute grows quadratically with sequence length, tokens at 256K cost roughly 1000x more per sequence than tokens at 8K. Spending the bulk of a 3.1T-token budget on maximally expensive sequences would be a catastrophic misallocation - and pointless, because the vast majority of what a model needs to learn (grammar, facts, local reasoning, world model) is learnable in short windows.
So the schedule is staged. The overwhelming majority of tokens are consumed at short context (we anchor the bulk of pretraining at 8K), where the model builds its core competencies cheaply. Only in the final phase do we ramp the window - 8K to 32K to 128K to 256K - applying YaRN-style RoPE scaling at each step and fine-tuning on a curated diet of genuinely long documents: full codebases, legal corpora, multi-document synthesis tasks. The late extension is deliberately data-curated: naively long web scrapes teach a model to attend locally inside a long buffer, which is exactly the failure mode you are trying to avoid.
- Stage 1 - bulk pretraining at 8K: cheap tokens, core language and reasoning competencies, the vast majority of the 3.1T budget.
- Stage 2 - progressive extension 8K to 256K: RoPE frequencies remapped per step, short fine-tuning passes to adapt the model to each new phase regime.
- Stage 3 - long-context SFT: curated long documents where the target genuinely requires cross-document synthesis, not local pattern completion.
The KV-cache blowup at 256K
Training is not where long context hurts most at inference time - the KV cache is. At generation, every previously seen token's key and value vectors are cached so they are not recomputed. That cache scales linearly with context length, and at 256K it becomes the dominant memory consumer, frequently dwarfing the model weights themselves.
The arithmetic is unforgiving. KV-cache bytes = 2 (K and V) x layers x kv_heads x head_dim x seq_len x dtype_bytes. For a dense decoder at bb-core's scale, a full 256K-token cache in fp16 runs to tens of gigabytes for a single sequence. This is precisely why grouped-query attention (GQA) is not optional at this context length: by sharing key/value projections across a group of query heads, GQA cuts the kv_heads term severalfold, and it is the single highest-leverage decision for making 256K inference affordable. Combine it with paged KV allocation to eliminate fragmentation and quantized KV where the eval budget permits, and 256K generation moves from research-cluster-only to deployable.
# KV-cache memory: why GQA is mandatory at 256K, not a nicety.
def kv_cache_gb(layers, kv_heads, head_dim, seq_len, dtype_bytes=2):
bytes_total = 2 * layers * kv_heads * head_dim * seq_len * dtype_bytes
return bytes_total / 1e9
# Multi-head (no sharing) vs grouped-query at 256K, illustrative config:
mha = kv_cache_gb(layers=48, kv_heads=48, head_dim=128, seq_len=262144)
gqa = kv_cache_gb(layers=48, kv_heads=8, head_dim=128, seq_len=262144)
print(round(mha, 1), 'GB ->', round(gqa, 1), 'GB') # ~154.6 GB -> ~25.8 GBIngest is not reason: the necessary-not-sufficient problem
Here is the claim we hold ourselves to most tightly. A model that can ingest 256K tokens is not the same as a model that can reason across them. Ingestion means the forward pass runs and the loss is finite at 256K. Reasoning means an answer at position 250K correctly depends on a premise at position 3K - that information propagated through a quarter-million positions of attention and survived.
Needle-in-a-haystack - planting a random fact in a long filler document and asking the model to retrieve it - is the standard long-context test, and it is necessary but badly insufficient. It measures single-hop retrieval: can the model find one token span given an exact-match query? It says nothing about multi-hop reasoning, aggregation over many spans, or resistance to distractors. A model can score 100% on needle retrieval and still be unable to answer a question that requires joining a definition on page 2 to a constraint on page 400. Passing needle tests and calling it long-context reasoning is the most common honest-looking dishonesty in the field.
Honest long-context evaluation
Our eval harness is built to defeat our own optimism. We report accuracy as a function of both context length and, critically, the depth at which the relevant information sits - the well-documented lost-in-the-middle degradation means a model can be strong at the start and end of the window and weak in the middle, and a single averaged number hides that entirely. So we publish the full length-by-depth grid, not a headline score.
- Multi-hop over distance: questions whose answer requires composing two or more facts placed tens of thousands of tokens apart, so retrieval alone cannot solve them.
- Length-by-depth grids: accuracy reported across the full 0-256K range and across insertion depth, exposing lost-in-the-middle rather than averaging it away.
- Aggregation and counting: tasks requiring the model to integrate evidence spread across many spans, not locate a single one.
- Adversarial distractors: near-duplicate decoys planted throughout the context to punish lazy lexical matching over genuine comprehension.
- Real workloads: full-repository code reasoning and multi-document synthesis, because that is what clients actually run, not synthetic needles.
An eval you cannot fail is marketing. We build long-context evals we routinely fail, because the failures are the only part of the number that tells the truth.
What client-owned means here
bb-core 16B ships as a genuine 256K-window dense 16.4B decoder, and it ships owned. The client receives the weights, the tokenizer, the staged training recipe including the exact RoPE-extension schedule, and the eval harness with its full length-by-depth grids - not an API key and a rate limit. Long context is not a feature you rent behind an endpoint; it is an engineered property of a specific model that you can host, audit, fine-tune and extend yourself. That is the difference between renting intelligence and owning it.