bb-core is a dense 16.4B-parameter transformer. In FP16 that is roughly 33GB of weights before you account for KV cache, activations, or the CUDA graphs vLLM allocates for continuous batching. On a single 80GB accelerator that leaves uncomfortably little headroom for a production batch, and it puts a hard floor under your cost-per-million-tokens. The instinct is to reach for a smaller checkpoint. The discipline is to shrink the one you have while holding its behavior - because the behavior is the product, and a 7B off-the-shelf model that scores well on MMLU will still fail the three internal evals your customer actually cares about.
Compression is not one lever. It is three families of technique - knowledge distillation, quantization, and pruning - that attack different parts of the cost equation and interact in ways that are easy to get wrong. This is how we reason about each of them, how they compose, and how we measure whether we broke the model. We don't rent AI. We engineer it, and that starts with refusing to trust perplexity as a proxy for whether the thing still works.
Distillation: teaching a smaller student the dark knowledge
Knowledge distillation transfers behavior from a large teacher into a smaller student. The naive version is data distillation - generate outputs from the teacher, fine-tune the student on them. It works, but it throws away most of the signal. The teacher's argmax token is a single label; its full output distribution over the vocabulary is a rich, soft target that encodes what Hinton called dark knowledge - the relative probabilities the teacher assigns to the tokens it did not pick. That a legal-drafting model puts 12% on "shall" and 3% on "must" tells the student something the hard label never will.
Logit distillation minimizes the KL divergence between teacher and student next-token distributions, usually with a temperature applied to soften both. The temperature is doing real work: it flattens the distribution so the student sees the tail, then you scale the gradient back by T-squared. But there is a subtler choice that dominates outcomes on generative models - where the training sequences come from.
On-policy vs sequence-level distillation
Sequence-level (offline) distillation trains the student on teacher-generated text. It is cheap and stable, but it creates exposure bias: at inference the student conditions on its own tokens, which drift off the teacher's distribution, and error compounds. On-policy distillation samples sequences from the student, then scores those sequences with the teacher's logits. The student learns to correct its own trajectory, which is exactly the distribution it will face at deployment. It is more expensive - you need the teacher in the loop during training - but for reasoning models it is the difference between a student that mimics and a student that generalizes.
Distilling reasoning, not just answers
The hard case is chain-of-thought. A 16B teacher that reasons through a multi-step problem is not just producing a better answer; it is producing a better trajectory. If you distill only on final answers, the student learns to guess. You have to distill on the intermediate tokens - the reasoning trace itself - and weight them. In practice we generate long CoT traces from the teacher, filter them by correctness (reject the traces that reach the wrong answer, even if the prose looks fluent), and do on-policy KL on the surviving trajectories. The filtering matters more than the loss function; a student trained on confidently-wrong reasoning becomes confidently wrong faster than the teacher.
Distillation moves capability, not weights. If you can't articulate which capability you're moving, you're just fine-tuning on synthetic data and calling it compression.
Quantization: the same weights, fewer bits
Quantization reduces the numerical precision of weights and/or activations. FP16 to INT8 halves the memory and roughly doubles throughput on hardware with INT8 tensor cores; INT4 halves it again. The question is never "can I quantize" - you almost always can - but "where does the error go and does it land on a token my eval measures."
PTQ vs QAT
Post-training quantization (PTQ) takes a trained model and rounds it, using a small calibration set to fit scales. It is fast - hours, not days - and needs no gradients. Quantization-aware training (QAT) simulates quantization in the forward pass during fine-tuning so the weights learn to be robust to rounding, using a straight-through estimator to pass gradients through the non-differentiable round. QAT recovers more accuracy at aggressive bit-widths (INT4 and below) but costs a training run. Our default is PTQ to INT8/INT4 for weight-only, and we escalate to QAT only when the task evals say PTQ left accuracy on the table we can't afford.
Weight-only vs weight-and-activation
Weight-only quantization (INT4 weights, FP16 activations) is the workhorse for memory-bound decode. Autoregressive generation is bandwidth-bound - you read the entire weight matrix per token - so shrinking weights directly buys latency, and the FP16 activations keep quality high. Weight-and-activation quantization (both in INT8, or FP8) is what you want when you are compute-bound: large prefill, big batches, throughput-oriented serving. FP8 is increasingly our sweet spot on Hopper-class hardware - it keeps a floating-point exponent, so it tolerates the dynamic range of activations far better than INT8 does, which sidesteps the single hardest problem in activation quantization.
The activation-outlier problem, and how GPTQ and AWQ handle it
Transformer activations develop systematic outliers - a handful of feature dimensions with magnitudes 10-100x the rest, concentrated in specific channels. Quantize activations naively with a single per-tensor scale and those outliers stretch the range so far that the normal values collapse into a few quantization bins. The model degrades sharply. Every serious quantization method is, at heart, a strategy for outliers.
GPTQ is a weight-quantization method that quantizes columns one at a time and, crucially, updates the remaining unquantized weights to compensate for the error introduced - using second-order (Hessian) information estimated from the calibration set. It is an approximate solver for "given I must round these weights, how do I adjust the others to minimize output error." Per-channel scales and a good calibration distribution matter enormously: calibrate on data that looks like production, not on a generic web dump, or your scales fit the wrong outliers.
AWQ (Activation-aware Weight Quantization) takes a different route. It observes that not all weights matter equally - the weights connected to high-magnitude activation channels are salient. Rather than keep those weights in higher precision (which breaks hardware kernels), AWQ scales them up before quantizing and scales the corresponding activations down to compensate, mathematically preserving the product while shifting the salient weights into a range that quantizes cleanly. It is a per-channel scaling search driven by activation statistics. In our experience AWQ is often more robust than GPTQ at INT4 for instruction-tuned models and is friendlier to fused kernels; we benchmark both per model rather than picking a favorite.
- Weight-only INT4 (GPTQ/AWQ): best for memory-bound decode and single-stream latency
- INT8 weight+activation (SmoothQuant-style): throughput serving where activations behave
- FP8 e4m3: high-batch prefill on Hopper; tolerates activation dynamic range natively
- Calibration set must mirror production traffic - outliers are data-dependent
Pruning: removing weights the model isn't using
Pruning zeroes out weights. Unstructured pruning removes individual weights by magnitude or by a saliency criterion (Wanda, SparseGPT), producing a sparse matrix that is theoretically smaller but, on most hardware, not faster - dense tensor cores don't skip zeros. It shines only where you have hardware or kernels that exploit it. Structured pruning removes whole units - attention heads, MLP channels, even layers - yielding a genuinely smaller dense model that runs faster on stock kernels with no special support. The catch is that structured pruning is coarser and hurts more, so it almost always needs a recovery fine-tune, which is where it folds neatly back into distillation.
The one structured-sparsity pattern worth calling out is 2:4 (semi-structured) sparsity, where two of every four contiguous weights are zero. Ampere and later accelerators have sparse tensor cores that give a real ~2x on 2:4 patterns, so it is the rare unstructured-ish scheme with hardware behind it. We use it selectively, and only after confirming the accelerator in the target deployment actually supports the sparse path.
How they compose - and the order that matters
These techniques are not independent knobs you set in isolation; they interact. Our default pipeline is: distill first, prune second, quantize last. Distillation sets the capability ceiling. Structured pruning then removes the capacity the distilled student demonstrably isn't using, with a short distillation-based recovery pass to heal the wound. Quantization goes last because it is the most reversible and the cheapest to re-run, and because you want to quantize the final weight geometry, not a geometry you're about to prune away.
The failure mode is stacking degradations blindly. INT4 quantization might cost you 1% on a task; structured pruning another 2%; a distilled student another 1.5%. Individually acceptable; compounded, you have shipped a model that lost 4.5% on the exact eval the customer signed for. Each stage must be measured against the deployment target, not against the previous stage.
# Composition sketch: distill -> prune -> recover -> quantize -> eval gate
student = distill(
teacher=bb_core_16b,
student=bb_core_pruned,
mode="on_policy", # student samples, teacher scores
loss="kl", temperature=2.0,
cot_filter=lambda tr: tr.final_correct, # drop wrong-answer traces
)
student = structured_prune(student, heads=0.15, mlp_channels=0.20)
student = distill(teacher=bb_core_16b, student=student, steps="recover")
q = awq_quantize( # activation-aware, INT4 weight-only
student, bits=4, group_size=128,
calib=production_sample(4096), # mirror real traffic
)
report = eval_suite(q, tasks=CONTRACTED_EVALS)
assert report.min_task_delta > -0.01, report # gate on the worst task, not the meanMeasuring degradation honestly
Perplexity is the liar's metric. A quantized model can hold perplexity nearly constant while its accuracy on structured reasoning, tool-calling, or long-context retrieval falls off a cliff - because perplexity averages over every token and the tokens that matter are a tiny fraction. We gate compression on a task-eval suite: the specific capabilities the model was contracted to deliver, scored the way the customer scores them, plus stress evals for the failure modes compression is known to introduce (long-context degradation, refusal drift, format compliance, and - for reasoning models - CoT faithfulness). We gate on the worst-performing task, not the average, because the average hides exactly the regression that gets you a support ticket.
Deployment economics: the number that actually decides it
Compression is a means; the end is cost-per-quality. A 4-bit, structurally-pruned, distilled bb-core that fits comfortably on one accelerator with room for a large batch can cut serving cost 4-6x versus FP16 - through higher batch density, lower memory pressure, and faster decode. We serve these with vLLM for high-throughput batched inference, Triton where we need multi-model or ensemble orchestration, and ONNX Runtime for edge and CPU targets where the accelerator isn't a given. The right operating point is not "smallest model" or "highest quality" - it is the point on the cost/latency/quality surface where the quality still clears the eval gate and the cost clears the business case. Everything above is just how we find that point without deluding ourselves about what we gave up.