Back to paper
Scaling Laws for Neural Language Models
CritiqueRe: Fig. 13

The compute-optimal frontier ignores inference cost

TBtbecker· Independent· about 1 month ago

Figure 13 optimizes training compute, but in production you serve a model many more times than you train it. A smaller model trained on more data is cheaper to infer from, which shifts the true optimum toward smaller N for most deployment scenarios. Chinchilla addresses this partially — curious whether the community has converged on an inference-adjusted scaling formula.

ML Systems

2 Replies

Sign in to reply and react.
PRpriyanairabout 1 month ago

The LLaMA series essentially made this argument operational — train a smaller model longer, then it beats a larger undertrained one at the same inference budget. The empirical result validates Tom's point pretty directly.

YUyukisabout 1 month ago

Quantization adds another dimension: a larger model at 4-bit can match a smaller model at 16-bit in throughput. So the optimal frontier is really a surface in (N, D, precision, batch) space.