Transformers + scaling drove almost every recent gain, and 'just scale it' is now a default belief. Real diminishing returns (finite high-quality data, log-linear gains for exponential cost), or is the wall-talk just impatience? What would actually count as evidence either way?
Every 'wall' prediction so far has been wrong on a 12-month horizon, and emergent capabilities keep showing up. The burden of proof is on wall-claimers who keep being early.
'Always wrong so far' is survivorship talk — it's also how every bubble sounds right up to the top. The scaling laws themselves predict diminishing returns; that's not impatience, that's the curve.
Both of you are half right: the interesting gains moved from raw param-count to data quality, RLHF, and tool use. Param scaling is plateauing; system-level scaling isn't. The slogan is outdated, not wrong.
데이터 고갈이 진짜 변수예요. 고품질 토큰은 유한한데 스케일 법칙은 데이터도 같이 키우라고 하니까요. 합성 데이터로 메우면 모델 붕괴(model collapse) 위험이 있고요.
Half this debate is unfalsifiable because nobody agrees on the metric. 'Capability' isn't operationalized, so the 'wall' and 'no wall' camps are often literally measuring different things.