Transformers replaced recurrence/convolution with pure attention + positional encodings. RNNs/CNNs bake in locality and sequential order; attention has to learn them from data. Is the lesson 'inductive biases don't matter at scale', or did we just relocate the bias into positional encodings and pretraining?
I'd say we traded a strong fixed bias for a weak learnable one — and that only pays off at scale. On small datasets, convolutional/recurrent biases still win; plenty of low-data benchmarks where Transformers underperform without heavy pretraining.
Fair, but 'at scale' is doing a lot of work — almost every deployed model now is at that scale. So practically the bias-free bet won, even if it isn't universally true.
저는 위치 인코딩이 사실상 순서 편향을 다시 넣은 거라고 봐요. 진짜 '편향 없음'이 아니라 어떤 편향을 줄지를 데이터/설계로 옮긴 거죠. RoPE 같은 상대 위치 인코딩이 잘 되는 게 그 증거 같습니다.
Agreed — RoPE/ALiBi succeeding is evidence the order bias is still essential; we just made it softer and tunable. What we genuinely dropped is hard locality, which attention re-learns as a soft version (induction heads, local patterns).
A middle position: attention is a more flexible prior, not the absence of one. 'Any token may attend to any token, weighted by content' is still a strong assumption — just a different one than locality.
Late to this, but a near-controlled experiment: vision Transformers needed huge data (JFT-300M) to beat CNNs, then ViT + strong augmentation closed the gap on ImageNet alone. That's almost a clean demonstration that the inductive bias matters exactly when data is limited.
@yukis 좋은 예시네요. 결국 'scale이 편향을 대체한다'가 아니라 'scale이 있으면 편향 없이도 버틴다'가 더 정확한 표현 같아요.