BERT-style bidirectional encoders were SOTA on NLU (classification, NLI, extraction). The field then went all-in on decoder-only autoregressive LLMs, which are left-to-right. For pure understanding tasks (no generation), are we leaving accuracy on the table for the convenience of one architecture that does everything?
For many extraction/classification tasks a fine-tuned encoder still beats a much larger decoder-only model per parameter and per dollar. The pivot was about generality and scaling, not because bidirectional understanding got worse.
Right — so it's an efficiency/generality win, not a quality win. I just worry people default to a 70B decoder for a task a 110M encoder nails.
There's an evaluation bias too: decoder-only models are easy to prompt and benchmark zero-shot, so they look more general even when a small encoder would win a fair fine-tuning comparison.
Encoder/bidirectional objectives didn't die either — they moved into retrieval (embeddings) and some multimodal stacks. The 'one architecture' story is cleaner in tweets than in production.
One more wrinkle: instruction-tuned decoders now beat fine-tuned encoders on some classification tasks once you allow chain-of-thought. So 'encoders win on NLU' keeps getting narrower — it's becoming a cost argument, not an accuracy one.
Honest update: as the quality gap closes, the case for encoders is increasingly 'same accuracy, 100x cheaper' rather than 'better'. Still a very strong case in production, just a different argument than 2019.