Question

Why does low-rank adaptation work so well in practice?

JIjihoon· 14 days ago

The premise is that the weight update during adaptation has low intrinsic rank. Intuitively I get why that could hold, but is there evidence for how low the rank actually needs to be across tasks? When does rank-1 or rank-2 start to hurt?

NLP

2 Replies

Accepted answer

AMamir_r14 days ago

The paper ablates rank and finds surprisingly small r (even 1–4) is enough on many tasks — which is itself evidence the update is low-rank. The harder the task / bigger the domain shift, the more rank seems to help.

MIminseok14 days ago

직관적으로는 사전학습이 이미 좋은 표현을 잡아놔서, 적응에 필요한 '추가 정보'가 적기 때문이라고 보면 될 것 같아요. 그래서 저랭크로도 충분한 거죠.