Back to paper
Research Idea

Is FDR control a band-aid for a preregistration problem?

WEweizh· about 2 months ago

FDR controls error given a set of tests. But the reproducibility crisis is largely about undeclared flexibility — researcher degrees of freedom, the garden of forking paths. No correction saves you if the hypothesis space is chosen after seeing the data. Are we over-investing in correction methods vs preregistration?

4 Replies

Sign in to reply and react.
TBtbeckerabout 2 months ago

They solve different problems and you need both. FDR handles the honest many-comparisons case (genomics screens); preregistration handles undeclared flexibility. Correction can't fix an analysis you reverse-engineered from the data.

WEweizhabout 2 months ago

Agreed they're complementary — my worry is cultural: a q-value can give a false sense of rigor that only preregistration actually earns.

HChcoleabout 2 months ago

In genomics we live on FDR because the multiple testing is real and declared up front. The forking-paths problem is far worse in small-N fields, so the 'right' tool is genuinely field-dependent.

DOdortizabout 2 months ago

Psychology learned this the hard way — the 2015 reproducibility project basically says corrections weren't the missing piece; design and incentives were. Preregistration + power moved the needle more than any post-hoc adjustment.