Principles and Flexibility in Multiple Testing

Abstract

Closed testing is known to underlie many multiple testing methods. Any method controlling a tail probability of a number or proportion of errors is either equivalent to a closed testing procedure or is uniformly improved by one. We extend closed testing to a novel necessary and sufficient principle for all multiple testing methods controlling any expected loss. This e-Closure principle asserts that every such multiple testing method is a special case of a generalized closed testing procedure based on e-values. It applies to a large class of error rates — in particular to the false discovery rate (FDR). We show that any procedure controlling FDR is either a e-Closure method or is uniformly improved by one. By writing existing methods as special cases of this procedure, we show uniform improvements, as we demonstrate for the Benjamini-Hochberg and the Benjamini-Yekutieli procedures. We also show that methods derived using our novel e-Closure Principle generally control their error rate not just for one rejected set, but simultaneously over many, allowing post hoc flexibility for the researcher. This flexibility is important in many application areas including neuroimaging and genomics. Moreover, we show that because all multiple testing methods for all error metrics are derived from the same procedure, researchers may even choose the error metric post hoc, allowing post hoc switching from e.g. familywise error to FDR. Under certain conditions, this flexibility even extends to post hoc choice of the nominal error rate.