I enjoyed a lot reading a "discussion" prepared by Maury Goodman on the value of "confidence level", discovery thresholds, and what physicists believe or not. If you are a HEP physicist and you want to widen your horizons on the value of statistical claims in experimental results, you are bound to read it. But you might find it thought-provoking and enlightening even if you are a layman, provided you can use three neurons in a row.
A few random excerpts should convince you to read the whole piece:
Manjib: No reasonable high energy physics will believe a two sigma effect.
Karana: I’ve believed several two sigma effects. I canname several (first SAGE indication of a solar neutrino deficit in Gallium, nutau appearance in Super-K, MINOS’ measurement of theta-13). I believed the LMA before KamLAND when global fits favored the LMA by 2 sigma.
Manjib: I calculate that the chance of that distribution is 6e-04, less than the claimed non-zero value.
Karana: Your calculation is meaningless. There are an infinite number of tests of the null hypothesis. Your test is a-posteriori. I could make a similar calculation about every right result
Karana: Let’s say someone claims a three sigma effect. It could be 1) something new; 2) an unlikely statistical fluctuation; 3) A systematic effect that was not taken into account, or 4) and most frequently, it wasn’t an (a-priori) three sigma effect.
Manjib: People can handle that with trials factors.
Karana: It is virtually impossible. People make multiple plots and cuts, and they are usually correlated. People often combine two uncorrelated probabilities P1 and P2 as the product P1*P2. That is an unreasonable test of the null hypothesis. A more reasonable one is to limit the product, and that probability is P1*P2 *(1-ln[P1*P2])
