If I keep writing, maybe this post will become significant

2 thoughts on “If I keep writing, maybe this post will become significant”

  1. The cartoon is terrific, and your comments spot-on.

    When I was a grad student (in stats), one of the tasks we did was to run a sort of consulting shop for the other disciplines. We provided free drop-in advice for researchers or students with questions, and most often, had med school, psychology, or B-school students come by. My recollection at the time was, “uh-oh. As the folks at SAS include more canned routines and make it easier to use, there is going to be trouble.”

    The “XKCD” example you cite, we used to try to impress upon (more often than not), the psychology department folks, using a variation of the famous “drunkard’s walk.” We modified it slightly by making the bar the only absorbing state. Thus, the person will *always* wind up at the bar.

    Beyond the obvious problem of multiple testing and variable stopping rules, one massive problem I encounter is the focus on p-value and the disappearance of its evil twin, statistical power. Even if one pre-specifies a sample size and thus reduces the possibility of an experiment ending once a result of p<0.05 (or, 0.01, or whatever level is chosen), the reality is that one can easily reject the null with a large enough sample.

    In the case of Phase 3 trials, if a molecule has reached that level, the chances that it has absolutely no impact is pretty small. Therefore, with a massive enough sample, you will reject the null, and your beans will show quite 'effective' in this case. The "fix" that has been attempted has been to pre-specify minimal clinical difference. I think anyone who reads your blog will be able to see the inherent problems.

    I've found (and been heartened) by the increasing acceptance of Bayesian methods in health research. The battle is not easy, but I have some confidence that it will eventually bring more order here.

    Whether that confidence is 95%, I can't say with any real power.

    1. Thanks! The stats “consulting shop” sounds like a fascinating experience. When I was an undergrad at Berkeley in the mid-1990s I remember seeing flyers for some similar service, and not knowing anything at all about the subtleties of statistics then (being a physics major), the flyers seemed completely mysterious to me.

      Coincidentally to your note on Bayesian methods, a recent post on Andrew Gelman’s really blog (http://andrewgelman.com/) points to slides on a talk: “Can we use Bayesian methods to resolve the current crisis of unreplicable research?” Gelman’s blog is consistently fascinating — one of the best out there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s