Monday, February 18, 2008

A "Thorny Ethical Question"!?

After discovering a stupid mistake in what's supposed to be a trivial matter of my Me,Y & Z paper in preparation, which fortunately didn't result in any qualitative changes in my results (things that had been significant/meaningful were still significant meaningful), we discovered that the same error was in Y & Z (2007). This time the re-analysis was bad; what had been reported as significant was in fact not quite significant. The trend was there, my results showed the effect was real, but the published data set would normally have been presented as "not meeting conventional levels of significance" and so wouldn't have gotten published.

Y, my co-author and lead author on the first paper, had the following response:

"What do you think? Are we obligated to point out the PS flaw in the earlier paper? Would doing so help or hurt the current paper? A thorny ethical question..."

And I went off and had a mild conniption fit.

I was fully on-board with Y's attempts to find a spin to put on the sudden non-significance of the results. The finding is valid, check this other paper; there was this or that confound so we can explain why those results didn't actually turn out. But the suggestion of just ignoring the problem, even on the basis of "more recent/better studies do show the effect is real..." entirely set me off.

The problem is that the original paper, and our summary of the original paper, make a great deal of "is significant even controlling for PS". And it turns out that there was no control of PS at all, there was just control for some nonsensical number value they/we thought was PS. The stats show the important effect is still significant when PS isn't controlled, but all those arguments about how it's beyond and better than this trivial measure are suddenly invalid.

Z, the established name on both papers and my advisor, has not yet weighed in on this issue. The two dozen emails we sent on this matter had panic only in the body, not the subject lines, so s/he prioritized other things and may not even know about the problem yet. I'm hoping, almost desperately, that Z will state that they must issue a correction about the recently-published paper and address this in the current manuscript. I don't care if our almost-ready-for-submission paper has to be changed drastically over another three months of drafts in some attempt to salvage the original set of results. I am going to have a mild nervous breakdown if my advisor decides to let wrong results stand. It's bad enough that the possibility of errors in the papers I read looms large in my mind every time I even glance at a journal. I will lose all my faith in science if faced with respected members of the field explicitly deciding not to correct mistakes they find after publication.

The arguments against correction would boil down to this: that original data set isn't significant, but we can look at Me, Y & Z to show that the effect is real, and the original data would have been significant if we'd kept the experiment going a little longer. To me, this is an excellent basis for the letter of correct, just lacking the specific details, but is completely unsatisfactory for not sending a correction at all. It's the slippery slope of ethics. If people don't correct mistakes that change the meaning of the results for a "good" reason, they might decide not to correct results for a "bad" reasons. Full disclosure is one of the foundations of science, that allows us to decide whether to trust the results (of a specific paper or an author's body of work) and figure out for ourselves what might really be going on instead of taking the author's word on it.

Now, if my advisor argues against making a formal correction, do I go to someone about it? That's where the nervous breakdown comes in...

No comments: