Done?

As of shortly before 4:30 pm Mountain time, my dissertation is written. The data reflect the most recent analysis, my advisor's final comments have all been addressed, the arduous process of revising my thinking based on one single new reference is complete. I even created a special committee copy, without the pages of of Tables of Contents and such required by the school. It's sitting there, a lovely .pdf of 116 pages (including references), in my dissertation folder. And yet I have not emailed it out. It isn't due until tomorrow, and I can't quite conceive of what it will mean to have submitted my dissertation to my committee. It is the sensation I have had several times since my defense was announced to the department, and people started congratulating me and/or wishing me good luck when we passed in the halls. It doesn't quite seem real. I can't really be this close to being done.

Trouble by the Numbers

A rant from elsewhere, back in the Thesis Defense phase:

Multiple choice question. Graphs:

a) may or may not show all the work you put into creating them
b) present data much more coherently and completely than words and averages
c) are the product of THE DEVIL or his minions intent on creating chaos and frustration
d) all of the above

Let's consider these different options.

A) This is correct. Excel will automatically put standard error bars on any graphs, but as far as can be determined it pulls these numbers out of its electronic posterior. SPSS is supposed to have a button to add standard error bars, but this doesn't work. I spent two hours Tuesday manually entering standard error bars in Excel, and I will probably spend less than two minutes talking about each one.

B) Also correct. My powerpoint presentations contain few words, and dozens of images. Only one slide in my practice talk contained a verbal explanation of a finding rather than a visual one, and that was because I hadn't had time to create that graph yet. I promptly went and created a very pretty scatterplot showing the relationship between two measures in my experiment in hopes of impressing my committee that my research wasn't an entire null result. Looking at that pretty scatterplot, I discovered an undetected outlier, which let me remove it and make re-do the analyses before sending everything off to my committee.

C) Oh, is this ever correct. Thanks to that outlier, not only did I have to trudge through re-typing a lot of obscure numbers throughout my methods section, the entire meaning of the results for any analyses related to that measure changed, and I spent an entire evening re-writing chunks of my results and discussion section and reformulating my
conclusions.

Therefore...

D) All of the above. As in all such multiple choice questions, this is the correct answer. As thrilled as I am to discover that my experiment just became more meaningful in its "marginally significant" way, and that the error was discovered before sending a draft to my committee (they might read it, and changing conclusions for the defense probably wouldn't go over well), I spent most of a day fixing something that was supposed to have been finished. I fully intend to irrationally blame the graph for all that frustration. Ignorance is bliss; knowledge is a re-write.

A "Thorny Ethical Question"!?

After discovering a stupid mistake in what's supposed to be a trivial matter of my Me,Y & Z paper in preparation, which fortunately didn't result in any qualitative changes in my results (things that had been significant/meaningful were still significant meaningful), we discovered that the same error was in Y & Z (2007). This time the re-analysis was bad; what had been reported as significant was in fact not quite significant. The trend was there, my results showed the effect was real, but the published data set would normally have been presented as "not meeting conventional levels of significance" and so wouldn't have gotten published.

Y, my co-author and lead author on the first paper, had the following response:

"What do you think? Are we obligated to point out the PS flaw in the earlier paper? Would doing so help or hurt the current paper? A thorny ethical question..."

And I went off and had a mild conniption fit.

I was fully on-board with Y's attempts to find a spin to put on the sudden non-significance of the results. The finding is valid, check this other paper; there was this or that confound so we can explain why those results didn't actually turn out. But the suggestion of just ignoring the problem, even on the basis of "more recent/better studies do show the effect is real..." entirely set me off.

The problem is that the original paper, and our summary of the original paper, make a great deal of "is significant even controlling for PS". And it turns out that there was no control of PS at all, there was just control for some nonsensical number value they/we thought was PS. The stats show the important effect is still significant when PS isn't controlled, but all those arguments about how it's beyond and better than this trivial measure are suddenly invalid.

Z, the established name on both papers and my advisor, has not yet weighed in on this issue. The two dozen emails we sent on this matter had panic only in the body, not the subject lines, so s/he prioritized other things and may not even know about the problem yet. I'm hoping, almost desperately, that Z will state that they must issue a correction about the recently-published paper and address this in the current manuscript. I don't care if our almost-ready-for-submission paper has to be changed drastically over another three months of drafts in some attempt to salvage the original set of results. I am going to have a mild nervous breakdown if my advisor decides to let wrong results stand. It's bad enough that the possibility of errors in the papers I read looms large in my mind every time I even glance at a journal. I will lose all my faith in science if faced with respected members of the field explicitly deciding not to correct mistakes they find after publication.

The arguments against correction would boil down to this: that original data set isn't significant, but we can look at Me, Y & Z to show that the effect is real, and the original data would have been significant if we'd kept the experiment going a little longer. To me, this is an excellent basis for the letter of correct, just lacking the specific details, but is completely unsatisfactory for not sending a correction at all. It's the slippery slope of ethics. If people don't correct mistakes that change the meaning of the results for a "good" reason, they might decide not to correct results for a "bad" reasons. Full disclosure is one of the foundations of science, that allows us to decide whether to trust the results (of a specific paper or an author's body of work) and figure out for ourselves what might really be going on instead of taking the author's word on it.

Now, if my advisor argues against making a formal correction, do I go to someone about it? That's where the nervous breakdown comes in...

Data Panic Attack

Lab meetings are very useful things. Yesterday we presented our paper (X, Y, & Z, in preparation) to the lab, my fellow graduate students and an assortment of undergraduate RAs. And one of the graduate students asked a very simple question that has snowballed into massive implications for the published paper (Y & Z, 2007). Meaning, the published data may turn out to be wrong. What was reported and published as significant, and thus meaningful, may turn out to be beyond the accepted cutoff for significant, and thus not meaningful at all.

What exactly is wrong? It was really quite basic. To answer the key question in both of these papers, something has to be factored out. I got quite sick of writing "even controlling for [average z-score]", but it was really an important point. The grad student asked a very simple question about what the difference between groups on this z-score was. And we discovered that the z-score measure was wrong.

For those not in the know, a z-score makes data relative. It tells us how far from the average a certain score is. If the z-score is -2, it's very far smaller than average; if it's 0, it's exactly average; if it's +1.5, it's higher than average. The problem is that I told my statistics package, "give me the z-score for Measure1. give me the z-score for Measure2. give me the average of those z-scores", not realizing that "smaller than average" is good for Measure1 but bad for Measure2. So this average z-score is 0 if both scores were good, or if both scores were bad. In sum, were weren't "controlling for [average z-score]" at all, we were controlling for something entirely nonsensical.

The good news it that this doesn't affect my data. I fixed the z-scores. The statistics actually turn out slightly better for all the critical measures, so I just had to update a lot of post-decimal point numbers to reflect the changes.

The bad news is that, as far as I can tell, this really does affect the original, published study. The statistics have gone from being "significant" to being what we grad students usually term "marginally significant" when forced to present something, anything, to the rest of the department. It's enough to make us think the effect is real, but nowhere near strong enough for publication.

I could be wrong. I hope I'm wrong. I am waiting confirmation from Y that I understood his near-incomprehensible column headings on the original data file. With any luck, I pulled the wrong column for the raw data, and the right column will magically leave the important effect as significant. Maybe I just have too much of an ego to doubt my interpretation of the column names, but I don't think I'm that lucky.

This is too much for a graduate student working on a first publication. There aren't even any poster presentations on my CV, and now I may be about to find out what happens when you find a massive error in data after something has already been published. This strikes me as being rather backward in training. I'd prefer to have the confidence of an actual publication before discovering that publications can be just plain wrong and perhaps shouldn't be trusted.

At least I can console myself that the "obvious" error in the z-score calculation was also made by Y, who was an experienced post-doc at the time (now an assistant professor). I'm sure this will be great comfort as I spend the next 24 hours stepping through each data calculation and analysis for the nth time just to make sure that all the statistics are as accurate as we can make them.

Praise

Three-way debates on whether to proceed with an analysis (can it tell us anything? would it be extra work for little gain?), a bug in the script (not mine, though I caught it), and three attempts at analyses (give us graphs, what if we include these trials, what about this interaction...) can all be made worthwhile with a single, one-word reply from my advisor:

"beautiful!"