On Samizdata, the comment was made that the UEA has confessed to having “lost” all the original data. This was a revelation known some time ago, as a result of the FOI responses that have been received. It’s not quite that straightforward, I suspect. I commented there, but this is an issue that is not in the forefront of the coverage yet, so I thought was worth repeating here.
When they say "lost", they mean lost in a special way. Imagine you are collecting a database of temperature statistics. You input the data, and dispose of the no longer needed input media. You are tweaking and correcting them as you go. Now you run a program to collect a summary, and publish a paper. And then you carry on adding (and tweaking) statistics to the database. Adjusting and improving your programs, and so on.
Ten years later, somebody asks for the data you used to generate the original paper. Problem is, you’ve got no idea of what the state of the database was at the time. That was thousands of additions and hundreds of tweaks ago. You have no regular backups, no archives, no records of what you’ve done. Because after all, why should you keep a record of the old data when you know the new version is so much better? It makes no sense, and would be a waste of valuable storage space.
And so you tell the enquirers to get knotted. You know what you’re doing, why should these amateurs be allowed to come in and rip 25 years of work to shreds?
But then some fool politician passes the FOIA, and you’re suddenly told you have to share your data. "Oh God!" What are you going to do?! You don’t have it any more, and it’s going to be a nightmare to reconstruct.
This, I believe, is the desperate genesis of the HARRY_READ_ME.txt file. Some poor sod (called Ian "Harry" Harris, it is generally suspected) was given the miserable job of going back through the stinking tangled mass of code and intermediate results to try to reconstruct what was done. We gather that several years of late nights and working weekends were involved. Most of the software people who have looked at the file have expressed deep sympathy for the poor fellow. Many of us have been there.
You are utterly trapped! On the one hand, reconstructing the data is impossible, on the other, admitting what you’ve done is unthinkable, and increasing legal pressure is getting ever closer to forcing a choice, and the politics in the run-up to Copenhagen is doubling and redoubling the stakes. And yet at no point in the whole process did you do anything you consider to be wrong.
I wonder, did the release of the emails maybe even come as something of a relief? Yes, you are now going to be pilloried before the entire world, but at least the agonising wait is over. And in a few months time, it might even pass. And you can relax in your enforced retirement in that little cottage with roses over the door, instead of that sword of Damocles. Looking at it on a human level, wouldn’t it be nice if that was so?