I just came back from visiting Sarah in Stockton for a few days — so now, in quick succession, we have gotten a much better look at each other’s lives, which is nice. Forbes Magazine ranked Stockton as America’s second most miserable city, after Detroit, but apparently Forbes doesn’t place as much emphasis as I do on the presence of a wide variety of good ethnic food, and, even more importantly, on the presence of Sarah. We did some fun cooking together during my visit. I took home some chili as a souvenir.
But the title of this post has nothing to do with that.
One of my fantastic graduate students is currently writing a paper, with me and an optical astronomer (call him X) as co-authors*. The paper involves data taken with an orbiting x-ray observatory. With most of these satellites (and with many ground-based, optical observatories), the person who proposed the observation gets the data all to his/herself for one year, and then the observatory releases the data to the general public. So if you want to be sure that you get to publish the results of your own idea, you must analyze your data and write your paper in less than a year. Normally this isn’t a problem, but if you are encouraging a new graduate student to do the work, and to write the paper, then she has to learn a lot about the observatory, about x-ray astronomy techniques in general, about the scientific question and scientific background, and about how to write a paper for a journal. Even for a fantastic grad student, that’s a lot to do in less than a year, taking classes at the same time.
We have just about made it. The paper is only a few days from being ready to submit to a journal, and we have 3 more weeks until the data become public. But there is still a problem — it takes a couple of months for the paper to be reviewed by an expert referee and accepted by the journal. In that time, someone else (someone who is very practiced and quick at data analysis) could download the data, analyze them, and write a paper of their own. “But,” you say, “they will have to wait for the journal too, and so you will still be first.”
Sound reasoning, but there is another factor: the “arXiv”. The arXiv (pronounced “archive”) is a hugely valuable service provided by Cornell University to the community of astrophysicists and other scientists. When I have a paper accepted by a journal, I also upload a copy of it to the arXiv and it is immediately available for anyone to look at on the Web. But I could also upload a copy of the paper even before the journal accepts it, or without submitting it to a journal at all. So an ambitious person could post a paper on the arXiv while the referee is still reading our paper. That person would end up getting primary credit for the results. The only defense against this is for us to upload our paper to the arXiv at the time we submit it to the journal, instead of at the time the journal accepts it. This is not immoral, but it isn’t desirable. If we have to change the paper significantly because of the referee’s comments, then there will be two versions of it rattling around out there.
I asked collaborator X for his advice, and he reluctantly agreed that we should upload early, even though neither of us has ever done that before. He said,
I have never posted a paper [to arXiv] without formal acceptance, and always felt very reluctant posting papers before having a (mostly) positive review. But I’ve never been in a case like this before. People don’t jump like hungry jackals on optical data the day they go public!

Photo by Megan Bradfield from the South African National Parks website. There's some choice data in that bush.
Before you judge the jackals too harshly, please note a few things. First, the system of 1-year proprietary data is designed to make sure that data get used, which is a good thing, since these observatories are paid for by taxpayers. Sometimes people get lazy, and don’t have a pure, virtuous excuse like I do (training a student) for letting their year elapse. Some people also have very good ideas and are very good at analyzing data but haven’t been able to get observing proposals accepted (sometimes because their country hasn’t contributed to building the satellites).
Finally, note also that X and I recently started working on some data on another object that we are interested in — data that just went public a month ago, for which the “owner” never published anything. Sometimes the jackal eats you, and sometimes the jackal is you
* “Optical” here means astronomy done with wavelengths near or in the range visible to the human eye. This is in contrast to x-ray astronomy, which is my specialty. X-rays are about a thousand times more energetic than the particles of light that our eyes (and normal telescopes) can see. X-ray telescopes have to be in orbit because the x-rays don’t penetrate the atmosphere.