This is a great effing question! The textbook species definition is that proffered by Ernst Mayr: populations of actually or potentially interbreeding individuals, capable of producing viable (and fertile) offspring. Cool, so a dog and a cat are different species because if they mated (ew) no ungodly animal would come from this monstrous union. Expensive high-tech multivariate Scientific reconstruction simulations show the abomination would probably look like this:
But there are many "good" plant and animal species that do mate and reproduce successfully ('hybridize'). Very often these hybrids are sterile, but then very often they're not. This has led researchers to come up with scores of other ways to define species (Holliday (2003) has a great discussion on the matter).
Worse, there's no way to measure, genetically or morphologically, just how different things should be before they can be called different species. The late Morris Goodman and others (Wildman et al. 2003) argued that humans and chimpanzees are so genetically similar that chimps, now in the genus Pan, should be moved to our genus Homo to denote how similar we are. But any other, non-genetic comparison would put our chimp cousins in a very different group from us. Moreover, the effects of hybridization seem, to me at least, to be fairly unpredictable, at least superficially. That is, the outcome of hybridization is highly contingent on what animals are hybridizing, and on these lineages' own evolutionary histories (this is the intractable problem that made me abandon doing hybrid work for my dissertation. Some day though...).
A major issue relates to what I blogged about yesterday: both 'species' and 'hybrid' are terms we've found ourselves with, but they have no inherent meaning in themselves, other than whatever we give them. So it's funny to read this from Gibbons' story:
In the real world, [Jean-Jacques Hublin] says, Mayr's concept doesn't hold up: "There are about 330 closely related species of mammals that interbreed, and at least a third of them can produce fertile hybrids."
But is it Mayr's species concept that's flawed, or was it misguided to have put these hybridizers into different species in the first place? Should we delineate species based on our a priori conception about whether two things are different, or should a definition of 'species' determine what we call them? Or does it even matter?
To this end, Gibbons's other story describes the morphologically-unremarkable Denisova fossils as belonging to "a new type of human." Well, now what the eff does that mean? We're back to "The Species Problem" (the title of Gibbons's article), but with a new term. And pretend for a moment that the Denisovan fossils didn't yield DNA: the pinky and tooth probably would not have made headlines. Pretend they did have diagnostic cranial remains - would we have recognized them as being so distinct as their genes indicate?
For that matter, I wonder how many arguably 'modern' human fossils would still retain the modern moniker if we could analyze their genes...
Gibbons, A. (2011). The Species Problem Science, 331 (6016), 394-394 DOI: 10.1126/science.331.6016.394
Gibbons, A. (2011). A New View Of the Birth of Homo sapiens Science, 331 (6016), 392-394 DOI: 10.1126/science.331.6016.392
Holliday, T. (2003). Species Concepts, Reticulation, and Human Evolution Current Anthropology, 44 (5), 653-673 DOI: 10.1086/377663
Wildman, D. (2003). Implications of natural selection in shaping 99.4% nonsynonymous DNA identity between humans and chimpanzees: Enlarging genus Homo Proceedings of the National Academy of Sciences, 100 (12), 7181-7188 DOI: 10.1073/pnas.1232172100