[Taxacom] Reproducibility of phylogenetic analysis

[Peter Hovenkamp]

It is curious to see that scientists are worried by these objections to 
computerized methods raised by lawyers. They should rejoice.

It has long been recognized by philosophers of science that exact 
reproduction of observations does little to further scientific knowledge 
(see, e.g., Collins 1984, Franklin & Howson 1984, but similar statements 
can be found throughout the literature - although many seem to find this 
so self-evident that they think it unnecessary to state it explicitly). 
The argument, basically, is that an exact (I mean exactly exact) 
repetition of an observation can hardly count as an independent 
replication and is therefore, uninformative about anything except, 
perhaps, about the question whether the measuring apparatus (or computer 
algorithm, in this case) is working properly. Therefore, every single 
deviation, however slight, in the course of an analysis, takes us one 
step in the direction of an independent (and therefore informative) 
replication.

Illustrative examples are easy to find, but "subjective judgments about 
similarities and relationships" are not part of them.

Collins, H.M.  1984. When do scientists prefer to vary their 
experiments? Stud.Hist.Phil.Sci. 15, 169-174.

Franklin, A., Howson, C.  1984. Why do scientists prefer to vary their 
experiments? Stud.Hist.Phil.Sci. 15, 51-62.


Peter Hovenkamp


Bob Mesibov wrote:
> This article
>
> http://arstechnica.com/science/news/2010/01/keeping-computers-from-ending-sciences-reproducibility.ars
>
> is relevant to systematics, as it highlights an aspect of algorithm-based phylogenetic analysis which troubles some systematists.
>
> In the past, some Taxacomers have worried aloud that phylogenetic inference is a house of cards, implying (incorrectly, IMO) that if even one of the analytical steps is flawed, then the whole process is worthless. I and others have suggested here that algorithm-based phylogenetic inference is a closed, self-consistent process whose results are not tested by other means. Richard Zander has proposed an explicit statistical procedure to clarify the nature of Bayesian results, but I suspect his paper has been widely ignored.
>
> I'm not sure that any of this skepticism will make systematists any less enthusiastic about filling the literature with analyses, because trees are very useful and systematists are keen to build them using whatever tools are acceptable to their peers and their editors. What's worth keeping in mind is that most (imagine your own percentage here) of animal and plant taxonomy has been built by morphologists who made 'non-reproducible' and subjective judgements about similarities and relationships.
>
> Some of those judgements were wrong, and it's easy to feel superior to systematists who didn't have today's analytical tools. It would be a mistake, though, as the Ars Technica article implies, to try to escape responsibility for bad judgements by saying, 'Well, we used these data and these methods and got these results, so if they're wrong it's not our fault.'
>