fear and loathing in academentia

OK, I'm mad and probably shouldn't write the following today. Oh well.

Now, here is why I am mad. I've had a paper rejected by the Journal of Biomedical Informatics on the basis of one review. It took two months to get this review. The review does not seem fair and certainly does not provide useful guidance about how to improve the quality of the science described in the paper. Here is JBI's response in its totality, with some embedded reactions from me in red.

Ms. No.: JBI-08-163
Title: OntoLoki: an automatic, instance-based method for the evaluation of biological ontologies on the semantic Web
Corresponding Author: Dr. Mark Denis Wilkinson
Authors: Benjamin M Good; Gavin Ha; Chi Kin Ho;

Dear Dr. Wilkinson,

I requested that my advisor be the corresponding author on the paper because I would be traveling right after the submission and am hoping to relocate soon.

Experts in the field have now reviewed your paper, referenced above. Based on their comments, we regret to inform you that we are unable to accept your manuscript for publication in the Journal of Biomedical Informatics.

We have attached the reviewers' comments below to help you to understand the basis for our decision. We hope that their thoughtful comments will help you in future submissions to the JBI and in your future studies.

Sincerely,

Janine Burch
Journal of Biomedical Informatics, Editorial Office

Elsevier
525 B Street, Suite 1900
San Diego, CA 92101-4495
USA
Phone: (619) 699-6392
Fax: (619) 699-6211
E-mail: jbi@elsevier.com

Reviewers' comments:

Reviewer #2:

Its a little odd that we only got to see Reviewer 2's comments. I don't know if anyone else reviewed it or not.

Good and his colleagues present OntoLoki, a very interesting approach for data-driven ontology evaluation. The novel idea is that the quality of ontologies can be measured automatically although ontologies without or with very few formal restrictions on class membership are used. For poly-hierarchically organized classes suitable datasets with positive examples - i.e. instances with properties - as well as negative examples are composed. Machine learning algorithms are used to determine empirically those rules (patterns of properties) that allow predicting class membership reliably. With other words: The ideal situation is to find instances of classes like "Cat" with properties like "furry" allowing their consistent assignment to the class "Cat" and discrimination to neighbour classes like "Bird".

Yep, that pretty much sums up the general idea. So far, so fair.

There are a lot of inherent challenges with this approach that are addressed by the authors, e.g.
- the dependence on the context (chapter 1) and on the way of determining instances and their properties (chapter 1.2.1)
- the problem of sufficient number of instances for every class for estimating a class predictor (chapter 1.2.4)

These are common problems when using empirical approaches. However, the reviewer has doubts about the suitability of the OntoLoki approach for evaluating ontologies. The authors themselves admit that especially the results of the Cellular Component experiment are suboptimal, see chapter 3.2.1 (only 17% is evaluated) and chapter 3.2.3 (results are not overwhelmingly illuminating).

OK. At this point the reviewer has pointed out that we correctly identified challenges with empirical approaches to ontology evaluation and discussed them in respect to our approach in the paper. Both of these challenges, context-sensitivity and data dependency, are fundamental to any methodology that is based on the use of data to help answer a question. Keep in mind that the main point of the paper is to describe and evaluate a method. To do so, we explain it and then test it out in a variety of different scenarios (different ontologies and different datasets). In some cases it is successful and others it is not. By describing the results from all of these experiments we faithfully represent the realities of applying the method.

The reviewer criticizes the method by pointing out our own admissions regarding problems encountered with the dataset assembled for the evaluation of the cellular component branch of the gene ontology without actually saying anything about the method itself. Perhaps, criticism could fairly be placed on our data collection methods for that particular ontology. However, the point was not to evaluate that ontology it was to evaluate the proposed method. That 17% number resulted because we didn't collect enough instances to evaluate the other classes. If we collected more data, the number would have been higher, but that is completely irrelevant to the utility of the method and our evaluation of it. In fact, by including data like that, we much more accurately present both the positive and negative aspects of the method. Perhaps next time we should simply obscure any negatives to avoid such criticism.

The MAIN PROBLEM the reviewer has with this approach:
OntoLoki tries to solve a structural classification problem empirically that originates in poor defined ontologies. Instead of (suboptimally) trying to determine the consistency of ontologies with no formally defined restrictions on class membership the ontologies should be enriched by such formal definitions on class membership, see http://bioinformatics.oxfordjournals.org/cgi/content/abstract/22/14/e530.

Now, here is where this starts to get ridiculous. "the ontologies should be enriched by such formal definitions". Well, we couldn't agree more! That is one of the main reasons we did this! OntoLoki provides a starting point for doing just that!

As the reviewer seemed to understand in the summary of the paper above, the method is intended to be applied to ontologies (or whatever you want to call class polyhierarchies used in classification situations) that aren't necessarily formally defined. The rules that are learned could be used to suggest possibilities for formal class restrictions that are based on the data the classes are already associated with.

As a matter of fact, the OntoLoki method can actually be used on formally defined ontologies to identify candidate expansions of other definitions. We recognize the importance of these definitions and the reasoning they allow for, that is why the reference so generously provided above was one of the main citations in the paper! In fact, the ontology described in that paper was used as a benchmark of quality for other ontologies - thus providing us with a means to evaluate our method.

In the introduction Jeremy Rogers and later on Barry Smith are referenced as proponents of ontology evaluation. However, these and other researchers in the field of biomedical ontology are mostly concerned with the quality of explicit formal definitions and the structure of ontologies, see http://ontology.buffalo.edu/evaulation.html.

What exactly does the "however" mean here? Indeed, both of these scholars are involved in ontology evaluation and I would say are, in fact, proponents of the idea. Why the contrasting "however"? The quality of formal definitions and the structure of ontologies (which can of course result directly through inference applied to those formal definitions) are certainly aspects of relevance to the domain of ontology evaluation.

The structure of ontologies must certainly have something to do with the inferred or asserted class hierarchies they produce. The OntoLoki method is designed for evaluating these hierarchies. So.. why this statement ? Perhaps you could argue that the method is not useful in achieving the task, but it doesn't make any sense to say that the task is irrelevant as seems to be implied here.

The very idea of ontologies is to have explicit criteria for deciding class membership of instances opposed to ambiguous language terms denoting those classes. If there are artefacts with no formally defined restrictions they should not be called ontology.

Alright, now we've got to essence of this so-called "review". The reviewer doesn't believe that the things the method was built to evaluate should be called ontologies. So they don't believe the Gene Ontology is an ontology and they don't believe that most of the ontologies in the OBO foundry are ontologies. OK, fine. Perhaps the reviewer should have suggested that we change the title and used a different word to describe whatever it is these things are. The complaint has absolutely nothing to do with the manuscript! The maddening thing is that we have been (sometimes very lonely) proponents of the expanded use of axiomitized, property-based definitions in biological ontologies for years and are still very much of this view. To be criticized for the community's fairly slow uptake of these methods makes my head feel like its going to explode.

However, the machine learning methods are very interesting for supporting different purposes in the context of REAL ontologies WITH formal restrictions on class membership", see chapter "Making use of OntoLoki" in the discussion section. The whole paper should be rewritten oriented to those other supporting purposes in the context of developing, using and evaluating ontologies.

Well thanks. It seems that some of the applications of the method (and the software we developed) are "very interesting" but only in the context of "REAL ONTOLOGIES". As it turns out, the method and implemented code could be applied directly to REAL ONTOLOGIES without alteration. (Note that the capitalization is from the reviewer).

This paper, submitted as a paper in the Biomedical Informatics Journal, is a copy of a Technical report, see http://bioinfo.icapture.ubc.ca/bgood/OntoLoki_14.pdf.

That this is even mentioned as a presumed negative is outrageous. The report (which does in fact contain the same content as the submission) is not a peer-reviewed publication, it is simply a very informal pre-print. Posting it is perfectly in accordance with Elsevier's rules when it comes to pre-prints, rules that it is clear the reviewer is not aware of.

It is far to long and should conform to the editorial guidelines of the journal.

First, I actually agree that it is probably a bit too long. We discussed this at some length before deciding to submit the full version and, in the end, decided that the length was warranted in this case in order to present the argument and experiments in completion. We could shorten it, and likely will when we resubmit to a different (open-access) journal, but that was actually one of the reasons we chose JBI - they explicitly state that there is no "arbitrary limit on the length of individual articles". The submission was well within the editorial guidelines of the journal - guidelines which the reviewer, again, was clearly not familiar with.

Ok, my rant is over now, the red has drained out of my face and I can no longer hear my heart beating in my ears, so I will switch back out of the red to conclude.

So, Reviewer #2, who are you?

One of the more impressive people I met at SciFoo told me that he has been signing his reviews for years to "keep himself in check". If reviewers had to sign their reviews it seems that perhaps they might be forced to do a better job. Good quality reviews (either arguing for reject or accept) would provide another form of publication - another way for scientists to get credit for the work that they do. Are you up to it? Sign your next review.

official reviews of E.D.

After more than three months, I've just received notification that the E.D. manuscript has been rejected for publication in the semantic mashup edition of JBI. I provide the reviews below and pose the question to you, the ether - what should I do now? Should I carry out some user-studies and resubmit? Should I make ammendments to the text as suggested by the second reviewer and resubmit? Should I send it to a different a journal? Should I give up on it and finish other pending projects?

?

Dear Mr. Good,

Experts in the field have now reviewed your paper, referenced above. Based on their comments and the number of submissions, we regret to inform you that we are unable to accept your manuscript for publication in the Special Issue "Semantic BioMed Mashup" of the Journal of Biomedical Informatics. One of the major concerns is that more work (e.g., a better use case) is needed for increasing the substance of the paper. You may consider revising the paper according the reviewers' comments and re-submit it to a regular JBI issue in the future.


We have attached the reviewers' comments below to help you to understand the basis for our decision. We hope that their thoughtful comments will help you in future submissions to the JBI and in your future studies.

Sincerely,
JBI Editorial Office

Reviewers' comments:

Reviewer #1: This paper could be a good workshop paper but it is not suitable for journal publication. The paper describes a prototype interface for a system that could potentially be useful, but provides no evaluation whatsoever. It doesn't even say if there are any users of the system. As the authors rightly note, there are many open questions, and even a simple user evaluation would have taken some steps in addressing those questions. Otherwise, it looks like an ad hoc exercise.

For instance, do the users use the suggested tags correctly, or is not being able to see the context for definition makes them select wrong terms? What happens if there are several tags from different vocabularies? Is the extra selection step too cumbersome and users won't bother? How is the agreement between users? How about the agreement with manually generated tags such as MeSH headings?

Without at least some evaluation, I don't think the paper can be a journal paper.

HOwever, the research goal is worthwhile and the approach interesting, so I would strongly encourage the authors to pursue it!

Additional comments: you compare the number of MeSH tags and Connotea annotations and suggest that the difference on the number of tags per item is somehow an indication of quality. I have a hard time understanding how the number of tags corresponds to teh *quality* of those tags. All this shows are the difference in scale.

I think the section motivating adding controlled vocabularies to social tagging systems (:Linking taggers and their tags...) is too one-sided. No potential drawbacks are discussed. What if users don't understand the tags from these controlled vocabularies and use them incorrectly? Would it be worse than not using them at all? Do users need to understand the vocabularies? Will non-rpofessional users know the vocabularies enough to use them without any special training? All this discussion must be present in the paper.

You say that you couldn't use the NCI Thesaurus and WordNet because they are too big for "Semantic Web technologies" This is not true. Many Semantic Web tools can process these easily; so it is a limitation of your technology.


Reviewer #2: This paper presents an application, the Entity Describer (ED), for
generating and storing controlled semantic annotations on biomedical
resources, as an extension of the Connotea social tagging system.
The authors briefly review semantic annotation (i.e., professional
indexing) in biomedicine and social tagging of Web resources, before
comparing the two. While professional annotation results in more
complete, standard and accurate sets of annotations, it is also not
sustainable due to its cost. The authors argue that the quality of
annotation through social tagging would improve if the taggers used
standard terminologies rather than homegrown tags. In order to explore
this hypothesis, they combined an existing social tagging system,
Connotea, with some controlled terminologies, including MeSH and GO. The
application is built -- using Semantic Web technologies -- as a mashup
of Connotea, terminologies and a database of annotations. The ED
modifies the Connotea interface to help users select terms from
controlled vocabularies and stores these annotations in a database,
while maintaining the usual features of Connotea. A prototype of this
application has been developed. The authors propose this annotation
model as an alternative to professional indexing and automatic
indexing. Future work includes making additional terminologies available
and applying ED to other tagging systems than Connotea.


This paper on enriching social tagging with controlled terminologies
through Semantic Web technologies is undoubtedly relevant to this
special issue. The paper is interesting and clearly written, easily
accessible to a readership that would not be familiar with the Semantic
Web. The references are appropriate.
This reviewer has essentially minor reservations about this manuscript
regarding the overall organization, statement of objectives, and the
discussion. These points could be addressed easily. The only major
reservation is the absence of proper discussion of the limitations of
this work.

Overall organization
The paper is composed of ten sections. Although logically flowing, this
succession of small sections might be distracting to the reader as it
fails to reflect the overarching organizational structure of the
paper. I would recommend grouping the first 3 sections under
Introduction/background and the next 4 under Materials and
Methods. Discussion and future work could be grouped.

Statement of objectives.
Again, it does not become clear until section 4 what the objectives of
this work are. I would recommend adding a short introduction to present
the issues of professional indexing and social tagging and stating that
the application presented proposes to reconcile them.

Discussion and future work.
In the future work section, rather than a litany of issues, it would be
useful to regroup the issues around terminology-related and
system-related issues. The issue of extension to other terminological
systems is presented in a somewhat naive manner. For example, it does
not look like the authors have fully appreciated the issues in making
the UMLS available through this system (e.g., size, lack of explicit
subclass relations, intellectual property restrictions, etc.)

Insufficient discussion of the limitations of this work.
The discussion is extremely short. The limitations of this work are not
clearly mentioned (lack of an evaluation [or even a metric for an
evaluation], scalability issues, etc.) Another limitation of this
approach is that you never make the point that professional indexing
relies not only on a controlled terminology, but also on a set of
indexing rules, used to further control the use of the indexing
terms. Finally, in your OWLization of MeSH, you briefly mention
converting broader/narrower links into subClassOf properties, without
raising any issues. What about Liver subClassOf Abdomen? I understand
this shortcut helps you meet the requirement that OWL be used in the
framework of this mashup. This is nonetheless highly inappropriate and
deserves being addressed in the discussion. The short paragraph about
using SKOS instead of OWL should be expanded and moved to the
discussion. The work of Guus Schreiber's group on representing MeSH in
SKOS should be acknowledged. http://thesauri.cs.vu.nl/eswc06/

Technical comment about MeSH.
Figure 7 shows hippocampus in MeSH identified by
"A08.186.211.577.405". Using tree numbers instead of the unique ID
(D006624) is bad practice. In this example, it so happens that
hippocampus occurs in only one hierarchy and has therefore only one tree
number. Most MeSH descriptors, however, have several tree numbers.
A side effect of this practice is that URI based on tree numbers would
result in multiple, non-reconcilable identifiers for the same MeSH
descriptor, leading to seemingly distinct annotations for the same
descriptor.


Minor comments
- Introduction: The sentence "Examples of semantic annotation
... UniProt [1-3]." would fit better between the first two sentences,
that at the end of the first paragraph.
- Introduction, 2nd paragraph: Arguably, the semantic annotation of
MEDLINE citations with MeSH describes *topics* more than it described
*entities*.
- p. 3: "The act of adding a resource to a social tagging collection" Do
you mean "The act of adding a tag to a resource"? The tagging *event*
is a process and cannot be composed of entities such as a tagger,
etc. please rephrase.
- p. 8: "formal training in classification" Do you mean "formal training
in *annotation* (or indexing)"? Classifying resources is a different
issue.
- p. 8: "main subject descriptors". The "official" MeSH terminology
refers to "main headings" (or "descriptors"), with the "major
descriptors" (marked by an asterisk) denoting the main topics in the
article. It is probably safer to stick to this terminology. In this
case, 12.7 must be the average number of descriptors, not major
descriptors.
- p. 9: Since your goal is to compare dispersion of the number of
descriptors in MEDLINE and Connotea, where the means are different,
the coefficient of variation should be used instead of (or in
addition to, for the purpose of the comparison) the raw standard
deviation values. For details, see:
http://en.wikipedia.org/wiki/Coefficient_of_variation
- p. 10: This section should introduce the notion of "controlled
vocabularies" or "controlled terminologies".
- p. 10: Arguably, what will decrease is not so much the quality of the
annotations as it is their *homogeneity*.
- p. 11: Please some background on GreaseMonkey.
- p. 12: It is unclear why the term "Hip" in MeSH (D006615)is not
retrieved as part of the list of terms suggested for the entry "hip".
- p. 15, first line: "if a non-annotation property and was used..."
Remove and.
- p. 15, later: "it would render the knowledge base OWL-Full". you
probably mean: "it would require OWL-Full for the representation of
the KB"
- p. 17, bullet 1: It is unclear what is the justification for
suggesting the addition of these particular terminologies.
- p. 18, bullet 3: tree-like interfaces would be extremely inconvenient
to render biomedical terminal terminologies with a high degree of
multiple inheritance.