Strange reality of academic workspace

My sister forward this job posting to me.

We are looking for a researcher or consultant to participate in the
collaborative development of Neural ElectroMagnetic Ontologies (NEMO). This
NIH-sponsored position would involve developing ontologies for
representation of patterns in event-related brain potentials (ERP) data
that reflect various aspects of language processing.

The Associate would work with members of our International NEMO Consortium
(experts in EEG and MEG studies of language) to develop, manage, and curate
ontology and database structures for this project. Please see the website
for more information, and contact Gwen Frishkoff if you have any questions.

The ideal candidate would have a background in cognitive neuroscience,
experience with database design and curation, and/or familiarity with
Protege/OWL ontology development software...

Sounds like an interesting job that matches my background pretty well, but read on..

Salary will be commensurate with experience in the range of $30-$40K per year.

Perhaps I'm just greedy...

shiny new pen

Another page of conclusions, about 17 corrected typos, and about 7 more forms later and I have reached the penultimate experience of academic student life: a free pen from the university!  After that comes the funny hat and then thats all she wrote.

For those silly people that were asking - assuming I did manage to find and sign all the needed forms - my dissertation should be available online within a few days.  

Now...  hmm  

So long 23rd grade!

After all the stress and the worry of the last few months I finally made it to the other side.  Now, I suppose its ok to call me Dr. Feel Good if you insist ;)

I will refrain from doling out advice to my formerly fellow 23rd graders and just give a couple highlights for posterity:

Best advice I received before the defense: "Have fun with it" - Nicole Quenneville.

Worst beginning to an answer I gave: "That is a really hard question!"  (laughter and heckling emerging immediately from the committee and the audience) - (Note that I did subsequently answer the question).  

Questions I had the hardest time answering: 

1. "What is bioinformatics?" - Francis Ouellette
   
2. "What emerging technologies do you think might have the biggest impact on the future of your research?"  - Wyeth Wasserman

Best response from an examiner when I said I was still deciding about my next career steps (hint hint), "No way, I'm crowdsourcing everything from now on!" - Paul Pavlidis

Thanks one more time to everyone that has helped and encouraged me along the way - you all rock!

non-anonymous peer review

I spent this afternoon acting as a voluntarily non-anonymous peer reviewer - its scary.  I ended up advocating rejection of the article I was reading and I have to say that Vince Smith(see end of linked post) was absolutely right that the act of signing your review "keeps you in check".  Knowing from the outset that your words are going to be linked to your name can really change what you have to say - it certainly makes you think about it for a while longer.  It is scary though - I hope that I managed to convey enough of my reasoning and suggestions for ways to improve the article that the authors don't despise me and attempt to ruin my life...  I also hope that the editors of the journal manage to acquire at least one additional reviewer for this manuscript - safety in numbers! Or perhaps the editors will strip my name from my comments?  Time will tell I guess.

getting attention

I sent the first complete draft of my dissertation to my committee last week.  Yay!  

At my last meeting with them they promised a quick response if I got it to them that day - even saying that they would reserve time specifically for reading it.  Unsurprisingly, most have not responded as they had promised - in fact most have not said anything.  Boo!  This means that either I will submit the thesis without having the benefit of their comments or I will miss my window for graduating this spring - double boo.

As many of you are no doubt aware, this is not an atypical situation.  Whether it is a 200 page thesis document, a 2 page report for the local hospital, or a technical article being prepared for publication, it can be very difficult to get essential feedback.  As Michael Nielsen has succinctly put it, the critical resource in science today is the attention of scientists.  My committee isn't ignoring me out of spite, they are just unbelievably overworked people with limited amounts of attention to meter out and I happen to be lower on their personal totem poles than their grant applications, their manuscripts, their other students, their husbands, their wives, (and probably their dogs and cats).  

The question this brings up is, when its critical that you receive some attention from a scientist or two, how do you go about getting it?  Lets make this more specific and ask, when you need some one to review one of your papers (prior-to, or rather-than a journal), how do you go about acquiring that needed attention?  

The only way that I have addressed this problem is by asking friends and family for help.  This works well (depending on your friends and family) up to a certain extent, but has some significant shortcomings:

1. they like you and don't want to make you feel bad - which may influence their assessment of your work
   
2. they may reside within the same information cocoon that you do - which means they may have little additional knowledge to contribute 
3. they eventually get tired of helping you out because they have their own problems to deal with

What other options are there? I suggest two that both revolve around markets.  The first market exists and the second is yet to be created.

$Market #1$

In discussing whether it would be worth my time to take a free scientific writing course offered at my institute, a professor who had taken the course really encouraged me to take it.  When I asked what she got out of it, she said that the most important thing that she learned was the value of working with a professional editor.  She now pays an editor to review every research paper and every grant that she submits.  I found that a little strange.  The most valuable product of a class is to learn that you need to pay some one to help you do what the class was trying to teach you?  Weird.  I would have dropped it there, but in another conversation with a very talented and well respected author, the same advice appeared.  To write at a professional level, getting professional help appears to be a vital component.

Now, as a student or a post-doc making lets just say not a lot of money, this advice is about as valuable as another suggestion that I love to hear from friends that actually have savings and real jobs (or rich parents), "oh, you should really try to buy a house, its such an important investment and now is such a great time to buy".  Great, thanks.  As soon as my scholarship check comes in I'll head out to the real estate agent...   Lacking funds to actually pay money to an editor, what could I possibly provide in exchange for some scientific attention?

Market #2

Well, according to some definitions, you might actually call me a scientist.  In fact, several journals have successfully taken my scientific attention from me (without any form of compensation) and handed it out to other scientists in the form of peer reviews.  Maybe I could claim greater control over this process?  Maybe there is a way to generate a market within which I could pay for attention when I needed it with my attention at other times.  I'm not referring to the perhaps more exciting 'collaboration markets' that Dr. Nielsen discusses, at least not yet, I'm simply referring to a market for the direct exchange of literary review in scientific contexts.  

Here is the essence of the deal; I will exchange my attention in reading and commenting on your paper in exchange for your attention on mine.  

Here are some of the additional complexities that might make an implementation of this idea interesting;

1. the chance to accumulate 'reviewer points' so that the system could go beyond barter and towards a more complete kind of market
2. the opportunity for anonymity for authors and reviewers to ensure that you can always say what you think you should say
3. the opportunity for the lack anonymity - for reviewers to be acknowledged in future iterations of the work that they review
4. the chance for participants in the system to establish levels of trust - some reviews really are more valuable than others and this should be recognized

I think something like this is vital.  It opens up a wide range of new opportunities for improving the way science works.  Papers could be 'published' within this system and gradually accumulate findability-enhancing credibility (and improvements).  Such assessments of credibility could be used to form a continuous rating scale for 'publications' that would replace the unnecessarily binary nature of journal-based publishing without losing the filtering effect touted by its proponents.

Such a system might improve on preprint archives like Nature Precedings by both providing a direct incentive for scientists to contribute comments on papers (most papers are never commented on at all at the moment) and providing a very direct approach to the filtering problem.  

If anyone is interested in creating something along these lines, let me know.  I hope that I will be needing a job soon.

p.s.  Thanks to Mikele Pasin for thoughts we shared on the 'Paper Demolisher' at KCAP 2007 that are directly related to this post.

fear and loathing in academentia

OK, I'm mad and probably shouldn't write the following today. Oh well.

Now, here is why I am mad. I've had a paper rejected by the Journal of Biomedical Informatics on the basis of one review. It took two months to get this review. The review does not seem fair and certainly does not provide useful guidance about how to improve the quality of the science described in the paper. Here is JBI's response in its totality, with some embedded reactions from me in red.

Ms. No.: JBI-08-163
Title: OntoLoki: an automatic, instance-based method for the evaluation of biological ontologies on the semantic Web
Corresponding Author: Dr. Mark Denis Wilkinson
Authors: Benjamin M Good; Gavin Ha; Chi Kin Ho;

Dear Dr. Wilkinson,

I requested that my advisor be the corresponding author on the paper because I would be traveling right after the submission and am hoping to relocate soon.

Experts in the field have now reviewed your paper, referenced above. Based on their comments, we regret to inform you that we are unable to accept your manuscript for publication in the Journal of Biomedical Informatics.

We have attached the reviewers' comments below to help you to understand the basis for our decision. We hope that their thoughtful comments will help you in future submissions to the JBI and in your future studies.

Sincerely,

Janine Burch
Journal of Biomedical Informatics, Editorial Office

Elsevier
525 B Street, Suite 1900
San Diego, CA 92101-4495
USA
Phone: (619) 699-6392
Fax: (619) 699-6211
E-mail: jbi@elsevier.com

Reviewers' comments:

Reviewer #2:

Its a little odd that we only got to see Reviewer 2's comments. I don't know if anyone else reviewed it or not.

Good and his colleagues present OntoLoki, a very interesting approach for data-driven ontology evaluation. The novel idea is that the quality of ontologies can be measured automatically although ontologies without or with very few formal restrictions on class membership are used. For poly-hierarchically organized classes suitable datasets with positive examples - i.e. instances with properties - as well as negative examples are composed. Machine learning algorithms are used to determine empirically those rules (patterns of properties) that allow predicting class membership reliably. With other words: The ideal situation is to find instances of classes like "Cat" with properties like "furry" allowing their consistent assignment to the class "Cat" and discrimination to neighbour classes like "Bird".

Yep, that pretty much sums up the general idea. So far, so fair.

There are a lot of inherent challenges with this approach that are addressed by the authors, e.g.
- the dependence on the context (chapter 1) and on the way of determining instances and their properties (chapter 1.2.1)
- the problem of sufficient number of instances for every class for estimating a class predictor (chapter 1.2.4)

These are common problems when using empirical approaches. However, the reviewer has doubts about the suitability of the OntoLoki approach for evaluating ontologies. The authors themselves admit that especially the results of the Cellular Component experiment are suboptimal, see chapter 3.2.1 (only 17% is evaluated) and chapter 3.2.3 (results are not overwhelmingly illuminating).

OK. At this point the reviewer has pointed out that we correctly identified challenges with empirical approaches to ontology evaluation and discussed them in respect to our approach in the paper. Both of these challenges, context-sensitivity and data dependency, are fundamental to any methodology that is based on the use of data to help answer a question. Keep in mind that the main point of the paper is to describe and evaluate a method. To do so, we explain it and then test it out in a variety of different scenarios (different ontologies and different datasets). In some cases it is successful and others it is not. By describing the results from all of these experiments we faithfully represent the realities of applying the method.

The reviewer criticizes the method by pointing out our own admissions regarding problems encountered with the dataset assembled for the evaluation of the cellular component branch of the gene ontology without actually saying anything about the method itself. Perhaps, criticism could fairly be placed on our data collection methods for that particular ontology. However, the point was not to evaluate that ontology it was to evaluate the proposed method. That 17% number resulted because we didn't collect enough instances to evaluate the other classes. If we collected more data, the number would have been higher, but that is completely irrelevant to the utility of the method and our evaluation of it. In fact, by including data like that, we much more accurately present both the positive and negative aspects of the method. Perhaps next time we should simply obscure any negatives to avoid such criticism.

The MAIN PROBLEM the reviewer has with this approach:
OntoLoki tries to solve a structural classification problem empirically that originates in poor defined ontologies. Instead of (suboptimally) trying to determine the consistency of ontologies with no formally defined restrictions on class membership the ontologies should be enriched by such formal definitions on class membership, see http://bioinformatics.oxfordjournals.org/cgi/content/abstract/22/14/e530.

Now, here is where this starts to get ridiculous. "the ontologies should be enriched by such formal definitions". Well, we couldn't agree more! That is one of the main reasons we did this! OntoLoki provides a starting point for doing just that!

As the reviewer seemed to understand in the summary of the paper above, the method is intended to be applied to ontologies (or whatever you want to call class polyhierarchies used in classification situations) that aren't necessarily formally defined. The rules that are learned could be used to suggest possibilities for formal class restrictions that are based on the data the classes are already associated with.

As a matter of fact, the OntoLoki method can actually be used on formally defined ontologies to identify candidate expansions of other definitions. We recognize the importance of these definitions and the reasoning they allow for, that is why the reference so generously provided above was one of the main citations in the paper! In fact, the ontology described in that paper was used as a benchmark of quality for other ontologies - thus providing us with a means to evaluate our method.

In the introduction Jeremy Rogers and later on Barry Smith are referenced as proponents of ontology evaluation. However, these and other researchers in the field of biomedical ontology are mostly concerned with the quality of explicit formal definitions and the structure of ontologies, see http://ontology.buffalo.edu/evaulation.html.

What exactly does the "however" mean here? Indeed, both of these scholars are involved in ontology evaluation and I would say are, in fact, proponents of the idea. Why the contrasting "however"? The quality of formal definitions and the structure of ontologies (which can of course result directly through inference applied to those formal definitions) are certainly aspects of relevance to the domain of ontology evaluation.

The structure of ontologies must certainly have something to do with the inferred or asserted class hierarchies they produce. The OntoLoki method is designed for evaluating these hierarchies. So.. why this statement ? Perhaps you could argue that the method is not useful in achieving the task, but it doesn't make any sense to say that the task is irrelevant as seems to be implied here.

The very idea of ontologies is to have explicit criteria for deciding class membership of instances opposed to ambiguous language terms denoting those classes. If there are artefacts with no formally defined restrictions they should not be called ontology.

Alright, now we've got to essence of this so-called "review". The reviewer doesn't believe that the things the method was built to evaluate should be called ontologies. So they don't believe the Gene Ontology is an ontology and they don't believe that most of the ontologies in the OBO foundry are ontologies. OK, fine. Perhaps the reviewer should have suggested that we change the title and used a different word to describe whatever it is these things are. The complaint has absolutely nothing to do with the manuscript! The maddening thing is that we have been (sometimes very lonely) proponents of the expanded use of axiomitized, property-based definitions in biological ontologies for years and are still very much of this view. To be criticized for the community's fairly slow uptake of these methods makes my head feel like its going to explode.

However, the machine learning methods are very interesting for supporting different purposes in the context of REAL ontologies WITH formal restrictions on class membership", see chapter "Making use of OntoLoki" in the discussion section. The whole paper should be rewritten oriented to those other supporting purposes in the context of developing, using and evaluating ontologies.

Well thanks. It seems that some of the applications of the method (and the software we developed) are "very interesting" but only in the context of "REAL ONTOLOGIES". As it turns out, the method and implemented code could be applied directly to REAL ONTOLOGIES without alteration. (Note that the capitalization is from the reviewer).

This paper, submitted as a paper in the Biomedical Informatics Journal, is a copy of a Technical report, see http://bioinfo.icapture.ubc.ca/bgood/OntoLoki_14.pdf.

That this is even mentioned as a presumed negative is outrageous. The report (which does in fact contain the same content as the submission) is not a peer-reviewed publication, it is simply a very informal pre-print. Posting it is perfectly in accordance with Elsevier's rules when it comes to pre-prints, rules that it is clear the reviewer is not aware of.

It is far to long and should conform to the editorial guidelines of the journal.

First, I actually agree that it is probably a bit too long. We discussed this at some length before deciding to submit the full version and, in the end, decided that the length was warranted in this case in order to present the argument and experiments in completion. We could shorten it, and likely will when we resubmit to a different (open-access) journal, but that was actually one of the reasons we chose JBI - they explicitly state that there is no "arbitrary limit on the length of individual articles". The submission was well within the editorial guidelines of the journal - guidelines which the reviewer, again, was clearly not familiar with.

Ok, my rant is over now, the red has drained out of my face and I can no longer hear my heart beating in my ears, so I will switch back out of the red to conclude.

So, Reviewer #2, who are you?

One of the more impressive people I met at SciFoo told me that he has been signing his reviews for years to "keep himself in check". If reviewers had to sign their reviews it seems that perhaps they might be forced to do a better job. Good quality reviews (either arguing for reject or accept) would provide another form of publication - another way for scientists to get credit for the work that they do. Are you up to it? Sign your next review.

Tick Tock

I've just returned from watching my second PhD defense in as many weeks. The first one was my younger sister's, and this one was a good friend of mine who started at UBC at the same time as me. I'm reminded of the time in my mid twenties when friends started getting married and disappearing, one by one. The obvious response is to wonder, "when will I be next?". Well it looks like it will be some time yet for me; however, I think I had a little bit of a conceptual breakthrough the other day that may help straighten my path a bit.

I've spent a bit more than half of the past year working with a professor from the library and information sciences on ways to generate useful comparisons of different "types" of information organization system. Throughout this collaboration we've struggled to both communicate with one another and write collaboratively because of the vastly different training and work practices of the social sciences versus the biological sciences.  The realization I had was that his concern is completely to do with information systems at the level of the Type - social tagging, semantic web, subject indexing, and so on, while my concern should really be with information systems at the level of the Instance - Connotea, the Gene Ontology, MEDLINE..   I think this is really the mark that distinguishes bioinformatics from information science,  at its heart it is a completely applied science.  I hope that keeping that up front in my mind will help me keep on track going forward.

For this particular project, now 4 months overdue, I'm now planning to change from building a general purpose framework for defining axes of similarity of semantic indexing systems to conducting a straighforward comparison of Pubmed and Connotea.  This comparison will be specifically designed only to identify ways that the data from Connotea might be used to enhance user's experiences with Pubmed.  I think this will make for more definable goals and easier justifications for the work though I will be using some of the more generic code and concepts we came up with along the way.  

In any case, well done Obi and Erin!

Ack .. PhD comic hits home again..

 As time goes on I'm shocked to see myself transition fully into Mike from the PhD comic..  Thanks (I think) for spotting this one Richard!

graddatical

Graddatical : a short period of study-related travel taken by a graduate student.

During the long and winding course of graduate school, I've been lucky to have the opportunity to take two graddaticals - the second of which is just finishing up today.  As I've defined it above, a graddatical is an opportunity for a graduate student to, in increasing order of importance:

1. take advantage of the main perk of student life - the relative freedom from strict time and location constraints (which is typically discussed in relation to the lack of money with which to frivolously enjoy that freedom)
2. take a break from your current surroundings which, particularly in the Nth years, may start to have the look and feel of a prison chamber regardless of the beauty of the campus or the character of your associates
3. spend time working closely with experts in your field who you would normally only ever interact with over email (if they bothered to respond to you)

The value of this last attribute of the graddatical really can't be overstated.  Though my personal path through the PhD jungle has probably been even more meandering than most, I think its safe to say that many students may find that their research takes them in many unexpected directions that can end up leaving them feeling somewhat, shall we say, intellectually lonely.  For example, you may come into and even select your graduate program thinking that you are going to be working on computational vaccine design, then follow a trail that leads you through machine learning through the semantic Web into classification theory; and from there wind up working on things that only people who manage libraries for a living actually care or know anything about.  Needless to say, its likely that you are now the only one in your lab and possibly your department and possibly even your city that cares about the semantic underpinnings of the Dewey Decimal system and thus find yourself with no one to talk to about your work.  A graddatical is what you need.

To have a successful graddatical:

1. identify some one who seems to be interested in what you are interested in and seems to know a lot more about it than you do
2. find a way to get to know them - ideally through a strong network connection like your supervisor
3. ask if you can come visit their lab/office for a week (most people will be happy to let you come hang out - if they aren't, you don't want to waste your graddatical on them)
4. make cheap travel arrangements..  this may be tricky but can almost always be done.  Keys for getting there are to either a) pick  somewhere that isn't too far or b) pick somewhere that is not too far from a conference you can get funded to attend somehow.  Another financial and fun related consideration is to request lodging on the couch of graduate students in the lab you are visiting.  Again, you will be amazed how welcoming people can be and it will make your experience both cheaper and more enjoyable.

Now, students of the world, go forth and get graddatical!