Molecular Predictor Repository? (not gene set repository)

I have a simple question.  Say that I have the results from a gene expression analysis done in my laboratory or pulled from a public repository.  Say the sample has something to do with cancer (or I think that it might).  Say I read about so called 'signatures' that have been found to be associated with key phenotypes related to cancer.  (Here is a list of 13 signatures like this).

How do I now test to see which, if any, of these signatures are showing up in my sample?

I have my input, (e.g. the Affy CEL file from my experiment), how do I get the output that indicates that my sample shows an active wound response, suggests poor outcomes in breast cancer patients, looks like lung-specific metastasis, etc. etc.

This should be relatively easy, no?  I've got data about human gene expression, these people have made useful predictive models that take human gene expression as input.  Where is the website?

Some people have directed me to useful resources like GeneSigDB that provide curated repositories of "gene signatures".  However, these "signatures" are just sets of genes, they are not predictive models.  If all that we needed were gene sets, no one would ever need to train a random forest classifier or a support vector machine on the data associated with those gene sets.  Sets of phenotypically related genes are great, but I need the full predictive model.

The only system that I know of that seems to have the capacity to answer my question (had the model builders used it) is the Synapse platform.  For example, if you are good at R, you should be able to use Synapse to execute any of the models submitted to the recent breast cancer prognosis challenge.  This is a great step forward for the community (though it recapitulates pretty much everything from the more generic world of scientific workflow systems like Taverna).

But still.. a) comparatively very few published predictive models are in Synapse and b) should I really have to know R to answer that question?

respond!

Human Guided Forests - HGF

Yesterday I posted some slides about an idea I had recently which I call Human Guided Forests or HGF for short.  This an attempt to marry crowdsourcing with machine learning to produce better class predictors for datasets with very large features spaces.  Specifically, the idea is to replace the 'random' in the Random Forest algorithm with 'human'.

Random Forests basically work like this:
Given a labeled dataset with M input variables and N samples,

• Choose m as the number of input variables allowed per tree in the forest
• For X iterations:

1. Choose a subset of n samples from the training set
2. Select m random input variables
3. Build a decision tree using the randomly selected variables, all n samples (the 'bootstrap' or 'in bag' sample), and standard induction techniques (e.g. C4.5)
4. Measure the error rate for that tree on the samples not used to train it (the 'out of bag' or 'oob' samples)
5. Save the tree

After the forest of decision trees has been constructed, classify new samples by running them through all the trees and choose the class that is predicted most frequently.  (This is a very successful kind of 'ensemble classifier' that is similar to one that I, because of my ignorance, reinvented as one of my first projects in bioinformatics.)

This algorithm has been shown to be very effective at extracting good classifiers from datasets automatically.  However, as the random forest authors say:

 "But the cleverest algorithms are no substitute for human intelligence and knowledge of the data in the problem."  Leo Breiman and Adele Cutler

So, the question I'm posing is this: by inserting humans into the learning process can we improve it using their reasoning and background knowledge?

For HGF we replace the random input variable selection at step 2 above with expert guided variable selection.  (We may also let people guide the inference of the decision tree.)  The hypothesis is that the experts will choose feature sets that are better than the randomly selected ones - that will generalize to novel datasets with less error and produce more easily understood classifiers.  

As with standard RF, we need the HGF to produce many trees for each dataset with each tree having high classification performance and low overlap with the rest of the forest.  Ideally we would have each bootstrap of the training data converted into a tree by a different expert where the experts were drawn from a pool with highly diverse expertise.  This would require a significant investment of work from a large collection of expensive people..

Now, the next question is how on earth are we going to get a very large pool of skilled professionals to contribute their expertise to this project?  The answer we have been gravitating towards is games.  We hope to translate the feature selection problem into a game that knowledgable biologists and interested lay people will play for fun.

The formulation of the game(s) that will be used to drive an HGF implementation is very much a work in progress.  At the moment, the basic structure of our candidate games is that of a card game.  One way or another, players compose 'hands' of cards that correspond to features in a particular dataset.  For example, cards might correspond to genes from a gene expression dataset.  Hands are scored by testing the predictive performance of classifier trees inferred using the features in the hand and the training data (like one cycle of a random forest run).

Relation to Network Guided Forests

This idea is highly related to the concept of 'Network Guided Forests' (NGF) described by Dutkowski and Ideker in a PLoS paper last fall.  In that approach, the features used to build decision trees are constrained to related nodes within protein-protein interaction networks.  Features are selected for a given tree by picking one at random and then walking out along the network to bring in others in close proximity in the network.  The algorithm did not improve on classification performance in comparison to standard random forest as measured in cross-validation, but it did result in much more stable and coherent feature selection across several datasets.  It tended towards choosing genes that were known to relate to the phenotype of interest (e.g. breast cancer prognosis) much more often than random methods.  In comparison, NGF has the huge advantage that it can be used immediately based on data in databases without any dependence on human intelligence.  HGF has the theoretical advantage of tapping into a much broader collection of knowledge that is not limited to interaction data.  

Call for comments

At this point, this is just a nascent idea.  I have no evidence beyond intuition that it will succeed and there is a quite a bit of difficult work ahead to find out.  Any thoughts on it at this early point in time are most welcome!

freebase ED and sparql

So what do you do when the two papers you would like to finish and submit are sitting in the hands of co-authors?  Kayaking? Sleeping?  Surfing? today, no.  Hacking? today, yes.

While I wait, I decided to finally start working on bridges between freebase and our semantic tagging repository for ED for use after the data is collected. To get started, I wrote the code to answer this question: "what URIs have been tagged with the organism classification X or any of the sub classifications of X".  For example, has anyone tagged anything with magnoliopsida or any of its lower classifications, such as arabidopsis?

To do this, I needed to utilize the 'Higher classifications' (or of course 'Lower classifications') property of the Organism Classification Type.  Unfortunately, there is, thus far, no such thing as a generic transitive property in freebase as far as I can tell, so I built a brute force, recursive query that implements it myself.  I send the following with the '???' replaced with my starting point (e.g. 'magnoliopsida') to freebase as the rest of the URL starting with http://www.freebase.com/api/service/mqlread?queries= .

{"q1":{"query":
 [{
 "higher_classification" : "???",
 "name" : null,
 "guid" : null,
 "type" : "/biology/organism_classification"
 }]
}}

Freebase responds with the lower classifications of my query and then I repeat the process with these until either a maximum depth is reached or it bottoms out. If you know a better way to do this please let me know.

Once I have all of the guids for all of the lower classifications of my query, I send these over to get URIs tagged with any of them via a SPARQL query like this:

prefix tag: 
prefix rdfs: 
select ?tagging ?tag
where {
{
?tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f8000000000516f8d
}
UNION
{
?tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f800000000003be00
}
UNION
{?
tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f800000000572e4660
}
}

The query has as many UNIONs as topics to check for. (Note that you have to put URIs in SPARQL queries inside angle brackets - blogger was making this difficult for me to include). It works well enough, but if there are too many, I hit the max URL size limit (HTTP 414) so I set it up to send them in chunks and then reassemble the results.

Hacky? Yes. Successful for demo purposes? so far..

Any ideas about optimizing such activities most appreciated.

On the todo list:

1. Assemble the must-tag list of web services for the upcoming biomoby/ED jamboree
2. Build up an API-like library of queries like the above and normal queries like 'get all the URIs tagged by user X' so that we can more easily put up reasonable human interfaces for users of ED2.0. (Thanks to those that have already started using it!).  Note that any developers out there already have access to all of the data needed to build ED applications via HTTP calls to freebase and to our repository.  The library I speak of will be used by us and probably made public, but the real idea is for external developers to utilize SPARQL/MQL directly as that provides the most flexibility.
3. Create mappings between bio-ontology classes and freebase topics.  Likely follow Shawn Simister's model for approaching this integration.  (He has some excellent ideas about SPARQL/MQL integration).
4. Prepare for kayaking trip tomorrow
5. Graduate before they cut off my funding...

Back from Backpacking

Just returned from a much needed 3-day walk through the Glacier Peak wilderness area in the Cascades of norther Washington. Un-edited pics up here with some more to come from the other camera.
Thoughts about the trip:

1. pay whatever you have to for boots that fit and socks that wick or tragedy awaits


2. must make a habit of doing this sort of thing more often - good for body, mind, and soul

Thoughts that surfaced while walking:
Every time I do this sort of thing, I am amazed at the diversity of life I see and am curious to know more about it. I thought again about how cool it would be to have a handheld sequencing device + blast database so that I could identify whatever I saw. A great application of this would be to identify what I should and should not be eating... I tasted some excellent berries on the trip, but shied away from the plentiful mushrooms because of my ignorance and fear of death by poisoning. The handheld sequencer remaining impracticable at the moment I thought, hey - maybe we could do something with image recognition! I know a few people that are quite good at that sort of thing, maybe we could put together a database, train up some classifiers, and stick it into an iPhone or something. Finally, I realized that most of the functionality that I'd like to see could be achieved with a simple (though vast) set of classification rules that could definitely be stored in lots of different handheld devices. I like this a lot - though not as Star Trek as the tricorder type devices above, this would still appear to be pretty clever, but would take advantage of things that people are good at (seeing, labeling), things that machines are good at (storing and accessing vast amounts of information), and thus have the advantage of definitely being possible in the immediate future. If I ever have enough time, maybe I'll see if the folks at the Global Biodiversity Initiative have anything like this yet or if I might be able to use their data to build it.

Lets see.. anything else.. Ahh yes, I thought of a name for my next project for school - stay tuned for the P.R. machine...