Thursday, April 10, 2014

Microtask crowdsourcing and biocuration

Thanks to the hard work of my coauthors @x0xMaximus and @andrewsu , I was able to nab the award for the best presentation at The Seventh International Biocuration Conference from the International Society of Biocuration.   The slides for the presentation and the poster are available from slideshare.

Yay team!

I think the presentation garnered the interest it did because many of the people in the audience had heard the term "crowdsourcing" before, but had never seen a real example of a specific application - let alone one in science.  I was surprised by the number of people that I spoke to that had no idea what the Amazon Mechanical Turk was - nevermind that it might be applicable to some of the problems they were working on.  We had a decent result to talk about, but much more importantly, we taught the audience about a powerful new tool that they might be able to use in their own work.

For those that do want to try scientific applications of microtask crowdsourcing I'd like to emphasize that its probably not going to be an easy process.  The result we presented was from the third iteration of our system and represents several months of developer time.  While resources are emerging that should make this process much faster to get started (e.g. [1-4]), expect to engage in an iterative cycle to get your system dialed in!

If you do want to give crowdsourcing a try for biocuration or other scientific objectives, (1) we would love to hear about it! and (2) it might be worth a quick look at our review of the domain [5].  Microtask systems such as the one we worked with here are just one of many ways that scientific challenges can be opened up to much broader communities.

References
  1. Our code: mark2cure 
  2. Soltilab mention tagger for crowdflower
  3. GATE crowdsourcing plugin 
  4. Crowd Watson from IBM
  5. Good, Benjamin M., and Andrew I. Su. "Crowdsourcing for bioinformatics" Bioinformatics 29.16 (2013): 1925-1933.

3 comments:

Anonymous said...

Really interesting Benjamin, thanks​!​

I think that you would be really interested in some of the most cutting-edge research that I have come across explaining crowds, open innovation, and citizen science.​

http://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=1919614

And you may also enjoy this blog about the same too:
https://thecrowdsociety.jux.com/

Powerful stuff, no?

Anonymous said...

congrats!

> I'd like to emphasize that its probably not going to be an easy process
I concur (we've figured that out the hard way)

Benjamin Good said...

Yeah.. we are on the 8th iteration of this experiment right now. I think people should view this as the time when the Web-lab protocols are being worked out. Eventually I'm guessing that there will be 'kits' that make this sort of thing routine.