Google buys Metaweb

If there was any previous doubt about whether 2010 will be known as the year that the Semantic Web came of age, this acquisition seals the deal.

I wonder if my life would be different if I had listened to my wife's advice and walked up to Danny Ayers at sci foo in 2008 and asked for a job... His three word intro "just, an, engineer" still inspires me.

Well done Danny and friends!

Freebase authors semantic web book

I just noticed this post on the freebase dev blog - announcing the release of a new book about "programming the semantic web" written by freebasers Toby Segaran, Colin Evans, and Jamie Taylor. I haven't picked it up yet, but based on the description there and in the amazon reviews, it looks like a(nother) nice explanation of the concepts involved in the semantic web as well as a set of practical programming examples based on the W3C standards (OWL, RDF, SPARQL, etc.).

I found it surprising that freebase was not mentioned anywhere in the brief description given in the post or on Amazon.

tightening the knowledge cycle

I've recently returned from a trip to the Leiden University Medical Center where I met with Barend Mons, Marco Roos, and Eric Schultes regarding the formation of what Barend is calling the Concept Web Alliance (CWA) (as well as some open positions within his groups at the LUMC and the University of Rotterdam).

In preparing for my presentation I read the WikiProteins article again (from Barend's group) and was struck with the similarity of the pattern I saw there with some of my old work and things that are emerging right now at Freebase and elsewhere. It seems that there is a very clear uptake of the following basic cycle in the context of building large knowledge bases on the Web.

1. Datamining (from text or other sources) generates many many candidate assertions
2. The assertions are presented to (many) people who manually correct/refine them

These two basic steps are being united everywhere right now. In WikiProteins we see natural language processing techniques seeding a semantic protein Wiki where people (presumably scientists) can correct/extend the predictions. In Google search we see the familiar products of datamining in the ranked result lists but these are now coupled with interfaces that allow users to 'vote-up' specific results. Freebase, which makes extensive use of automated knowledge acquisition techniques, is now unveiling a series of 'games with a purpose' like TypeWriter and Genderizer that are being used to validate the predictions of their algorithms (see a nice post about the Freebase process).

A somewhat more hidden indicator of this trend is that Dolores Labs - a company that seems to be entirely devoted to the application of Amazon's Mechanical Turk - is now hiring. Its hard to tell what they are doing internally, but it would be surprising if the interplay between datamining and the wisdom of crowds didn't play a strong role.

These trends will place increasing pressure on the scientific/tech community to come to a better understanding of the processes involved in motivating, coordinating, and aggregating the knowledge of many millions of minds in the formation of structured knowledge.

Authority, ontology, and Freebase

Something I knew about but hadn't thought through until Kirrily Robert pointed it out to me yesterday at the Vancouver Freebase meetup is the importance of the 'as_of_time' parameter in freebase.  This allows developers to access views of freebase data at any point in time.  The important consequence of this is that it makes it easy to create authoritative versions of data in freebase.  This means that if, for example, the gene ontology were loaded into freebase on a particular day by a particular trusted authority, users of freebase could access versions of the GO topics exactly as they appeared at that moment.  This answers the common worry from biologists about trust and authority.  The 'oh no, I wouldn't use anything that could be edited by just anyone' phenomenon goes away and the best of both worlds becomes a possibility.  With an open app like freebase you get the wikipedian potential for rapid, parallel, collaborative knowledge formation.  With this kind of versioning, it should be possible to create trusted, authoritative versions at any point in time.

huge.

freebase ED and sparql

So what do you do when the two papers you would like to finish and submit are sitting in the hands of co-authors?  Kayaking? Sleeping?  Surfing? today, no.  Hacking? today, yes.

While I wait, I decided to finally start working on bridges between freebase and our semantic tagging repository for ED for use after the data is collected. To get started, I wrote the code to answer this question: "what URIs have been tagged with the organism classification X or any of the sub classifications of X".  For example, has anyone tagged anything with magnoliopsida or any of its lower classifications, such as arabidopsis?

To do this, I needed to utilize the 'Higher classifications' (or of course 'Lower classifications') property of the Organism Classification Type.  Unfortunately, there is, thus far, no such thing as a generic transitive property in freebase as far as I can tell, so I built a brute force, recursive query that implements it myself.  I send the following with the '???' replaced with my starting point (e.g. 'magnoliopsida') to freebase as the rest of the URL starting with http://www.freebase.com/api/service/mqlread?queries= .

{"q1":{"query":
 [{
 "higher_classification" : "???",
 "name" : null,
 "guid" : null,
 "type" : "/biology/organism_classification"
 }]
}}

Freebase responds with the lower classifications of my query and then I repeat the process with these until either a maximum depth is reached or it bottoms out. If you know a better way to do this please let me know.

Once I have all of the guids for all of the lower classifications of my query, I send these over to get URIs tagged with any of them via a SPARQL query like this:

prefix tag: 
prefix rdfs: 
select ?tagging ?tag
where {
{
?tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f8000000000516f8d
}
UNION
{
?tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f800000000003be00
}
UNION
{?
tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f800000000572e4660
}
}

The query has as many UNIONs as topics to check for. (Note that you have to put URIs in SPARQL queries inside angle brackets - blogger was making this difficult for me to include). It works well enough, but if there are too many, I hit the max URL size limit (HTTP 414) so I set it up to send them in chunks and then reassemble the results.

Hacky? Yes. Successful for demo purposes? so far..

Any ideas about optimizing such activities most appreciated.

On the todo list:

1. Assemble the must-tag list of web services for the upcoming biomoby/ED jamboree
2. Build up an API-like library of queries like the above and normal queries like 'get all the URIs tagged by user X' so that we can more easily put up reasonable human interfaces for users of ED2.0. (Thanks to those that have already started using it!).  Note that any developers out there already have access to all of the data needed to build ED applications via HTTP calls to freebase and to our repository.  The library I speak of will be used by us and probably made public, but the real idea is for external developers to utilize SPARQL/MQL directly as that provides the most flexibility.
3. Create mappings between bio-ontology classes and freebase topics.  Likely follow Shawn Simister's model for approaching this integration.  (He has some excellent ideas about SPARQL/MQL integration).
4. Prepare for kayaking trip tomorrow
5. Graduate before they cut off my funding...

ED2.0!

The Entity Describer is back and, we think, much better.  Give it a try from its new home at entitydescriber.org and please do let us know what you think. 

There is more information on the website, but briefly, the Entity Describer enhances the Connotea experience by enabling Semantic Tagging with Freebase.  Essentially it makes it both easy and enjoyable to say what you mean when you tag something.

 +  = 

centralized content and decentralized control

No time for depth of thought here, just wanted to quickly jot down some current observations and recent links to ideas related to the future of the semantic Web.

In chronological order

1. Early January 2008, the annual NAR database issue comes out listing more than 1000 distinct databases in molecular biology.
2. Shortly after, Duncan Hull complains that essentially none of the "dark data" residing in these databases will ever be used because of the near complete lack of both syntactic and semantic interoperability between these isolated, decentralized silos.
3. I participate in writing up a paper about the Banff Manifesto, which, among other things seeks to improve cross-database interoperability through the introduction of a single, open, centralized, resource for defining public namespaces for use in the construction of unique identifiers.
4. I am forwarded a link to a Wired blog post about Google base - Google providing a centralized repository for scientific data.
5. the Freebase dev blog quotes an article about Wine tasting websites that indicates that Freebase is an example of the semantic Web (which they also refer to as Web3.0) and that everyone that creates a community-directed database should be using Freebase to do so - because of the dramatic advantages of centralization.

Notice anything interesting here?

It seems that many people think that success in achieving the goals of the semantic Web is really all about centralization of content. This of course, seems to smack against the decentralized roots of the first Web and is, in that way, rather disappointing. On the other hand, it also seems that success on the semantic Web is really all about decentralization of content control - which was, IMHO, the fundamental characteristic of the first Web that made it such a success (providing it with the coveted capital letter).

Is it possible to achieve a useful giant global graph without centralizing content?

Not as far as I can tell right now.

In fact, the current Web would hardly be very useful without massive efforts to centralize its content. How useful would the Web be if Google and others stopped downloading it and indexing its content on their servers, thus depriving us all of our god given right to find out anything about anything in milliseconds??

Extreme Writing for ISMB

Over the last 48 hours or so I participated in a Google-Doc enabled Extreme Writing session as the lead (English-speaking) author of an article submitted to ISMB. The article describes a new area of the Freebase knowledge space meant, in addition to quite a number of other things, to capture knowledge about the interconnectivity of different biological databases. (See the temporary development version in the Sandbox or the eventually more permanent URI). The idea for this project, which has evolved a bit from the description in the Wiki available today, and the actual work of building this resource came almost entirely from Francois Belleau of the Bio2RDF project - he invited me to help with the writing and to add my two cents about knowledge gardening on the Web, a subject I am supposed to be becoming an expert in somehow.

Here are a couple snippets from the paper:

"... a grander vision of integrative bioinformatics. In this vision, researchers or their computational agents not only discover and access all the databases that they require, but also clearly understand how each entity in each database relates to the entities in all the others. The information required to realize this vision can be conceptualized as a map that, rather than describing the interconnectivity of points in physical space, describes the interconnectivity of biological entities in the context of the Web..."

"...Right now, this meta-database makes it possible to answer queries about the connectivity of the various data sources but does not yet enable queries of the connectivity of their individual components. Before it is possible to 'zoom in' in on the lowest level entities in the global map of bioinformatics data, it is first necessary to establish a consistent strategy for their identification. The bioinformatics meta-database presented here provides a centralized, community-governed repository of public namespaces that we propose might serve as the foundation of a global unique identifier system for bioinformatics on the Web. This identifier system is based on principles outlined in the Banff Manifesto (BM) instigated at the 2007 World Wide Web conference and presented here for the first time..."

It was an exciting, intriguing, sometimes frustrating experience simultaneously co-authoring this document with 5 other people (Francois, Michel Dumontier, Mark Wilkinson, Marc-Alexandre, Jamie Taylor). Overall, Google docs handled the experience surprisingly well. A quick look indicates that it captured 5925 revisions over the past three days. My only major complaint with it was that the screen tended to jump all over the place when lots of people working at the same time. I guess this may be one mechanism they use to keep people from editing in the same text area before its had a chance to sync everyone up for that area (otherwise it wouldn't know what to display). To avoid this problem to some extent and to make the work a bit more efficient we evolved strategies for social locking of different sections of the document. It would be great if support for this could be added to G-Doc - for example, a user could select a segment of text and request a lock on it that was enforced by the Google system.

In the end, the paper really could have been much a better with a some more time to put it together. That being said, it does have some interesting ideas in it and so might just squeak into the conference. If it does, Francois and co. will certainly produce a very exciting presentation by the time the conference finally comes to pass. If not, we'll try try again.

ED2.0 underway

Manuscripts and academentia aside, we want to make E.D. better. To do so, we're planning to take advantage of the FreeBase type-ahead box pictured here.

I've been thinking about some sort of integration with FreeBase since I saw Jamie Taylor's "Stone Soup" presentation at WWW2007 and now the time seems to be ripe. "We" now includes Francois Belleau of bio2rdf fame. Francois is working on integrating bio2rdf with FreeBase and thus taking advantage of, among other things, their extremely fast graph query engine. By embedding the source vocabs (e.g.MeSH, GO..) in FreeBase, we will be able to use their (also extremely fast) type-ahead. This should eliminate the slowness and the memory problems we've had in the current implementation as well as providing a nice way to access ~way more terms. With Francois' work to map back and forth between freebase and RDF, I think this will end up being a much more killer application than v1.0.