Semantic media retrieval service? please?

Here is an application of semantic Web technologies that I would like to have. Please make it for me so that I don't have to. When I finish writing this post,

1. I would like to press a button that said "Enhance?".
2. When I pressed the button, the application would read through the text and identify terms, phrases, or other conceptual nuggets that it 'understood'.
3. These concept nuggets would then be used to find stock / open access images (and videos, etc.)
4. Where a likely candidate set of images was identified, they would be displayed such that I could quickly choose which, if any, that I liked
5. When I agreed to keep one, it would be embedded in a reasonable location in the text and I would very rapidly go on with my life, but with the added joy of having authored a much more entertaining piece of online personal history.

This thought crept into my mind after reading through Joey de Villa's post about joining Microsoft which is shot full with entertaining media enhancements to the text - which likely took a non-insignificant amount of time for him or his team of personal assistants to put together.

Pictures are indeed worth many words, but how many $$$'s? Perhaps you might even be able to make money with such an app by using it to sneakily sell professional photos and other content.

While you are at it, could you please provide the same text-to-media service in a non-embedded application so that when I needed a clever portrayal of a concept like 'failure', 'success', or 'mass collaboration', for a presentation I could quickly look one up. I might even be willing to by it if the content was good and the price was reasonable -> in a world where I could almost certainly find what I needed by spending a little more of my own valuable time looking for it.

BioMoby Annotation Jamboree

If you happen to be a BioMoby developer, a bioinformatician, or just interested in helping us out and possibly making $20 of gift certificate remuneration, come have a go at annotating some BioMoby web services with a specialized version of the Entity Describer.  You will need to use Firefox, Safari, or Opera - no Explorer support so far.. and jam on over to the jamboree website.  

Warning, it is seriously difficult..

We've been working hard to produce a system that makes this task (a) possible and (b) as intuitive as possible. However, even when you know what you are doing and have figured out how to use the interface, the task of describing things you don't necessarily know intimately with a vocabulary you are also unfamiliar with is a real challenge.  I'm very curious to see what people end up doing and to assess how well we can pull their collective efforts together.

Working on this project has reminded me just how difficult interface design (and implementation in javascript) really is.  It is absolutely stunningly surprising to watch people try to use a tool like this for the first time.  They simply never do what you expect them to, nearly everyone seems to react differently, and no one reads the instructions (even while I stand right beside them encouraging them to do so).  The challenge of building software to aid in accomplishing a task that most users will never have done before to satisfy a purpose they ubiquitously have difficulty grasping has been quite an adventure in iteration.  We think we have something that works, finally, but we won't know until you give it a try now will we ;).

freebase ED and sparql

So what do you do when the two papers you would like to finish and submit are sitting in the hands of co-authors?  Kayaking? Sleeping?  Surfing? today, no.  Hacking? today, yes.

While I wait, I decided to finally start working on bridges between freebase and our semantic tagging repository for ED for use after the data is collected. To get started, I wrote the code to answer this question: "what URIs have been tagged with the organism classification X or any of the sub classifications of X".  For example, has anyone tagged anything with magnoliopsida or any of its lower classifications, such as arabidopsis?

To do this, I needed to utilize the 'Higher classifications' (or of course 'Lower classifications') property of the Organism Classification Type.  Unfortunately, there is, thus far, no such thing as a generic transitive property in freebase as far as I can tell, so I built a brute force, recursive query that implements it myself.  I send the following with the '???' replaced with my starting point (e.g. 'magnoliopsida') to freebase as the rest of the URL starting with http://www.freebase.com/api/service/mqlread?queries= .

{"q1":{"query":
 [{
 "higher_classification" : "???",
 "name" : null,
 "guid" : null,
 "type" : "/biology/organism_classification"
 }]
}}

Freebase responds with the lower classifications of my query and then I repeat the process with these until either a maximum depth is reached or it bottoms out. If you know a better way to do this please let me know.

Once I have all of the guids for all of the lower classifications of my query, I send these over to get URIs tagged with any of them via a SPARQL query like this:

prefix tag: 
prefix rdfs: 
select ?tagging ?tag
where {
{
?tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f8000000000516f8d
}
UNION
{
?tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f800000000003be00
}
UNION
{?
tagging tag:associatedTag ?tag .
?tag rdfs:isDefinedBy http://www.freebase.com/view/guid/9202a8c04000641f800000000572e4660
}
}

The query has as many UNIONs as topics to check for. (Note that you have to put URIs in SPARQL queries inside angle brackets - blogger was making this difficult for me to include). It works well enough, but if there are too many, I hit the max URL size limit (HTTP 414) so I set it up to send them in chunks and then reassemble the results.

Hacky? Yes. Successful for demo purposes? so far..

Any ideas about optimizing such activities most appreciated.

On the todo list:

1. Assemble the must-tag list of web services for the upcoming biomoby/ED jamboree
2. Build up an API-like library of queries like the above and normal queries like 'get all the URIs tagged by user X' so that we can more easily put up reasonable human interfaces for users of ED2.0. (Thanks to those that have already started using it!).  Note that any developers out there already have access to all of the data needed to build ED applications via HTTP calls to freebase and to our repository.  The library I speak of will be used by us and probably made public, but the real idea is for external developers to utilize SPARQL/MQL directly as that provides the most flexibility.
3. Create mappings between bio-ontology classes and freebase topics.  Likely follow Shawn Simister's model for approaching this integration.  (He has some excellent ideas about SPARQL/MQL integration).
4. Prepare for kayaking trip tomorrow
5. Graduate before they cut off my funding...

Peter (Google) and Christine (the librarians)

I was very lucky to have my new wife with me at SciFoo for many reasons, not the least of which is that she is much better at socializing than I am and thus managed to introduce me to many people I would never normally have met.  One of those people was Christine Borgman, Professor & Presidential Chair in Information Studies at UCLA.  While I was struggling to explain my work on the Entity Describer project to her, she noticed Peter Norvig walk by and dragged him over to join our conversation.  I guess she must have known him from somewhere but I'm not sure where.  Anyway, I didn't realize this at the time, but Peter is head of research at Google.  Ahem... did I mention that SciFoo interactions could be somewhat intimidating?  So there I am, standing between two giants of modern information science trying to explain what it was I was doing there but mostly trying to get some insight into their respective thoughts on the organization of the world's information. It wasn't a long discussion, but here are the basics.

The main question that I posed to them was whether or not and how semantic tagging (a la ED) is or might be useful.  On the surface, the answer from Peter was no and the answer from Christine was yes.  However, the truth of the matter is that they were really talking about supporting different functions for the end user - though this fundamental difference became a little lost during the conversation.  Google is principally focused on providing the best possible results, to the most people, given the least amount of information in the query - that is, keyword based search of the entire Web.  Library-science is typically much more concerned with providing the capacity for people to make very specific requests using much more sophisticated queries that operate over much smaller collections of information (e.g. the library of congress). The fact that there is some overlap in the information needs of the users of these different kinds of systems often brings up the desire for combative  comparison, but I think that, in reality, there is clearly no need for combat because they are simply too different in the functions that they intend to provide.

My interpretation is that Google isn't really concerned with intentionally provided meta-data in the name of end-user, full-Web search because the scale that they operate on seems to render any such indexing by one or even a number of parties almost laughably shallow in its characterization of both the nature of any particular item and its expected relevance to a query.  When you have literally millions of people passively voting and indexing every item of the Web through their decisions to link to it or not, you have very sophisticated algorithms for understanding the text in the pages generating and receiving those links and to top it off you record and process millions of people's behavior when faced with your search results, why should you care what some person or institution says the item is about?  The fact that they (among other search engines) beat out the directory-based approach to finding information on the Web is a clear demonstration that automatic indexing and link based relevance ranking do a better job than meta-data based classification - for the problem of Web scale search.  Google clearly doesn't need human semantic indexers to succeed, though, as Peter said, they certainly use all of the information that exists.  If there happen to be good indexes online (as Connotea turned out to be be for a fairly brief window), then their algorithms will certainly find them and use them - if not, no worries, the algorithms will take advantage of the 'normal' data on the Web and do just fine thank you. 

From the library-science professional perspective, this attitude is clearly annoying.  If human indexing isn't really necessary to find things, what is the point of the field that has devoted itself to the creation of effective ways for people to categorize things for retrieval?  For example, there is a lot of annoyance that the Google Books initiative seems to ignore most, if not all of the meta-data already associated with the books that they are scanning and indexing.  This means that meta-data, even as basic as volume numbers, is inaccessible for searching.  For the library professional that is trained to both search through and construct careful and precise classification structures,the inability to even search for a specific volume of a book is infuriating - particularly knowing that it is well within Google's power to incorporate such abilities into their system.

So on the one side we have the perspective that there is still value in the careful, intentional use of meta-data in the search and retrieval process while on the other side we are quite happy to let the intersection of algorithm and massive passive indexing do the work.  I guess, as is the usual answer, I'd suggest that both sides provide useful functions that are both worth keeping and advancing.  A detailed classification system, either constructed intentionally through the work of professional labor or semi-intentionally through the work of social taggers, provides functionality that is clearly different than what can be achieved by automatic indexing; however, it may not provide any help whatsoever in improving a full-Web-scale keyword-based search.  The essence of the power of intentional classification is the precision of the queries that it enables.  For example, if I want only version 3 of "The Devil's Rights and the Redemption" and thats it or I want only those items that have been tagged as with bioinformatics and to_read by Jaa, there is really no way ( AFAIK) to accomplish this without the intentional recording and utilization of meta-data about those resources.

So, though Peter and Google may have little direct use for ED and its semantic meta-data generating and consuming brethren emanating from the library and information sciences, there are still clearly meaningful applications of such work.  It just happens that providing effective search over the contents of the entire Web based on a string like 'Britney Spears' isn't really one of them.

I'm ok with that.

updates to ED

Since the last post, we've made a couple minor changes to the ED interface based on feedback we've received. Changes visible in the current version were intended to meet these objectives

1. clear separation between normal user created tags and semantic tags 
2. easier access to previously added user tags
3. easier to add many semantic tags to the same post

We also noticed that several new users were tagging exclusively with private posts.  To clarify the fact that private posts don't go into our database and that the semantics of the tags added to private posts are therefore inaccessible, we've added a warning message that appears whenever private bookmarks are posted through the system.  In the words or Eddie, the javascript guru of this project, 

"... go private they will,

annoy them we must ..."

We may at some point expand the ED functionality to support private bookmarks, but for now, we think its fair that if people would like to use a free, social system they share openly.  Private bookmarks will still be posted to Connotea as they currently are without ED and the non-sharing users will still benefit from the semantic tags during the process of tagging.

I've also gone ahead and added ED to the growing list of semantic tagging projects that I am beginning to compile.

Semantic Tagging Projects

As I mentioned in the last post, I've recently discovered a few other social bookmarking services that utilize semantic tags. (Though they all have their own take on what "semantic" means of course). Here are some links to the ones I've found so far, please let me know if I've missed any.

1. ZigTag is ".. an intelligent, semantic, social tagging and bookmarking service". ZigTag is a new company, based in Edmonton, that seems to be seeking to replace Delicious as the de facto standard for social bookmarking on the web. Its semantic tags are drawn from its own database, culled automatically from public sources and soon to be made API accessible. The service works as a FireFox (Explorer in development) side-bar extension and as a bookmarklet. They are currently in private beta.
2. Fuzzy is "a Web 2.0 organic ontology collaborative socio-semantic polyscopic web research project". It is currently the product of Roy Lachica, a graduate student at the University of Oslo. The description of the project on the about page of the website is, intentionally I believe, fuzzy... but I did gather that the representation used for the semantic tags is based on topic maps and that the tags are created by the users using editing tools available on the website. I found it a bit strange that the tagging activity seems to be separate from the bookmarking activity. When I went to add a bookmark, the bookmarklet let me specify the URL, a name, whether it was private, a description, what kind of resource it was {webpage, tool, video, etc.}, geographic context, mood {fact, fun, business, or compassion}, knowledge type {why and if, how, what where who when} and details level {overview, detailed}. However, there wasn't any option to tag the post. This happens later on, within the context of the fuzzzy website. This seems to be part of their drive to get users actively editing their "folktology" of tags. For more info, have a look at a fuzzzy conference paper.
3. If Fuzzzy is semantic tagging, then I think Bibsonomy must also be classified as such because it does allow its users to establish relationships between the tags. Bibsonomy is also an academic project and one that has been operational for several years.
4. SemKey is (was?) an Italian academic project that utilizes WordNet and Wikipedia topics as its sources of semantic tags. It was actually presented at the 2007 World Wide Web conference while I was attending a different workshop.. I can't believe I missed it! A quick search didn't turn up any working versions, but they may be lurking out there somewhere.
5. Faviki is the latest semantic tagging project to emerge. It is a google-app engine project that utilizes DBpedia as its source of semantic tags. One thing that I didn't like about it was that there currently doesn't appear to be any way to use tags that aren't in the database. Given my experience so far with the entity describer, which accesses about a million more topics then faviki, I think this is a mistake. I often want to tag with terms that I can't find for some reason such as "social semantic tagging"..
6. MOAT looks like yet another semantic tagger, but it seems to be pushing to be more of a software framework than a distinct application. Interesting that another PhD student that seems to be about the same academic age as me came up with a nearly identical idea. Clearly the time has come! Looks like a nice clean, swebby implementation. Uncertain if anyone is using it.
7. RichTags started as Masters thesis project and now seems to be incorporated specifically into a project to improve the navigation of digital repositories. Users create tags and link them up via SKOS relations (synonymy, broader, narrower, etc.).
   
8. The two versions of the entity describer are both semantic tagging extensions built on top of Connotea. The first, now deprecated, version was a greasemonkey script that accessed an RDF database (that we assembled) of ontologies. The second, currently operational, version replaces the connotea bookmarklet with a new one and uses Freebase as its source of semantic tags.

There are some examples of full text annotation tools, that basically take the semantic tagging concept inside the document, but I think, for now, I'll keep these in their own class separate from the other plain old URI taggers. One of the oldest and most written about of these tools is Annotea by Marja-Riitta Koivunen.

ED2.0!

The Entity Describer is back and, we think, much better.  Give it a try from its new home at entitydescriber.org and please do let us know what you think. 

There is more information on the website, but briefly, the Entity Describer enhances the Connotea experience by enabling Semantic Tagging with Freebase.  Essentially it makes it both easy and enjoyable to say what you mean when you tag something.

 +  = 

seeing is believing

Wow, its been nearly a month since I wrote here.  I've been busy running in 5 directions at once - no surprise I haven't gotten anywhere lately.  But, just to get back in the habit, here is one of the tangential  things I've been thinking about when I should have been running in one direction as fast as I could.

Web3.0 needs a face.

Web2.0 has the profile page, the social network graph, and the tag cloud, what is the face of Web3.0?

The most popular and functional candidates thus far appear to be the products of the Simile group from MIT.  The main view I've seen appearing all over is the faceted browser (Longhorn, Exhibit) with fun enhancements like the Timeline component.  

These are great, but to me seem like fairly small steps from what we have already (at least from the user perspective, the code that runs them is pretty much magic).

Something new .

I wonder if something more along the lines of PhotoSynth will someday be making its way into our day to day desktop experience.  The thing that I see Photosynth doing, in a very intuitive way as it cheats by using reality, is to give huge collections of data such as that which can already be drawn from the giant global graph a nicely navigable, beautiful shape.  I wonder if there is some way to map large collections of other kinds of data such that the form underlying them is a) rendered in some beautiful way and b) rendered such that it actually contributes to the understanding and navigation of the collection.  [That will probably make more sense if you have seen this amazing demonstration of photosynth].  

the structure is the thing,

and it needs to be pretty!

ED2.0 underway

Manuscripts and academentia aside, we want to make E.D. better. To do so, we're planning to take advantage of the FreeBase type-ahead box pictured here.

I've been thinking about some sort of integration with FreeBase since I saw Jamie Taylor's "Stone Soup" presentation at WWW2007 and now the time seems to be ripe. "We" now includes Francois Belleau of bio2rdf fame. Francois is working on integrating bio2rdf with FreeBase and thus taking advantage of, among other things, their extremely fast graph query engine. By embedding the source vocabs (e.g.MeSH, GO..) in FreeBase, we will be able to use their (also extremely fast) type-ahead. This should eliminate the slowness and the memory problems we've had in the current implementation as well as providing a nice way to access ~way more terms. With Francois' work to map back and forth between freebase and RDF, I think this will end up being a much more killer application than v1.0.

E.D. update

In response to some complaints about slow load time, we've implemented a temporary fix that should make you happier. Basically, your E.D. vocabularies are now only loaded a) after the basic add to connotea page is loaded and b) if you request at that time that you want to use them.

This way, if you just want to tag something to_read quickly and move on with your life, you can do so without any extra delay, but, if you want to use the controlled vocabularies, you can do so with only one more button click. In addition, as we implement improvements to the script you will see them via the auto-update feature.

So.. if you were one of those that un-installed the E.D. script because you got tired of waiting for the vocabs to load, please don't be afraid to re-install it and use the semantic tagging capabilities when and how you see fit.

install the add to connotea E.D. script
install the user library browsing E.D. script

How to make the web safe for (g)Monkeys ?

Thanks to the superhuman efforts of my collaborator Eddie, ED survived a nasty scare today..

As foreshadowed in a recent blog post , Connotea added a new *feature today that rendered both of ED's (1000+ line) GreaseMonkey scripts totally useless. The problems, now fixed, were cause by the addition of javascript elements that now connect Connotea to adds and links from proximic.com . This brings up the question,

what might websites like Connotea that seem to be designed to be mashed up do to improve the stability of the external programs that enhance them?

In earlier writing, I suggested that some of the solution to this problem might be a focus first on the provision of data in a standardized, presentation agnostic form (e.g. RDF) or/and via API access. Connotea provides RDF via a nice RESTful API, so I couldn't really ask for more... but, there really is more to share than just the data. User-scripts can and do benefit from the presentation information on the websites that they process. So I wonder to myself, "wouldn't be cool if the presentation elements of web pages could be separated from the data they display so that mash monkeys could use both as they see fit?" Best of both worlds - sharable, mergable, mashable data chunks and presentation chunks!

Of course ;) those bright folks at places like MIT have already been working on this sort of thing for years... So far, they've come up with Fresnel, a standardized display vocabulary for RDF and have used it inside several semantic web browsers. When I first started thinking about this, I naively thought we were going to see general purpose semantic web browsers that could consume and somehow beautifully display pure RDF.. This is of course totally impossible - there is simply way, way, way more than one way to present a piece of data. Semantic Web Browsers won't do all the work - we'll still need good designers. What we need to do, I think, is work out the kinks of binding RDF to flexible, standardized presentation elements like those in Fresnel. Only with the two separate, symbiotic standards in place, one for the data and one for the presentaiton of that data, will it be possible to generate a successful, general purpose browser for the semantic web.

* p.s. it bums me out a little that I've put so much work into a project that, if successful, will end up benefiting a for-profit company that I don't work for and that now collaborates closely with a company that wants to "provide the Internet economy with greater control, precision, and profitability of content monetization"...gross. If only CiteULike had produced an RDF-serving API when I was first looking at these things...

Why you might want to start using ED in addition to your normal tags

The images in the picture are from the first page of results for a Google image search for "hippocampus", which, to the best of my knowledge, relies exclusively on automatically extracted tags from text surrounding the image or from the tags collected by the Google image labeling game. The yellow box in the middle represents the normal tag "hippocampus" and the blue boxes represent the equivalent ED tags.

Things to notice:

With ED tags, the three different senses of the word are captured through reference to the appropriate ontology.

The context and the meaning of these 'semantic' tags are created through the links between the terms.

Using the structure found in the ontologies, the ED tagged images can be found in a number of ways aside from querying by the exact tag. E.g. queries for "Mythical Creatures", "Brain", and "Chordata" can all be answered effectively by the ED system.

Seems pretty cool doesn't it? Why not give it a try and let me know what you think? Thanks for any thoughts...