Python-YQL: client library for YQL | Comments (0)
Posted in Code, Python on 24th November 2009, 9:52 pm by Stuart
You’d of have to have been under a rock to not have heard about Yahoo Query Language which provides a “SQL-like” abstraction to Yahoo’s own APIs as well as a growing number of third party APIs.
I’ll be honest; for a while I was quite dismissive of YQL thinking that if I was using those APIs then I’d query them directly rather than use a 3rd party abstraction. But then I saw a recent post from Christian Heilmann that touched on making YQL calls from the server that began to pique my interest. It made me start to think about all the data from the web that could be accessed from just some simple SQL-like queries.
So I decided in order to try out YQL I’d build a dedicated client library. This also provided me with an opportunity to explore OAuth properly as I’d need to make the client work with both Two and ThreeLegged Oauth so it’s also relatively straightforward to query Private tables from a web application.
The result is Python-YQL which you can install from PyPi with:
sudo easy_install yqlThe documentation is available at http://python-yql.org/ and branches are available at Launchpad.net or you can get a branch with the following:
bzr branch lp:python-yqlBasic usage is along the following lines:
>>> import yql
>>> y = yql.Public()
>>> query = 'select * from flickr.photos.search where text="panda" limit 3';
>>> y.execute(query)For details on more advanced usage see the docs at python-yql.org
Patches Welcome!
So go and check it out and kick the proverbial tyres. It’s early days and there’s bound to be some bugs and missing features but this is where you come in. I’ll happily expedite provision of fixes as they come in. Patches/Bug reports/branches are welcome!
Python Oauth2
For oauth integration, I started out using python-oauth until Cyril mentioned that Joe Stump had created a new oauth library based on various forks of the original python-oauth. Python-oauth2 provides an easy to use well-tested library that is a natural progression of python-oauth.
Once I knew about it I couldn’t resist switching python-yql over to using it. At the moment I’m bundling python-oauth2 as I needed to patch in support to to make it work with python versions < 2.6. However hopefully in due course this will cease to be necessary.
Future
There’s lots I can improve upon but for now I’m planning to spend some time using the library in order to work out what can be better. I’m thinking one of the first things I’d like to do is make what’s returned an object rather than just simply returning the data. This will mean that I can provide useful attributes such as the query that was called to return that dataset for example. This is now a reality.
The version in the trunk branch now returns a YQL object which provides a much more awesome API for accessing the data returned by YQL queries. This will be released as 0.3 and at that point I’ll update the docs. Until then here’s a taster:
>>> import yql
>>> y = yql.Public()
>>> res = y.execute("use 'http://yqlblog.net/samples/search.imageweb.xml' as searchimageweb; select title from searchimageweb where query='pizza' limit 3")
>>> print res
<yql.YQL object at 0xb77adc2c>
>>> res.rows
[{u'title': u'<b>Pizza</b> Hut'}, {u'title': u"Domino's <b>Pizza</b> -
Official Site"}, {u'title': u"Papa John's"}]
>>> res.rows[0]
{u'title': u'<b>Pizza</b> Hut'}
>>> res.rows[1]
{u'title': u"Domino's <b>Pizza</b> - Official Site"}
>>> res.query
u"use 'http://yqlblog.net/samples/search.imageweb.xml' as searchimageweb; select title from searchimageweb where query='pizza' limit 3"
>>> res.uri
u'http://query.yahooapis.com/v1/yql?q=use+%27http%3A%2F%2Fyqlblog.net%2Fsamples%2Fsearch.imageweb.xml%27+as+searchimageweb%3B+select+title+from+searchimageweb+where+query%3D%27pizza%27+limit+3'
>>> res.count
3