Jun 22, 2012

Web API and Metaprogramming

In the last week I hacked a little with the preview of Diablo 3 web API . My aim was to produce a simple python client. I started the old way, parsing the JSON data the service returned and building Classes for each element. Then I realized I wasn't doing it not pythonic nor DRY. So I started to build generic objects that could hold the data and take the shape of each element. The basic idea is to read the maps and convert them in to python object structure. The fulcrum is this:

 class ApiObject(object):  
   def __init__(self, arg, host, battle_id):  
       self.__dict__.update(self.fetch_map(arg))  

In this way I can update the dict internal representation of the newly created object. the fetch_map method
recursively create other ApiObject for nested dictionaries, lists of ApiObject for... lists or plain attibutes for key-value elements.
An issue I early discovered is that some data were ref to other elements or resources exposed through a different service. I didn't want to fetch tons of may be useless data so I extended the ApiObject to implement a lazy behavior for some attributes:
 class LazyObject(ApiObject):  
   hydrate = False  
   http_client = requests  
   def http_client_callback(self):  
     pass  
   def __getattribute__(self, name):  
     if not object.__getattribute__(self, 'hydrate') and name in object.__getattribute__(self, 'lazy_load_attrs')():  
       data = self.http_client_callback()  
       self.__dict__.update(self.fetch_map(data))  
       object. __setattr__(self, 'hydrate', True)  
     return object.__getattribute__(self, name)  

So that if the client code tries to access a lazily loaded attribute and the object is not yet 'hydrated'  the remote call is performed and lazy data loaded. It keeps the concrete implementation of the object simple and tight to his own needs

 class Hero(LazyObject):  
    def lazy_load_attrs(self):  
     return ['skills', 'items', 'followers', 'progress']  
   def http_client_callback(self):  
     return load_hero(self.host, self.battle_id, self.name, http_client=self.http_client)  

Final mention to the "special cases". Now...
Special cases are never special enough to break the rules 
so, lets create rules for special cases!

The rule is:
If the ApiObject implementation has a method named "manage_[key in the dict]" is  as special cases and that method should be used to decode/deserialize that entry. 

I'm happy of the code right now. It has decent integration test against specs from the Diablo 3 web API, is flexible enough is those specs change in near future and is totally PEP-8 compliant.

And checked by Travis-CI too!

You can see, get and fork the code on Github.

Jun 14, 2012

Mining the Social Web: for fun , profit and satisfaction

I have recently read "Mining the social web" By Matthew A. Russell. It was part of my "big data & Natural Language" master plan, and, since the example were in python it looked as a good read. And it is!

This is a review I completly agree with:
Mining the Social Web is a good start for anyone is going to create scripts to analyze patterns in Social Networks. I've to say that this book consider that the reader already masters Python. I think that should be written directly on the title (ie: "Mining the Social Web with Python").
I liked the really fast approach to the Social Networks, even if a lot of times I wanted more; for this reason I consider it only a start, not a complete book.
The part I really enjoyed was the one about the HTML5 microformats. This is the only book that cover the topic from the data mining point of view, as far as I've seen.
I had the pleasure to read this book in the e-book version, and I've to admit that O'Reilly did a really good job in linking the different parts of the book through hyperlinks.
The biggest part of the book is focused more on what text is important rather that why it is important. In the book is touched the why speaking about the Semantic Web processing, but this is beyond the goal of this book, I suppose.

There are also a number of areas out of date. Twitter API data has been changed considerably and these are covered in the errata on the website. In other cases, a whole chapter is dedicated to Google Buzz which no longer exists. But it doesn't matter since the most interesting things are the techniques explained in the whole book. Another aspect I appreciated was that the code examples were all in a github repo. I found some performance / memory issue in one of those so I forked the repo , did my patch and submitted a  pull request that has been accepted! The author of the book honored me with a very kind comment:
It's a nice and elegant solution. Wish I'd have thought of it myself :)
So fun, profit and satisfaction!

Jun 10, 2012

My Trip To Djangocon 2012 Zurich

I'm back from my trip in Zurich for the 2012 european Django Conference. It was great! Great talcks, great organization, free snacks, good WIFI and amazing location.

The venue


Stadion Letzigrund... a conference in a stadium foyer? YES great idea the location was perfect may be a bit cold in te morning but great talks and a hundreds of laptops wormed it up easly.

The topics

The trendy topics of the conf has been:
  • Databases: how to deal with them.. do we need NoSql.. should NoSql support be in the core etc... I think hybrid solutions could work at the moment... may be in the future full NoSql solution will become the standard, and standard support from frameworks too.
  • Javascript: the so called "Real time web" which, IMHO, is not just yet another buzz word. Things like google docs collaboration are sooo innovative! We need more of them, and we need stuff to build them.
  • ClassBasedViews: A stream of in the hallway was about CBV. A mandatory talk was given by BRUNO RENIÉ I had a really log chat about the topic in the ALMOSTINFINITEQUEUEFORTHEMOLTENCHEESE. I will write a dedicated post on that soon

My lightning talk!

Completely unplanned I got a lightning talk about my orl project Django-IDS. Inspired by ERIK ROMIJN talk: Building secure Django websites I hacked long hours in the evening and ported it in version 1.4. But You have to plan your talk well if they had to last just 5 minutes. And I crashend in the  "stop now" gong. I was almost done so it's ok anyway. A couple of people came to me asking details so I'm glad to have the chance to speak ad DjangoCon

The trip 


The trip was amazing. I traveled by car across the alps seeing beautiful Swiss landscapes. In on the way back we did the Gotthard Pass with now clouds and everything. But that did not stop the coding. I was in good company of my Incode friends and this is Magnum pushing a django app on Heroku