Travis, give me my data back!

Inspired from this blog post about "Using Travis-CI With Python and Django" I started playing around to make this recipe fit my expectation from a CI enviroment. Running tests is the bare bottom of the whole thing. Email notifications and the handy badge to display in the project webpage are ok, but if you want to get somewhere metrics and coverage reports are the key to get the most out of the Continuous Integration approach.

So I started creating a Django project on Github with this Travis configuration

 language: python  
 python:  
  - "2.6"  
  - "2.7"  
 env:  
  - DJANGO=1.4  
 install:  
  - pip install -q Django==$DJANGO --use-mirrors  
  - pip install pep8 --use-mirrors  
  - pip install https://github.com/dcramer/pyflakes/tarball/master  
  - pip install coverage  
  - pip install django_coverage  
 before_script:  
  - "pep8 --exclude=migrations --ignore=E501,E225,E261,W191,E101 ."  
  - pyflakes -x W .  
 script:  
  - python manage.py test_coverage  

Basicaly I added coverage and django_coverage to the depandecy and then ran the tests with coverage report enabled. ( in real world I would use a requirement file but I will keep dep. installation this way for the sake of clarity) This is the result:

 Name            Stmts  Miss Cover  Missing  
 ---------------------------------------------------------  
 trango.molten_core.models    0   0  100%    
 trango.molten_core.views    0   0  100%    
 ---------------------------------------------------------  
 TOTAL              0   0  100%   

Ok this is just and empty app with one test always passing. But my point is to put this data somewhere outside Travis ( and gh-pages seemed a very good place ) and have an easy way to see how coverage increase or decrease in particular builds ( and who is guilty for that!)

First we have to find an easy way to identify the build commit. I found that something like
export GIT_COMMIT_ID=`git rev-parse HEAD` works just fine. Then I started speculating what was the optimum ( or at least possible) way to move away the data and I ended up with these three solutions:

HUMAN PALADIN: Make Travis-CI push back the reports

This is the first, clean , way that came to my mind. I created the gh-pages branch on my repo ( if you are unfamiliar with this please read here ) and I set django_coverage to create the html reports in a specific folder (in settings.py file)

 if os.getenv('GIT_COMMIT_ID'):  
   COVERAGE_REPORT_HTML_OUTPUT_DIR = os.path.join("reports", os.getenv('GIT_COMMIT_ID'))  
 else:  
   COVERAGE_REPORT_HTML_OUTPUT_DIR = 'reports'  

Then I tried to meke Travis-CI to push back the reports

 language: python  
 python:  
  - "2.6"  
  - "2.7"  
 env:  
  - DJANGO=1.4  
 install:  
  - pip install -q Django==$DJANGO --use-mirrors  
  - pip install pep8 --use-mirrors  
  - pip install https://github.com/dcramer/pyflakes/tarball/master  
  - pip install coverage  
  - pip install django_coverage  
 before_script:  
  - "pep8 --exclude=migrations --ignore=E501,E225,E261,W191,E101 ."  
  - pyflakes -x W .  
  - mkdir reports  
  - export GIT_COMMIT_ID=`git rev-parse HEAD`  
  - mkdir reports/$GIT_COMMIT_ID  
 script:  
  - python manage.py test_coverage  
 after_script:  
  - cp -pR reports/$GIT_COMMIT_ID /var/tmp  
  - rm -rf reports/$GIT_COMMIT_ID  
  - git branch gh-pages  
  - cp -pR /var/tmp/$GIT_COMMIT_ID reports  
  - git add reports/*  
  - git commit -m "reports for $GIT_COMMIT_ID"  
  - git push origin gh-pages  

I had to move the reports in a temp directory to quietly switch from the master branch to the gh-pages branch. And sadly this is the output of the build

 $ git branch gh-pages  
 103  
 104$ cp -pR /var/tmp/$GIT_COMMIT_ID reports  
 105  
 106$ git add reports/*  
 107  
 108$ git commit -m "reports for $GIT_COMMIT_ID"  
 109[detached HEAD 5b018bc] reports for 4b01202c546546bc424af1828eb211a011bcc278  
 110 Committer: vagrant <vagrant@nettuno.(none)>  
 111Your name and email address were configured automatically based  
 112on your username and hostname. Please check that they are accurate.  
 113You can suppress this message by setting them explicitly:  
 114  
 115  git config --global user.name "Your Name"  
 116  git config --global user.email you@example.com  
 117  
 118After doing this, you may fix the identity used for this commit with:  
 119  
 120  git commit --amend --reset-author  
 121  
 122 3 files changed, 384 insertions(+), 0 deletions(-)  
 123 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/excludes.html  
 124 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/index.html  
 125 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/modules/trango.molten_core.models.html  
 126  
 127$ git push origin gh-pages  
 128fatal: remote error:   
 129 You can't push to git://github.com/sammyrulez/django-travis-demo.git  
 130 Use git@github.com:sammyrulez/django-travis-demo.git  
 131  
 132  
 133  
 134after_script: 'git push origin gh-pages' returned false.  
 135Done. Build script exited with: 1  

As you can see when Travis-CI tries to commit we got evidence that git is not configured with a proper user nor have keys or Github user to add as collaborator to my repo. So also the push fails. I think up to now this is a dead end. Possible solutions are to run Travis-CI localy ( but I would go straight in Jenkins arms instead:  the good point in favor of Travis-CI is his cheap and cloudy nature ) or that Travis-CI allows you to add specific deployments keys to each building repository.

FORSAKEN WARRIOR: Use a curl friendly file host

Searching for options of  "push-back" I found a message in the Travis-CI mailing list suggesting to use a file hosting service to send your data via curl. For example ge.tt. Now the problem is that you have to put your API Key, username and clear text password in the .travis.yaml file commited in a public repository. Definitively not a secure way to store files online. One solution would be to create a proxy service that uses commit hashes as ONE TIME PASSWORD. But building such a thing would require a lot of time and the result would not be "vandal proof". Maybe, someday....

GNOME ROGUE: Ajax scraping

I think another option is to add to my gh-pages some javascript that reads tha Travis-CI api to list the builds and then perform some scraping to extract data from the dom of the build details. 100 Haking ponts, 10 Fun points, 0 usability points.

Conclusion

Travis-CI is an amazing project but without a feature to export build results I find hard to think it would fit a project with many developers and aiming a high code quality and test coverage. I found historical CI data very important and in some case also inspiring ("Look how crappy the project were a month ago!"). Travis-CI is handy and sleek but having reports on the project web page gives an instant direction on how the project is managed. Well also the Travis badge is a really really good indicator.

I will keep it failing just to remember to fix it!