Inspired from this blog post about "
Using Travis-CI With Python and Django" I started playing around to make this recipe fit my expectation from a CI enviroment. Running tests is the bare bottom of the whole thing. Email notifications and the handy badge to display in the project webpage are ok, but if you want to get somewhere metrics and coverage reports are the key to get the most out of the Continuous Integration approach.
So I started creating a
Django project on Github with this Travis configuration
language: python
python:
- "2.6"
- "2.7"
env:
- DJANGO=1.4
install:
- pip install -q Django==$DJANGO --use-mirrors
- pip install pep8 --use-mirrors
- pip install https://github.com/dcramer/pyflakes/tarball/master
- pip install coverage
- pip install django_coverage
before_script:
- "pep8 --exclude=migrations --ignore=E501,E225,E261,W191,E101 ."
- pyflakes -x W .
script:
- python manage.py test_coverage
Basicaly I added coverage and django_coverage to the depandecy and then ran the tests with coverage report enabled. ( in real world I would use a requirement file but I will keep dep. installation this way for the sake of clarity) This is the result:
Name Stmts Miss Cover Missing
---------------------------------------------------------
trango.molten_core.models 0 0 100%
trango.molten_core.views 0 0 100%
---------------------------------------------------------
TOTAL 0 0 100%
Ok this is just and empty app with one test always passing. But my point is to put this data somewhere outside Travis ( and gh-pages seemed a very good place ) and have an easy way to see how coverage increase or decrease in particular builds ( and who is guilty for that!)
First we have to find an easy way to identify the build commit. I found that something like
export GIT_COMMIT_ID=`git rev-parse HEAD` works just fine. Then I started speculating what was the optimum ( or at least possible) way to move away the data and I ended up with these three solutions:
HUMAN PALADIN: Make Travis-CI push back the reports
This is the first, clean , way that came to my mind. I created the gh-pages branch on my repo ( if you are unfamiliar with this please read
here ) and I set django_coverage to create the html reports in a specific folder (in settings.py file)
if os.getenv('GIT_COMMIT_ID'):
COVERAGE_REPORT_HTML_OUTPUT_DIR = os.path.join("reports", os.getenv('GIT_COMMIT_ID'))
else:
COVERAGE_REPORT_HTML_OUTPUT_DIR = 'reports'
Then I tried to meke Travis-CI to push back the reports
language: python
python:
- "2.6"
- "2.7"
env:
- DJANGO=1.4
install:
- pip install -q Django==$DJANGO --use-mirrors
- pip install pep8 --use-mirrors
- pip install https://github.com/dcramer/pyflakes/tarball/master
- pip install coverage
- pip install django_coverage
before_script:
- "pep8 --exclude=migrations --ignore=E501,E225,E261,W191,E101 ."
- pyflakes -x W .
- mkdir reports
- export GIT_COMMIT_ID=`git rev-parse HEAD`
- mkdir reports/$GIT_COMMIT_ID
script:
- python manage.py test_coverage
after_script:
- cp -pR reports/$GIT_COMMIT_ID /var/tmp
- rm -rf reports/$GIT_COMMIT_ID
- git branch gh-pages
- cp -pR /var/tmp/$GIT_COMMIT_ID reports
- git add reports/*
- git commit -m "reports for $GIT_COMMIT_ID"
- git push origin gh-pages
I had to move the reports in a temp directory to quietly switch from the master branch to the gh-pages branch. And sadly this is the output of the build
$ git branch gh-pages
103
104$ cp -pR /var/tmp/$GIT_COMMIT_ID reports
105
106$ git add reports/*
107
108$ git commit -m "reports for $GIT_COMMIT_ID"
109[detached HEAD 5b018bc] reports for 4b01202c546546bc424af1828eb211a011bcc278
110 Committer: vagrant <vagrant@nettuno.(none)>
111Your name and email address were configured automatically based
112on your username and hostname. Please check that they are accurate.
113You can suppress this message by setting them explicitly:
114
115 git config --global user.name "Your Name"
116 git config --global user.email you@example.com
117
118After doing this, you may fix the identity used for this commit with:
119
120 git commit --amend --reset-author
121
122 3 files changed, 384 insertions(+), 0 deletions(-)
123 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/excludes.html
124 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/index.html
125 create mode 100644 reports/4b01202c546546bc424af1828eb211a011bcc278/modules/trango.molten_core.models.html
126
127$ git push origin gh-pages
128fatal: remote error:
129 You can't push to git://github.com/sammyrulez/django-travis-demo.git
130 Use git@github.com:sammyrulez/django-travis-demo.git
131
132
133
134after_script: 'git push origin gh-pages' returned false.
135Done. Build script exited with: 1
As you can see when Travis-CI tries to commit we got evidence that git is not configured with a proper user nor have keys or Github user to add as collaborator to my repo. So also the push fails. I think up to now this is a dead end. Possible solutions are to run Travis-CI localy ( but I would go straight in
Jenkins arms instead: the good point in favor of Travis-CI is his cheap and
cloudy nature ) or that Travis-CI allows you to add specific deployments keys to each building repository.
FORSAKEN WARRIOR: Use a curl friendly file host
Searching for options of "push-back" I found a message in the Travis-CI mailing list suggesting to use a file hosting service to send your data via curl. For example
ge.tt. Now the problem is that you have to put your
API Key, username and
clear text password in the .travis.yaml file commited in a public repository. Definitively not a secure way to store files online. One solution would be to create a proxy service that uses commit hashes as
ONE TIME PASSWORD. But building such a thing would require a lot of time and the result would not be "vandal proof". Maybe, someday....
GNOME ROGUE: Ajax scraping
I think another option is to add to my gh-pages some javascript that reads tha
Travis-CI api to list the builds and then perform some scraping to extract data from the dom of the build details. 100 Haking ponts, 10 Fun points, 0 usability points.
Conclusion
Travis-CI is an amazing project but without a feature to export build results I find hard to think it would fit a project with many developers and aiming a high code quality and test coverage. I found historical CI data very important and in some case also inspiring ("Look how crappy the project were a month ago!"). Travis-CI is handy and sleek but having reports on the project web page gives an instant direction on how the project is managed. Well also the Travis badge is a really really good indicator.
I will keep it failing just to remember to fix it!
