how to execute tests on a bazaar pre-commit hook

20 01 2008
you should always execute the test suite from a project before committing the source code to a version control system. everybody knows this but sometimes we forget… so lets have a pre-commit hook that will automatically execute the test suite. i am using bazaar.

in bazaar you write hooks as plugins.

how it works

the “hook” is simple python function that will be executed before really committing when bzr commit is called . it is very simple, it just run an executable and check its result. if it is successful the commit operation is executed. if it fails the commit operation is cancelled.

plugins in bazaar are not project specific. so you cant control in which projects (branches) your plugin will be applied (it will be applied to all). so, first the plugin checks if the branch contains a “precommit” executable. it must be named “precommit” and must be in root path of your branch.

the plugin file should be placed in ~/.bazaar/plugins/. the name of the module doesn’t matter (i called it “check_pre_commit.py”). putting the boilerplate code together i got this:

"""this is a plugin/hook for bazaar. just add this file to ~/.bazaar/plugins/"""

from bzrlib import branch

def pre_commit_hook(local, master, old_revno, old_revid, future_revno, future_revid, tree_delta, future_tree):
    """This hook will execute precommit script from root path of the bazaar
    branch. Commit will be canceled if precommit fails."""

    import os,subprocess
    from bzrlib import errors

    # this hook only makes sense if a precommit file exist.
    if not os.path.exists("precommit"):
        return
    try:
        subprocess.check_call(os.path.abspath("precommit"))
    # if precommit fails (process return not zero) cancel commit.
    except subprocess.CalledProcessError:
        raise errors.BzrError("pre commit check failed.")

branch.Branch.hooks.install_hook('pre_commit', pre_commit_hook)
branch.Branch.hooks.name_hook(pre_commit_hook, 'Check pre_commit hook')

thats all for the hook.

now we just need to create an executable that will run the tests and return something different from zero if it fails. and put it on the root path of the branch.
in this example i execute my tests using nose.

precommit

#!/bin/sh
nosetests

ok. everything is working. but who remember to run the tests should be “punished” and will have to wait for the tests to run twice (once manually before committing and once automatically as a pre-commit hook)? no. if you remember to run the tests you can also remember to skip the plugin :)

bzr commit --no-plugins

Update: You can put your tests in a smart build-system (like doit) to keep track of the modifications in your source code and re-execute only the required tests (the ones that depends on the modified files).

caveats

  • the precommit executable is part of the branch but the hook is not. so just doing a branch on the project is not enough. every user has to install the plugin. but just once for all projects.

  • the hook is executed after you supply the commit message. it is annoying to write a message and than loose it because the commit failed. or maybe it is a good punishment for trying to commit an untested code :)

  • the output gets a bit messed up. but not a big deal…

    eduardo@eduardo-laptop:~/work/doit$ bzr commit
    Committing to: /home/eduardo/work/doit/
    modified precommit
    [==========      ] Running pre_commit hooks [Check pre_commit hook] - Stage 3/5....
    ----------------------------------------------------------------------
    Ran 4 tests in 0.268s
    
    OK
    Committed revision 2.

thats it.





python testing + folder structure – import issues

19 01 2008
whenever i start coding something in python i use the following folder structure:

my_project
        ./foo
             ./__init__.py
             ./bar.py
        ./tests
             ./test_foo.py
             ./test_bar.py

but i used to get confused on how to organize the import(s) and when to use PYTHONPATH for my tests.

my requirements are:

  • i want to be able to run my tests both from my_project folder and from the tests folder
  • i don’t want to set the PYTHONPATH manually (and i don’t want to add my development path to PYTHONPATH.
  • i want to import the lib modules on the test on the same way i would import after the lib is deployed.

(yes i am lazy)

bar.py:

def dumb_true():
    return True

1st try

the code for test_bar.py should like something like this:

import unittest

from foo import bar

class TestBar(unittest.TestCase):
    def test_bar_true(self):
        self.assertTrue(bar.dumb_true())

if __name__ == '__main__':
    unittest.main()

the problem here is that you need to set the PYTHONPATH manually.

2nd try

the first thing that comes in my mind is to add ‘foo’ folder to PYTHONPATH on the test module

...
import sys,os
sys.path.insert(0,os.path.abspath(__file__+"/../.."))
from foo import bar
...

not bad. but you need to add this to every single test module. and hope that your folder structure wont change.

3rd try

i guess we need a smarter test runner (like nose).
get your code back to just:

...
from foo import bar
...

you can just run “nosetests” from the project folder (‘my_project’ in the example above) and it will just work.

the same doesn’t happen if run “nosetests” from the tests folder.

from nose homepage:
When nose imports a module, it adds that module’s directory to sys.path; when the module is inside of a package, like package.module, it will be loaded as package.module and the directory of package will be added to sys.path.

so the trick here is to make the tests folder a package (adding __init__.py to it). so in the tests folder you will find:

./tests
     ./__init__.py
     ./test_foo.py
     ./test_bar.py

now from the tests folder you can:

  1. execute all tests doing “nosetests”
  2. execute just one file “nosetests test_bar.py”
  3. check nose docs for more options

and you don’t have to worry about PYTHONPATH at all :)

more nose

nose also free you from all the unittest boilerplate code. the test file could be just:

from foo import bar

def test_bar_true():
    assert bar.dumb_true()

and thats it.





running selenium tests from the command line

12 01 2008

I wrote a tutorial on how to run selenium tests from the command line. it is very handy you plan to include it on your continuous integration process.

intro

selenium is great tool for writing tests to web applications. and with Selenium IDE, it is a breeze to write tests. here i will describe how to run your selenium tests from the command line (instead of manually starting the server, opening a browser, selecting the test file…). if you dont use selenium yet you should take a look at it first.

i will not go over the advantages over having regression tests… but to be effective it should be easy to run the tests. it is very boring to manually run the tests, so once in a while when i decide to run it i found out that some regression bugs appeared, and they were not easy to spot because they might got into the code a while ago. even worse is when somebody else on team introduced the bug ;)

complete article





firefox profiles & batch friendly options

10 01 2008

on my previous post i showed how to avoid gui windows popping up on your screen. now i am gonna show how to configure firefox to be less interactive (you dont want it to be interactive when you are executing background tasks.)

first you need to create a new profile. see how to manage profiles. you need a different profile so it wont mess with your default profile, and will allow two independent instances of firefox.

in order to have two simultaneous windows with two different profiles you need to run it using the following command line:

$ firefox -no-remote -P testProfile

if you dont give the -no-remote optional parameter it will ignore the profile parameter and will open a new tab on the currentelly open firefox (if you have one already open).

now that you have a new profile and know how to start it, you just need to configure some options to disable confirmation dialogs.

1) On the menu go to: Tools->Options…->Security->Warning Messages->Settings. Disable everything.

2) one very useful feature that you dont want in your test profile is crash recovery(or session restore). to get rid of it type ‘about:config’ on the location bar (a.k.a. where you type the url). set browser.sessionstore.resume_from_crash to false.

thats all. on the next post we will get some real action…





running gui apps on the background (invisible)

9 01 2008

here i will show how to execute an app that requires GUI (X window) to run on the background. i mean you wont be able to see the screen.

you may wonder why someone would want this since without the GUI you cant interact with the app. the test suite for the application that i am working on requires running some javascript test on firefox. i managed to write a script that automatically starts firefox run the tests and than kills the firefox process. but the problem is that while i run the tests i would like to keep working on another stuff but the windows keep popping up and bugging me.

this problem is simple to solve. i used a vncserver. a vnc (virtual networking computing) server will create a virtual X server. so you actually doesnt need to have a real display. usually, vncserver is used to display an application on a different machine. a remote machine would connect to the vncserver using a vncviewer…

you will need to install a vncserver. on ubuntu i am using the package vnc4server.

after installing, the first step is to create a password using ‘vncpasswd’. this password will be used to authenticate any vncviewer trying to connect to this server. we wont setup any viewer but vncserver requires this password anyway. so just type:

$ vncpasswd

now start the server

$ vncserver

if you dont specify a display number it will get one for you. it will probably be ‘:1’

our vncserver is up and running. to use it you need to change the env variable DISPLAY to ‘:1’ (or whatever display number you got).

now any process started on this shell will use this display. since it is a virtual display you wont see anything. just what we were looking for.

some applications also have a ‘display’ parameter, so you dont need to set the env variable.

$ firefox –display=:1

thats all.