doit – a build-tool tale

14 04 2008

doit is a built-tool like written in python. In this post explain my motivation for writting yet another buil tool. If you just want to use it. Please check the website

Build-tool

I started working on a web project… As a good TDD disciple I have lots of tests spawning everywhere. There are plain python unittest, twisted’s trial tests and Django specific unit tests. That’s all for python, but I also have unit tests for javascript (using a home grown unit test framework) and regression tests using Selenium. Running lint tools (JavaScriptLint and PyFlakes) are as important.

So I have seven tools to help me keeping the project healthy. But I need one more to control the seven tools! Actually there are more. I am not counting the javascript compression tool, the documentation generator…

I am not looking for a continuous integration (at least right now). I want to execute the tests in a efficient way and get problems before committing the code to a VCS.

- What tool do we use to automate running tasks?
- GNU Make. Or any other build tool.

SCons

I had the misfortune to (try to) debug some Makefile‘s before. XML based was never really an option to me. Since I work with python SCons looked like a good bet.

SCons. Writing the rules/tasks in python helps a lot. But the configuration (construct) file is not as simple and intuitive as I would expect. Maybe too powerful for my needs. Thats ok I don’t have to write new “Builders” that often.

Things went ok for a while… but things started to get too slow. Normal python tests are fast enough not to bother about it. But Django tests using postgres execution time do bother. The javascript tests run on the browser. So it needs to start the server, launch the browser, load and execute the tests… uuoooohhhh.

Most of the time i really need to execute just a subset of tests/tasks. The whole point of build tools is to keep track of dependencies and re-build only what is necessary, right? The problem with tests is that actually i am not building anything. I am executing tasks(in this case tests). Building something is a “task” with a “target” file(s), but running a test is a “task” with no “target”. The problem is that build tools were designed to keep track of target/file dependencies not task dependencies. Yes I know you can use hacks to pretend that every task has a target file. But I was not really willing to do this…

I was not using any of the great SCons features. Actually at some point I easily substitute it to a simple (but lengthy) python script using the subprocess module. Of course this didn’t solve the speed problem.

doit

doit. I want a tool to automatically execute any kind of tasks, having a target or not. It must keep track of the dependencies and re-do (or re-execute) tasks only if necessary, (like every build tool do for target files). And of course it shouldn’t get on my way while specifying the tasks.

Requirements:

. keep track of dependencies. but they must be specified by user, no automatic dependency analysis. (i.e. nearly every build tool supports this)
. easy to create new task rules. (i.e. write them in python)
. get out of your way, avoid boiler-plate code. (i.e. something like what nose does to unittest)
. dependencies by tasks not on files/targets.

The only distinctive requirement is item 4. I guess any tool that implements dependency on targets could support dependency on tasks with not so much effort.
You just need to save the signature of the dependent files on successful completion of the task. If none of the dependencies changes the signature the task doesn’t need to be executed again. Since it is not required to have a target the tasks needs to be uniquely identified. But thats an implementation detail…

So how does it look like?

look at the tutorial:

About these ads

Actions

Information

8 responses

28 08 2009
Godfryd

It looks promising.

3 12 2009
Adrian Silva

I’m currently using paver for project automation and throw-away continous integration.

Any idea on how doit compares to paver?

4 12 2009
schettino72

Make and other traditional build-tools focus on 2 things:
* efficient compilation for developers
* deployment

doit focus in on the first point. I was actually looking for a more “efficient” test runner. I figured out the way to go was to create a more generic build tool.

I have not used paver though I’ve read their docs a long time ago. As far as I could get, it focus more on deployment and tight integration to some other python specific tools.

It would be cool if you could give it a try to doit and post an impartial comparison (of course I can’t do that myself). If you need some help just send a email to doit mailing list of what you are trying to do. I will definitely look into it and give some hints.

7 06 2010
akira

Hi there,

DoIT looks and sounds nice, how does it compare to fabric(http://www.nongnu.org/fab/). It missed commands that allow SSH commands.

Thanks

5 01 2011
Elena

Congratulations! Your tool follows the KISS principle very closely. I always wondered why build tools had to be that complicated.

Your reasoning about avoiding decorators sounds very convincing.

I have a suggestion: why is “python-action” a tuple? By using a list you could avoid the ugly “,” when there are no arguments.

10 01 2011
schettino72

Thanks.

> I have a suggestion: why is “python-action” a tuple?
I guess thats the standard way people reference (callable, args, kwargs) in python :)

> By using a list you could avoid the ugly “,” when there are no arguments.
If there is no arguments you can pass just the callable, no need to use a tuple. Said that, it wouldnt be a problem for me to also support lists…

10 01 2011
Elena

> If there is no arguments you can pass just the callable, no need to use a tuple.

Perfect. No need to change that.

13 06 2012
yoyo

Btw, just changed:
@echo off
c:\python32\python.exe “%~dpn0″ %*

in the doit.bat

and also, there was some binary junk in the docs/index.rst file that I deleted. Now doit works on windows7 with python3 for me. :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

%d bloggers like this: