How to auto-reload a server on changes with doit

5 03 2012

Another day my friend told me he is using wsgiref.simple_server and asked if doit could be used to auto-reload the server. My first answer was do not use wsgiref :)

But you might prefer to use wsgiref & doit for for auto-reload for two reasons. doit can be bundled in a single file and included in the project. You might want to have explicit control on when the server should be reloaded.

server.py:

from wsgiref.simple_server import make_server, demo_app

httpd = make_server('', 8000, demo_app)
print "Serving HTTP on port 8000..."

# Respond to requests until process is killed
httpd.serve_forever()

dodo.py:

import subprocess
import glob
import os
import signal

def start_server(server_cmd, pid_filename, restart=False):
    # check the server is running
    if os.path.exists(pid_filename):
        if restart:
            stop_server(pid_filename)
        else:
            msg = "It seems the server is already running, check the file %s"
            print  msg % pid_filename
            return False

    # start server
    process = subprocess.Popen(server_cmd.split())

    # create pid file
    with open(pid_filename, 'w') as pid_file:
        pid_file.write(str(process.pid))
    return True

def stop_server(pid_filename):
    # check server if is running
    if not os.path.exists(pid_filename):
        return
    # try to terminate/stop server's process
    with open(pid_filename) as pid_file:
        pid = int(pid_file.read())
        try:
            os.kill(pid, signal.SIGTERM)
        except OSError:
            pass #ignore errors if process does not exist
    # remove pid file
    os.unlink(pid_filename)


########################################

DOIT_CONFIG = {'default_tasks': ['restart']}

PID_FILENAME = 'pid.txt'
START_SERVER = 'python server.py'

def task_start():
    return {'actions': [(start_server, (START_SERVER, PID_FILENAME,))]}

def task_stop():
    return {'actions': [(stop_server, (PID_FILENAME,))]}

def task_restart():
    return {'actions': [(start_server, (START_SERVER, PID_FILENAME, True))],
            'file_dep': glob.glob('*.py'),
            'uptodate': [False],
            }

In order to do a auto-reload/restart the server we need to be able to start and stop the server with two independent commands.
So when starting the server we create a text file containing the PID of the process server.

The start_server funcion contains a boolean parameter ‘restart’ to control the behaviour when there is already a pid file.

task_restart

Usually ‘file_dep’ is used to indicated when a task is up-to-date but in this case we use it just to trigger a re-execution of the task in the ‘auto’ mode.

So apart from the action to restart the server the task’s file_dep controls which files to watch for modifications. Since we want always want to start the server when the task is called we need to add the ‘uptodate’ parameter to be false.

To use it just type:

$ doit auto restart




doit – a build-tool tale

14 04 2008

doit is a built-tool like written in python. In this post explain my motivation for writting yet another buil tool. If you just want to use it. Please check the website

Build-tool

I started working on a web project… As a good TDD disciple I have lots of tests spawning everywhere. There are plain python unittest, twisted’s trial tests and Django specific unit tests. That’s all for python, but I also have unit tests for javascript (using a home grown unit test framework) and regression tests using Selenium. Running lint tools (JavaScriptLint and PyFlakes) are as important.

So I have seven tools to help me keeping the project healthy. But I need one more to control the seven tools! Actually there are more. I am not counting the javascript compression tool, the documentation generator…

I am not looking for a continuous integration (at least right now). I want to execute the tests in a efficient way and get problems before committing the code to a VCS.

– What tool do we use to automate running tasks?
– GNU Make. Or any other build tool.

SCons

I had the misfortune to (try to) debug some Makefile‘s before. XML based was never really an option to me. Since I work with python SCons looked like a good bet.

SCons. Writing the rules/tasks in python helps a lot. But the configuration (construct) file is not as simple and intuitive as I would expect. Maybe too powerful for my needs. Thats ok I don’t have to write new “Builders” that often.

Things went ok for a while… but things started to get too slow. Normal python tests are fast enough not to bother about it. But Django tests using postgres execution time do bother. The javascript tests run on the browser. So it needs to start the server, launch the browser, load and execute the tests… uuoooohhhh.

Most of the time i really need to execute just a subset of tests/tasks. The whole point of build tools is to keep track of dependencies and re-build only what is necessary, right? The problem with tests is that actually i am not building anything. I am executing tasks(in this case tests). Building something is a “task” with a “target” file(s), but running a test is a “task” with no “target”. The problem is that build tools were designed to keep track of target/file dependencies not task dependencies. Yes I know you can use hacks to pretend that every task has a target file. But I was not really willing to do this…

I was not using any of the great SCons features. Actually at some point I easily substitute it to a simple (but lengthy) python script using the subprocess module. Of course this didn’t solve the speed problem.

doit

doit. I want a tool to automatically execute any kind of tasks, having a target or not. It must keep track of the dependencies and re-do (or re-execute) tasks only if necessary, (like every build tool do for target files). And of course it shouldn’t get on my way while specifying the tasks.

Requirements:

. keep track of dependencies. but they must be specified by user, no automatic dependency analysis. (i.e. nearly every build tool supports this)
. easy to create new task rules. (i.e. write them in python)
. get out of your way, avoid boiler-plate code. (i.e. something like what nose does to unittest)
. dependencies by tasks not on files/targets.

The only distinctive requirement is item 4. I guess any tool that implements dependency on targets could support dependency on tasks with not so much effort.
You just need to save the signature of the dependent files on successful completion of the task. If none of the dependencies changes the signature the task doesn’t need to be executed again. Since it is not required to have a target the tasks needs to be uniquely identified. But thats an implementation detail…

So how does it look like?

look at the tutorial: