appengine & virtualenv

UPDATE: appengine 1.6.1 & uses gaecustomize.py

This article will explain how to setup Google AppEngine (GAE) with virtualenv.

GAE does not provide a “setup.py” to make the SDK “installable”, it is supposed to be used from a folder without being “installed”. GAE actually forbids the use of any python library in the site-packages folder. All included libraries must be in the same folder as your application, this allows GAE to automatically find and upload third-party libraries together with your application code when you upload the code to GAE servers.

So what would be the advantages of using of using virtualenv with GAE? The main reason is to have an environment to run unit-tests and functional tests. It will allow us to use the interactive shell to make operations on DB. And it also enforce you are using the correct python version.

Step 0 – install App Engine SDK

Make sure you use 1.6.1 or later, an important bug was fixed on this release. As described on official docs. Tested with virtualenv 1.6.4.

Step 1 – create and activate a virtualenv

Same as usual…


$ virtualenv --python python2.5 --no-site-packages gae-env
$ source gae-env/bin/activate


Step 2 – add google_appengine path

Add a path configuration file named “gae.pth” to the virtualenv site-packages with the path to google_appengine. This way google_appengine will be in sys.path enabling it to be imported by other modules.

You will need to adjust the content of the file according to where you created your virtualenv and google_appengine location. Mine looks like this:


$ cat gae-env/lib/python2.5/site-packages/gae.pth
../../../../google_appengine


Simple test to make sure your gae.pth is correct:

(gae-env)$ python
>>> from google import appengine


If you did not get any exception you are good to go on.

Step 3 – fix path for third-party libs

The AppEngine SDK comes with a few third-party libraries. They are not in the same path as google’s libraries. If you look at dev_appserver.py you will see a function called fix_sys_path, this function adds the path of the third-party libraries to python’s sys.path. One option would be to add these paths to gae.pth… But I prefer to use the function fix_sys_path so we have less chances of having problems with future releases of the SDK.

Note that this will not look for your config in app.yaml. So you might need to add some extra imports. The example below is using webob version 1.1.1 instead of the default one.

Path configuration files can also execute python code on if the line starts with import. Add a module gaecustomize.py to site-packages:

gae-env/lib/python2.5/site-packages/gaecustomize.py

def fix_sys_path():
    try:
        import sys, os
        from dev_appserver import fix_sys_path, DIR_PATH
        fix_sys_path()
        # must be after fix_sys_path
        # uses non-default version of webob
        webob_path = os.path.join(DIR_PATH, 'lib', 'webob_1_1_1')
        sys.path = [webob_path] + sys.path
    except ImportError:
        pass

And modify gae.pth it calls the above module:

gae-env/lib/python2.5/site-packages/gae.pth

../../../../google_appengine
import gaecustomize; gaecustomize.fix_sys_path()

For some unknown reason gae.pth is being processed twice and on the first time google_appengine is not added to sys.path. Thats why I explicitly call the function fix_sys_path.

Check if it is working fine:


(gae-env)$ python
>>> import yaml


Again. You should not any exceptions on this…

Step 4 – add dev_appserver.py to bin

Not really required but handy.


gae-env/bin $ ln -s ../../google_appengine/dev_appserver.py .


Conclusion

Now you have an isolated environment running AppEngine! But pay attention libraries used your production code should not be installed in your virtualenv, you should do “GAE way” and link them from your application folder. You should install on virtualenv only stuff used on your tests. Check site.py docs for more details on using .pth files.