appengine & virtualenv

21 11 2010

UPDATE: appengine 1.6.1 & uses gaecustomize.py

This article will explain how to setup Google AppEngine (GAE) with virtualenv.

GAE does not provide a “setup.py” to make the SDK “installable”, it is supposed to be used from a folder without being “installed”. GAE actually forbids the use of any python library in the site-packages folder. All included libraries must be in the same folder as your application, this allows GAE to automatically find and upload third-party libraries together with your application code when you upload the code to GAE servers.

So what would be the advantages of using of using virtualenv with GAE? The main reason is to have an environment to run unit-tests and functional tests. It will allow us to use the interactive shell to make operations on DB. And it also enforce you are using the correct python version.

Step 0 – install App Engine SDK

Make sure you use 1.6.1 or later, an important bug was fixed on this release. As described on official docs. Tested with virtualenv 1.6.4.

Step 1 – create and activate a virtualenv

Same as usual…


$ virtualenv --python python2.5 --no-site-packages gae-env
$ source gae-env/bin/activate

Step 2 – add google_appengine path

Add a path configuration file named “gae.pth” to the virtualenv site-packages with the path to google_appengine. This way google_appengine will be in sys.path enabling it to be imported by other modules.

You will need to adjust the content of the file according to where you created your virtualenv and google_appengine location. Mine looks like this:


$ cat gae-env/lib/python2.5/site-packages/gae.pth
../../../../google_appengine

Simple test to make sure your gae.pth is correct:

(gae-env)$ python
>>> from google import appengine

If you did not get any exception you are good to go on.

Step 3 – fix path for third-party libs

The AppEngine SDK comes with a few third-party libraries. They are not in the same path as google’s libraries. If you look at dev_appserver.py you will see a function called fix_sys_path, this function adds the path of the third-party libraries to python’s sys.path. One option would be to add these paths to gae.pth… But I prefer to use the function fix_sys_path so we have less chances of having problems with future releases of the SDK.

Note that this will not look for your config in app.yaml. So you might need to add some extra imports. The example below is using webob version 1.1.1 instead of the default one.

Path configuration files can also execute python code on if the line starts with import. Add a module gaecustomize.py to site-packages:

gae-env/lib/python2.5/site-packages/gaecustomize.py

def fix_sys_path():
    try:
        import sys, os
        from dev_appserver import fix_sys_path, DIR_PATH
        fix_sys_path()
        # must be after fix_sys_path
        # uses non-default version of webob
        webob_path = os.path.join(DIR_PATH, 'lib', 'webob_1_1_1')
        sys.path = [webob_path] + sys.path
    except ImportError:
        pass

And modify gae.pth it calls the above module:

gae-env/lib/python2.5/site-packages/gae.pth

../../../../google_appengine
import gaecustomize; gaecustomize.fix_sys_path()

For some unknown reason gae.pth is being processed twice and on the first time google_appengine is not added to sys.path. Thats why I explicitly call the function fix_sys_path.

Check if it is working fine:


(gae-env)$ python
>>> import yaml

Again. You should not any exceptions on this…

Step 4 – add dev_appserver.py to bin

Not really required but handy.


gae-env/bin $ ln -s ../../google_appengine/dev_appserver.py .

Conclusion

Now you have an isolated environment running AppEngine! But pay attention libraries used your production code should not be installed in your virtualenv, you should do “GAE way” and link them from your application folder. You should install on virtualenv only stuff used on your tests. Check site.py docs for more details on using .pth files.

About these ads

Actions

Information

5 responses

24 03 2012
ssspiochld

I believe that your post is currently the best gae+virtualenv HOWTO out there. Thanks!

I created a project a while ago https://github.com/ssspiochld/googleappengine-python
which:
1) mirrors+tags it in git
2) in pip branch automates that all in one step using setup.py+MANIFEST.in
maybe U will consider it interesting ;)

10 06 2012
Configuring virtualenv to run Google appengine samples « AutoShop

[…] bit of Googling turned up this post which details all of the necessary steps. In step 2 the path I used for google_appengine […]

12 09 2012
Chris

Trying to run the dev server gives me the error “google.appengine.tools.dev_appserver.AppConfigNotFoundError”. Adding a print statement in dev_appserver.py LoadAppConfig to show appinfo_path indicates the GAE is using /local/lib/python2.7/site-packages as the project root, which is incorrect.

14 09 2014
Bolutife

Nice. I think this might have been the inspiration for how gae-init handles virtual environments.

7 11 2014
houmie

Unfortunately step 2 already failed with Virtualenv and Python 2.7. Are you sure the path is loaded this way?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Follow

Get every new post delivered to your Inbox.

%d bloggers like this: