Skip to content

Avoiding buildout failure when pypi is down using collective.eggproxy

September 1, 2009

In my last post in the continuing series about deploying and optimizing Plone sites, we learned how to make a buildout for Plone. A buildout is a set of instructions for how to create a Zope instance with all of the dependencies to run our Plone site.

In this post, we will learn how to:

  1. Configure buildout to fetch Plone add-ons and add them to our Zope instance
  2. Configure buildout to use a local egg proxy server
Note: in this blog post, I’m using the terms egg = add-on = product interchangeably. There are of course differences between these names, but for the purposes of this article, we will just assume they mean the same thing.

Fetch Plone add-ons and add them to our Zope instance

In order to add a Plone add-on to our Zope instance, we must add a couple lines to the buildout.cfg file. Let’s say that we wanted to add PloneFormGen, a popular Plone add-on for making forms through-the-web, to our Plone site. We would simply add the bold line below to our buildout.cfg file:

[instance]
...
eggs =
    PIL
    Plone
    Products.PloneFormGen

And then we need to re-run buildout so that this new add-on is downloaded from pypi, and added to our Zope instance.

$ bin/buildout -v

This will use the default egg server at pypi.python.org/simple/ to download the egg.

Now we need to restart our Zope instance since we have added a new product that must be loaded.

$ bin/instance fg

If all goes well, you should see this:

2009-08-25 21:04:30 INFO Zope Ready to handle requests

Navigate to the Add/Remove products page, and you should see PloneFormGen available for install.

http://localhost:8080/Plone/prefs_install_products_form

Or click in the upper right hand corner of the screen on Site Setup and then select Add/Remove Products.

If we want to add an add-on product that is not in the Products.* namespace, then we need to add a zcml declaration to our buildout.cfg. Let’s say we want to add collective.flowplayer for handling of Flash video files and playback of MP3 files on our Plone site.

[instance]
...
eggs =
    PIL
    Plone
    Products.PloneFormGen
    collective.flowplayer

zcml =
    collective.flowplayer

Again re-run the buildout and restart Zope as per the instructions above. As with PloneFormGen, you should see collective.flowplayer as one of the installable products on the Add/Remove products screen.

Congratulations – you have successfully installed two Plone add-on products into your Plone site!

Using a local egg proxy server to avoid buildout failure

Normally, when you run buildout everything will go smoothly and buildout will have no problem connecting to pypi.python.org and downloading the eggs you need, but what happens if pypi (or “the cheeseshop” as it is affectionally called) goes down? Yes, it does happen on occasion, and you don’t want it to go down right when you are in the middle of a deployment to production.

If the cheeseshop is down when try to run the buildout, things can go wrong, and you can end up with a broken buildout, and not be able to start up your Zope instance. Not good.

So the solution is to run a local egg proxy which fetches all the eggs from the cheeseshop and stores them on your local machine or if you are working with a team, on a server that you control. That way, even if the cheeseshop is down, buildout will still be able to get the eggs it needs from your local egg cache.

Thanks to Tarek Ziade, he has created collective.eggproxy which does all the hard work for us. Watch a little screencast he made which shows how it works. Also see his excellent slides from Plone Conference 2008.

Another bonus of using collective.eggproxy, is that you can run buildout even if you don’t have an Internet connection – when you can a plane, on a bus, in a cafe. Buildout

Add the following lines in bold to the buildout.cfg:

[buildout]
...
parts =
    zope2
    instance
    console_scripts
    configuration

...

[console_scripts]
recipe = zc.recipe.egg
eggs = collective.eggproxy

[configuration]
recipe = collective.recipe.template
input = etc/eggproxy.conf.in
output = etc/eggproxy.conf

Then you need to make an /etc dir and add the eggproxy.conf.in file to it:

$ cd budapesttraining/buildout
$ mkdir etc
$ [mate|vi|emacs] etc/eggproxy.conf.in

Create the following file etc/eggproxy.conf.in:

[default]
eggs_directory = ${buildout:directory}/var/cache
#update_interval = 24
#index = http://pypi.python.org/simple
#port = 8888

Re-run the buildout to create the eggproxy scripts:

$ bin/buildout -v

Then you’ll also need to make the var/cache directory before starting up the egg proxy:

$ mkdir var/cache

Then we modify the buildout.cfg to use this local cache proxy instead of the pypi.python.org.

[buildout]
extends = http://dist.plone.org/release/3.3/versions.cfg
versions = versions
find-links = http://dist.plone.org/thirdparty
index = http://localhost:8888/
allow-hosts = *localhost*

Now we can start up the egg proxy with this command:

$ bin/eggproxy_run
serving on http://127.0.0.1:8888

After we re-run our buildout, we can see that the buildout/var/cache directory is replicating a subset of the eggs on the cheeseshop. So the next time we run buildout, it will first look in this cache directory for the eggs, rather than going out to the cheeseshop.

Conclusion

We have learned how to add eggs to our buildout, both eggs with the Products.* namespace as well as add-ons which require adding a zcml declaration to our buildout.cfg file.

We have also set up a local egg proxy which will save us in the event that the cheeseshop goes down – our buildout will still run and use the locally cached eggs.

References

Reinout van Rees’ blog post on collective.eggproxy

Thanks Reinout for your improvements to collective.eggproxy!

Get hands-on training from Plone experts

If you are attending the Plone Conference in Budapest, and would like to participate in our comprehensive training course on Best Practices for Deploying and Optimizing Plone, the two day class costs only $300 USD. Read the full agenda – suggestions for other topics are most welcome!

Seating is limited, so I encourage you to sign up today to reserve your place.

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: