Avoiding buildout failure when pypi is down using collective.eggproxy
In my last post in the continuing series about deploying and optimizing Plone sites, we learned how to make a buildout for Plone. A buildout is a set of instructions for how to create a Zope instance with all of the dependencies to run our Plone site.
In this post, we will learn how to:
- Configure buildout to fetch Plone add-ons and add them to our Zope instance
- Configure buildout to use a local egg proxy server
Fetch Plone add-ons and add them to our Zope instance
In order to add a Plone add-on to our Zope instance, we must add a couple lines to the buildout.cfg file. Let’s say that we wanted to add PloneFormGen, a popular Plone add-on for making forms through-the-web, to our Plone site. We would simply add the bold line below to our buildout.cfg file:
[instance]
...
eggs =
PIL
Plone
Products.PloneFormGen
And then we need to re-run buildout so that this new add-on is downloaded from pypi, and added to our Zope instance.
$ bin/buildout -v
This will use the default egg server at pypi.python.org/simple/ to download the egg.
Now we need to restart our Zope instance since we have added a new product that must be loaded.
$ bin/instance fg
If all goes well, you should see this:
2009-08-25 21:04:30 INFO Zope Ready to handle requests
Navigate to the Add/Remove products page, and you should see PloneFormGen available for install.
http://localhost:8080/Plone/prefs_install_products_form
Or click in the upper right hand corner of the screen on Site Setup and then select Add/Remove Products.
If we want to add an add-on product that is not in the Products.* namespace, then we need to add a zcml declaration to our buildout.cfg. Let’s say we want to add collective.flowplayer for handling of Flash video files and playback of MP3 files on our Plone site.
[instance]
...
eggs =
PIL
Plone
Products.PloneFormGen
collective.flowplayer
zcml =
collective.flowplayer
Again re-run the buildout and restart Zope as per the instructions above. As with PloneFormGen, you should see collective.flowplayer as one of the installable products on the Add/Remove products screen.
Congratulations – you have successfully installed two Plone add-on products into your Plone site!
Using a local egg proxy server to avoid buildout failure
Normally, when you run buildout everything will go smoothly and buildout will have no problem connecting to pypi.python.org and downloading the eggs you need, but what happens if pypi (or “the cheeseshop” as it is affectionally called) goes down? Yes, it does happen on occasion, and you don’t want it to go down right when you are in the middle of a deployment to production.
If the cheeseshop is down when try to run the buildout, things can go wrong, and you can end up with a broken buildout, and not be able to start up your Zope instance. Not good.
So the solution is to run a local egg proxy which fetches all the eggs from the cheeseshop and stores them on your local machine or if you are working with a team, on a server that you control. That way, even if the cheeseshop is down, buildout will still be able to get the eggs it needs from your local egg cache.
Thanks to Tarek Ziade, he has created collective.eggproxy which does all the hard work for us. Watch a little screencast he made which shows how it works. Also see his excellent slides from Plone Conference 2008.
Another bonus of using collective.eggproxy, is that you can run buildout even if you don’t have an Internet connection – when you can a plane, on a bus, in a cafe. Buildout
Add the following lines in bold to the buildout.cfg:
[buildout]
...
parts =
zope2
instance
console_scripts
configuration
...
[console_scripts]
recipe = zc.recipe.egg
eggs = collective.eggproxy
[configuration]
recipe = collective.recipe.template
input = etc/eggproxy.conf.in
output = etc/eggproxy.conf
Then you need to make an /etc dir and add the eggproxy.conf.in file to it:
$ cd budapesttraining/buildout $ mkdir etc $ [mate|vi|emacs] etc/eggproxy.conf.in
Create the following file etc/eggproxy.conf.in:
[default]
eggs_directory = ${buildout:directory}/var/cache
#update_interval = 24
#index = http://pypi.python.org/simple
#port = 8888
Re-run the buildout to create the eggproxy scripts:
$ bin/buildout -v
Then you’ll also need to make the var/cache directory before starting up the egg proxy:
$ mkdir var/cache
Then we modify the buildout.cfg to use this local cache proxy instead of the pypi.python.org.
[buildout] extends = http://dist.plone.org/release/3.3/versions.cfg versions = versions find-links = http://dist.plone.org/thirdparty index = http://localhost:8888/ allow-hosts = *localhost*
Now we can start up the egg proxy with this command:
$ bin/eggproxy_run serving on http://127.0.0.1:8888
After we re-run our buildout, we can see that the buildout/var/cache directory is replicating a subset of the eggs on the cheeseshop. So the next time we run buildout, it will first look in this cache directory for the eggs, rather than going out to the cheeseshop.
Conclusion
We have learned how to add eggs to our buildout, both eggs with the Products.* namespace as well as add-ons which require adding a zcml declaration to our buildout.cfg file.
We have also set up a local egg proxy which will save us in the event that the cheeseshop goes down – our buildout will still run and use the locally cached eggs.
References
Reinout van Rees’ blog post on collective.eggproxy
Thanks Reinout for your improvements to collective.eggproxy!
Get hands-on training from Plone experts
If you are attending the Plone Conference in Budapest, and would like to participate in our comprehensive training course on Best Practices for Deploying and Optimizing Plone, the two day class costs only $300 USD. Read the full agenda – suggestions for other topics are most welcome!
Jazkarta Blog 