KCRW is Southern California’s flagship National Public Radio affiliate, featuring an independent mix of music, news, and cultural programming supported by more than 55,000 member/subscribers. The non-commercial broadcast signal is simulcast on the web at KCRW.com, along with an all-music and an all-news stream and an extensive selection of podcasts. KCRW was a pioneer in online radio, they’ve been live streaming on the web since 1995. The station has an international listening audience, and is widely considered one of the most influential independent music radio stations around the globe.
KCRW has used the Plone CMS for 7 years and for the last 6 the website has been running on Plone version 2.5, which has been increasingly showing its age. Over a year ago KCRW embarked on a project to redesign the site and upgrade the software and functionality, and they selected Jazkarta for the technical implementation. It was an amazingly fun and challenging project, with lots of interactive multimedia functionality and a beautiful responsive design by the New York City firm Hard Candy Shell. We’re very excited to announce that the site launched last Monday!
Making the Most of the CMS
Plone is a robust and feature rich enterprise CMS with many features and add-ons that were invaluable for developing the KCRW site. Some highlights:
- Flexible theming – Using Diazo, a theme contained in static HTML pages can be applied to a dynamic Plone site. For KCRW, Hard Candy Shell created a fully responsive design with phone, tablet and desktop displays. Jazkarta applied the theme to the site using Diazo rules, making adjustments to stylesheets or dynamically generated views where necessary so the CSS classes matched up.
- Modular layouts – We used core Plone portlets and collective.cover tiles to build a system of standard layouts available as page fragments. Many custom tiles and portlets are based on the same underlying views so editors can easily reuse content fragments throughout the site. Plone portlets are particularly handy because of the way they can be inherited down the content tree – for example allowing the same promotion or collection of blog posts to be shown on all the episodes and segments within a show.
- Managing editors – Plone provides granular control over editing permissions. For KCRW, this means that administrators can control what different types of users are allowed to edit in different parts of the site.
- Customizable forms – We created PloneFormGen customizations to track RSVPs, signups, and attendance at events.
- Salesforce integration – Plone has an excellent toolkit for integration with the Salesforce.com CRM. For this phase of the project we implemented basic integration. Stay tuned for additional KCRW member features to be added this fall that take advantage of the Plone-Salesforce connection.
Supporting a Radio Station
KCRW is a radio station, and we developed some cool features to support their content management process and all the rich media available on the site.
- A set of custom content types (shows, episodes, segments, etc.) and APIs for scheduling radio programs and supporting live and on demand audio and video. The APIs provide all sorts of useful information in a consistent way across lots of contexts, including mobile and tablet applications.
- An “always on top” player built using AJAX page loading – as you navigate around the site it just keeps playing and the controls continue to show. This works equally well on mobile devices and desktops.
- Central configuration of underwriting messages in portlets using responsive Google DFP tags.
- Integration with many external services like Disqus for threaded comments, Zencoder for audio encoding, Ooyala for video hosting and encoding, and Castfire for serving podcasts and live streams with advertising.
- An API for querying data about songs played on the station – live or on demand. The API is built on the Pyramid framework and queries a pre-existing data source.
A Robust Deployment Platform
More than any other client, KCRW’s site provided the impetus for us to adopt AWS OpsWorks. KCRW.com had been hosted on a complex AWS stack with multiple servers managed independently. We needed an infrastructure that was easier to manage and could even save KCRW money by being easily scaled up and down as needed. Another major concern was high availability and we tried to eliminate single points of failure.
To accomplish this we made sure everything on OpsWorks was redundant. We have multiple instances for nearly every piece of functionality (Plone, Blob Storage, Celery, Memcached, Nginx, Varnish and even HAProxy), and the redundant servers are in multiple Availability Zones so the website can withstand the failure of an entire Amazon AZ. The layers of functionality can be grouped onto a single instance or spread across multiple instances. It’s easy to bring up and terminate new instances as needed; this can be done manually, or automated based on instance load or during specific times of day. Time-based scaling is particularly relevant to KCRW and we are still experimenting with how best to schedule extra servers during popular weekday listening hours. Amazon’s Elastic Load Balancer and Multi-AZ RDS services give us the ability to deploy resources in multiple Availability Zones and eliminate single points of failure.
Dynamic Client, Dynamic Site
All of these technical details are fun for developers to talk about, but what’s really impressive is how much fun the site is to look at and use. Kudos to KCRW for having the vision to create such a great site, and to KCRW staff for the appealing new content that appears every day.
Here at Jazkarta we’ve been creating repeatable deployments for Plone and Django sites on Amazon’s AWS cloud platform for 6 years. Our deployment scripts were built on mr.awsome (a wrapper around boto, Python’s AWS library) and bash scripts. Our emphasis was on repeatable, not scalable (scaling had to be done manually), and our system was a bit cumbersome. There were aspects that were failure prone, and the initial deployment of any site included a fair amount of manual work. When we heard about AWS OpsWorks last summer we realized it would be a big improvement to our deployment process.
OpsWorks is a way to create repeatable, scalable, potentially multi-server deployments of applications on AWS. It is built on top of Chef, an open source configuration management tool. OpsWorks simplifies the many server configuration tasks necessary when deploying to the cloud; it allows you to create repeatable best practice deployments that you can modify over time without having to touch individual servers. It also lets you wire together multiple servers so that you can deploy a single app in a scalable way.
OpsWorks is a bit similar to PaaS offerings like Heroku but it is better suited for sites that need more scalability and customization than Heroku can provide. The cost of Heroku’s point and click simplicity is a lack of flexibility – OpsWorks lets you change things and add features that you need. And unlike the PaaS offerings, there is no charge for OpsWorks – you don’t pay for anything besides the AWS resources you consume.
Briefly, here’s how it works.
- First you create a stack for your application, which is a wrapper around everything. It may include custom chef recipes and configuration and it defines the AMI that will be used for all the instances. There are currently 2 choices, latest stable Ubuntu LTS or Amazon Linux. (We use Ubuntu exclusively.)
- Within the stack you define layers that represent the services or functionality your app requires. For example, you might define layers for an app server, front end, task queue, caching, etc. The layers define the resources they need – Elastic IPs, EBS volumes, RAID10 arrays, or whatever.
- Then you define the applications associated with the layers. This is your code, which can come from a github repo, an S3 bucket, or wherever.
- Then you define instances, which configure the EC2 instances themselves (size, availability zone, etc.), and assign the layers to the instances however you want. When you define an instance it is just a definition, it does not exist until it is started. Standard (24-hour) instances are started and stopped manually, time-based instances have defined start and stop days and times, and load-based instances have customizable CPU, load or memory thresholds which trigger instances to be started and stopped. When an instance starts, all the configuration for all the layers is run, and when it is stopped all the AWS resources associated with it are destroyed – aside from persistent EBS storage or Elastic IPs which are bound to the definition of the instance in OpsWorks instead of being bound to an actual instance.
For more details and a case study about switching from Heroku, see this excellent introduction to OpsWorks by the folks at Artsy.
What We Did
OpsWorks has native support for deploying Ruby on Rails, NodeJS, Java Tomcat, PHP, and static HTML websites, but no support for Python application servers (perhaps partly because there is no standard way to deploy Python apps). This was a situation we thought needed to be remedied. Furthermore, few if any PaaS providers are suitable for deploying the Plone CMS which many of our clients use. Because OpsWorks essentially allows you to build your own deployment platform using Chef recipes, it seemed like it might be a good fit.
Chef is a mature configuration management system in wide use, and there are many open source recipes available for deploying a variety of applications. None of those quite met our needs in terms of Python web application deployment, so we wrote two new cookbooks (a bundle Chef recipes and configuration). We tried to structure the recipes to mimic the built in OpsWorks application server layer cookbooks.
The repository is here: https://github.com/alecpm/opsworks-web-python. Each cookbook has its own documentation.
- Python Cookbook – provides recipes to create a Python environment in a virtualenv, to deploy a Django app, and to deploy a buildout
- Plone Cookbook – builds on the Python and buildout cookbooks to deploy scalable and highly available Plone sites
The Plone cookbook can handle complex Plone deployments. An example buildout is provided that supports the following layers:
- Clients and servers and their communication
- Load balancing
- Shared persistent storage for blobs
- Relstorage – either via Amazon RDS or a custom database server in its own OpsWorks layer (there is a built in layer for MySQL)
- Redis and Celery
- Auto-scaling the number of Zeo clients from the AWS instance size
The recipes handle automatically interconnecting these services whether they live on a single instance or on multiple instances across different Availability Zones. For more information, see the README in each cookbook.
We’ve used OpsWorks with our custom recipes on a few projects so far and are quite happy with the results. We have a wishlist of a few additional features that we’d like to add:
- Automated rolling deployments – a zero down time rolling deployment of new code that updates each Zeo client in sequence with a pause so the site doesn’t shut down.
- Native Solr support – use the OS packages to install Solr (instead of buildout and collective.recipe.solr) and allow custom configuration for use with alm.solrindex or collective.solr in the recipe.
- Integration of 3rd party (New Relic, Papertrail, …) and internal (munin, ganglia, …) monitoring services.
- Better documentation – we need feedback about what needs improvement.
If you’d like to contribute new features or fixes to the cookbooks feel free to fork and issue a pull request!
For over a year a Jazkarta team consisting of David Glick, Carlos de la Guardia, Cris Ewing, and me has been working on the development of a new website for The Mountaineers, the premier outdoor education non-profit in the Pacific Northwest. We are very excited to announce that it just launched today!
The Mountaineers has over 10,000 members and over a thousand volunteers leading hundreds of activities and classes in hiking, backpacking, climbing, skiing, sea kayaking and much more. Their old website had become dated and no fun to use, with convoluted processes and an ad hoc structure, so the organization embarked on a major project to create something better. The goal was a beautiful, modern, and engaging website that simplified as many things as possible – making it easy for leaders to create new activities, for volunteers to volunteer, for members to sign up and donate, and for everyone to find what they’re looking for in The Mountaineers’ vast portfolio of outdoor knowledge.
After kicking the tires on lots of systems, The Mountaineers settled on the technology for their new website: a partnership between Plone, the leading open source enterprise CMS, and Salesforce.com, the leading software-as-a-service CRM. Plone+Salesforce is a dynamite combination for non-profits because of extensive integration possibilities and deep license discounts. For The Mountaineers, it meant that their oceans of data about members, courses, activities, rosters, routes and places, committees, etc. could be seemlessly synchronized between these two best-of-breed systems.
Implementation of the site was a partnership as well – Jazkarta doing the Plone front end and Percolator Consulting, engagement strategists and CRM experts, doing the Salesforce backend. Darrell Houle provided user experience design for the many complex course and activity creation, scheduling, search, and registration processes, and Neal Maher turned Darrell’s wireframes into a gorgeous graphic design.
Fourteen months and 673 github tickets later, the result is a content-rich site with curation and management shared by a community of hundreds of collaborators. The functionality visible on any given page is determined by the role (member, leader, staff) of the person who is viewing it. A new Plone add-on, collective.workspace, provides roster functionality for courses, activities, and committees, allowing each participant’s role, registration status, and other information to be managed. A lightweight shopping cart integrated with Stripe payment processing allows users to easily pay for memberships and courses, or make a donation. Custom content types provide extensive metadata that enables as many kinds of faceted searches and dynamic listings as staff can dream up – like the activity faceted search shown below.
All of this is the result of an incredible amount of work by a lot people. Thanks to the Jazkarta and Percolator team members and the many Mountaineers staff and volunteers who tested, made tickets, and tested again!
We at Jazkarta have worked on CMS-powered websites for many years. We’ve built a lot of sites and worked with a lot of customers and we think we know what makes a good CMS for the kinds of projects we do – substantial websites with lots of members and lots of content. Our go-to CMS is Plone – we’ve been working with it since 2003! – but we’ve also tried out Django CMS, Mezzanine, and have managed our share of WordPress blogs.
As part of our 3 day rural sprint, we decided to pool our experiences and write down our thoughts about what makes a good CMS. What features are valuable and what features can be skipped? Because Plone is so feature-rich and we know it so well, it made sense to frame our ideas in terms of those features.
Our group consisted of Nate Aune, Carlos de la Guardia, Alec Mitchell, Cris Ewing, David Glick, Chris Rossi, and Sally Kleinfeldt. First we brainstormed lists of features in the 3 categories below. Once we had the lists, each person got 12 red sticky dots to place on any item to indicate how important they thought that item was. We could place all 12 dots on one item, 1 dot each on 12 items, or any combination thereof. (If these red dots look familiar, you may remember this process was used extensively at the Plone Strategic Summit in 2008.)
Here are our lists, annotated with asterisks to represent the red dots. We hope this information will be useful to anyone writing a new CMS or extending an existing one.
What does Plone do well that our customers use?
- User Management **
- Pluggable authentication system, configurable
- LDAP, SSO integration
- Easier integration with social is desirable ***
- Search **
- Built-in is good
- Configurable to different back-ends is vital
- Plugins/Add-ons ***
- Content Placement/Management **
- Orderable *
- Cut/Copy/Paste *
- Versioning *
- Workflow ****
- Usually simple
- Occasionally internal/external
- Business process/government can be complex
- Staging or Working Copies *
- Especially useful for government or education customers
- Granting of local (to a portion of the site) roles/permissions to groups/users ****
- Roles/Permissions and Users/Groups
- Content Types **
- Pages *
- Folders (default page problematic) *
- Events (but should they be a core feature?)
- Content Lead Images *
- Collections **
- WYSIWYG **
- Files/Images (thumbnail generation, basic image editing) **
- A few default views *
- Selectable views
- Analytics integration
- In-place editing/creation/management *****
- Important for users with little training
- Site admins could be comfortable with separate backend-style configuration/editing
- Should be cleanly separated for ease of theming ***
- Categorization/Redirection/Link integrity
- Inherited down the tree of content ***
- Would be nice to have it merge toward tiles
- Usually customized
What does Plone do well that our customers do not use?
- Related items
- Complex workflow
- Internationalization (i18n)/Multi-lingual (because we work primarily in the U.S.)
- Content Rules **
What does Plone not do well?
For this category we always have to install an add-on or fix something.
- Tagging/Categorization especially management of tags
- Default Pages
- Previewing changes before save
- Default folder/collection listings are unattractive and not easily configurable *
- Form builder *** (PloneFormGen is reasonable)
- Faceted search/navigation
- Videos/embedded media ***
- Page layout customization **
- Mobile-first/Responsive **
- Well-defined API ************
- Hosting/Repeatable deployment *****
- Theming and template overrides ****
What we need
At the end of this exercise, we totaled up all the red dots to see which features we had voted most important. If there were red dots on sub-items, they were rolled up to the parent item. Here’s the list – the number of red dots the item received is indicated in parentheses. We added a few essential things that didn’t get any votes but that a serious CMS can’t do without.
- (12) Well-defined API
- (11) Content Types
- (8) In-place Editing
- (5) Content Placement/Management
- (5) Hosting
- (5) Search
- (5) User Management
- (4) Theming/Template Overrides
- (4) Workflow
- (4) Local Roles/Permission
- (3) Form Building
- (3) Video/Embedding
- (3) Inheritable Portlets
- (3) Plugins
- (2) Content Rules
- (2) Mobile First/Responsive
- (2) Page Layout Customization
- (2) Views – Attractive Defaults
- (1) Staging/Working Copies
The clear winner in this exercise was “well-defined API”, which is not a big surprise given that a bunch of developers made the list! Although our customers never ask for this, it is essential for doing the type of highly customized websites that they demand.
Over the last few months the community has really picked up the pace on Plone 5. Supported by the Plone Foundation and numerous sponsors, there have been a series of productive sprints. The Emerald Sprint’s focus was on user management, registration, and login. A robust system of user permissions, groups, and roles is one of Plone’s most notable (and oldest) features and the concepts and underpinnings are still solid. However the UI is overdue for an overhaul and the old implementation layers have gotten pretty crufty.
The sprinters, led by David Glick, took a UI-first aproach which was fantastic. Before cracking open the laptops and diving into the code, we developed mockups of the new registration and login process and user management screens. It really helped that 2 of the 17 sprinters were UI/UX designers. Always try to get designers to come to your sprints!
The sprint gathered together a fantastic set of Plone gurus who were able to have in depth discussions of some of the thornier technical problems associated with users. For example, should user objects work like content objects? This discussion resulted in a concrete list of pros and cons and a better understanding of how to ameliorate potential problems if and when we decide to move to “contentish” users.
And of course software got developed. Teams worked on the Users and Groups control panel, member profile design and implementation, registration, log in, and forgotten password dialogs, and more. Read the summary report on plone.org for more details.
For the last three days, the Jazkarta team has been enjoying delicious meals lovingly prepared from local farm ingredients by Sally and her husband Doug, walks in the thawing New England woods, and even a visit to a farm to meet the baby goats who were especially attracted to Carlos for reasons unknown, and showed their affection by nimbling at his pants and shirt.
We usually work together from our home offices, and “meet” each other in the IRC channel, so when Sally invited us all to her place for a mini-sprint prior to Pycon, we jumped at the opportunity to meet up in person.
While we have the tradition of monthly “happy hours” in which we all get on Google Hangout with our beverage of choice and rap for an hour, it’s not the same as hanging out in real life, eating and drinking together, sprinting together, and yes, playing with baby goats together. ;)
Tomorrow we’re getting up early, piling into the rented mini-van and driving 5 hours to Montreal for PyCon! We’ll be staying in a shared apartment just a few blocks away from the conference center. If you want to get away from the PyCon crowds and chill out, you’re welcome to stop by our pad and say hi. Who knows – Sally might even treat you to some of her amazing home cooking!
See some photos from the sprint!
A big thank you to the Salesforce Foundation for inviting me to contribute a post on Plone+Salesforce integration to their blog. It was fun to reacquaint myself with all the tools and learn what’s new. Timely too, since we have just started work on a new Plone site for The Mountaineers which will include integration of all their custom content with Salesforce.com. We are working on this project in partnership with Percolator Consulting. They are engagement strategists and Salesforce specialists and are creating the CRM back end while we are creating the Plone site.
In the process of writing and discussing this post, I discovered that David Glick, who is working on The Mountaineers project with us, and Jason Lantz, from the Innocence Project, were having similar thoughts about the Plone+Salesforce toolkit. This week we had a chat to explore ways in which we might collaborate.
There are two main areas in need of improvement.
- Salesforce.com API integration – The current Python library used by Plone is Beatbox, which is a wrapper for the Salesforce.com SOAP API. We now have Simple Salesforce, a Python library providing a low level interface to the Salesforce.com REST API (it returns a dictionary of the API’s JSON response). There is also a new socket-based streaming API. David and Jason are evaluating replacing Beatbox with one of these more modern components.
- Message queuing – The current set of Plone+Salesforce tools rely on real-time communication with Salesforce.com, which slows performance and does not recover gracefully from network errors. A message queuing approach is sorely needed, where API requests can be queued for delivery and automatically retried. David and Jason are evaluating our options for Python message queuing. This includes but is not limited to plone.app.async. We would like the Plone implementation to build on a pure Python layer that could be used in any Python web framework (such as Django or Pyramid).
These items are critical to the success of The Mountaineers’ and the Innocence Project’s Plone+Salesforce integrations. In both cases the volume of API calls will be great enough that asynchronous communication is a must-have. Over the next few weeks we hope to create a plan that our two projects can collaborate on, for drastically improving the Plone+Salesforce infrastructure. This will benefit the entire Plone+Salesforce ecosystem – for example, the message queuing system could be used as the basis for a form-to-Salesforce solution that is more responsive and robust.
We would also like to improve the administrative tools that are available. Site admins will need a control panel that allows them to monitor and retry messages in the queue. Down the road it would be great to provide a way for integrators to define Plone to Salesforce content mappings through the web. Today it is necessary to do Python development to implement anything other than a fairly simple Plone+Salesforce web-to-lead form.
Our goal is to flesh out a plan by June 5th, when David, Jason and I will all be at Plone Symposium Midwest. Please contact me if you’d like to join our conversations, and maybe even sprint on an improved Plone+Salesforce infrastructure!