Over the last few months the community has really picked up the pace on Plone 5. Supported by the Plone Foundation and numerous sponsors, there have been a series of productive sprints. The Emerald Sprint’s focus was on user management, registration, and login. A robust system of user permissions, groups, and roles is one of Plone’s most notable (and oldest) features and the concepts and underpinnings are still solid. However the UI is overdue for an overhaul and the old implementation layers have gotten pretty crufty.
The sprinters, led by David Glick, took a UI-first aproach which was fantastic. Before cracking open the laptops and diving into the code, we developed mockups of the new registration and login process and user management screens. It really helped that 2 of the 17 sprinters were UI/UX designers. Always try to get designers to come to your sprints!
The sprint gathered together a fantastic set of Plone gurus who were able to have in depth discussions of some of the thornier technical problems associated with users. For example, should user objects work like content objects? This discussion resulted in a concrete list of pros and cons and a better understanding of how to ameliorate potential problems if and when we decide to move to “contentish” users.
And of course software got developed. Teams worked on the Users and Groups control panel, member profile design and implementation, registration, log in, and forgotten password dialogs, and more. Read the summary report on plone.org for more details.
For the last three days, the Jazkarta team has been enjoying delicious meals lovingly prepared from local farm ingredients by Sally and her husband Doug, walks in the thawing New England woods, and even a visit to a farm to meet the baby goats who were especially attracted to Carlos for reasons unknown, and showed their affection by nimbling at his pants and shirt.
We usually work together from our home offices, and “meet” each other in the IRC channel, so when Sally invited us all to her place for a mini-sprint prior to Pycon, we jumped at the opportunity to meet up in person.
While we have the tradition of monthly “happy hours” in which we all get on Google Hangout with our beverage of choice and rap for an hour, it’s not the same as hanging out in real life, eating and drinking together, sprinting together, and yes, playing with baby goats together. ;)
Tomorrow we’re getting up early, piling into the rented mini-van and driving 5 hours to Montreal for PyCon! We’ll be staying in a shared apartment just a few blocks away from the conference center. If you want to get away from the PyCon crowds and chill out, you’re welcome to stop by our pad and say hi. Who knows – Sally might even treat you to some of her amazing home cooking!
See some photos from the sprint!
A big thank you to the Salesforce Foundation for inviting me to contribute a post on Plone+Salesforce integration to their blog. It was fun to reacquaint myself with all the tools and learn what’s new. Timely too, since we have just started work on a new Plone site for The Mountaineers which will include integration of all their custom content with Salesforce.com. We are working on this project in partnership with Percolator Consulting. They are engagement strategists and Salesforce specialists and are creating the CRM back end while we are creating the Plone site.
In the process of writing and discussing this post, I discovered that David Glick, who is working on The Mountaineers project with us, and Jason Lantz, from the Innocence Project, were having similar thoughts about the Plone+Salesforce toolkit. This week we had a chat to explore ways in which we might collaborate.
There are two main areas in need of improvement.
- Salesforce.com API integration – The current Python library used by Plone is Beatbox, which is a wrapper for the Salesforce.com SOAP API. We now have Simple Salesforce, a Python library providing a low level interface to the Salesforce.com REST API (it returns a dictionary of the API’s JSON response). There is also a new socket-based streaming API. David and Jason are evaluating replacing Beatbox with one of these more modern components.
- Message queuing – The current set of Plone+Salesforce tools rely on real-time communication with Salesforce.com, which slows performance and does not recover gracefully from network errors. A message queuing approach is sorely needed, where API requests can be queued for delivery and automatically retried. David and Jason are evaluating our options for Python message queuing. This includes but is not limited to plone.app.async. We would like the Plone implementation to build on a pure Python layer that could be used in any Python web framework (such as Django or Pyramid).
These items are critical to the success of The Mountaineers’ and the Innocence Project’s Plone+Salesforce integrations. In both cases the volume of API calls will be great enough that asynchronous communication is a must-have. Over the next few weeks we hope to create a plan that our two projects can collaborate on, for drastically improving the Plone+Salesforce infrastructure. This will benefit the entire Plone+Salesforce ecosystem – for example, the message queuing system could be used as the basis for a form-to-Salesforce solution that is more responsive and robust.
We would also like to improve the administrative tools that are available. Site admins will need a control panel that allows them to monitor and retry messages in the queue. Down the road it would be great to provide a way for integrators to define Plone to Salesforce content mappings through the web. Today it is necessary to do Python development to implement anything other than a fairly simple Plone+Salesforce web-to-lead form.
Our goal is to flesh out a plan by June 5th, when David, Jason and I will all be at Plone Symposium Midwest. Please contact me if you’d like to join our conversations, and maybe even sprint on an improved Plone+Salesforce infrastructure!
Last week Marianne Kay tweeted that the Dumbarton Oaks site was a “beautiful example of responsive design”. (Thanks Marianne!) Kudos for the beautiful Dumbarton Oaks graphic design go to the design firm Might & Main. But kudos for the responsive behavior go to Plone and the Plone community, which made implementation easy. Here’s how we did it.
A responsive design provides an optimal viewing experience across a wide range of devices, from large desktop monitors to mobile phones. It uses CSS3 media queries, fluid, proportion-based grids, and flexible images to adapt the layout to the browser width. Normally, adding responsiveness to a website’s design means multiplying theming time by a factor of 2 or 3. But for the Dumbarton Oaks site responsiveness didn’t add much time because we used Plone’s built-in Diazo theming tool and started with a theme that provided responsive behavior out of the box. We didn’t have to write the code that (for example) makes images re-size, portlets change position, and menu tabs transform into drop downs for smaller screens. The Plone UI/CSS integration was already done for us.
Diazo is a fantastic tool that lets you map dynamic web page content from any CMS or web application into an HTML design. The mapping is done via a set of rules that define how to move elements from the content page into placeholders in the theme. Plone provides nicely integrated Diazo theming, soon to get even better with a through-the-web theme editor that will come with Plone 4.3. You can see a preview of it in this screencast made by Eric Steele.
Here is our recommended Diazo theming process.
- Choose a Diazo Plone theme that provides the desired responsive behavior. For the Dumbarton Oaks project we used beyondskins.responsive by Simples Consultoria.
- Have the designer base his or her HTML templates on the ones provided in the responsive theme.
- Add CSS files to the static theme that provide the responsive UI (from the responsive theme package), the Plone editing interface, IE fixes, and any special features provided by add-ons that are used on the site. Extract the contents of these files from the CSS that Plone serves with all products installed. Plone’s large CSS payload can be significantly pruned, only provide what’s needed.
- To style the HTML, have the designer supplement this CSS in a way that doesn’t interfere with necessary UI elements or harm the responsiveness. It is helpful to add this graphic-design-specific CSS in a separate file.
- Apply the static theme to the Plone site with Diazo rules.
- Plone’s markup consistency and quality is a huge benefit during this process. Just give the designer sample HTML and let them do what’s needed in the theme’s CSS.
When we engaged Might & Main to do the Dumbarton Oaks design, we forgot to tell them it was going to be responsive. They were very perturbed when they found out, since they had not budgeted to do the extra work that is normally required. We told them not to worry because that extra work had already been done in beyondskins.responsive. And sure enough, Dumbarton Oaks got a responsive design with very little effort from Might & Main.
Here is what the site looks like without the Diazo theme applied.
Here is what the themed site looks like on the desktop.
And here is what it looks like on a mobile device.
We recently helped the Innocence Project finish collective.salesforce.fundraising, an open source fundraising package designed to help non-profit organizations raise money online. The system is designed to function either as a standalone donation system or to be integrated into a larger Plone website. The Innocence Project is currently using it as a standalone donation system.
The brainchild of the Innocence Project’s Jason Lantz, who developed the package with help from Groundwire and Jazkarta, this plugin adds significantly to the Salesforce.com functionality that can be tapped from the Plone content management system. The Plone CMS/Salesforce.com CRM integration capabilities previously consisted of:
- Submitting form information from a Plone form into Salesforce.com (web-to-lead forms)
- Adding a new lead to a Salesforce.com account when a new Paypal payment is made
- Making Events in your Plone site “RSVP-able” with data saved directly to Salesforce.com campaigns
- Authenticating in Plone with account information (login, password and properties) which is from Salesforce.com
- Dynamically displaying Salesforce.com data in Plone
That’s already a darn good package, but collective.salesforce.fundraising adds features that take things to a whole new level:
- The ability to take one time and recurring donations through an uncluttered, modern UI. All donations are sent directly to the payment processor (Authorize.net or Recurly), no credit card data is stored or transmitted.
- Integration with Salesforce.com campaigns. They can be created in Salesforce.com or Plone, and all changes and donations made in Plone are added to the Salesforce.com campaign.
- The ability to create campaign timelines and goals. Progress indicators in the UI show progress towards the goal and the time remaining.
- Personal fundraising, which allows users to create personal campaign pages for any campaign. Users can set a goal and promote their campaign to their friends; all donations roll up to the parent campaign in Salesforce.
- Social integration via Janrain, a social login and sharing service. This is important for personal fundraisers, since it allows login through a variety of social accounts, and sharing on multiple social networks to help spread the word about personal campaigns.
And there are many smaller features, like memorial donations, donor testimonials, fundraising “seals” providing third party analysis and endorsements, and an easy way for users to share campaigns with their social networks via “share messages” which can be tracked for effectiveness.
How Jazkarta Helped
The Innocence Project engaged Jazkarta to help with the donation and personal fundraising capabilities. This included selling tickets to events, adding campaign landing pages and personal fundraising pages, and working out some of the details of the social login process. Personal fundraising is a compelling addition to the package. The ability it provides for individuals to set themselves up as fundraisers under an umbrella project puts collective.safesforce.fundraising head and shoulders above many of the other solutions available to non-profits.
Convio Common Ground Customers Take Note
The addition of personal fundraising to collective.salesforce.fundraising comes at a particularly opportune moment. Only days after Jazkarta’s work was done, Blackbaud announced it was discontinuing Convio Common Ground, a fundraising and donor management application built on top of Salesforce.com and used by many small to mid-sized non-profits. A Plone+Salesforce.com powered fundraising site offers a great option for the 400 or so Common Ground clients that have to make a decision and migrate to a new system within the next 19 months. The Plone CMS and all of the plugins mentioned in this post are open source software with no licensing fees, and Salesforce.com is a great deal for small to mid-size non-profits. Through the Power of Us program, the Salesforce Foundation will donate 10 enterprise licenses worth $1600/month to 501(c)3 organizations, and any additional licenses are 80% off. There is also a 50% discount for in-person training and the customer portal is priced at $24/user/year.
Contact Jazkarta if you’d like more information about Plone-Salesforce powered online fundraising solutions.
We’ve been having classic summer weather here in New England – gardens, fields and forests are exploding with lush vegetation. It brings out my inner botanist – something I actually have advanced training in. These days botany is usually a hobby for me, but two years ago my Jazkarta world and my botanist world came together on an amazing project – Go Botany.
In 2009 the New England Wild Flower Society was awarded a major grant from the National Science Foundation to develop an innovative suite of tools to teach botany and plant identification. This multi-year project has many deliverables, but the overarching goal is to develop radical new web-based tools for identifying plants that will improve botanical knowledge and science education for novices and citizen scientists. In other words, get more people hooked on plants!
NEWFS selected Jazkarta to be their technology partner and we worked on the project throughout 2010. We started with a discovery phase, during which we sorted out the various tools, prioritized requirements, defined the technology platform, and created a plan for tackling development. Our plan relied on an agile, iterative approach; the project owner – Elizabeth Farnsworth, NEWFS Senior Research Ecologist and PI on the NSF grant – was always in charge of what we delivered next. This was essential since at the beginning of the project no one had a very clear idea of what the tools should look like. By working in an agile way we were able to easily adjust the plan as we learned more. This quote from the book The Art of Agile Development, by James Shore and Shane Warden (O’Reilly) summarizes the situation nicely.
The beginning of every software project is when you know the least about what will make the software valuable. You might know a lot about its value, but you will always know more after you talk with stakeholders, show them demos, and conduct actual releases. As you continue, you will discover that some of your initial opinions about value were incorrect. No plan is perfect, but if you change your plan to reflect what you’ve learned – if you adapt – you create more value.
Python was a natural choice for implementation language because NEWFS IT staff were already using it. In addition to being mature, powerful, secure, open source, and object oriented, Python has a wide array of libraries available for scientific computing and GIS.
We considered several Python web frameworks that were popular at the time. Plone (the CMS in use at NEWFS) is not well suited to building the sort of custom applications we needed, but some of the other frameworks (like Grok and repoze.bfg) didn’t have enough functionality. We chose Django, which had gained traction in the Python community because of its simplicity, admin features, form technology, and the availability of numerous add-ons. Two of those add-ons were particularly critical to the success of the project: GeoDjango for adding GIS features, and Pinax for adding social networking features.
We had a lengthy in-house debate over the choice of database technology. PostGIS (the spatially enabled version of PostgreSQL, an industrial-strength open source relational database) was the clear winner for its compatibility with the Django ORM, and it was required for GeoDjango. But the data model for the botanical information was essentially a star schema (one table in the middle looking up information in dozens of auxiliary tables for attributes and vocabularies), which is much more efficient to implement in an object database than in a relational database. In the end, we chose to use PostgreSQL/PostGIS for everything, mostly because we knew it would make life easier for future developers and sysadmins on the project.
We needed one more component to implement the Go Botany tools, which were going to rely heavily on various types of search. We chose Solr, an open source search server based on the Lucene Java search library, to provide indexing and search services for all tools. It provides cross database search functionality over a REST interface, including features like custom query structures, common document type extraction, geographic searches, and search facets.
By May we had put together a team and launched into release planning and implementation. The team consisted of Jazkarta developers and NEWFS developers working together on the project, with me as the PM and Elizabeth as the project owner. We like to have joint, collaborative teams with our clients whenever we can, it’s a great way to get their IT staff up to speed quickly. We also spent some time teaching Elizabeth about agile development practices, which helped her do a great job supplying us with the information we needed, when we needed it (user stories, feedback, etc.)
Database and API
The database plus API was the most technologically challenging part of the project. It had to provide a flexible and fast foundation for the entire family of Go Botany tools. It was also interesting to write “user stories” for this part of the project – the “users” were all the Go Botany tools, which would need to get information from the API.
Once the database and API foundation was laid, we began work on the Simple Key – an interactive, online guide to identifying 1,200 of the more common New England plants. This was the most challenging part of the project from a user interface point of view. It needed to be fun and exciting to seduce novices into learning about plants and botany, while at the same time being useful to pros. NEWFS engaged user experience designer Matt Belge (Visionlogic.com) to help them define the tool we would be building. Based on lots of interviews and other research, Matt produced wireframes for successive bits of Simple Key functionality, and we figured out how to implement them.
The result is unlike any previous plant identification tool. The Simple Key presents the user with a series of questions about their plant’s characteristics that are designed to home in on the species identification as efficiently as possible. It does this based on the questions that have already been answered and the features that the user is able to observe. The secret behind this behavior is an information gain algorithm that generates optimal decision trees.
When the user has identified their plant, they can go to a species page with a wealth of information – including maps of its geographic range, diagnostic characteristics, memorable facts, and gorgeous color photographs of the plant and its leaves, flowers, bark, etc. in different seasons. A fitting end to a successful quest.
And It’s Open Source
Because taxpayer dollars through the National Science Foundation supported this project, the software and data on plant characteristics will be open sourced. Details are still being worked out, but the goal is to enable others to contribute and improve the code over the long-term. Stay tuned to the Go Botany site for an announcement of the details. NEWFS staff are also engaged in adapting the database to allow other botanical institutions to customize the data for their particular region.
But This Is 2012!
Yes, all of our Go Botany work was done in 2010, so why am I blogging about this now?
All of this hard work came to fruition in April: the Go Botany website is now live at http://gobotany.newenglandwild.org/!
So if you live in the northeast and you sometimes wonder what the name of a plant is, please try it out! Go Botany works best with Firefox, Safari, and Chrome on iPads, tablet and desktop computers (and the UI is being adapted for phones) so you can take it with you on walks and have an expert assistant for your botanizing. If you have comments, NEWFS would love to get your feedback at firstname.lastname@example.org.
At the sprint following Plone Symposium East, I helped Cris Ewing write documentation for Templer, a Python code generation system that was derived from the older ZopeSkel application. I knew very little about Templer before the sprint, but the experience taught me that it is an incredibly useful tool for Python developers working with Plone, Django, Pyramid, or any other Python framework. Whether you are a freelancer or part of a larger web development organization, Templer can make you more productive.
Templer is a general-purpose system for generating code skeletons from pre-defined templates. End users provide information through an interactive interface, which is used to generate a skeleton of files and folders that make up a Python project. The projects that can be built with Templer today are:
- Python namespace and nested namespace packages
- Buildouts and recipes to extend the zc.buildout system
- Namespace and nested namespace packages for Zope
- Add-on packages for the Plone CMS
There are also commands (based on Python Paste) for adding features to Plone software projects that were created with Templer, such as browser views and Archetypes content types.
The functionality that Templer provides is very handy, but the thing that makes it a Python developer’s secret weapon is the fact that it can be extended and used to build applications focused on particular problems. ZopeSkel, which provides a suite of templates for generating Zope and Plone projects (buildouts, themes, add-ons, etc.), is one example of a Templer application. During the sprint, Ian Anderson began work on templer.django, a new Templer package to provide templates for Django projects. This package could form the basis of DjangoSkel, an application to quick-start Django projects. If you are in the business of creating Plone or Django websites, you can quick-start your own projects with a Templar based application. For example, MyPloneSkel could set up your production and development buildout configs just the way you like them, plus optional packages for Generic Setup policies, a Diazo or standard theme, and custom content types.
Thanks to everyone who has worked on ZopeSkel and Templer over the years to make such a useful suite of tools.