Agile Development with Plone, Revisited

The announcement of the 2012 Plone Symposium East (thank you Weblion! always a great gathering) is reminding me of the talk I gave there last year on Agile Development with Plone. Here at Jazkarta we’re engaged in another large project, similar in many ways to the one that inspired that talk. Because our project management tools and techniques have evolved a bit since last year, I figured it was time for an update.

The Projects

Last year at this time, we were just getting started on a major website redesign for the University of Minnesota Press. There’s lots of information about this project in my Plone Conference presentation How To Get a Fabulous Website on a Modest Budget Using Plone. Briefly, it involved a major data conversion from a FileMaker database into Plone, a small number of complex custom content types, extensive design work, and a featured “explore” section powered by eea.facetednavigation.

Our new project is a major website redesign for Dumbarton Oaks Research Library and Collection, a research institute of Harvard University and the home of a little known but outstanding museum and garden in Washington, D.C. We began with a discovery phase last summer, and have been working on implementation since November.

What’s the Same

Both of these are fixed budget projects with a large number of desired features, and at the outset many of these were inadequately defined for development purposes. In both cases there are multiple stakeholders with differing priorities, but there is an intelligent, proactive project owner who is trusted and empowered to make decisions. The Jazkarta team is the same – Carlos de la Guardia in Mexico, Alec Mitchell in California, and me in Massachusetts – and none of us ever work full time on any of our projects.

This all adds up to the same basic project management challenge: how to adapt agile techniques to manage a fixed price, flexible scope project with a distributed, part time team.

What’s Different

Happily, the Dumbarton Oaks project does not involve a complicated data import from a FileMaker database. For UMP this required a collective.transmogrifier script, many many many trial and error imports, and over 60 hours of Jazkarta development time. Dumbarton Oaks currently has a static HTML website, which is much simpler to import. Funnelweb made this easy – only a few trial imports and some content massaging were required. We’re not a quarter through the project but all the old content is already on staging. Total development effort: 10 hours.

Because the content import was relatively easy and the project mostly requires custom features and content types that are independent of each other, we have been able to divide up the work into several releases. This gives us a series of smaller planning horizons and allows us to be more agile (more on this later). Each release can potentially be deployed to production – and even if we decide not to, new content is developed on staging which gives the new features a thorough vetting. No matter how much you thought you tested, there are always problems that don’t get discovered until the software sees serious use. Having a series of small production deployments avoids piling all the bug fixes and theme tweaks into a single mega deployment phase at the end of the project.

Our Evolving Agile Development Practices

Our basic development process remains the same:

1. Discovery
2. Define schedule and resources (how the budget will get spent)
3. User story development
4. Story estimation (we use planning poker)
5. A series of development iterations alternated with evaluation periods
6. Production deployment

During discovery the project owner and I distributed the feature requirements into a series of releases, based on a combination of priorities and keeping all the stakeholders happy. (Everyone got something in almost every release.) This high level release plan allows us to do our story development and estimation one release at a time, which saves time and money – especially considering that all the developers participate in planning poker sessions. Since we know the customer wants more than their fixed budget will get them, and we also know they will change their mind about what they want as the project progresses, it makes sense to postpone the effort of writing and estimating stories we may never get to.

As in previous projects, I took the first cut at turning the project requirements into stories, then discussed and revised the stories with the project owner. Because I have intimate knowledge of Plone, I can translate requirements into stories that make sense to develop. For example, we don’t need to spend time defining stories for CMS functionality or for things that Plone has great add-ons for (like staff directories and bibliographies). Some things may turn out to need tweaking, but it is better to give the customer stock Plone and add-on features to try out first. They may decide to live with the stock features and put their development dollars into new features instead of making changes.

Our communication tools remain the same: Skype, IRC, chat, Google Calendar, and Google Documents. This combination is the secret sauce that allows a distributed team to work as if they were in the same room. We did switch from ClueMapper (based on svn and trac) to github for source code repository and ticket tracking. We miss a few management features, but the developers are much happier working in a dvcs.

Our development iterations remain the same – one week each, followed by a week of evaluation during which the customer has time to test and reflect on what they want to do next. We kick off each iteration with a planning meeting, which surfaces any questions about the stories we’re tackling and allows us to do task breakdown and assignment while we are all in the virtual room of a Skype conference call. During the iteration we have a short standup meeting every day to keep things on track – everyone says what they did yesterday, what they’ll do today, and raises any blocking issues. My notes from the standups are written in a Google spreadsheet which is shared with everyone. This gives the project owner access to detailed information about how things are going, if they miss a standup meeting.

The biggest change we’ve made for this project is to adopt ScrumDo for story development and iteration planning. ScrumDo is an open source agile project management tool that’s written in Python. We are using the hosted, SaaS version that’s at scrumdo.com – accounts are free for up to 10 users and 10 projects. ScrumDo has a friendly drag and drop UI that makes it easy for for project owners to plan iterations, edit stories, add comments, and see results. Unlike Pivotal Tracker, it allows flexible iteration scheduling and story and task assignments. Google Docs provide better simultaneous editing for multiple users and more flexible formatting, but ScrumDo makes it really easy to add and change tasks, assign them, and mark them as complete. If it managed estimated and actual hours I’d totally love it.

ScrumDo added a Planning Poker tool last March, and this has been a real time saver. It’s a web page where the scrum master selects a story to size and everyone puts down a story point card. Once you play your card you can see the others, and after discussion and reaching consensus, the scrum master assigns the winning value to the story. What set of cards to use is one of the ScrumDo project settings (we use modified Fibonacci).

ScrumDo Planning Poker interface

The Value of Agile Project Management

The planning and standup meetings required by an agile approach do take time. For the UMP project, the meetings and project management activities combined used almost 40% of the total budget. But what is the alternative? Written requirements “thrown over the wall” to developers are a recipe for failure. The customers and the developers have to talk to each other to understand the project and its development trade offs. I’ve never had a customer complain that I’m using too much project management time when I’m running a project in this way. The value of open and frequent communications and iterative development cycles is very clear to them. They get full ownership of the project, choose exactly what it will deliver at every step along the way, and in the end they are delighted to get the system they envisioned.