Batucada – Tools & Vision

I mentioned in the announcement of batucada, that we are dropping Drupal for the next phase of development on the Drumbeat website project. We’ve decided to build Batucada with the Python web framework django. There are a bunch of reasons why I think django is a good choice, but the primary one is that people are already using it for a number of Mozilla web projects, including AMO and SUMO. Having other people in the organization who are familiar with the framework is helpful, of course, and it means we can reuse work done elsewhere within the webdev team (particularly in fairly generic areas like localization, caching, etc). It also means that our IT department is already familiar with deployment issues.

Django is also simply a great framework. The ability to package up reusable pieces of a project as applications means that we should be able to take advantage of a lot of great work done by folks working on similar web applications, and where we build applications ourselves, we’ll be able to release those as separate reusable components. If the work we do can help other people build Django based social web applications that use open web technologies, then we will be further working towards Mozilla’s mission of promoting openness, innovation and opportunity on the web.

In order to stay on track, I’m working on a roadmap and a set of core principles that will guide our development of batucada. The roadmap is a work in progress, but I think we’ve got a good set of principles hashed out:

  1. Social – Batucada will be a social application. People and activities will be at the centre of the experience.
  2. Intelligent – Batucada should be able to suggest connections based on a users activity and profile information.
  3. Data Ownership – Users will control who has access to their personal data and will be able to export their data easily.
  4. Open Web – Batucada will be built using Open Web technologies and will use open standards to integrate with other applications whenever it makes sense to do so.

It goes without saying that Batucada will be Open Source as well of course!

Batucada – Marching towards Drumbeat.org 1.0

I’m excited to announce a new phase of development for the Drumbeat website project. We are about to begin development on a new project, code named batucada, that will eventually power drumbeat.org. With batucada, the future of drumbeat.org will be more social and will have the open web at its core.

To understand how we got here, it’s important to give a brief history of the drumbeat website. The current alpha and beta phases of development were very experimental. We didn’t have a clear idea of what the end result would be. We wanted something that would allow us to experiment and iterate quickly and so building on a CMS was a natural decision. Drupal was a good choice for an open source CMS product. Building the site on Drupal allowed us to get something up and running quickly and gave us a good chance to see how people interacted with the site. We always knew that the technology choices we had made were probably not going to be permanent. Any CMS product is going to introduce tight coupling between code and content changes, which means that developers need access not just to code, but also to the configuration data stored in a database. Providing this in a way that does not compromise users privacy has been a challenge and a barrier for people wanting to contribute code to the project. Additionally, while Drupal’s module system is very handy for getting something up and running quickly, at times it makes adding custom functionality and integrations more difficult than if we were using a standard web framework. The lack of unit testing in Drupal also means that we can’t create a robust set of test cases for each feature we add to the site, and can’t take advantage of things like Continuous Integration, which we’re big fans of. I’m aware that many of these issues are going to be further addressed in Drupal 7, but given the amount we’re looking to change in the next major release, waiting and upgrading presents its own set of challenges.

We now have dedicated technical staff and a much clearer idea of what drumbeat.org should be, so it’s a good time to take the user feedback we’ve collected from the alpha and beta versions of the site, and apply our experimental results to a production ready, 1.0 product. Batucada will represent a radical shift for drumbeat.org. The vision is a much simpler, user centric site that integrates well with other web applications.

Social and the Open Web

Thinking about where we want to take the site next, one thing is really clear to me, drumbeat.org is a great use case for a social web application: connecting people with projects and local events. Furthermore, if we are to promote the open web, we should be using open web technologies wherever they make sense. Batucada will use open web standards to integrate with other web applications whenever possible. Users should not have to do everything on drumbeat.org. Instead, drumbeat.org should be a “launch pad” for projects, and activities performed on other sites should make their way back to drumbeat.org.

The road-map hasn’t been completed yet, but we’ll be examining everything from discovery (WebFinger), real-time activity streaming (Activity Streams, PubSubHubbub), distributed updates (OStatus / Salmon, Portable Contacts) and decentralized identity (OpenID).

The next version of drumbeat.org will have social at its core and will be built on Open Web goodness. Where the current version is pretty document centric, the new version will be entirely people centric.

Global Community

The Drumbeat community is not centered in any one country or region. Drumbeat participants come from all corners of the globe. Events have been held in Brazil, Canada, Germany and community members are currently organizing events in France and the United States. Given the global nature of the movement, we will be building in support for localization and internationalization from day one and will be seeking help from localizers who want to contribute to the project. The code name for the project, batucada, is a nod to Drumbeat community members in Brazil who kicked off the first two Drumbeat events.

Open Source and Community Driven

Another thing that is clear is that drumbeat.org should be an open source project. We are Mozilla, we make open source software. For a variety of reasons, we failed to fully capitalize on this with the current version of the site. The next version will be launched as an open source project and we will be actively encouraging ideas and contributions. We want to build a community of people actively building the software behind the project. If this sounds exciting to you, join us! We’ll need software developers, front end designers, UX engineers, QA testers, localizers and a whole host of other skill-sets.

I’ll be publishing an initial roadmap in the near future and will be looking for feedback.

Using Drumbeat to build the next Drumbeat

Drumbeat.org is all about projects. It makes perfect sense to us to make batucada itself a Drumbeat project. Using existing tools to build new one will allow us to step into the shoes of project maintainers on the site – a perfect way to experience first hand how a future version of the site could better serve the needs of our users. Plus, as a programmer, I like recursion! The batucada project page on Drumbeat is a work in progress, but will be used to facilitate work on the project. Join up on the project page to help build this, and share ideas and feedback.

So there you have it! I’m really excited to kick off this next phase of development. This is the first of a series of posts, in the near future I’ll be outlining some of the technical choices we’ve made (language, framework, etc) as well as a set of values I’m going to pitch for us to follow while we work on this project. Much more to come.

Motivation

Reading posts like this make me feel pretty damn good about decisions I’ve made recently.

“And in this mission, I guess, is the key to what makes Drumbeat special: rather than an initiative, it felt more like there’s a movement building up.”

Joining Mozilla

I’m very happy to announce that I’ve taken a position as a Senior Web Developer for the Mozilla Foundation. I’m sure I’ll be getting into all sorts of things, but job number one is going to be helping out with the Drumbeat community. If you’ve never heard of it, Mozilla Drumbeat is a launch-pad and playground for every day web users working on preserving the openness of the web. It’s kick-ass and extremely important and I’m proud to be a part of it.

Deciding to leave FreshBooks was a tough call to make. I worked with some amazing people, and I’ve been able to build Some Really Cool Things. FreshBooks is truly a cool place to work, and I continue to encourage people to check them out. In the end though, the idea of working for a public benefit organization was just too good to pass up. Mozilla is an organization that I’ve always had a profound amount of respect for and I look forward to helping them promote openness, innovation and opportunity on the web.

I won’t be moving (a number of people have asked me this). I’m going to be working out of the Toronto office, which I’m very glad to say has gone through some serious beautification since being written about by Valleywag. (No seriously, it’s a lot nicer now. There’s even an espresso machine!)

On Implementing WebHooks

I wrote about our experiences implementing WebHooks at FreshBooks yesterday. You can read all the gory details here. Quick summary is: use a Message Queue, send HTTP requests asynchronously, have a retry queue, verify ownership of URIs.