Back in 2007, agile software development was not brand-new anymore. Still, some clients had their reservations. We thought: What if we could prove that Test-Driven Development and Pair Programming were better? There were a couple of good books about TDD and there was “Pair Programming Illuminated” from Laurie Williams, but many studies boiled down to software engineering professors dividing 20 students into a test group and a control group where e.g. the test group was instructed to work in pairs while the participants of the control group had to work individually on a given problem in a given programming language.
We (= me and Christian Federspiel) pondered about a better setting for such a study
- with more people
- with people of different age groups and experience levels (not just students, but also pupils and practitioners)
- with no restrictions to coding languages
- where the participants didn’t think too much about how they worked (TDD or PP) but were focused on the result
Our best idea was to “disguise” the experiment as a public coding contest:
- As for working in pairs: software developers could participate individually or in teams of 2 or 3 people (one notebook + 1 person to the left + 1 person to the right)
- As for TDD: participants could decide to get a.) the entire description of the problem with the final tests upfront or b.) level by level: we broke down the challenge into several levels that had to be solved one after the other; each level with its description and the acceptance tests. In the first case, the participants could work however they liked, in the latter case, they automatically followed a test-driven approach.
We talked with software engineering professors and leading experts in the field of agile software development and testing. We integrated their advice and took their quotes to advertise our first coding contest in July 2007 at the Johannes Kepler University in Linz/Austria. We managed to have more than 50 participants on a Friday afternoon from 2 to 6 p.m., some pupils, many students, and some practitioners.
We collected data from the participants, namely
- their interaction with our contest server (the results submitted, whether they were correct or wrong), and
- their source code (via a script that ran every couple of minutes).
We partnered with a professor to post a diploma thesis to analyze the data and prove our hypothesis regarding TDD and Pair Programming. We looked for a student. But we didn’t find anybody who was interested enough in the topic.
So we gave it another try in 2008, organized the 2nd CCC, looked for a student again, found one, handed over all the collected data, and waited for his insights.
However, after a few months, there still was no progress, so we cancelled the thesis.
Over the years we were less and less interested in the original research question, but we kept organizing coding contests.
Why? We got in contact with so many brilliant software developers who were as passionate about developing software as we are.
That same passion kept us going for 10 years. In March 2017 the 25th CCC took place in 18 locations worldwide. It has been quite a journey already and we still have big ideas for the future.
In retrospect, an academic question made us stumble upon coding contests. We liked them and turned them into an event for thousands of software developers. As a nice side-effect, Catalysts was able to grow from 2 people in 2007 to 200 in 2017.