Yes indeed, Catalysts has moved, but that doesn’t mean we are gone, we just moved to our new website
together with our partners from Crisp Research to do the same things we used to do in an improved way. We still do great software and we are still great people, more or less only the color has changed except from our domain 😉

Scientific Workshop on Large Scale Experiments on Agile Programming Techniques (LSExp)

Active at Research

Catalysts and the Vienna University of Technology will organize together a workshop on this year’s XP Conference on Agile Software Development in Malmö/Sweden.

During one day between 21st and 25th of May 2012 (the exact day is not assigned yet) we will present the past Catalysts Coding Contests (CCC) and their results to experts of the agile community and discuss how they can provide valuable input data to scientist and practitioners.

Call for paper – PDF Version

Themes & Goals

For agile researches it is often difficult to find statistically significant evidence for their hypotheses like “test-driven development leads to shorter development times and to fewer errors”. For their theses they need experiments with at least 40-50 participants. By organizing a programming contest and enforcing specific rules, researches can easily motivate many programmers to participate in the experiment and use the results for their work.

Large scale coding contests with hundreds of participants provide an opportunity to observe the participants – how they approach on solving the problem as well as their solution (i.e. source code).

In 2007, Catalysts, an Austrian company and corporate member of the AgileAlliance, organized a coding contest for the first time with the goal to investigate whether TDD is beneficial and whether teams win over individuals. 180 persons participated at the most recent contest in December 2011. The challenge was specified in six consecutive levels, where only 12 participants solved all levels within three hours.

These contests are now held three times a year – people can participate “on site” or remotely via internet. Information about the progress of the participants has been collected as well as the source codes (level by level) and some profile data of the participants. This information is already publicly available via an Open Data Server.

This workshop shall bring researchers and practitioners together to:

1. exchange experience of coding contests

2. brainstorm possible experiments

3. brainstorm how to perform those experiments in the context of coding contests

The main outcome of this workshop shall be to increase the amount of usable data for agile researchers, so that they can easier make empirical studies. Catalysts organizes three large coding contests per year and they are willing to adapt the context of future contests on the outcome of this workshop.

Other organizations could join in as well and set up appropriate contexts for such experiments. So there are several opportunities to run such large scale experiments.

Workshop’s Relevance

Coding contests with hundreds of participants provide a lot of data points:

  • participants with various backgrounds
  • different experience levels (number of years)
  • pupils, students, practitioners
  • individuals and/or teams
  • various programming languages
  • history of source code level by level
  • etc.

Such data points allow comparative analyses as well as analysis of the evolution of programs.

Admittedly, the data is not collected in a completely clean laboratory environment. However, the context of a coding contest allows to set up the necessary rules to make the experiment “as controlled as necessary” – much more controlled than the unrestricted open field.

Agile researchers can provide valuable contributions on how to evolve coding contests in the future. They can ask and answer concrete questions about the current state of the art and discuss future publications in this topic. Further, they can design contests in other environments and contexts in regard to receive usable data.

Call for papers

We encourage agile researchers to present a case study or an experience report of an experiment, how it was set up, which rules were tried to enforce, what data was collected, how the data was analyzed, etc. We will select three papers to be presented (15-20 minutes) during the workshop, more papers can be shared with the participants in advance to foster a lively and diverse discussion.

The papers have to be in English using the Springer LNBIP, or similar style. Please submit only PDF files, not longer than 10 pages, on or send it to

The deadlines are:

Submission due April 7, 2012

Notification of Acceptance April 21, 2012

Final manuscripts due May 11, 2012

The reviewers are:

  • Stefan Biffl (Vienna University of Technology)
  • Barbara Weber (University of Innsbruck)
  • Ralph Miarka (Agile & Systemic Coach)
  • Paul Grünbacher (Johannes Kepler University Linz)
  • Dietmar Winkler (Vienna University of Technology)
  • Christoph Steindl (Catalysts)
  • Martin Kitzler (Vienna University of Technology, Catalysts)

The Agenda

  • Presentation of the coding contest background (slides with pictures from the contest) – 15 min
  • Presentation of the collected data – 15 min
  • Slot 1: Case study or experience report of an experiment – 20 min
  • Slot 2: Case study or experience report of an experiment – 20 min
  • Slot 3: Case study or experience report of an experiment – 20 min
  • Brainstorming ideas for experiments – 20 min
  • Work out the ideas in groups of 3-5 people – 30 min
  • Present the worked out ideas – 30 min
  • Prioritizing ideas – 10 min
  • Brainstorming how to set up the experiments – 20 min
  • Work out the rules in groups of 3-5 people – 30 min
  • Present the rules – 30 min
  • Develop an action plan – 60 min
  • Quick retrospective (how to improve the workshop in the future) – 30 min
  • Breaks & Buffer – 60 min

Short bio of each organizer

Dipl.-Ing. Dietmar Winkler

Institute of Software Technology & Interactive Systems

Vienna University of Technology


Dietmar Winkler is researcher and lecturer at the Vienna University of Technology. Since 2010 he is working as project assistant in the Christian Doppler Laboratory “Software Engineering Integration for Flexible Automation Systems”. In 2007 he worked as a guest researcher at Czech Technical University, Department of Cybernetics in Prague (CZ). In 2008 he received a scholarship (by Siemens Austria and the Faculty of Informatics, TU Vienna) for a 4 months 2 Large Scale Experiments on Agile Programming Techniques research stay at the Fraunhofer Institute of Experimental Software Engineering in Kaiserslautern (DE).

Martin Kitzler, BSc

Vienna University of Technology and Catalysts GmbH

Web: and

Martin Kitzler studies the master’s program “Software Engineering and Internet Computing” at the Vienna University of Technology and currently writes his master thesis on various agile development hypothesizes by using coding contest data.

In 2011 he joined Catalysts and works as a SW developer and tester.

Dr. Christoph Steindl

Catalysts GmbH


Lecturer at the Vienna University of Technology, at the Johannes Kepler University Linz and at the University of Applied Sciences in Hagenberg. Christoph Steindl has studied Computer Science at the University Linz, worked there as University Assistant with Professor Mössenböck, then joined IBM and became an IT Architect before he founded Catalysts. He has been an agile practitioner for many years, has coached many teams and set up an agile company. He has been organizing coding contests for more than 5 years.

He is a member of the ACM and OCG (Austrian Computer Society).

Scientific Workshop on Large Scale Experiments on Agile Programming Techniques (LSExp)
Previous Post
Workshop 2011 – The Logical Thinking Process
Next Post
Catalysts ist wieder im CCC Fieber

Related Posts

No results found

Leave a Reply

Your email address will not be published. Required fields are marked *

Fill out this field
Fill out this field
Please enter a valid email address.
You need to agree with the terms to proceed