After my comments a few days ago, a few people asked me how to do data driven design in practice. I’m working on a specific post about process, but here’s a teaser: a rant against whiteboards.

People throw around the word “iterate” a lot. What can it actually mean, apart from the normal agile definition?

Whiteboard driven iteration

Whiteboard driven iteration involves thinking of ideas and drawing stuff on a whiteboard or in power point, going home, sleeping on it, thinking of more ideas, and then drawing more stuff on the whiteboard.

Once the idea is perfect, you give it to developers and they implement it. By the time that they’re done, you’ve drawn more stuff on the whiteboard, so there are lots of necessary changes that the developers must make before release, plus it must have a good design, which must conform to your vision.

Once there’s a design and implementation, you spend several days/weeks making sure that the implementation and design match what you thought of. Since the design on the whiteboard was a design on a whiteboard, everybody had a different idea of what it would look like and how it would behave, so this takes a while. Plus, your vision is always changing, so the design and implementation must constantly change.

Finally, you release the feature or website. Since the feature is released, you’ve accomplished something and can feel good.

In whiteboard driven iteration, you iterate with your designers and developers over ideas on the whiteboard and ideas in your head until the feature is done. The point of release is to achieve success.

1
2
3
4
5
6
7

while(feature != complete)
  developer.implement!
  whiteboard.draw
  (design + idea).change!
  time++
end

After several months of whiteboard driven iteration, your team gets better at releasing code. Your developers switch to agile development, because they got tired of changing specifications. Your developers get everyone to switch to Pivotal Tracker so that it’s easier to make it clear that spec changes mean extra dev time. You feel good because features get released faster.

The only problem is that your conversion rate has been basically flat since launch. You’re running out of money, and your burn rate has doubled since you had to hire more designers and developers in order to turn more stuff on the whiteboard into code. Time to call some VCs!

Customer data driven iteration (evolution?)

Customer driven iteration takes customer validation rather than released features as its core unit of progress. It assumes that you have not accomplished anything and therefore cannot feel good until your metrics tell you that your market will use and pay for your stuff.

It assumes that both the problem and the solution are unknown, no matter what your whiteboard says. Customers don’t look at your whiteboard any more than they look at your business plan (i.e. Never).

Since you are probably wrong, you collect data that will at least let you learn as much as possible from each minute of developer time, designer time, and customer time.

In customer data driven iteration you iterate over a perceived customer need until your metrics show you that your solution is effective, or that your customers do not have the problem you thought they did. The point of a release is to learn.

1
2
3
4
5
6
7

while(customer_metric != goal)
  learning << try(solution)
  break unless goal.makes_sense_with?(learning)
  time++
end

After several months of customer driven iteration, your company gets better at delighting customers and making money. Since you always assume that your solution may not work, your releases become much smaller and you iterate much faster, because any single idea is never worth more than a few days to try.

Furthermore, since your developers and designers are now focused on improving customer metrics rather than meeting your whiteboard requirements, they are empowered to be masters of their own time.

False success with Whiteboard iteration

Teams using whiteboard iteration sometimes see a small amount of success. A feature that started on the whiteboard, took a month to implement and design, and was finally released to customers appears to actually work. Great!

But, what specifically worked? Was it this thing up on the top right of the whiteboard that pushed customers to convert? Or was it the thing over there? Unfortunately, since you were so sure that the whiteboard design would work, you didn’t measure any of that stuff as you were going, and you did it all at once so there’s no way to know now without extra work.

Could you have implemented 10% of the whiteboard feature and seen the same success? That would have been great, since you would have saved 90% of your team’s time/money. But nobody knows. Nobody learns anything specific about what drives your customers.

I might be wrong

So, guess which type of iteration I prefer.

But, as with everything else, since I don’t know for sure that whiteboard driven iteration cannot lead to success, I have to assume that I’m wrong.

Has anybody had $uccess with features implemented in that way? If so, why?

c.f. Built to learn