The Myth of Intelligent Design

pieterhpieterh wrote on 10 May 2012 21:10

chart2.png

The dominant theory of design is that you take smart, creative people and money, and produce amazing products. The smarter the people, the better the results. I'm going to claim that theory is bogus and based on a quasi-religious model of the "inventor" and "invention" as a function of individual minds. As an alternative I'll present the Theory of Heuristic Innovation, which states roughly that we do not invent solutions, we discover them, and that discovery process can be highly automated.

Why Discuss This?

Presenting ZeroMQ at Mix-IT in Lyon a couple of weeks ago, I was asked several times for the "roadmap". My answer was, roadmaps are bad for several reasons. First, they make promises we can rarely keep, which causes problems for our users. Second, they claim territory and make it harder for others to participate. Lastly, they preempt the thinking process of the community. The audience didn't really like my answer. So un-French. Software engineers don't like the notion that powerful, effective solutions can come into existence without an intelligent designer actively thinking things through. And yet no-one in that room would question evolution. A strange irony, and one I want to explore further.

The Theory of Individual Creativity

In the dominant theory, brilliant individuals reflect on large problem sets and then carefully and precisely create a solution. Sometimes they will have "eureka" moments where they "get" brilliantly simple answers to whole large problem sets. The inventor, and the process of invention are rare, precious, and can command a monopoly. History is full of such individuals.

Looking closer, however, the facts don't match. History doesn't show lone inventors. It shows lucky people who steal or claim ownership of ideas that are being worked on by many. It shows brilliant people striking lucky once, and then spending decades on fruitless and pointless quests. The best known large-scale inventors like Thomas Edison were in fact just very good at systematic broad research done by large teams. It's like claiming that Steve Jobs invented every device made by Apple. It is a nice myth, good for marketing, but utterly incorrect.

Recent document history shows this very well. The Internet is surely one of the most innovative and fast-moving areas of technology. It has no inventor. Instead it has a large economy of people who have carefully and progressively solved a long series of immediate problems, documented their answers, and made those available to all. The innovative nature of the Internet comes not from a collection of Einsteins. It comes from RFCs anyone can use and improve. It comes from open source software anyone can use and improve. It comes from sharing, scale of community, and the continuous accretion of good solutions and disposal of bad ones.

The Theory of Heuristic Innovation

Here is the Theory of Heuristic Innovation (v0.2):

  1. There is an infinite problem/solution terrain.
  2. This terrain changes over time according to external conditions.
  3. We can only accurately perceive problems we are close to.
  4. We can rank the cost/benefit economics of problems using a market for solutions.
  5. There is one temporarily optimal solution to any solvable problem.
  6. We can approach this optimal solution heuristically, and mechanically.
  7. Our intelligence can make this process faster and more accurate but does not replace it.

It's an approximation. Feel free to send me patches.

What does This Mean?

First, and most broadly, it means that we do not invent solutions, but we discover them. Given a well-defined problem, there is exactly one optimal solution, and our work as software engineers is to climb the solution terrain until we find it. There may be local peaks. We may have to back-track. We make mistakes. But if we have a good process for climbing that terrain and we apply that process mechanically, we will eventually find the optimum.

Second, this means that individual talent is helpful only in the speed with which we move. Intelligence moves us faster but does not make any fundamental difference in the outcome. The process is much more significant than talent. Which is why so many talented engineers produce trashware, and why teams working in the same space systematically seem to discover the same solutions.

Thirdly, it means that old solutions to old problems can become obsolete as the terrain shifts. Technological and environmental changes are the main factors here.

Finally, it means that our primary goal in producing software should be to define the best possible process, and then learn to apply it mechanically and systematically.

As a correlation, we can bootstrap this by applying any given process to the process itself.

Elements of a Mechanical Process

We need several elements:

  • A starting point in the problem/solution terrain.
  • A way to collect and measure problems in the near space around us.
  • A way to experiment with solutions cheaply and rapidly.
  • A way to measure these experiments against our problems.
  • A way to discard the failures and propagate the successes.
  • A way to isolate layers so we can review and re-solve old problems.

Let's apply the mechanical process to software design and see what we get. This will sound repetitive to readers of my blog.

  • Starting point: identify a single interesting problem and make a minimal, plausible prototype. Time taken: 3-5 days maximum.
  • Collecting and measuring problems: publish your prototype on github.com and ask people to play with it. Use it yourself and collect problems with it. Write these down as issues.
  • Experimenting with solutions: pick out the most critical ones and figure out "good enough" solutions to these. Make patches and merge them.
  • Measuring solutions against problems: try the results. You should always have executable, usable product that can be shipped if needed (even as a prototype).
  • Discarding failures and propagating successes: if you don't like the solutions, revert the patches. If you like them, close the issue. Use the LGPL to ensure your successes get the widest trade with other projects.
  • Isolating layers: use ZeroMQ to isolate layers so that you can throw away old pieces and start again without bringing your whole product into question.

Conclusions

Talent lets you move faster, and process gets you to the right place. The ideal process seems to be one based on highly incremental small steps, each step solving one identifiable problem with a "good enough" solution. Over time, the collection of small accurate solutions to real problems, and removal of mistakes, will give you a highly valuable product that people will love.

Comments

Add a New Comment
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License