Progress is not the same thing as complexity increase. While I agree that there can be upper bounds on complexity increase in evolution, this doesn’t really have that much relevance to the question of whether evolution has any cumulative direction. The latter is something I consider to be an open question.
As any engineer knows, great increases in capability often occur through the removal of complexity, not its addition. This happens more frequently than you’d think. It’s part of why, for instance, computers have gotten better, faster, and cheaper. If you look at a computer today you’ll see that it has less components than a computer did a decade or so ago. Evolution can of course do the same thing. Fitness can increase through loss of complexity. The simplest and most straightforward example would be the knockout of a detrimental gene, while a more complex example would be the construction of a complex genetic framework or “scaffolding” followed by the emergence of a simpler system within that framework followed by the dissolution of the framework’s no-longer-necessary parts.
So while Williams’ work may show the existence of upper bounds on complexity, it does not disprove the idea that there might be some kind of directionality that could be called progress in evolution. If it exists, it seems to be something that we haven’t learned how to quantify or explain quite yet.
Allow me to clarify douglas a bit if I can. Correct me if I’m wrong.
What douglas is (I think) invoking here is a phenomenon called the evolution of evolvability. Essentially the idea is that evolution is not quite as blind or random as pure classical Darwinism would have it, but that it evolves. Evolution evolves, recursively. Lineages that do a better job exploring fitness landscape space do a better job surviving, and so therefore their genes tend to do a better job surviving as well. Evolution therefore favors the emergence of genetic systems that aid evolution.
Competent cells are an example of this. Competence (the ability to take up naked DNA) is likely an evolvability adaptation. Having it turned on in all cells would be disasterous since the entire population would be virus fodder. But having genes in there that cause this phenomenon to happen and having them activate occasionally is good for all genes involved since under stress it greatly increases the likelihood of major discontinuities that might propel the lineage out of a valley in fitness landscape space.
If you want a really far out and extreme take on this, read this:
http://users.tpg.com.au/users/jes999/gencog.htm
Stewart crosses over into evolutionary romanticism on occasion, so I don’t buy everything he says. But he does have a grasp of just how big an idea the evolution of evolvability is. I admire visionaries with the courage to write like this, even if some of what they write strays a little into la-la-land. That the price you pay for getting excited about the new. We have far too few of such people these days.
Evolutionary theory with the evolution of evolvability is to classical Darwinism what Einsteinian and Quantum mechanics are to classical Newtonian physics. All the responders are right in that this stuff is a part of modern evolutionary theory, but it’s not really a part of “Darwinism.” Darwin didn’t predict this. Calling modern evolutionary theory “Darwinism” is like calling physics “Newtonism.” Darwin was Newton, but evolutionary theory did not end with him.
Now for an annoying Google suggestion: go to scholar.google.com or arXiv and search for “evolution of evolvability” as a phrase.
http://scholar.google.com/scholar?as_q=&num=10&btnG=Search+Scholar&as_epq=evolution+of+evolvability&as_oq=&as_eq=&as_occt=any&as_sauthors=&as_publication=&as_ylo=&as_yhi=&as_allsubj=all&hl=en&lr=