(This is just a string of poorly organized random thoughts about an analogy I've had in mind for a few years; not sure if it is very good or anything, but I find it interesting. It was brought to mind by recent spam filter breakdowns. Warning, very long, something approaching 3000 words.)
Like an awful lot of people, I’ve been fascinated by the Internet for a long while, though in my case, it is from perhaps a slightly different perspective. My best academic background was actually pretty heavy on evolutionary theory, and over the last 20 years I have kept up to a degree, as an enthusiastic amateur I suppose. I am one of that very small number of people that have actually read all of and probably understood a great deal of Stephen J. Gould’s The Structure of Evolutionary Theory
(yes I'm a fanboy), a tome of approximately the same thickness, thoroughness, and page turning excitement as Das Kapital
(which I’ll have you know I have also actually read most of, to mixed feelings).(Mind you, whenever I think I understand something, I am reminded of this bit from A Fish Called Wanda:Wanda: I’ve known sheep that could outwit you. I've worn dresses with higher IQs. But you think you're an intellectual, don't you, ape?Otto: Apes don't read philosophy!Wanda: Yes, they do, Otto. They just don't understand it.)
I remember way back when, when Big Bro
was cutting his chops on internet search technology and starting to think about markup languages seriously, he once asked me what I thought the internet should look like. My immediate visual was biological, I suggested that the analogy to surfing was that of a python snaking along the branches in a gigantic complex jungle. He didn’t go with it, I believe he was thinking of Hong Kong at the time, big freewheeling marketplace with an insane amount of stuff packed in, and all that and surely he was right, then he ultimately he went with the sterility of Antarctica
cuz nobody ownded it. But me, I still kind of think of tech and internet things in a biological/ evolutionary sort of way.
It has become such an interesting place, these days, you have bots and spam, viruses and worms, spiders and scrapers, splogs and spamindexing, and even pump & dump spam (which is surprisingly sophisticated these days) and, mon dieu, right out of Monty Python, spammers spamming us offering spam services
And on the other side of things, you’ve got the running battle against all that vermin and parasites: sophisticated filters, user hygiene through rueful experience, what amount to reverse Turing tests (the machine tests the human, really), and so on. To say nothing of the incredible sophistication of Google algorithms, heck, search alone is a whole little ecosystem, complete with its parasites and symbiots and whatever the heck meta search engines
could be likened to (and boy was I ever wrong about that, I figured metasearch was going to be the way to go, and it’s still around, but I suppose ultimately it is too parasitical really).
And look at the creation of new species and phyla: we’ve got blogs in all their many flavours and indexes and webmail and creaky old bulletin boards and RSS/Atom and IM and streaming video and P2P and BitTorrent and wikis and newspapers and MMOG
s in all their MMORPG, MMORTS, MMOFPS, etc. varieties. To say nothing of weird massively successful phenoms like Wikipedia, Project Guttenberg, eBay, CraigsList, and so on.
I look at all that and all the rest of it, and speed and unpredictability at which it is all moving, well, I think in terms of ecology and evolution.
Like with evolution, in retrospect each development seems so obvious. But if you went back to the mid 90s, when people were at least aware that there was an internet and that it might be able to do new things, there was absolutely nobody that could have predicted what would happen. Back then, in fact, with the dotcom boom and bust, you had something pretty analogous to the Precambrian explosion and subsequent mass extinction, all these weird hopeful forms that mostly disappeared, replaced by the descendants of the most unremarkable candidates.
Which is why I am such a deep fan of Home of the Underdogs (link troubles, I do wish some white knight would show up and save Sarinee Achavanuntakul)
; back before anybody “knew” what computer games were supposed to be like, there was a much greater variety on display, and it is a pity that some of them have been lost. It kind of reminds me of the Burgess Shale and all its very weird creatures. Nowadays games are bigger, flashier, and technically more sophisticated, but they are all also much much more alike in look and content and mechanics and thinking, which is a pity.
The pattern is very common in evolution; by pure dumb luck some species stumbles into a new niche, evolves at hyperspeed until it becomes so ubiquitous as to be unremarkable, and thus stops changing much except through what would amount to genetic drift. Look at something like text processing—it is a classic example of Gould’s punctuated equilibrium (we seem to be at equilibrium right now, but I suspect it may get punctured pretty soon by what are currently being called webservices—in any case, there is certainly a nibbling around the edges; I am typing this in my browser, for instance). In fact the effect of punctuated equilibrium actually looks a lot like the paradigmatic 80/20 tipping point
I think this is because evolution and internet development are being driven by a very analogous underpinning: DNA works because it is very simple and stupid, and because it is very good at storing, copying, and transmitting staggering amounts of information. Sounds familiar? Mind you, there are limits to analogy, in that code is pretty Lamarckian, and the driving force of genetic evolution is data corruption (mutation); but then perhaps the DNA analogy is not so much with code, as the algorithms used by programmers, and I suspect that from time to time a creatively misunderstood algorithm might have gone a long way.
And my take on what have been successful on the internet is the same as in the biological world: simple, small, and stupid, beats specialized, sizeable, and sophisticated almost every time—as the saying famously has it, God surely must love beetles.
Here are two examples of how hard this is to predict: way back when, everyone thought that the big money was to be made as a portal, essentially corralling your customers into something a lot like tv, and AOL was the way to go (sadly, that kind of thinking is still with us, even if AOL won't be much longer). Other than that, money might be had through reduced infrastructure and efficiencies, so maybe a retailer might be able to use it a little.
Well, no one foresaw the Long Tail
, which goes well beyond Amazon—eBay is the ultimate Long Tail. But my favourite example is this: one of the most profitable services turned out to be dating sites (the industry may well beat porn for all I know—I surely don’t trust the figures that bandied about on porn sales, they make no sense at all and I smell propaganda all over it). So now the services are competing away with ever more specialized niches and ever more sophisticated matching systems and add-on services (and torrents of near-spam if they happen to get ahold of your email address—hate someone? Sign them up as a lonely heart and watch their in-box explode with NSFW embarrassing missives). And guess what is winning in the matchmaking department? Good old stupid simple CraigsList.
I sometimes think that Open Source is almost the equivalent of the development of sexual reproduction in evolutionary history. Sure, the proprietary type asexual reproduction will always be with us, but in terms of adaptability, transmission of useful traits, and so on, well, sex is hard to beat. (Mind you, I’m trying to figure out what evolutionary analogy there would be for the wretched US Patent Office; so far, the best I can come up with is non-scientific diabolic intervention, sort of the opposite of the Intelligent Designer I guess. Or perhaps that the PO is anti-sex?)
Anyway, the internet provides a clear and concrete example, as if one were needed, of why the Intelligent Design guys (they do tend to be guys) are complete idiots at best. Their basic claim is that there is such a thing as irreducible complexity, usually in interdependent parts. Well, the internet was not intelligently designed, it was stupidly
designed. Yet it has become extraordinarily complex and yes, even beautiful sometimes. And one would be tempted to look at certain phenomena as apparently irreducibly interdependent.
One of the key ideas in evolution is “functional shift,” where say a small dinosaur’s thermal cooling/heating vane becomes useful in controlling stability in tree-leaping (a la today's flying squirrels), to the point that it starts looking more and more like a wing, and yes, you eventually get a bird. The computer’s function was originally to handle large calculations, literally to compute. No one ever thought it was there to process text or play music. The web browser was to share files; no one imagined it could be used to buy things or display multi-sensory art (turn on your sound for that link, and it's unpredictably and mildly interactive and best enjoyed in a calm atmosphere)
Stephen J. Gould championed several important ideas in evolution, and many other things as well; the Mismeasure of Man is all you need to refer to when racist nonsense like The Bell Curve comes up; his ruminations on the separate Magesteria of Science and Religion were gentlemanly, if somewhat unrealistic. Incidentally, his baseball statistical work was fascinating in itself, but was primarily used to illustrate his more important points:Punctuated equilibrium
, which while still not totally accepted, and argued about in its details, is nevertheless pretty much part of the cannon now. I do think that this has been pretty clearly the pattern of IT development over the long view (for example, a wordprocessor now is almost identical to the first GUI/WYSIWYG ones in use before the word blog had even been coined, just a good deal more stable).
The metaphoric idea of spandrels (very briefly, characteristics that are or were originally side effects and not true adaptations to the environment, which can then be exploited for a use of their own). In hacker terms these would be “misbugs” perhaps, or concrete examples like animated ascii art or some of the weirder developments in HTML/Web page design.
Seriously thinking about what levels evolution/competition operates on (DNA/ organism/ gene-pool/ species/ biosphere/ phyla, etc.) These days, the confusion between OS, browser, platform, app, etc. might be thought of similarly. In any case, you can look at code & apps, interactions, user use, types of sites, search technology, commerce, server architecture, etc., each of which would be a different ‘level’ I suppose.
The great debate over whether there is an inherent directionality towards complexity in evolution: a complex and subtle debate led by Gould vs E O Wilson; Gould thought not, where Wilson does, and I rather think Gould pretty much put it away with the “drunkard’s walk” argument, but Wilson is still alive and publishing; in any case the debate like the Gould/Dawkins debate got greatly distorted in the press and by various jerks with axes to grind). The struggle between the two approaches is certainly alive and well in the tech world I gather.
And by extension, that evolution is contingent, not teleological. This was laid out in Wonderful Life: The Burgess Shale and the Nature of History (I remember big bro raised an eyebrow at the apparent grandiosity of the title—but biologists tend to work from the bottom up, the particular to the universal, and in fact the book really is making a fundamental claim about the nature of history). Essentially, what he was after was that in spite of the human propensity to think of things in terms of Aristotle’s Final Cause, history demonstrates that it just ain’t so. Evolution is not directed, and not predictable, it does not have a direction, goal, or purpose; as he put it, rewind the tape and play it again, and you get something quite different. (He was very big on the graphical display of evolutionary information, not being overly fond of the traditional tree-trunk for instance.) It’s a hard concept to accept, actually, but as he and Darwin put it (in the final words of The Origin of the Species “there is a grandeur to this view of nature.”)
Warren Buffet has the same idea: he once said something to the effect that we are driving into the future only being able to look in the rear-view mirror. Someday some
Linnaeus will come along and inadvertently sort it all out for us I would hope.
Which is why I love internet prognosticators (I love “futurists” for much the same reason): they are essentially attempting the impossible. In fact, it is worse than that: you can pretty much bet that any prediction made at more than the purely technical level (like Moore’s Law) is going to be wrong (and Moore is looking a little shaky of late too). Now, evolution does proceed by fits and starts, in punctuated equilibrium, (“evolution by jerks” as it has been described by non-fans of Gould) and it may be that we are ending a period of massive explosion in diversity and adaptation. But one of the interesting things in evolution is how ecological niches are created again and again, spandrels themselves at one level or another, and it is amazing how something always fills them in and creates new ones. Macro change be slowing down, but I suspect that there is a lot of filling in to do, and internet ecology is going to grow a lot more complex in matters of detail, at the same time as it grows more uniform at a macro level.
But the only real bet I’d make is this: simple, small, and stupid, beats specialized, sizeable and sophisticated almost every time. God loves beetles, ascii, bacteria, and blogs; she isn’t so fond of dinosaurs, bloatware, condors, nor, I think at the end of the day, Microsoft and its products.