Note: I posted this to the extropians mailing list April 2000, to this blog six years later. Recordings of the symposium are online at TechNetCast.
…

Ken Clements wrote:
> Hofstadter admitted that he had stacked the panel by not asking anyone
> from the anti-technology movement (Bill made up that whole side).

Hofstadter didn’t invite anyone who believes that intelligence requires a biological brain, which is quite different from believing that technology is bad. Joy seems to believe some technology is bad, but he doesn’t seem to fall into the “intelligence requires biology” camp. (Offtopic aside: Searle sounds like a very reasonable classical liberal in a recent interview with Reason magazine. Just more proof, not that any was needed, that even reasonable people often take dumb arguments seriously.)

There were really two debates going on (though the atmosphere wasn’t contentious at all): rapid vs. slow development/evolution of human-level or greater machine “intelligence” (in quotes because what this means is nebulous and wasn’t discussed) (primarily Kurzweil and Moravec vs. Holland and Koza respectively) and “we must relinquish dangerous technology now or face catastrophy” (Joy, with support from Koza, vs. Merkle and Moravec, with support from Kurzweil).

Kurzweil and Moravec’s initial talks were quite boring, though their contributions to the discussion and Q&A periods were the most insightful of the group. After droning about exponential and double exponential increases in computational speed, Kurzweil did sneak in one gem: he indicated that Moore’s law, or something like it, also applies to software, of course very much contrary to most people’s intuitions. I was very eager to hear a rationale for this claim. Unfortunately when Holland asked about it at one point, Kurzweil only mentioned better development tools.

Joy seemed quite proud (in a very serious way) that the media is paying attention to him and that he is well read (or at least can scour books for emotional quotes supporting his argument, or at least pay someone to do so). His argument basically boiled down to this: supervirulent pathogens will be easily engineered and/or produced in crazy and/or sick people’s basements, and if only a few of the millions of certifiably crazy, evil people in the world do this, we’re all doomed. We must not allow the democratization of KMDs (Knowledge of Mass Destruction?). Oh yeah, and remember how bad the plague was in Greek times or the middle ages? Why, they catapulted plague-infected bodies over city walls, and people died horrible deaths and doctors couldn’t help at all. Clearly we have not evolved to the point where people can be trusted with knowledge of biology sufficient to engineer pathogens. And oh yeah, there are a bunch of famous people and books that agree with Joy, and he can quote them all (I think Einstein was probably most quoted).

Joy’s solution is “relinquishment”, though he didn’t really give any details of what this would involve, though he seems to think that arms control treaties and subsequent verification protocols point in the right direction. He also mentioned, once, strict corporate liability as a deterrent to corporations developing dangerous technologies. I got a tiny chuckle out of that, as strict liability is one of those libertarian catchall answers.

I believe Joy said that he thinks there is a 30-50% chance of human extinction (presumably with no posthuman successor), not including all the other horrible outcomes that are likely. I didn’t get the impression from the other panelists (I should have asked that question), not to mention reading this mailing list, that human extinction isn’t a real possibility. I’d say that many of his concerns are valid, though his scaremonger/authoritarian approach seems contrived to create fame for himself.

If Joy was “wrong” and annoying, Merkle was “right” and extremely annoying. I felt that Merkle came across as a (highly intelligent) pompous ass with a really bad sense of humor. He didn’t even attempt to address Joy’s points, not counting wisecracks (“Would those nanomachines be using the broadcast architecture, or some other architecture?” Ok, you had to be there. I cringed.) I got the same impression of Merkle when I saw him on stage with Michio Kaku at a “Next 20 Years” event. My tentative evaluation: brilliant researcher, rotten public spokesperson.

I hadn’t heard of the broadcast architecture before (I don’t attempt to keep current with nanotech research, though hardly anyone in the audience raised their hands when Merkle asked if anyone had heard of it, and I suspect many of them were imaginging some networking or distributed computing architecture, as I was when I considered half-raising my hand). The idea seems to be that nanobots would somehow be broadcast instructions, eliminating the need for them to act completely independently (an analogy with DNA was made — these broadcast architecture nanobots wouldn’t carry around a full complement of DNA) and making them much cheaper and more controllable. The last point was held forth as a promising means of preventing a runaway self-replicator catastrophe.

My intuition (and that’s all I have on this point) doesn’t find this one-sentence version of the broadcast architecture very compelling in terms of cost or danger. Embedding instructions in a nanobot seems really cheap, considering the capacity of nanotech storage. Would an embedded communications device be cheaper? Well, it may be in one sense at least: it would be much easier to program nanobots to do some very limited function and await instructions than it would be to program nanobots to do generalized tasks and to handle general contingencies. But then it would be even simpler (not to mention safer) to program nanobots to do one task, then “die” after doing that task a desired number of times. On controllability, it seems that if nanobots can be broadcast instructions, then they, having security bugs, can be broadcast bad instructions.

John Holland’s comments were all very brief and generally well spoken. He was highly skeptical of Kurzweil and Moravec’s predictions. Holland said that we have a very slight understanding of intelligence, and without much better theory we won’t get very far. He drew an analogy between machine intelligence and fusion power — he believes that we haven’t gotten very far in five decades with the latter because we don’t have sufficiently good theory, despite spending billions trying to make it work, and despite fusion power potentially being a really good thing.

Throughout the afternoon there were several comments that alluded to the need for better theory, or at least different approaches, in order to make breakthroughs. Or, as Jeff Davis’ Ray Charles signature quote says “Everything’s hard till you know how to do it.” Kurzweil and Moravec were asked whether if 100 years in the future we knew how to create machine intelligence, we couldn’t run such an intelligence on today’s computers (this followed someone mentioning a tinkertoy computer (but it doesn’t run Linux!)). Both seemed to indicate that today’s computers simply don’t have the storage or horsepower needed. I can understand storage, but given an intelligent program and glacially slow hardware, why can’t it just be really slow?

Another comment in this vein from the audience mentioned that someone (at Sandia?) had created a robot that could walk with only twelve transistors, involving an analog feedback system, wheras it has been extremely hard to get many-MIPS digital-brained computers to walk. Moravec seemed to say that because analog requires some bulk technology, digital nanocomputers would probably be more cost effective even if they must be really complex. Well, yeah, but we don’t have nanocomputers yet. There’s lots of cool stuff remaining to be done with old technology, and I bet it will sometimes be much more cost effective from a development perspective.

Kevin Kelly’s answer to the symposium’s title “Will spiritual robots replace humanity by 2100” was “NO WAY”. His argument, to the extent I caught it (I kind of zoned out for awhile do to extreme thirst) was that machine intelligence will fill lots of specialized niches, some of them niches previously filled by humans, but no machine will completely replace humans. He used as a calculator as a primitive example — it’s much better than any human at arithmatic, but not good for much else. I’m not making the point as eloquently as he did. Perhaps it was the graph with lots of little dots on it, all representing little niches for intelligent entities. At best, he seemed to say, intelligent machines will free humans from having to work.

I also remember Kelly being the first to mention that communicating with intelligent machines of our creation could be a very spiritual thing, much like communicating with “ET” would be. Kurzweil made a similar point several times.

Frank Drake came off as a mildly boring, mildly crackpot case. We’ll judge the aliens intelligence by the size of their radio telescopes, har, har, har.

John Koza said that in numerous attempts to have a genetic program learn to model some tiny aspect of human intelligence or perception, perhaps equivalent to one second of brain activity (I know this doesn’t really make sense, I’m fuzzy on the details and I don’t recall any of the specific cases) that he found he required 10^15 operations (requiring months on standard PCs). So, a “brain second” is 10^15 operations, and this huge number obviously poses a huge barrier to machine intelligence. Or something like that. I’ll have to watch the webcast when it is available, seemed like an interesting point.

Even while listening, I was confused concerning Koza’s argument vis-a-vis the hardness of machine intelligence. It seems (as Kurzweil later pointed out concerning his speech recognition software) that once a genetic program “learns” a desired behavior, it can be copied infinitely, so the operations required to get to a certain level of functioning are mostly irrelevant.

There was lots of good stuff in the discussion and Q&A sessions, but it’s mostly a blur to me. I’ll mention three things I remember:

Kurzweil said that he was using genetic programming to simulate stock traders (presumably using historical data?) Successful trader programs get to recombinate with other successful trader programs. He didn’t mention whether they were making real trades and if so, how successfully. I’m sure lots of people are doing similar research, given the potential payoffs.

A few people mentioned consciousness being a pattern that presumably could be mapped to any substrate. An example, given by either Kurzweil or Moravec, was that of a pattern in a river — the water molecules constantly change, but the pattern may remain for long periods of time. Moravec went even further, saying that perhaps conciousness is an interpretation of a pattern, so if you know what you’re looking for, you could perhaps find conscious patterns, say in rocks, to pick a cliche. Sure, this is run of the mill daydreaming for extropians, but somehow it’s pleasant to hear it in public.

In response to an audience question about spirituality, Joy said that he had read a book (of course!) by E.O. Wilson in which Wilson had hinted at explaining all beliefs, including spiritual beliefs, in physical terms. Joy said, roughly paraphrased, “the game’s changed, they [religious people] just haven’t been told yet.” See, he has some sense! Yeah, he wrote vi too.

After the event let out, I wondered around a bit and laid down under the pleasant sun in the deserted engineering quad. The cirrus clouds above were beautiful and the temperature perfect. The experience was giddy. I rededicated myself to experiencing the wonder of life, even as a mere human, and eagerly look forward to attaining ever giddier heights, perhaps with some technological assistance in the future.

Later I wondered around Palo Alto while waiting for the next Caltrain. I hadn’t been there in a few years. On a saturday night, it’s like fairyland. Healthy and obviously wealthy people literally spilling out of every immaculate restaurant. Someone went out of their way to pick up a pen I dropped in the bustle. Even the sole homeless man seemed to be doing pretty well. Reminded me of Santa Barbara, except that Stanford is where the ocean would be, and the workers aren’t mostly Mexican. Amazing what extraordinary wealth can do. Don’t imagine too many happy faces there today (NASDAQ selloff).