Mike Linksvayer

Most popular descriptions of why BitTorrent works so well are off the mark, The BitTorrent Effect in the current Wired Magazine included. Excerpts:

The problem with P2P file-sharing networks like Kazaa, he reasoned, is that uploading and downloading do not happen at equal speeds. Broadband providers allow their users to download at superfast rates, but let them upload only very slowly, creating a bottleneck: If two peers try to swap a compressed copy of Meet the Fokkers - say, 700 megs - the recipient will receive at a speedy 1.5 megs a second, but the sender will be uploading at maybe one-tenth of that rate.

…

Paradoxically, BitTorrent’s architecture means that the more popular the file is the faster it downloads - because more people are pitching in. Better yet, it’s a virtuous cycle. Users download and share at the same time; as soon as someone receives even a single piece of Fokkers, his computer immediately begins offering it to others. The more files you’re willing to share, the faster any individual torrent downloads to your computer. This prevents people from leeching, a classic P2P problem in which too many people download files and refuse to upload, creating a drain on the system. “Give and ye shall receive” became Cohen’s motto, which he printed on T-shirts and sold to supporters.

…

Sites like Kazaa and Morpheus are slow because they suffer from supply bottlenecks. Even if many users on the network have the same file, swapping is restricted to one uploader and downloader at a time.

Most home and many business broadband connections are asymmetric — the downstream pipe is much fatter than the upstream pipe. That’s a problem any net application that requires significant upstream bandwidth has to contend with. There is no protocol solution. A BitTorrent client can’t upload any faster than a Gnutella client.

Kazaa, eDonkey and various Gnutella clients (e.g., LimeWire) have incorporated multisource/swarming downloads for three years, and the latter two also use partial file sharing (I’m not sure about Kazaa and PFS). These two key features — download from multiple peers, and begin uploading parts of a file before you’ve completed downloading — don’t set BitTorrent apart, though it may be slightly ahead in pushing the state of the art (I haven’t examined the protocols side by side).

So why does BitTorrent work so well? Deployment.

Gnutella et al users start a client and typically download and share files in one or more local directories. All files a user has collected are shared simultaneously. A client connects to random peers and accepts random queries and requests to download any files the client is sharing. There are significant refinements, e.g., ultrapeers and supernodes.) Downloads will be spread across a huge number of files and peers downloading the same file won’t necessarily know about each other (and thus won’t be able to upload to each other while downloading). Again, there are significant refinements — Gnutella peers maintain a list of other peers sharing a given file — knows as an alternate location download mesh.

For BitTorrent such refinements are superfluous. A BitTorrent user finds a “torrent” for a file the user wants to download, a BitTorrent client is launched and connects to a tracker specified by the torrent. All clients connecting to a tracker via a single torrent are by definition all downloading the same file, and they know about each other — the ideal situation for swarming distribution. And here’s the key to BitTorrent’s success in spite of typically limited upload rates: Because users are sharing only one or a few files — the one(s) they’re downloading — their precious upstream bandwidth is used to enhance a virtuous cycle in which everyone downloading the same file downloads faster.

This ideal situation also allows BitTorrent to utilize tit-for-tat (PDF) leech resistance — “Give and ye shall receive” above. A typical filesharing client can’t effectively use this strategy as it is unlikely to have multiple interactions with the same peer, let alone simultaneous mutually beneficial interactions.

There are technologies (possibly Distributed Hash Tables), tweaks (a client giving preference to uploads of files the client has recently downloaded has been proposed) and practices (encourage filesharing client users to initiate downloads from editorially controlled lists of files rather than via ad hoc peer searches) that can be implemented in typical filesharing clients to make the average user’s download experience better, perhaps someday approaching the average BitTorrent user’s experience when downloading a popular file.

There’s also lots of talk about decentralizing BitTorrent. See eXeem, supposedly to be released in a few weeks. It appears that eXeem will attempt to keep BitTorrent’s beneficial characteristics by limiting files shared to those a user has obtained or created a .torrent file for — perhaps similar to a hypothetical Gnutella client that only shared files for which it had alternate sources. I don’t have high hopes for the decentralized bits of eXeem, whatever they turn out to be. It may serve as a decent standard BitTorrent client, but there’s no need for another of those, and eXeem will supposedly be riddled with spyware.

Votes for Washington state governor cast in November have now been counted three times. One candidate won the first two counts (first by 261 votes, then 42) , the other won the second recount by 129 votes. Dino Rossi (Republican), the candidate who won the first two counts, wants a revote and is threatening litigation. Christine Gregoire (Democrat), who won the second recount, says a revote would waste $4 million, according to Rossi urges revote to fix “mess” in the Seattle Times.

Why re-count, re-vote, or litigate over same at all? Change whatever rule determines that a recount is necessary to mandate a coin flip instead of a recount.

It turns out that many people have suggested flipping a coin in this election and in the past (e.g., Ralph Nader on Florida in 2000), including several stories and columns from the Seattle Times, but I don’t gather that anyone is seriously considering the option. “Counting every vote” is too sacred.

However, “counting every vote” is itself subject to random errors. If an election is close enough for a recount, the recount won’t necessarily be any more accurate than a coin flip.

Apparently many jurisdictions allow flipping a coin or drawing lots to determine the winner of close elections in low-stakes contests. Given that recounts are no more accurate than coin flips, chance should be allowed to determine the outcome of any close election, especially “high stakes” elections, as those will be the most costly to recount, revote, and litigate.

I’d be happy to replace most elections entirely with random selection, which in my view would be far more democratic and far less costly than the current system.

Cory Doctorow correctly lambastes those soft on DRM for the umpteenth time. The following excerpt sparked a thought:

DRM isn’t protection from piracy. DRM is protection from competition.

Reminds me of airport “security” and similar. In the essay IDs and the illusion of security Bruce Schneier makes a case (not nearly as forcefully as can be done) that

Identification and profiling don’t provide very good security, and they do so at an enormous cost.

I’d argue that most measures justified by “security” actually make us less secure, in part because of their enormous cost. Another time.

Anyway, I think there’s a nice (ugly) symmetry in the arguments of apologists for Digital Restrictions Management and the national security state. Both are really much about restricting competition.

[Schneier link via Anton Sherwood.]

Tyler Cowen cites a Harper’s Index factoid:

Number of American five-year-olds named Lexus: 353

One of them works at Raisins, featured in the first South Park episode I ever watched and still my sentimental favorite. Every kid should watch this episode. If it is available on DVD I can’t find it, but search for “South Park 714″ or “South Park Raisins” on any filesharing network — South Park episodes are among the most shared content.

Also see Christian Hard Rock, which tackles filesharing. Almost every episode is well worth watching for kids and adults. Skip the movie, it sucks ass.

Betterhumans cites a study that found curcumin (turmeric, the spice that makes curry yellow) inhibits the accumulation of destructive beta-amyloid plaques and breaks up existing plaques in genetically altered mice.

Perhaps this helps explain the possible very low incidence of alzheimer’s in India. Much more study needed.

For years I have eaten lots of turmeric, which may be added to just about any food. Yum. I doubt my father’s mother has eaten any apart from tiny amounts used in food coloring. Unfortunately she’s far past being helped by any crude dietary intervention. Happy holidays to the husk of Victoria Ulakey.

Nathan Yergler just cut ccPublisher 1.0, a Windows/Mac/Linux desktop app that helps you license, tag, and distribute your audio and video works. I’m very biased, but I think it’d be a pretty neat little application even if it weren’t Creative Commons centric.

• It’s written in Python with a wxPython UI, but is distributed as a native windows installer or Mac disk image with no dependencies. Install and run like any other program on your platform, no implementation leakage. Drag’n'drop works.
• Also invisible to the end user, it uses the Internet Archive’s XML contribution interface, ftp and CC’s nascent web services.
• RDF metadata is generated, hidden from the user if published at IA, or available for self-publishing, ties into CC’s search and P2P strategies.

Python and friends did most of the work, but the 90/10-10/90 rule applied (making a cross platform app work well still isn’t trivial, integration is always messy, and anything involving ID3v2 sucks). Props to Nathan.

Version 2 will be much slicker, support more media types, and be extensible for use by other data repositories.

Addendum 2005-01-12: Check out Nathan’s 1.0 post mortem and 2.0 preview.

Dear LazyWeb,

Bloglines, Technorati and probably others do a passable job of presenting direct references to a blog entry. (Minor complaints: With Bloglines you have to subscribe to a feed or preview with a “siteid” internal to Bloglines; if your blog has multiple duplicative feeds (e.g., rdf/rss2/atom) direct entry references only appear for one of the feeds; Bloglines makes no attempt to consolidate or allow feed owners to consolidate; Technorati appears to use screen scraping and picks up some garbage along the way.)

So here’s my LazyWeb request:

I want to know, without lots of extra clicking, not just resources that directly cite blog entry A, but resources that cite resources that cite blog entry A, and so on. In the context of Bloglines, instead of “2 references” I want to see “2 direct references, 3 level1 indirect references, 1 level2 indirect reference”, “6 references, 2 direct” or similar, and I want to be able to see all references, direct and indirect, on a single page. In the context of Technorati, indirect references could optionally be part of a “watchlist” feed.

Will Bloglines, Technorati, or some up and coming aggregation service please do this?

I’m not terribly interested in visualization of social networks implied by blogs (blogosphere visualization, blogversation maps?) or even blogthread visualization and the like here. Neat, but too heavyweight to use daily. I just want a small feature increment.

Many of John “Searchblog” Battelle’s predictions for 2005 seem like near certainties, e.g., a fractious year for the blogosphere and trouble for those who expect major revenues from blogging.

Two trends I hope 2005 proves that Battelle’s predictions missed:

Metadata-enhanced search. Will be ad hoc and pragmatic, pulling useful bits from private sources and people following officious Semantic Web and lowercase semantic web practices.

Proliferation of niche web scale search engines. Anyone can be a small-scale google, crawling the portions of the web cared about and offering search options specific to a niche. The requisite hardware and bandwidth are supercheap and the Nutch open source search engine makes implementation trivial.

The Creative Commons search engine is a harbinger of both trends.

Battelle’s look ahead spans the web, not just web search. Possibly the biggest trend missing from his list is the rise of weblications. Egads, I have to learn DHTML, and it isn’t 1997!

A few of my near certainties: lots of desktop search innovation, very slow progress on making multimedia work with the web and usable security, open source slogs toward world domination, and most things get cheaper and more efficient.

This evening I had the pleasure of attending an open house for the CommerceNet Labs center for decentralization or Zlab. I’ve been meaning to write about Zlab for awhile, and I’m taking advantage of tonight’s event to write without having anything to say.

If I may boil down Zlab’s aim to one paraphrase: Make software that works the way a fully decentralized society would work.

Check out their The Now Economy for a flurry of deep items concerning decentralized commerce and net infrastructure, lab projects that abet the above aim and publications. Of personal interest, see Nutch: A Flexible and Scalable Open-Source Web Search Engine, which uses the Creative Commons search engine to demonstrate how a Nutch plugin is implemented.

Over a month ago I attended Accelerating Change 2004. I agree with Peter McCluskey’s take: an unexpected but mostly well done and welcome focus on current developments and lots of excitement about virtual worlds, Second Life in particular. Virtual words offer a low cost platform for economic, social and even physical object experimentation, prototyping and more. Virtual worlds are the future! Pity I never was much of an enthusiast for MUDs or video games, so I have a couple decades’ worth of catching up to do.

Had it not been held across the continent (South Carolina) I would’ve preferred to attend the conflicting 3rd Annual Calorie Restriction Society Conference. The first two CR conferences were lots of fun, with a good mix of learning from CRONies (CRON is Calorie Restriction with Optimal Nutrition) far more serious than me and talks by academics studying CR and aging mechanisms.

Dean Pomerleau, whose site is well worth visiting, took notes on approximately every CR 2004 talk: day 1, day 2, and day 3.

My impression is that North Korea as the ultimate Stalinist freakshow really only entered western consciousness in the last few years as the Kim Jong-Il regime acquired nuclear weapons and long range missiles. Well known emblems of the freakish regime range from mass tragedy (famine), brutality (escapees strung together with a wire through their noses), to monumental failures and the amazing (a human video billboard). My personal favorite is a full page color ad placed in the New York Review of Books a few years ago for a book exalting Kim Jong-Il as the greatest human to walk the earth and then some (perhaps the book was Kim Jong-Il, A Brief History, brief indeed compared to the 2,161 page Complete biography of the Great Leader Kim Il Sung). The layout of the ad was sub-amateur and the hyperbolic language clearly written for a captive audience. I don’t believe I’ve ever seen a more hopeless advertisement in my life.

However, if two recent columns by Andrei Lankov are accurate, western perception may be trailing reality. December 7th’s Cracks in North Korean ‘Stalinism’ and December 14th’s Welcome to capitalism, North Korean comrades paint a picture of a regime that has lost control of the people’s minds and no longer runs the economy. Cheap used VCRs and transistor radios smuggled from China and soap operas broadcast from South Korea have helped, but the largest factor seems to be a total collapse of the state run economy over the last decade, leaving even the ultimate Stalinists bereft of carrots for party members and sticks for everyone else.

Total control of the economy is a rather impossible goal, though mere dictatorial political control should be no problem. However, Lankov argues that the regime’s myths are dissolving …

Perhaps few North Koreans believe that every South Korean family has its own car (even if it really is the case). But there is no doubt that it is dawning on them that the South is not exactly a land of hunger and desperation. This is certain to have political consequences in the not too distant future, since the myth of South Korean poverty has been fundamental to the survival of the North Korean state. Pyongyang has always based its claims for legitimacy on being a better type of Korean government, supposedly delivering the quality of life that would be unavailable in the “exploited” and “impoverished” South. If the North Korean populace learn about South Korean prosperity, then the Pyongyang government is in deep trouble - as the fate of the much more successful East German government demonstrated: the economic gap between North and South Korea is much greater than was once the case in Germany. According to current estimates, the per capita gross national product (GNP) in the South is 10 to 20 times higher than in the North.

… perhaps leading to a collapse not unlike that seen in Lankov’s native Soviet empire, and just as unexpected by most.

There are far more interesting observations in Lankov’s above columns and his Korea Times series on “Another Korea.”

While I have little stomach or interest in the most brutal atrocities of oppressive regimes, I find attempted total economic control perversely interesting. Here’s one illustrative quote from Lankov:

Unlike governments of other communist countries, until the late 1980s the North Korean government did not even allow its farmers to cultivate kitchen gardens - the individual plot was limited to merely 20-30 square meters, hardly enough to grow enough chili pepper. This was done on purpose. In many other communist countries, farmers had bigger plots and made their living from them, ignoring their work obligations to the state-run cooperative farms. Without their own plots, farmers would work more for the state - or so believed the North Korean government. In the utopia constructed by Kim Il-sung, every single man or woman was supposed to work for the state, and was rewarded for his and her efforts with officially approved rations and salaries.

This reminds me of a claim I once heard that tiny personal gardens made a substantial contribution, far out of proportion to space used, to the Soviet table, as gardeners had de facto property interest in production from such gardens, and no other production. If I ever recall and confirm details I may post them here.

[Welcome to capitalism, North Korean comrades link via the Mises Economics Blog.]

The esteemed Gary Becker and Richard Posner begin their new publishing venture with poor rationalizations of perpetual war for perpetual peace.

Becker’s very first sentence sounds suspect:

Combating crime mainly relies on deterrence through punishment of criminals who recognize that there is a chance of being apprehended and convicted-the chances are greater for more serious crimes.

Mainly? What of prevention (locks, alarms, guards and the like), social pressure and economic growth? I’m skeptical, but that’s another argument.

Fundamentally Becker argues that because weapons are more powerful and more available, the putative good guys must be less cautious about attacking suspected bad buys. In other words, 9/11 changed everything, a view which I’ve always thought doubly naive. First, proliferation of massive destructive power is inevitable, and anyone who didn’t think of that before 9/11 just wasn’t thinking. Secondly, and more apropos to this argument, it is not at all clear that lashing out at suspected enemies is a cost minimizing strategy in such an environment.

I just love this gem from Posner, which attempts to dismiss cost-benefit analysis of war:

But the appropriateness of thus discounting future costs is less clear when the issue is averting future costs that are largely nonpecuniary and have national or global impact.

Please! Perhaps the discount rate would be different, but it would exist. Time preference is fundamental to economic analysis, which is certainly not limited to financial concerns. Incredibly disingenuous coming from someone who certainly knows better.

But Posner can’t resist cost-benefit analysis anyway and sets up a scenario in which a preventive attack would, supposedly, be cost-justified:

Suppose there is a probability of .5 that the adversary will attack at some future time, when he has completed a military build up, that the attack will, if resisted with only the victim’s current strength, inflict a cost on the victim of 100, so that the expected cost of the attack is 50 (100 x .5), but that the expected cost can be reduced to 20 if the victim incurs additional defense costs of 15. Suppose further that at an additional cost of only 5, the victim can by a preventive strike today eliminate all possibility of the future attack. Since 5 is less than 35 (the sum of injury and defensive costs if the future enemy attack is not prevented), the preventive war is cost-justified.

This strikes me as a highly unrealistic scenario. Governments invariably overestimate the benefits of their actions and understimate the financial cost of war by a factor of ten. Did the overthrow of Saddam Hussein eliminate the threat of terrorists based in or sponsored by Iraq? Hardly. Given the rose-colored glasses worn by government planners, in Posner’s scenario above I’d expect a preventive attack to cost 50 and not change the expected damage from a terrorist attack. 70 is greater than 35, war is not cost-justified.

Posner makes many more assumptions in an alternative history example:

A historical example that illustrates this analysis is the Nazi reoccupation of the Rhineland area of Germany in 1936, an area that had been demilitarized by the Treaty of Versailles. Had France and Great Britain responded to this treaty violation by invading Germany, in all likelihood Hitler would have been overthrown and World War II averted. (It is unlikely that Japan would have attacked the United States and Great Britain in 1941 had it not thought that Germany would be victorious.) The benefits of preventive war would in that instance have greatly exceeded the costs.

Why would Hitler have been overthrown in all likelihood had France and Great Britain invaded? Unless they were dead set on regime change is isn’t hard to imagine Hitler surviving. We don’t have to look back far to see a dictator surviving an invasion and military defeat — Saddam Hussein in 1991.

Would destroying Hitler have averted World War II, and not only the one we know? Who knows what set of events an invasion of the Rhineland may have set off? It could be now seen as a the beginning of a tragedy that led to a communist revolution in Germany, the ascendancy of still-credible fascism and anti-semitism in France and Great Britain, the inevitable Fascist-Communist worldwide conflict, and the U.S. pulled mightly to adopt one or the other, leading to mass slaughter and the extinction of freedom worldwide. Strange things happen. See World War I.

Hindsight is wonderful, eh? Unfortunately there’s no reason to expect it to be 20-20 unless we hold nearly everything constant. Foresight is even harder. We desperately need tools that provide better estimates of the impact of policy than bogus intellectual handwaving and self-serving bureaucratic guesstimation. Conditional futures, which I’ve mentioned here and here may be one such tool. I don’t think conditional futures is quite the term of art, but see Robin Hanson’s page on policy markets for a good explanation and his pages on the Policy Analysis Market and idea futures for far more in depth treatment.