The real Open Source _ proliferation problem

The Open Source Initiative, best known for keeping a list of licenses compliant with its Open Source Definition, has hired its first-ever full time paid staffer, Patrick Masson as General Manager.

Masson’s blog has lots of good entries (if you just want to be amused, try a 10 year press release diff). One thing he bemoans repeatedly and pithily (It’s “many eyeballs…” not “many projects…”) is too much fragmentation and too little collaboration among open source projects. His most recent post, Joiners, Not Starters:

What’s painful is that there are already over 350 open source communities developing learning management systems. I find it frustrating and hypocritical to hear, “This is a great time to get involved for people who are interested in helping to shape this project…” from people who chose not to get involved–rather, choosing to do something on their own. Why is it a great time to join the Adapt project over any other existing effort looking to build community for support, contribution and collaboration? Why didn’t the folks who are developing Adapt take advantage of this great time in open source development and join an existing initiative? Indeed, couldn’t every current open source project (substituting out Adapt for their own name) use the above announcement to generate awareness and adoption of their own project?

Sounds just like the first question/advice for anyone looking to start a new organization. Economies of scale are hard to beat. But fragmentation is much worse for software, so much of its value coming from network effects. When lots of people, preferably most of the relevant population, are using a software application, it’s easy to find training, advice, employees, commercial support, and preinstalls of that software. It is easy to figure out which software is pertinent. Massively valuable stuff. Oh, and more people contributing to making the software better, if it is open source.

When there are lots of open source alternatives for a particular kind of software, and this contributes to none of them being dominant, the ability of open source to compete with proprietary vendors and deliver freedom to users and society, is severely hampered. More or less killed. (The inverse can also be true, but probably with many fewer instances.) The Linux desktop is probably an example. Further, public policy is negatively impacted: fragmented projects serve at best as existence proofs, dominant open projects powerfully shape the policy conversation, and the policy ecosystem — by wiping out the capitalization of entities aligned with rent seeking.

People who know the open source world well like to worry about about “license proliferation” (which the OSI’s license list mentioned at the beginning serves to throttle) and related, license incompatibility. I do too, including in and across nearby spaces, to the extent I think it is a minor tragedy that licenses first developed for software didn’t also come to dominate culture, data, hardware, etc, yet. But I’m pretty sure the open world could cope with each project adapting or developing a license just for its own use, though it would be hugely annoying. Fortunately, progress has often been in the right direction.

Project/program proliferation and related dwarfish network effects and collaboration are much, much bigger problems. There are cases where a dominant program has arisen from a highly fragmented field (eg WordPress among a mess of open source blog engines, Django among lots of Python web frameworks, git among a smaller number of distributed version control systems), but I’m not sure this has ever come about because people agitated against proliferation. There are standards-like collaborations among projects, such as, which can result in more sharing of code and collaboration among projects, but I’m not sure do much to enable mass adoption and network effects. What more can be done, given that of course it will always be acceptable, often educational, and very occasionally wildly successful to work on Yet Another Foo?

  • The low-hanging fruit is to help projects become easier for new contributors to get involved in, and friendly for staying involved in. Decrease the cost of contributing to existing projects, more will choose to do that rather than start, or leave to start, new projects.
  • I don’t have much insight into the politically charged process of picking winners and merging efforts. Distributions (which are themselves terribly fragmented) probably already do a lot. Could they do more? Could institutions broker mergers? Could OSI? Stun all by bringing LibreOffice and OpenOffice together. GNOME, KDE, and Unity as well. How about federated social web efforts?
  • Marketing, promotion, sales. These are what any large proprietary software company does (same outside software, for publishing, etc.), and what open source projects need a lot more of, both to compete directly with proprietary industry, and to help winners with huge network effects emerge.

Each of these points also apply very strongly to non-software projects.

Another thing Masson repeatedly bemoans on his blog, and that I very much agree with, is the lack of “open” advocates eating their own dogfood — using open things other than the one they’re promoting, or open things from fields other than the one they’re supposedly opening:

However with so little folks actually interested in openness, but rather promoting their open product, we just don’t see the level of adoption we should with all open initiatives. Basically, if I can be blunt, you’re a hypocrite if you get up in front of your peers to proclaim the superiority of your project because it embraces open principles and practices, arguing it is those principles and practices that yield better products, but you yourself have not adopted other open resources. “Hold on, let me open up PowerPoint to tell you about how bad commercial software is.”

This not only harms network effects (or rather, has “open” advocates contributing to the network effects of proprietary software, culture, etc), but reduces knowledge transfer across open projects and fields. Masson seems to come from the education technology world; if that is anything like the open education[al resources] world, I suspect he’s speaking from painful experience.

Congratulations to OSI and Masson. I look froward to amazing progress on the above problems and many others! You can support their work by joining OSI as an individual member. Of course I also recommend joining the Free Software Foundation as an individual member. Because open source means freedom.

10 Responses

  1. Aaron Wolf says:

    Thanks, Mike! Sums up my own concerns immensely. When I started volunteering and even trying to do anything, the fragmentation was (and remains) immensely frustrating. Freedom of choice is a potentially good freedom but choice in itself isn’t desireable. I only want choices when I don’t like the default; give me a good default, and I’m happy.

    The proprietary world is filled with fragmentation too (see the absurd number of iOS apps etc.), but with FLO, we *could* take all the best ideas and combine them (as long as licenses are compatible). We just need more people to do that.

    Incidentally, I’m writing this having taken a break from cataloging 440 (so far) crowdfunding websites that are almost all redundant (supported in part by some whitelabel proprietary services profiting from all the redundancy). Gah.

    Hail to Wikipedia!!

  2. There’s a common idea, reinforced in this post, that two projects trying to do a very similar things are a duplication of effort and therefore a bad thing for the products and our community. I’m not sure I buy it.

    First although there’s some duplication, projects are rarely the same in their goals or their methods. They are different in ways that (a) may, or may not, matter in terms of their success and (b) are mututally exclusive. Being written in Erlang instead of PHP might really matter and you can’t be written in both. Sometimes Feature A conflicts with Feature B.

    Clay Shirky has a chapter called “Failure for Free” that I really like where he argues (in part) that one reason FLOSS is successful is because the cost of failure is low. This low cost of failure means folks can try lots of things, in parallel, and they can fail. Very very often while doing the same thing. If we imagine the search for for the ideal learning management software as the search for a maximum in a highly jagged multi-dimensional space, simply trying many things in slightly different ways can help. It’s also the case that simply engaging in the development of software that fits your needs exactly is a benefit of FLOSS.

    Finding out ways to collaborate and work together more often and more effectively is something that we as a community living in freedom should strive to do. But knee-jerk reactions against “duplication of effort” rely a model of efficiency through mass-aggregation of software or community building effort that’s both not always true and throws out quite a bit of baby with the bathwater.

  3. Aaron Wolf says:

    I agree with Ben’s comment that seemingly duplicate efforts are not *necessarily* bad for the community. Part of the question is whether we are comparing a duplicate effort to nothing versus comparing a duplicate effort to collaboration. Each case will be different anyway.

    I support pursuing a social norm where it is expected for people to do their due diligence and research existing and past projects, learn from others’ successes and failures, and consider whether the goals can be achieved by collaboration and contribution to existing projects. If that is done and the choice is still to make a new competing project, that should be respected.

    I don’t want knee-jerk reactions against duplication, but I think it’s healthy to push people to do their due diligence. It’s reasonable to encourage people to provide some explanation and justification for their apparently duplicate work. But as long as such explanations are provided, we should avoid wasteful and discouraging arguments about whether the explanations are justified enough. It’s good just that we ask everyone to at least put some energy and thought into considering this issue when making decisions.

  4. Hi new and newly returned Cascadians,

    I skimmed “Failure for Free”; this seems a key quote:

    Has the press, then, gotten it wrong about open source? Has it mischaracterized the movement, based on the successes like Linux, when the normal condition of an open source effort is failure? The answer is yes, obviously and measurably yes. The bulk of open source projects fail, and most of the remaining successes are quite modest. But does that mean the threat from open systems generally is overrated and the commercial software industry can breathe easy? Here the answer is no. Open source is a profound threat, not because the open source ecosystem is outsucceeding commercial efforts but because it is outfailing them. Because the open source ecosystem, and by extension open social system generally, rely on peer production, the work on those systems can considerably more experimental, at considerably less cost, than any firm can afford. Why? The most important reasons are that open systems lower the cost of failure, they do not create biases in favor of predictable but substandard outcomes, and they make it simpler to integrate the contributions of people who contribute only a single idea.

    I suspect he’s overestimating the cost of failure in proprietary software by assuming it is developed in firms. That was never the case (shareware/freeware), though perhaps FLOSS had some cultural rather than structural advantage favoring more cheap experiments. Is that still the case? I doubt it: with apps, SaaS, and MVP as received wisdom, proprietary software may now have structural and cultural advantages over FLOSS for rapid and cheap exploration. Seems an open question to me. But I’d love to see evidence of “obviously and measurably yes” in favor of FLOSS.

    On to more pertinent questions…

    Cheap failure is unquestionably good, but so is a higher rate of success, both all-else-equal. It is the latter I want to increase. There are few FLOSS projects that would not, at least in theory, though they may be unprepared socially, benefit from more contributors. As I wrote in the post, it ought be possible to decrease the cost of joining an existing project, without increasing the cost of starting a new one (except of course viewed as an opportunity cost; but I don’t think we want to make it as costly as possible to join an existing project in order to minimize opportunity cost of starting a new one):

    The low-hanging fruit is to help projects become easier for new contributors to get involved in, and friendly for staying involved in. Decrease the cost of contributing to existing projects, more will choose to do that rather than start, or leave to start, new projects.

    But, my primary concern in this post is not contributor effort (duplicative or otherwise), but costs and value of adoption, and network effects of adoption. It seems to be there are huge costs to freedom, as I wrote (but perhaps should have phrased as questions):

    When there are lots of open source alternatives for a particular kind of software, and this contributes to none of them being dominant, the ability of open source to compete with proprietary vendors and deliver freedom to users and society, is severely hampered.

    It is possible that the existence of lots of alternatives never contributes to lack of a dominant FLOSS one, or even contributes to finding a dominant one through rapid exploration. It could even be that LMS is an example — I don’t know the space at all, but I gather Moodle is “the” usual FLOSS alternative. But, I doubt this is generally the case, just from my own experience looking for software. There are high search costs, for deployers/users as well as potential contributors. Really high search costs, as software evaluation is hard, and reviews and comparisons are generally both low quality when published, and out of date. It’s often hard to discern what features an application even has, without testing the application, and even then it can be hard to tell. And fragmented adoption is then further a killer for social support and knowledge sharing, and for commercial training, integration, etc.

    Then what about rapidly and cheaply exploring highly jagged multi-demensional spaces? My intended follow-up to this post is titled Where is the software avant-garde? I would love to see empirical characterizations, but my gut feeling is that software generally, including FLOSS, has done an appallingly bad job of exploration. I’d be very happy to lose, but my bet is, eg among those hundreds of LMSs, that variation is utterly dominated by considerations implementation and integration preferences and requirements (eg programming language/framework, licenses, database and other systems), institutional or business control, and lack of knowledge about other LMSs, rather than any more fundamental exploration. Of course many of these considerations are “valid” at least some of the time, but any correlation with exploration of what LMSs might be, I’d bet is coincidence.

    I wonder whether application proliferation doesn’t make exploration harder. It’s very difficult to tell what has been done; when there is no dominant application, it’s harder to explain how a new application is different, as there are many, only vaguely known, points of comparison; and it’s harder for a really different application to not be lost in the muddle.

    But, perhaps the software avant-garde ought not be mourned. If, as I suspect, much of the value of software comes from other people using the same software, the most relevant exploration then is not the jagged multi-dimensional space of features, but the relatively simple space of network effects.

    Anyway, I don’t want to throw the baby out with the dirty bathwater. I want demon children in an ocean of distilled water. Sometimes one of them boils the ocean.

  5. […] I recently wrote concerning open source project fragmentation: […]

  6. […] net the dominance of WordPress is probably good, but I also want to see more crazy blog/IndieWeb software, crazy meaning taking a very different […]

  7. […] systems, funders? A higher bar for user interface changes requiring translation updates? Fewer […]

  8. […] marginal — the result of failure to compete with proprietary/closed vendors on marketing (previously […]

  9. […] increase their contributions as more people contribute) could help create clear winners among the proliferation of such […]

Leave a Reply