Changing capital aggregation

The Rad Geek People’s Daily has an interesting comment on my post Capitalists vs. Entrepreneurs:
The only thing that I would want to add here is that it’s not just a matter of projects being able to expand or sustain themselves with little capital (although that is a factor). It’s also a matter of the way in which both emerging distributed technologies in general, and peer production projects in particular, facilitate the aggregation of dispersed capital — without it having to pass through a single capitalist chokepoint, like a commercial bank or a venture capital fund. Because of the way that peer production projects distribute and amortize their costs of operation, entrepreneurs can afford to bypass existing financial operators and go directly to people with $20 or $50 to give away and take the money in in small donations, because they no longer need to get multimillion dollar cash infusions all at once just to keep themselves running: the peer production model allows greater flexibility by dispersing fixed costs among many peers (and allowing new entrepreneurs to easily step in and take over the project, if one has to bow out due to the pressures imposed by fixed costs), rather than by concentrating them into the bottom line of a single, precarious legal entity. Meanwhile, because of the way that peer production projects distribute their labor, peer-production entrepreneurs can also take advantage of “spare cycles” on existing, widely-distributed capital goods — tools like computers, facilities like offices and houses, software, etc. which contributors own, which they still would have owned personally or professionally whether or not they were contributing to the peer production project, and which can be put to use as a direct contribution of a small amount of fractional shares of capital goods directly to the peer production project. So it’s not just a matter of cutting total aggregate costs for capital goods (although that’s an important element); it’s also, importantly, a matter of new models of aggregating the capital goods to meet whatever costs you may have, so that small bits of available capital can be rounded up without the intervention of money-men and other intermediaries.
I like the point about the improved aggregation of capital, that is quite possibly as important as the reduction in capital requirements.

This gets me thinking more about the importance of reduced coordination costs (aggregation of capital being a special case). Clearly computers and networks contribute enormously to improving the productivity of coordination. However I don’t think we have good models for the costs of coordination, or the effects of improved coordination, so its importance tends to be underestimated.

I guess cheaper / easier / faster coordination is a special case of (what economists call) “technology” — changes with big economic impact, that are outside the economic model. From another point of view, coordination costs and delays are a special case of (what economists call) “frictions” which are also hard for them to model well. So trends in coordination may well affect the economy in major ways, but are none the less mostly invisible in economic models.

And in fact we’d expect coordination costs to fall and speed and capacity to rise on an exponential trend, riding Moore’s law. Given that changes in coordination have significant economic impact (how could they not?) there’s a huge long term economic trend that’s formally invisible to economists.

One aspect of this that’s perhaps subtle, but often important, and that shows up strongly in capital aggregation, is how “technology” changes the risk of cooperation. One of the big sources of risk is that someone you’re cooperating with will “defect” (in the prisoner’s dilemma sense) — that the cooperative situation will give them ways to benefit by hurting you. In fund raising this risk is obvious, but working with people, letting others use your “spare cycles”, etc. have risks too. Ways of coordinating have been evolving to counteract or at least mitigate these risks — clearer norms for response to bad actors, online reputations, large scale community reactions to serious bad behavior, and so forth. Wikipedia’s mechanisms for quick reversion of vandalism is an example. Even spam filtering is a case in point, reducing our costs and the benefits to the bad actors. It is too early to know for sure, but right now there are enough success stories that I’d guess that the space of defensible and sustainable cooperation is pretty big — maybe even big enough to “embrace and extend” important parts of the current economy.

Turking! The idea and some implications

I recently read an edited collection of five stories, Metatropolis; the stories are set in a common world the authors developed together. This is a near future in which nation state authority has eroded and in which new social processes have grown up and have a big role in making things work. In some sense the theme of the book was naming and exploring those new processes.

One of those processes was turking. The term is based on Amazon’s Mechanical Turk. Google shows the term in use as far back as 2005 but I hadn’t really wrapped my mind around the implications; Metatropolis broadens the idea way beyond Amazon’s implementation or any of the other discussions I’ve read.

Turking: Getting bigger jobs done by semi-automatically splitting them up into large numbers of micro-jobs (often five minutes long or less), and then automatically aggregating and cross-checking the results. The turkers (people doing the micro-jobs) typically don’t have or need any long term or contractual relationship with the turking organizers. In many cases, possibly a majority, the turkers aren’t paid in cash, and often they aren’t paid at all, but do the tasks as volunteers or because they are intrinsically rewarding (as in games).

One key element that is distinctive to turking is some sort of entirely or largely automated process for checking the results — usually by giving multiple turkers the same task and comparing their results. Turkers who screw up too many tasks aren’t given more of those tasks. Contrast this with industrial employment where the “employer” filters job candidates, contracts with some to become “employees”, and then enforces their contracts. The relationship in turking is very different: the “employer” lets anybody become an “employee” and do some tasks, doesn’t (and can’t) control whether or how the “employee” does the work, but measures each “employee’s” results and decides whether and how to continue with the relationship.

This is an example of a very consistent pattern in the transition from industrial to networked relationships: a movement from gatekeeping and control to post hoc filtering. Another example is academic publishing. The (still dominant) industrial model of publishing works through gatekeeping — articles and books don’t get published until they are approved through peer review. The networked model works through post hoc processes: papers go up on web sites, get read, commented and reviewed, often are revised, and over time get positioned on a spectrum from valid/valuable to invalid/worthless. The networked model is inexorably taking over, because it is immensely faster, often fairer (getting a few bad anonymous reviews can’t kill a good paper), results in a wider range of better feedback to authors, etc.

It seems quite possible — even likely — that post hoc filtering for work will produce substantially better results than industrial style gatekeeping and control in most cases. In addition to having lower transaction costs, it could produce better quality, a better fit between worker and task, and less wasted effort. It also, of course, will change how much the results cost and how much people get paid — more on that below.

Amazon’s Mechanical Turk just involves information processing — web input, web output — and this is typical of most turking today. However there are examples which involve real world activities. In an extreme case turking could be used to carry out terrorist acts, maybe without even doing anything criminal — Bruce Sterling has some stories that explore this possibility. But there are lots of ordinary examples, like counting the empty parking spaces on a given block, or taking a package and shipping it.

Examples

  • Refugees in camps are turking for money. The tasks are typical turking tasks, but the structure seems to be some more standard employment relationship. If there were enough computers, I bet a high percentage of the camp residents would participate, after some short period in which everyone learned from each other how to do the work. Then the organizers would have to shift to turking methods because the overhead of managing hundreds of thousands of participants using contracting and control would be prohibitive.
  • A game called FoldIt is using turking to improve machine solutions to protein folding. Turns out humans greatly improve on the automatic results but need the machine to do the more routine work. The turkers have a wide range of skill and a variety of complementary strategies, so the project benefits from letting a many people try and then keeping the ones who succeed. (This is an example where the quality is probably higher than an industrial style academic model could generate.) The rewards are the intrinsic pleasure of playing the game, and also maybe higher game rankings.
  • There’s a startup named CrowdFlower that aims to make a business out of turkingrowdFlower has relationships with online games that include the turking in their game play. So the gamers get virtual rewards (status, loot). I can easily imagine that the right turking tasks would actually enhance game play. CrowdFlower are also doing more or less traditional social science studies of turking motivations etc. Of course the surveys that generate data for the research are also a form of turking.
  • Distributed proofreading. OCR’d texts are distributed to volunteers and the volunteers check and correct the OCR. (They get both the image and the text.) The front page goes out of its way to note that “there is no commitment expected on this site beyond the understanding that you do your best.” This is an early turking technology, and works in fairly large chunks, a page at a time. It may be replaced by a much finer grained technology that works a word at a time — see below.
  • Peer production (open source and open content). An important component of peer production is small increments of bug reporting, testing, code review, documentation editing, etc. Wikipedia also depends on a lot of small content updates, edits, typo fixes, etc. These processes have the same type of structure as turking, although they typically hasn’t been called turking. The main difference from the other examples is there’s no clear-cut infrastructure for checking the validity of changes. This is at least partly historical, these processes arose before the current form of turking was worked out. The incentive — beyond altruism and the itch to correct errors — is that one can get credit in the community and maybe even in the product.
  • I just recently came across another good example that deserves a longer explanation: ReCaptcha. It is cool because it takes two bad things, and converts them into two good things, using work people were doing anyway.

    The first bad thing is that OCR generates lots of errors, especially on poorly printed or scanned material — which is why the distributed proofreading process above is required. These can often be identified because the results are misspelled and/or the OCR algorithm reports low confidence. From the OCR failures, you can generate little images that OCR has trouble recognizing correctly.

    The second bad thing is that online services are often exploited by bad actors who use robots to post spam, abusively download data, etc. Often this is prevented by captchas, images that humans can convert into text, but that are hard for machines to recognize. Since OCR failures are known to be hard for machines to recognize correctly, they make good captchas.

    Recaptcha turns the user effort applied to solving captchas, which would otherwise be wasted, into turking to complete the OCR — essentially very fine grained distributed proofreading. Recaptcha figures out who’s giving the correct answers by having each user recognize both a known word and an unknown word, in addition to comparing answers by different users. Users are rewarded by getting the access they wanted.

    Note that if spammers turk out captcha reading (which they are doing, but which increases their costs significantly) then they are indirectly paying for useful work as well. Potentially Recaptcha could be generalized to any kind of simple pattern recognition that’s relatively easy for humans and hard for machines, which could generate a lot of value from human cognitive capacities.

    Some implications

    It seems that over time a huge variety and quantity of work could be turked. The turking model has the capacity to productively employ a lot of what Clay Shirky calls our “cognitive surplus”, and also whatever time surplus we have. Many unemployed people, refugee populations and I’m sure lots of other groups have a lot of surplus. As Shirky points out, even employed people have a discretionary surplus that they spend watching TV, reading magazines, playing computer games, etc. However right now there’s no way to bring this surplus to market.

    Switching from industrial relationships (heavyweight, gatekeeping and control) to networked relationships (lightweight, post hoc filtering) reduces per task transaction costs to a tiny fraction of their current level, and makes it feasible to bring much of this surplus to market.

    The flip side of that of course is that the more this surplus is available for production, the less anyone will get paid for the work it can do. Already in a lot of existing turking, the participants aren’t getting paid — and in many cases the organizers aren’t getting paid either. Also, more or less by definition, the surplus that would be applied to turking currently isn’t being used for any other paid activity, so potential workers aren’t giving up other pay to turk. Therefore, I expect the average payment for a turked task to approach zero, for both turkers and organizers. Usually there will still be rewards, but they will tend to be locally generated within the specific context of the tasks (online community, game, captcha, whatever). Often the entity that generates the rewards won’t won’t get any specific benefit from the turking — for example, in the case of ReCaptcha, the sites that use it don’t particularly benefit from whatever proofreading gets done.

    Mostly turking rewards won’t be measurable in classical monetary terms — in some cases rewards may involve “in game” currency but this doesn’t yet count in the larger economy. In classical monetary terms, the marginal cost of getting a job turked will probably approach the cost of building, maintaining and running the turking infrastructure — and that cost is exponentially declining and will continue to do so for decades.

    This trend suggests that we need to find some metric complementary to money to aggregate preferences and allocate large scale social effort. But I’m not going to pursue that question further here.

    Obviously it will be important to understand what types of work can be turked and what can’t. For example, could the construction of new houses be turked? That may seem like a stretch, but Habitat for Humanity and other volunteer groups do construct houses with a process very much like turking — and of course this has a long history in the US, with institutions like barn raising. Furthermore the use of day labor isn’t that different from turking. I’d guess that within ten years we’ll be turking much of the construction of quite complex buildings. It is interesting to try to imagine what this implies for construction employment.

    Realistically, at this point we just don’t know the limits of turking. My guess is that the range of things that can be done via turking will turn out to be extremely broad, but that it will take a lot of specific innovations to grow into that range. Also of course there will be institutional resistance to turking many activities.

    When a swarm of turkers washes over any given activity and devours most of it, there will typically be a bunch of nuggets left over that can’t be turked reliably. These will probably be things that require substantial specialized training and/or experience, relatively deep knowledge of the particular circumstances, and maybe certification and accountability. Right now those nuggets are embedded in turkable work and so it is hard or impossible to figure out their distribution, relative size, etc. For a while (maybe twenty years or so) we’ll keep being surprised — we’ll find some type of nuggets we think can’t be turked, and then someone will invent a way to make most of them turkable. Only if and when turking converges on a stable institution will we be able to state more analytically and confidently the characteristics that make a task un-turkable.

    Another issue is security / confidentiality. Right now, corporations are willing to use turking for lots of tasks, but I bet they wouldn’t turk tasks involving key market data, strategic planning, or other sensitive material. On the other hand, peer production projects are willing to turk almost anything, because they don’t have concerns about maintaining a competitive advantage by keeping secrets. (They do of course have to keep some customer data private if they collect it at all, but usually they just avoid recording personal details.) I’d guess that over time this will give entities that keep fewer secrets a competitive advantage. I think this is already the case for a lot of related reasons because broadly speaking “Trying to keep secrets imposes huge transaction costs.” Eventually keeping big secrets may come to be seen as an absurdly expensive and dubious proposition and the ability to keep big pointless secrets will become an assertion of wealth and power. (Every entity will need to keep a few small secrets, such as the root password to their servers. But we know how to safely give everyone read only access to almost everything, and still limit changes to those who have those small secrets.)

    There’s lots more to say, but that’s enough for now.

    The two faces of Avatar

    Avatar just keeps demanding a bit more analysis.

    To recap, I agree the story is an embarrassingly naive retread of the “white man goes native and saves the natives” plus gooey nature worship. But…

    I also believe the world Pandora, as shown to us in the movie, can’t be confined within that story. It keeps escaping and cutting across or contradicting the premises of the narrative, as discussed in many nerd posts including mine.

    So Avatar has two very different faces, and different personalities to go with them. And I think this goes back to the basic character of the social processes used to create Avatar. No, seriously, stay with me for a minute and I’ll explain.

    For our purposes we can say there are three modes of production in films and a lot of other activities: craft, industrial, and networked. Of course any real film is produced through a mix of these.

    A film made by a small team working on their own (with or without a presiding genius) is an example of craft production, just like similar teams producing ceramic tiles or houses.

    A film made in a “factory” environment along with many others (like The Wizard of Oz or Casablanca) is an example of industrial production.

    And a film made by multiple loosely coordinated groups with different expertise is an example of network production.

    Network production is now nearly universal in large films, but before Avatar I can’t think of any examples of network production driving the film content. Generally the network mostly fleshes out content dictated by a small team that is using craft production. (If you can think of good previous examples, please comment or email, I’d really like to know.)

    In Avatar, Cameron wanted a lot of depth in his world, and had the money and skills to pull together a network to produce it. Pandora was created by a huge collaboration between ecologists, biologists, linguists, artists, rendering experts and so forth. The collaboration also necessarily included software and hardware experts who built the computer networks, and project managers who shaped the social network, and these people were no doubt also very engaged with the ideas about Pandora and contributed to its character in significant ways. Cameron was of course involved, but the depth and complexity of the world (and the network) meant that most of the decisions had to be internal to the network.

    So Avatar inevitably has two faces. The plot arc, the characters and the dialog were crafted by Cameron. Much of the commercial success of the film no doubt is due to his judgements about what would work in that domain. But Pandora, and probably much of the human tech in the film was created by a social network that was focused on scientific (as well as artistic) verisimilitude, conceptual integrity across a wide range of disciplines and scales, and our best current skills for designing and managing big networks of people and machines. And a significant amount of the success of the film is due to the richness and coherence of the vision generated by the network.

    In some sense Cameron was responsible for both faces. In one case he was directly shaping the content. In the other, he was shaping and directing the social network that produced the content. But the two forms of production generate very different kinds of results, and those generate the divergent critical reactions that tend to focus either on the story or on the world.

    This analysis brings into focus a question on which I have no information, but which I think is important to our deeper understanding of Avatar and our thinking about the successors it will inevitably inspire. Who defined the parts of the world that bridge between the network and the story? For example, in Pandora, animals, Na’vi and trees can couple their nervous systems to each other. This coupling plays a role in the story, but it could have been avoided in some cases, and made less explicit and more “magical” in others. On the other hand this coupling mechanism is constitutive of key parts of Pandora such as the “world brain”, and it drastically affects our understanding of the nature of Pandora and its possible history.

    Did the network come up with this coupling as a way to make mind transfer — a part of the story that would otherwise have been magic — into science? Or was it somehow integral to Cameron’s vision of Pandora? Or — more likely — some combination of those.

    Untrustworthy by design? or incompetence? Just untrustworthy

    James Kwak has an interesting post Design or Incompetence? in which he discusses the ways banks are delaying and obstructing customer efforts to get benefits the banks have offered. In addition the banks are misrepresenting their own actions and obligations. As the title suggests he wonders whether this behavior is due to design or incompetence, and ultimately concludes (after some discussion of the internal institutional issues) that it doesn’t matter because it is a systemic consequence of the banks’ incentives.

    As in my previous post on “Lying or Stupid?”, I’d say this analysis is interesting and sometimes useful but that we should start by saying the institutions involved are untrustworthy which is true either way, and often we don’t need to look into the finer distinctions. Debating the details usually just gives these bad actors ways to muddy the water.

    More generally, we need to develop social sanctions — applied by governments, broad condemnation, boycott, and/or whatever else will work — to “adjust the incentives” of these bad actors so they either become trustworthy or are replaced by organizations that are trustworthy. These social sanctions worked with apartheid and to some extent with third world sweatshops, we can at least imagine them working with respect to untrustworthiness.

    Right now unfortunately there are many who argue that not only should corporations ignore this sort of issue, but even further that it would be immoral for them to take such considerations into account. Furthermore the general perception is that we can’t expect corporations to care about morality, as Roger Lowenstein discusses. I’ve been chewing on this issue in the comments to a couple of interesting posts by Timothy Lee and plan to summarize my resulting thoughts here soon. The good news from that discussion is that even some committed free market folks such as Timothy agree that we need to have corporations put moral obligations such as trustworthiness above profits. Now we need rough consensus on that…

    Of moose and men

    In a post several weeks ago, One Man’s Moose, Timothy Burke discussed the social tradeoffs between regulation and respect for individual desires and needs. That post came out of a larger web discussion on game management in Vermont, and resulting conflicts with people who keep moose as pets. Timothy summarized a key point:

    If you start cutting separate deals with everyone who pleads that their circumstances are special, that a legitimate attempt to safeguard the public shouldn’t apply to them, you’ll end up with a public policy that applies to no one.
    I reacted strongly to that general point; I’m posting a reworked version of my comments here.

    Of course Timothy’s summary is a more detailed statement of the classic bureaucratic argument, “If we let you do it, we’d have to let everyone do it”; living in a complex society we encounter this constraint on our liberty implicitly or explicitly many times a day.

    The basic point is tough to dispute. But the way it typically plays out, for example in Timothy’s quote, relies on an implicit assumption about the “cognitive” limitations of bureaucracy. We assume that the bureaucrat can only use fairly simple rules based on local information. As a practical matter this has been true of bureaucrats for the last several thousand years, so this assumption has gotten deeply embedded. But maybe it isn’t true anymore.

    Let’s suspend that assumption for a moment and instead, use the typical (crazy) assumptions of micro-economic models. Suppose all the bureaucrats enforcing a given policy (game wardens, medical referral reviewers, etc.) knew everything relevant to their decisions, including all the issues being considered by similar bureaucrats, and could see the implications of every choice.

    In this case, every bureaucrat could cut deals tailored to the individual circumstances of each moose owner, land owner, hospital, sick person, etc. while still preserving the effects of the global policy. Some otherwise unhappy citizens could be bought off with voluntary transfers (of money, services, alternative services, etc.) from others. Quite likely (but not necessarily) some people would remain dissatisfied, but surely far fewer. In addition, everyone could see that the decisions were closely tailored to circumstances, and if they viewed a range of decisions, they could probably see that it would be hard (by hypothesis, actually impossible) to improve on the local tradeoffs. So they’d be more inclined to accept their deals as “the best we all could do”.

    We know these assumptions are crazy. But they are crazy in exactly the same way as standard micro-economic models. To prove that markets “work”, the bodega operator on the corner is presumed to know everything relevant to his business and to fully calculate the effects of all his choices; we are just extending this generous assumption to bureaucrats. We now have the power to enrich our decision making with almost unlimited amounts of information, real time social networking, computation, etc. So maybe the micro-economic assumptions aren’t as crazy as they used to be.

    Thinking in these terms helps us see why a bureaucratic process, as Timothy says, “seems impoverished and cold compared to the vivid individuality of real people and real circumstances.” The problem isn’t mainly policy goals or the attempt to impose rational constraints on the situation. The problem is our (circumstantial) limits in matching up local variation with the demands of our global goal. This is not an in-principle problem of public vs. private, large vs. small etc. Instead it is basically a problem of data and computation, and perhaps techniques to prevent gaming.

    Intractability

    Conceivably we might run into in-principle problems of computational intractability, but that would need to be demonstrated and would be an interesting result. There are such intractability results for general micro-economic models, but it isn’t clear they apply to the much more limited cases of trying to manage moose, etc. Even if an exactly optimal outcome is intractable, likely we could find a tractable close approximation. If anyone cares I can dig up references; ask in comments or email me.

    Let’s compare this with a “free market” story, say about game management. In that story everyone could own their own moose, but we’d make sure they internalized all their externalities; if moose were giving each other diseases, we’d allocate the costs of diseases to the original sick moose, and so forth. This market story depends on just as many unrealistic assumptions about people, knowledge, calculation, etc. as “optimal bureaucracy” applied to game management — in fact it is arguably even less realistic, because we have to price the externalities correctly, impose the prices, and moose owners have to anticipate and respond to the possible costs correctly.

    So why do we like markets? To some extent they solve the information and calculation problem by aggregating choices into prices.

    When do markets work?

    The proof that free markets are optimal actually cheats by assuming every market participant knows everything and can calculate everything anyway! In many cases prices do usefully aggregate information and simplify calculation, but I don’t know of a strong analysis of where (and how) they actually work and where they don’t.

    More generally, though, markets create incentives for participants to locally optimize using abundant, cheap local data, and they aggregate those local optimizations (through prices) in ways that approximate a global optimum. (Of course often they totally screw things up in new ways, typically by incenting participants to pursue socially dysfunctional goals, some of which also systematically distort the social process to even more strongly favor dysfunctional ends. See ponzi schemes, patent medicines, marketing new drugs that are less effective than the old ones, lobbying and regulatory capture.)

    Happily we’re coming to understand how to do this sort of local optimization and aggregation without ownership and exchange. We all locally optimize and aggregate our ideo-dialects of our language, clothing styles, music choices, etc. The open source community has figured out how to locally optimize and aggregate software design and construction, and so forth. The web makes all of this easier and faster.

    Economic theory has focused on the exchange case, but markets are obviously derivative from the more general case. After all, markets arise from stable social arrangements, not the other way around, and these arrangements are stable because they have found local optima. In many ways exchange creates problems; for example, it creates opportunities to use bribes in one form or another.

    Given this analysis, how might we improve matters?

    How to get better at bureaucracy

    Historically we’ve found that large scale organizations, and setting and enforcing public policy gets us into these bureaucratic quandaries; but scale and public policy are unavoidable and we tend to figure we can’t do any better. If we realize the problem has been process limitations, and that now we can do better, we should devote more effort to process engineering. A better process would pull in more information and cognitive resources from the affected citizens and would organize their activities with constraints and incentives so they approximate the intended policy. We don’t (yet) have a good engineering approach to building and managing processes like this, but we surely we can improve current processes if we put our minds to it. One demonstration of the potential for improvement is the enormous differences in the effectiveness of complex organizations like hospitals — organizations which deliberately evolve their processes, monitoring and incorporating experience over time, can improve by orders of magnitude relative to those that don’t.

    Comment from T. Burke

    At this point I was very happy to get a comment from Timothy indicating we were in sync:

    I am really finding this a useful and thought-provoking way to circle back around the problem and come at it from some new angles. Thinking about open source as a generalized strategy or at least an insight to possible escapes from the public/private national/local is very stimulating. There’s something here about abandoning the kind of mastery and universalism that liberalism seems too attached to, while not abandoning a way of aggregating knowledge towards shared best practices (which include ethical/moral/social dispensations, not just technical ones).

    Maybe here it would help to think about why we keep getting stuck in this cul-de-sac. Bureaucracy is a highly evolved set of practices that maybe started in fertile crescent farm products management around 3,000 BCE.

    correction by G. Weaire

    Thanks to G. Weaire whose comment, in addition to raising fascinating issues, very gently corrected my overstatement of this period by 2,000 years.

    We’ve had plenty of time to figure out how to do things better but I can’t think of any historical societies that really got out of this bind. Even if some did, we have to grapple with why bureaucracy in basically all cultures today generates similar problems — of course with variations in corruption, efficiency, etc.

    The model for effective bureaucracy should perhaps be our other successful distributed negotiations. As I mentioned, we’re very good at “negotiating” changes in our language, social and cultural conventions, background assumptions, etc. etc. We’re so good at this that most of our negotiation is implicit and even unconscious.

    Is there theory?

    Elinor Ostrom analyzes the stable results of this sort of negotiation (as do Coase and others). But do we have any good models of the negotiation process itself? G. Weaire in his comment suggested “the sociolinguistics of politeness, esp. the still (I think) leading Brown-Levinson model. This tradition of inquiry is more-or-less entirely about trying to formalize an understanding of this sort of process at the level of conversational interaction.” He also mentioned “Michael Gagarin, Writing Greek Law… with its focus on highly formal public processes that aren’t bureaucratic but aren’t quite the village consensus either.” Luc Steels has simulated the negotiation of simple vocabularies during language formation…

    I believe these distributed negotiations are responsible for generating, shaping and maintaining essentially all of our institutions — replicated patterns of interaction — and thus our apparently stable social environment.

    So if we’re so good at this, why can’t we negotiate the enforcement of policy in the same way? I guess the main reason is that our negotiations operate in “consensus time” but bureaucratic processes have to operate in “transaction time”, and also need to maintain more detailed, reliable information than social memory typically does. When a farmer in Ur put grain into storage he needed a receipt right then, not when the village discussion could get around to it, and he needed a detailed stable record not whatever the members of the village could remember a few weeks or months later. So we got clerks making marks on a tablet, the rest is history.

    Could it really scale?

    G. Weaire commented that “the modern state has so much greater a bureaucratic capacity than any predecessor that it’s a difference of degree that adds up to a difference of kind, and that speaking of [5,000] years of bureaucracy maybe isn’t a helpful frame of reference.”

    He is right about a difference in scale of maybe six decimal orders of magnitude being certainly a difference in kind (from maybe 100 clerks in a city of several tens of thousands, to a hundred million or maybe even a billion bureaucrats of various flavors).

    However I think some important characteristics can persist even across such a great change. My own analogy here would be Turing’s original abstract machine compared with the one I’m using to write this. I’m sure the performance difference, storage capacity, etc. is at least as great. And Turing couldn’t anticipate huge differences in kind, such as the web (and its social consequences), open source, the conceptual problems of large scale software, etc. However even today everyone who works with computers, to a considerable extent, must learn to think the way Turing did.

    Similarly, the work of the clerk depended on social formations of fungibility of goods, identity of persons, standards of quantity and quality, etc. which are still the foundations of bureaucratic policy.

    So while it would be wrong to ignore this difference of kind, at the same time, I think there are important constraints that have stayed immutable from Ur until recently.

    I believe the limits on implementing a complex, widely distributed negotiation at transaction speed are mostly cognitive — humans just can’t learn quickly enough, keep enough in mind, make complex enough judgments, etc. As long as the process has to go through human minds at each step, and still has to run at transaction speed, bureaucracy (public or “private” — think of your favorite negotiation with a corporate behemoth) is the best we can do (sigh); we’re pretty much stuck with the tradeoff Timothy is talking about, and thus the perennial struggles.

    On the other hand, open peer production — open source, Wikipedia, etc. — seems to have partially gotten out of this bind by keeping the state of the negotiation mostly in the web, rather than in the participants heads.

    For example, on the web people negotiate largely through versioned (and often branching) repositories. These repositories can simultaneously contain all the alternatives in contention and make them easy to mutate, merge and experiment with. This option isn’t directly available to us for moose management

    check ‘em in!

    (though I enjoy the thought of checking all the moose, their owners, and the game management bureaucracy into git, and then instantiating modified versions of the whole lot on multiple branches)

    but examples like this suggest what may be possible going forward.

    The web also helps to make rapid distributed negotiation work through extreme transparency. Generally all the consequential interactions are on the public record as soon as they occur (in repositories or email archives). All the history is archived in public essentially forever, so is always available as a resource for analysis or bolstering or attacking a position. This has good effects on incentives, and also on the evolution of discourse norms.

    Evils of opacity

    The current financial system is pretty far from this, and is working hard to stay far away, by keeping transactions off exchanges, creating opaque securities, etc. As investigation proceeds, it seems more and more likely that the financial crisis would not have occurred if most transactions had been visible to other participants.

    We are in the process of generating transparency for a lot of existing bureaucratic processes and it probably can and should be made a universal norm for all of them (including game management). Note that simply having public records is not nearly enough — the records need to be on line, accessible without fees, and in a format consistent enough to be searchable. Then open content processes will tend to generate transparency for the process as a whole. There’s still a lot of contention around electronically accessible records — existing interests have thrown up all kinds of obstacles, including trade secrets (e.g. testing voting machines), copyright (e.g. building codes and legal records), refusal to convert to electronic form (e.g. legislative calendars), fees for access, etc. etc. But these excuses usually seem pretty absurd when made explicit, and they are gradually being ground down. Electronic transparency isn’t yet a social norm, but we seem to be slouching in that direction.

    My guess is that if we simply make any given bureaucratic process visible to all the participants through the web, it would evolve fairly quickly toward a much more flexible distributed negotiation. This would be fairly easy to try, technically; just put all the current cases, including correspondence etc. on a MediaWiki site, and keep them there as history when they are decided. The politics, privacy issues, etc. will be a lot more thorny. But it seems like an experiment worth trying.

    Open peer production also works because the payoffs for manipulating the system are generally very low. No one owns the content, and there’s no way for contributors to appropriate a significant share of the social benefits. There have been a few semi-successful cases where commercial enterprises manipulated open processes, such as the Rambus patent scam (essentially Rambus successfully promoted inclusion of ideas in standards, and only afterward revealed it had applicable patents). But these cases are rare and so far the relevant community has always been able to amend its practices fairly quickly and easily to prevent similar problems in the future.

    I’m much less clear how we can reduce the payoffs for manipulating social processes. In many cases (such as game management) payoffs are probably already pretty low. But in many important areas like finance and health care they are huge. My guess is that there are ways to restructure our institutions of ownership and control to improve matters but this will be a multi-decade struggle.

    Economic theory and policy failure

    Recently, we have seen repeated, massive failures of major policies focused on economic issues. For example, recent development policies advocated for countries in South America and Africa, the policies promoted by Western advisors during the conversion of the Soviet economies from communist to market-oriented, and the regulatory and risk management policies responsible for the still unfolding world-wide credit imbroglio.

    These policy failures have imposed immense suffering, excess death, and social and economic costs on their target populations. Furthermore, all of these policies were promoted as dictated by economic theory—and promoted not only by political snake oil salesmen, but by cadres of distinguished economic experts. Economic theory probably contributed significantly to the adoption of these policies by supporting passionate intensity, convincing rhetoric and intimidating mathematics. Complete lack of a theory for analyzing these policies, while far from ideal, would probably have saved many lives and much treasure.

    What was wrong? Economic theory in these cases actively suppressed consideration of institutional dynamics.

    These were not special cases. Classical economic theory is pervasively hostile to institutional dynamics:

    • It assumes a constant (and adequate) institutional backdrop. It assumes that transactions settle, contracts are enforced, theft is more costly than honest purchase, etc. It assumes this backdrop is not subject to manipulation by the market participants. Thus institutional dynamics are simply impossible in classical economic models.
    • It assumes transactions are evaluated in terms of one or a few scalar values — money, “utility”, etc. Basing all decisions on a scalar metric eliminates any “fine structure”, and participants in real transactions depend on this “fine structure” to negotiate their expectations. As a result, theoretical economic transactions lack any means to engage with institutional dynamics, although real economic transactions, of course, play a major role in institutional dynamics.

    These moves are justified as “idealizations” that make economic models tractable. However as with metaphors, idealizations must fit the task at hand or they are worse than useless. The policy failures mentioned above, as well as a wide range of other examples, indicate that the idealizations of classical economic theory are a very bad fit to many of the social policy tasks where economic models are typically invoked as justifications.

    Specifically, the effects of most public policy decisions depend critically on institutional issues — the institutions that will implement them, the institutions they will strengthen or weaken, and usually the institutional changes they will cause, intentionally or unintentionally.

    Economics aspires to judge the implications of a wide range of public policy proposals. But because it clings to a theoretical framework that idealizes away institutional dynamics, it actually cannot address critical aspects of any policy that is substantially affected by institutional issues—and that is most policies. Worse, economic models tend to distract from or entirely suppress deep analysis of institutional dynamics, so they actively damage policy discussions.

    One possible option for those who wish to preserve classical economic theory would be to restrict its application to cases where its idealizations do fit well enough to be useful. Certainly there are such cases. To exclude such domains as our modern financial systems from the purview of economic theory seems perverse, yet this would be an inevitable consequence of limiting current theory to domains where it works “well enough”.

    In practice institutional issues can’t be ignored in the vast majority of policy decisions. As a result, policy discussion tend to be defined in terms of economic “stories” about policy effects that are not theoretically valid, but that gain credence by invoking the impressive theoretical framework of classical economics. Actual arguments for or against a policy then resort to ad hoc models of institutional dynamics, retrofitted rhetorically and pragmatically to these economic “stories”. The dominance of the economic “story” obscures the ad hoc nature of the argument, avoids deep engagement with the models of institutional dynamics, and keeps the institutional analysis shallow and weak.

    Rhetorically effective but obviously mendacious examples of this are the policy arguments for “supply side economics”, school vouchers, increased “personal responsibility” in medical care, and unrestricted free trade. The arguments that led to disasters in recent development policy and financial market policy are more sophisticated but equally hollow.
    As mentioned above, the absence of a dominant theory would be less harmful in these cases than current use of economic theory. Unfortunately even the worst theories can only be displaced by other theories, so simply expelling classical economics from policy debates is not realistic. On the other hand, if we can find a theory that allows us to integrate and give proper weight to pragmatic arguments about institutional dynamics, and helps us develop stronger models as we go forward, then we have a reasonable chance to improve policy discussion.

    Happily, finding a theory that allows for institutional dynamics and still can match classical economics in its areas of strength is a realistic goal. We now have theoretical resources that allow us to subsume the results of classical economics models, and that can also be gracefully extended to model institutional dynamics. This is a strong claim, but it is justified by examples like Duncan Foley’s work in economics, and H. Peyton Young’s work in institutional dynamics.