Of moose and men

In a post several weeks ago, One Man’s Moose, Timothy Burke discussed the social tradeoffs between regulation and respect for individual desires and needs. That post came out of a larger web discussion on game management in Vermont, and resulting conflicts with people who keep moose as pets. Timothy summarized a key point:

If you start cutting separate deals with everyone who pleads that their circumstances are special, that a legitimate attempt to safeguard the public shouldn’t apply to them, you’ll end up with a public policy that applies to no one.
I reacted strongly to that general point; I’m posting a reworked version of my comments here.

Of course Timothy’s summary is a more detailed statement of the classic bureaucratic argument, “If we let you do it, we’d have to let everyone do it”; living in a complex society we encounter this constraint on our liberty implicitly or explicitly many times a day.

The basic point is tough to dispute. But the way it typically plays out, for example in Timothy’s quote, relies on an implicit assumption about the “cognitive” limitations of bureaucracy. We assume that the bureaucrat can only use fairly simple rules based on local information. As a practical matter this has been true of bureaucrats for the last several thousand years, so this assumption has gotten deeply embedded. But maybe it isn’t true anymore.

Let’s suspend that assumption for a moment and instead, use the typical (crazy) assumptions of micro-economic models. Suppose all the bureaucrats enforcing a given policy (game wardens, medical referral reviewers, etc.) knew everything relevant to their decisions, including all the issues being considered by similar bureaucrats, and could see the implications of every choice.

In this case, every bureaucrat could cut deals tailored to the individual circumstances of each moose owner, land owner, hospital, sick person, etc. while still preserving the effects of the global policy. Some otherwise unhappy citizens could be bought off with voluntary transfers (of money, services, alternative services, etc.) from others. Quite likely (but not necessarily) some people would remain dissatisfied, but surely far fewer. In addition, everyone could see that the decisions were closely tailored to circumstances, and if they viewed a range of decisions, they could probably see that it would be hard (by hypothesis, actually impossible) to improve on the local tradeoffs. So they’d be more inclined to accept their deals as “the best we all could do”.

We know these assumptions are crazy. But they are crazy in exactly the same way as standard micro-economic models. To prove that markets “work”, the bodega operator on the corner is presumed to know everything relevant to his business and to fully calculate the effects of all his choices; we are just extending this generous assumption to bureaucrats. We now have the power to enrich our decision making with almost unlimited amounts of information, real time social networking, computation, etc. So maybe the micro-economic assumptions aren’t as crazy as they used to be.

Thinking in these terms helps us see why a bureaucratic process, as Timothy says, “seems impoverished and cold compared to the vivid individuality of real people and real circumstances.” The problem isn’t mainly policy goals or the attempt to impose rational constraints on the situation. The problem is our (circumstantial) limits in matching up local variation with the demands of our global goal. This is not an in-principle problem of public vs. private, large vs. small etc. Instead it is basically a problem of data and computation, and perhaps techniques to prevent gaming.

Intractability

Conceivably we might run into in-principle problems of computational intractability, but that would need to be demonstrated and would be an interesting result. There are such intractability results for general micro-economic models, but it isn’t clear they apply to the much more limited cases of trying to manage moose, etc. Even if an exactly optimal outcome is intractable, likely we could find a tractable close approximation. If anyone cares I can dig up references; ask in comments or email me.

Let’s compare this with a “free market” story, say about game management. In that story everyone could own their own moose, but we’d make sure they internalized all their externalities; if moose were giving each other diseases, we’d allocate the costs of diseases to the original sick moose, and so forth. This market story depends on just as many unrealistic assumptions about people, knowledge, calculation, etc. as “optimal bureaucracy” applied to game management — in fact it is arguably even less realistic, because we have to price the externalities correctly, impose the prices, and moose owners have to anticipate and respond to the possible costs correctly.

So why do we like markets? To some extent they solve the information and calculation problem by aggregating choices into prices.

When do markets work?

The proof that free markets are optimal actually cheats by assuming every market participant knows everything and can calculate everything anyway! In many cases prices do usefully aggregate information and simplify calculation, but I don’t know of a strong analysis of where (and how) they actually work and where they don’t.

More generally, though, markets create incentives for participants to locally optimize using abundant, cheap local data, and they aggregate those local optimizations (through prices) in ways that approximate a global optimum. (Of course often they totally screw things up in new ways, typically by incenting participants to pursue socially dysfunctional goals, some of which also systematically distort the social process to even more strongly favor dysfunctional ends. See ponzi schemes, patent medicines, marketing new drugs that are less effective than the old ones, lobbying and regulatory capture.)

Happily we’re coming to understand how to do this sort of local optimization and aggregation without ownership and exchange. We all locally optimize and aggregate our ideo-dialects of our language, clothing styles, music choices, etc. The open source community has figured out how to locally optimize and aggregate software design and construction, and so forth. The web makes all of this easier and faster.

Economic theory has focused on the exchange case, but markets are obviously derivative from the more general case. After all, markets arise from stable social arrangements, not the other way around, and these arrangements are stable because they have found local optima. In many ways exchange creates problems; for example, it creates opportunities to use bribes in one form or another.

Given this analysis, how might we improve matters?

How to get better at bureaucracy

Historically we’ve found that large scale organizations, and setting and enforcing public policy gets us into these bureaucratic quandaries; but scale and public policy are unavoidable and we tend to figure we can’t do any better. If we realize the problem has been process limitations, and that now we can do better, we should devote more effort to process engineering. A better process would pull in more information and cognitive resources from the affected citizens and would organize their activities with constraints and incentives so they approximate the intended policy. We don’t (yet) have a good engineering approach to building and managing processes like this, but we surely we can improve current processes if we put our minds to it. One demonstration of the potential for improvement is the enormous differences in the effectiveness of complex organizations like hospitals — organizations which deliberately evolve their processes, monitoring and incorporating experience over time, can improve by orders of magnitude relative to those that don’t.

Comment from T. Burke

At this point I was very happy to get a comment from Timothy indicating we were in sync:

I am really finding this a useful and thought-provoking way to circle back around the problem and come at it from some new angles. Thinking about open source as a generalized strategy or at least an insight to possible escapes from the public/private national/local is very stimulating. There’s something here about abandoning the kind of mastery and universalism that liberalism seems too attached to, while not abandoning a way of aggregating knowledge towards shared best practices (which include ethical/moral/social dispensations, not just technical ones).

Maybe here it would help to think about why we keep getting stuck in this cul-de-sac. Bureaucracy is a highly evolved set of practices that maybe started in fertile crescent farm products management around 3,000 BCE.

correction by G. Weaire

Thanks to G. Weaire whose comment, in addition to raising fascinating issues, very gently corrected my overstatement of this period by 2,000 years.

We’ve had plenty of time to figure out how to do things better but I can’t think of any historical societies that really got out of this bind. Even if some did, we have to grapple with why bureaucracy in basically all cultures today generates similar problems — of course with variations in corruption, efficiency, etc.

The model for effective bureaucracy should perhaps be our other successful distributed negotiations. As I mentioned, we’re very good at “negotiating” changes in our language, social and cultural conventions, background assumptions, etc. etc. We’re so good at this that most of our negotiation is implicit and even unconscious.

Is there theory?

Elinor Ostrom analyzes the stable results of this sort of negotiation (as do Coase and others). But do we have any good models of the negotiation process itself? G. Weaire in his comment suggested “the sociolinguistics of politeness, esp. the still (I think) leading Brown-Levinson model. This tradition of inquiry is more-or-less entirely about trying to formalize an understanding of this sort of process at the level of conversational interaction.” He also mentioned “Michael Gagarin, Writing Greek Law… with its focus on highly formal public processes that aren’t bureaucratic but aren’t quite the village consensus either.” Luc Steels has simulated the negotiation of simple vocabularies during language formation…

I believe these distributed negotiations are responsible for generating, shaping and maintaining essentially all of our institutions — replicated patterns of interaction — and thus our apparently stable social environment.

So if we’re so good at this, why can’t we negotiate the enforcement of policy in the same way? I guess the main reason is that our negotiations operate in “consensus time” but bureaucratic processes have to operate in “transaction time”, and also need to maintain more detailed, reliable information than social memory typically does. When a farmer in Ur put grain into storage he needed a receipt right then, not when the village discussion could get around to it, and he needed a detailed stable record not whatever the members of the village could remember a few weeks or months later. So we got clerks making marks on a tablet, the rest is history.

Could it really scale?

G. Weaire commented that “the modern state has so much greater a bureaucratic capacity than any predecessor that it’s a difference of degree that adds up to a difference of kind, and that speaking of [5,000] years of bureaucracy maybe isn’t a helpful frame of reference.”

He is right about a difference in scale of maybe six decimal orders of magnitude being certainly a difference in kind (from maybe 100 clerks in a city of several tens of thousands, to a hundred million or maybe even a billion bureaucrats of various flavors).

However I think some important characteristics can persist even across such a great change. My own analogy here would be Turing’s original abstract machine compared with the one I’m using to write this. I’m sure the performance difference, storage capacity, etc. is at least as great. And Turing couldn’t anticipate huge differences in kind, such as the web (and its social consequences), open source, the conceptual problems of large scale software, etc. However even today everyone who works with computers, to a considerable extent, must learn to think the way Turing did.

Similarly, the work of the clerk depended on social formations of fungibility of goods, identity of persons, standards of quantity and quality, etc. which are still the foundations of bureaucratic policy.

So while it would be wrong to ignore this difference of kind, at the same time, I think there are important constraints that have stayed immutable from Ur until recently.

I believe the limits on implementing a complex, widely distributed negotiation at transaction speed are mostly cognitive — humans just can’t learn quickly enough, keep enough in mind, make complex enough judgments, etc. As long as the process has to go through human minds at each step, and still has to run at transaction speed, bureaucracy (public or “private” — think of your favorite negotiation with a corporate behemoth) is the best we can do (sigh); we’re pretty much stuck with the tradeoff Timothy is talking about, and thus the perennial struggles.

On the other hand, open peer production — open source, Wikipedia, etc. — seems to have partially gotten out of this bind by keeping the state of the negotiation mostly in the web, rather than in the participants heads.

For example, on the web people negotiate largely through versioned (and often branching) repositories. These repositories can simultaneously contain all the alternatives in contention and make them easy to mutate, merge and experiment with. This option isn’t directly available to us for moose management

check ‘em in!

(though I enjoy the thought of checking all the moose, their owners, and the game management bureaucracy into git, and then instantiating modified versions of the whole lot on multiple branches)

but examples like this suggest what may be possible going forward.

The web also helps to make rapid distributed negotiation work through extreme transparency. Generally all the consequential interactions are on the public record as soon as they occur (in repositories or email archives). All the history is archived in public essentially forever, so is always available as a resource for analysis or bolstering or attacking a position. This has good effects on incentives, and also on the evolution of discourse norms.

Evils of opacity

The current financial system is pretty far from this, and is working hard to stay far away, by keeping transactions off exchanges, creating opaque securities, etc. As investigation proceeds, it seems more and more likely that the financial crisis would not have occurred if most transactions had been visible to other participants.

We are in the process of generating transparency for a lot of existing bureaucratic processes and it probably can and should be made a universal norm for all of them (including game management). Note that simply having public records is not nearly enough — the records need to be on line, accessible without fees, and in a format consistent enough to be searchable. Then open content processes will tend to generate transparency for the process as a whole. There’s still a lot of contention around electronically accessible records — existing interests have thrown up all kinds of obstacles, including trade secrets (e.g. testing voting machines), copyright (e.g. building codes and legal records), refusal to convert to electronic form (e.g. legislative calendars), fees for access, etc. etc. But these excuses usually seem pretty absurd when made explicit, and they are gradually being ground down. Electronic transparency isn’t yet a social norm, but we seem to be slouching in that direction.

My guess is that if we simply make any given bureaucratic process visible to all the participants through the web, it would evolve fairly quickly toward a much more flexible distributed negotiation. This would be fairly easy to try, technically; just put all the current cases, including correspondence etc. on a MediaWiki site, and keep them there as history when they are decided. The politics, privacy issues, etc. will be a lot more thorny. But it seems like an experiment worth trying.

Open peer production also works because the payoffs for manipulating the system are generally very low. No one owns the content, and there’s no way for contributors to appropriate a significant share of the social benefits. There have been a few semi-successful cases where commercial enterprises manipulated open processes, such as the Rambus patent scam (essentially Rambus successfully promoted inclusion of ideas in standards, and only afterward revealed it had applicable patents). But these cases are rare and so far the relevant community has always been able to amend its practices fairly quickly and easily to prevent similar problems in the future.

I’m much less clear how we can reduce the payoffs for manipulating social processes. In many cases (such as game management) payoffs are probably already pretty low. But in many important areas like finance and health care they are huge. My guess is that there are ways to restructure our institutions of ownership and control to improve matters but this will be a multi-decade struggle.