Frustration Effects and Curse of Optimality

by Venkat on May 19, 2014

Let’s say you have a project to staff with three available roles: a leadership role P with power, a sexy role S with opportunity for high public visibility, and a grinder role G with a lot of tedious schlepping. For logistics reasons, the partitioning of the work is not negotiable. You have three people with whom to staff the project: Alice, Bob and Charlie.

You chat with each, and it’s clear they all have the same preference order of roles: P>S>G, which means there’s no way to satisfy them all perfectly. All three believe they can do all three roles well enough. So you sit back, think through how good each is at each role, make up a little table like the one below,  crunch some numbers and assign roles: Alice gets power, Bob gets the sexy role, Charlie gets the grinder role. Your configuration has a nominal value of 5+4+2=11 points, and is the best you can do among all possible configurations.

Skill\Person Alice Bob Charlie
Power  5  4  3
Sexy  3  4  1
Grinder  3  4  2

Unfortunately, each also has an unknown motivational drop-off element to their personalities, due to which their commitment and productivity drops by at least a certain fraction for every degree removed from most-preferred role. So how does that change the actual outcome?

Let’s make it specific.

  • Alice is the best leader, but also a trooper, and has a drop-off fraction of only 10%.
  • Bob is a power-hungry jerk and craves visibility, and his fraction is 70%.
  • Charlie’s is an average schmo with a drop-off rate of 50%.

Depending on these rates, every configuration will lead to a certain amount of net frustration which will cause the system to move into a modified configuration that shifts the global optimum. This happens via a process of “settling” along configuration dimensions that were unused in the design of the nominal optimum.  Here’s the “frustration map” I computed for our simple, 3-person, 3-known-factors, 1-unknown-factor system.

heatMap

The more red the row corresponding to a configuration, the more it will shift through settling. The greener it is, the less it will shift through settling. So your effective score for Row 1, after settling in response to frustration is no longer 11, but 5+4*0.3+2*(0.5)^2 = 6.7. (off by 4.3). If you did the calculation for all possible configurations, you’d find you now have a better possibility: if Bob gets power, Charlie gets the sexy role and Alice gets the grinder role, the nominal score is 9, but effective score 7.2 (I am assuming here the drift stops after one round of slacking, which is actually never the case: the settling/drift is an iterative process).

Your optimum has shifted due to compensating frustration effects triggered by implementing your nominally optimal solution. This is one effect of what I call the curse of optimality: high-dimensional systems drifting/settling off target in response to lower-dimensional optimization attempts, undermining the very optimality that was sought and hoped for. It is a particular consequence of the broader principle known as the curse of dimensionality.

The drift almost always makes the system worse in both a relative sense (underwhelming performance relative to design) and absolute sense (the pattern of under-performance has a decent chance of shifting the global optimum).  If the unmodeled dynamics involve agency, as in this case, the curse becomes exponentially worse. That’s not rhetorical adjective: modeling agency in even a minimally realistic way typically introduces exponential drift processes. So designed organizations don’t just “sag” a little as they settle, like buildings aging, but head off in pretty unpredictable directions. The qualitative reason is that “frustration” represents unsatisfied desires that find expression by using unused degrees of freedom to relieve the frustration at the expense of the optimality. In the case of intelligent agents, it can find creative expression, not just knee-jerk ones like slacking off.

In other words, dissatisfied people always at least slack off in proportion to being denied what they want. Smart and dissatisfied people work actively to subvert the system in the most creative ways they can think of. Smart and angry people might even work to blow things up entirely.

This is the absolute simplest illustration I could dream up, but the curse, frustration effects and settling patterns can get arbitrarily complex as you refine the model (something I’ve been working on off and on over the years).  The potential for complexity is so high that “organization design” in the naive sense the term is used these days is almost always a godawful joke. The more agency and autonomy there is in the organization, the less funny the joke and the more toxic the curse.

There are ways to turn the curse of optimality into a blessing. The trick is artistically devolving autonomy in ways that make people rein in their own frustration and actively seek out positive ways of using available slack in the system. This is, roughly speaking, the logic of the Blitzkrieg model of Schewrpunkt, Auftragstaktik, Fingerspitzengehful and Einheit.

But anybody who claims there is a systematic and simple process for achieving this devolution is either lying, mathematically challenged, or has never worked in a real organization.

You can get serendipitous over-performance via positive drift rather than underperformance and/or suboptimality through negative drift. But it is definitely an art rather than a science. One I spend a lot of time thinking about, and have idle hopes of some day turning into a science.

Share

Previous post:

Next post: