The Principal-Agent Problem

13 October 2013

No Developer is an Island

People behave with hidden motivations and flawed reasoning. 

Let's look at one of the underlying causes behind why many employees, teams, and companies don't behave the way we want: the principal-agent problem.

I win even when you lose

"It is difficult to get a man to understand something when his salary depends on his not understanding it" - Upton Sinclair

The principal-agent problem arises when the person/group asking for help (the principal) has different incentives from the person/group offering help (the agent). It is even more common in situations where the agent has more expertise, such as a hired professional or specialist.

The reason is that behavior, both individual and collective, changes to follow incentives. Individual will, morality, ethics, and integrity are all altered by circumstance and incentives.   The principal-agent problem is already part of your life:

  • DBAs are rewarded when they keep production stable, even if it means delaying needed features.
  • Developers are rewarded when they ship features, even if it makes an existing application/system less stable.
  • PMs are rewarded for being 'more visible' or 'done early', even if the product is not what customers want.
  • School administrators are rewarded for better test scores, even if their students are less prepared to be citizens, innovators, adults

Going further, you find examples of the principal-agent problem leading to control fraud:

I win because you lose

  • Bankers are rewarded when they fleece their customers instead of helping them.
  • Real-estate agents earn more when they sell a more expensive house, even if it's more than the buyer can afford.
  • Salespeople earn bonuses when they promise features that don't exist, infuriating customers and their product teams.
  • Marketers undercut peoples' confidence, making them emotionally vulnerable and easier to sell to.
  • Universities build fancy buildings, dorms and stadiums to be more 'prestigious', and then raise tuition and increase student debt.
  • The US Congress creating laws benefiting lobbyists more than their own constituents. The principal-agent problem is a structural problem that contributes to corruption, dead businesses, and bloated institutions.

It is so pervasive that economics majors study its theory. I see the principal-agent problem as a force to be fought and minimized. We each have great potential to change our situation and the world around us, especially if we work hard and think carefully.

Context Is King

"Know your enemy and know yourself and you can fight a hundred battles without disaster" - Sun Tzu

To solve a problem you must know your options. To know your options you must know the context.

  • People are authentic, and sympathetic. Companies are not.
  • The downside to good customer service is fixed (fixing a person's problem). The downside to bad publicity is potentially immense.
  • Principals, often individuals, have little power or money on their own. Collectively, they control the fate of agents, and thus have immense leverage.
  • Legal recourse is often ruinously expensive and full of fine print and loopholes that favor agents.
  • Agents remove value from their products/services by adding complexity: loopholes, bureaucracy, and fees. It tilts the information-asymmetry problem in their favor.
  • Monopoly companies are exceptionally bad: the barriers to change are so much higher. It is hard to fight ISPs, governments, and utility companies because sympathetic people can't vote with their feet.
  • "Policies" are a bland term that mean "some middle manager decided not to do X because it's in our best interests"

1. Fight with Data

"The best revenge is to not be like your enemy." - Marcus Aurelius

You can fight the principal-agent problem with data. Since it often involves information asymmetry , having more information is empowering.

We do this ourselves; it is why popular sites have feedback systems: Yelp, TripAdvisor,  Zillow and Amazon.

A powerful way to use data is comparison shopping. We have options to acquire goods & services, and vote with our wallets. Doing so in an informed way helps you choose a superior product/service and support companies whose practices you like.

Luckily for principals, the amount of data and number of skilled analysts are increasing rapidly. However, the vast majority of valuable data is not public. Imagine if detailed insurance company data was public: customers would rapidly switch to the most customer-friendly companies. Companies seeking to maximize profit will hide critical data. Therefore we must get information from unconventional sources or public institutions.

Health Care

A brilliant report by Stephen Brill found that the medical costs are insane. This has been corroborated by many other independent journalists and bloggers. Drug makers make deals to prevent generic (i.e. cheaper) versions of their drugs. It's sometimes cheaper to pay cash. Visitors from other countries are often appalled by our health care system.

It is easy to skew numbers and hide inefficiency when data isn't available. Conversely, it's easy to make systems more efficient when data is available. I'm a big fan of startups that are working on this problem, like Castlight. However, even then the data is not really public.

Data is potentially powerful, and therefore potentially risky. Any individual, group or company with sufficient skill can use data for their own ends. This can be for good reasons, like making health care affordable for the average person, or for nefarious reasons, like figuring out who is more likely to have health issues, so they can be discriminated against (ahem, "priced appropriately").

Any effort to expose, assemble, or publicize data should involve thought about how it can be used for good or evil. Powerful organizations can handle the risk of embarrassing disclosure. Individuals can't.

2. Fight with Publicity

One of the big changes in the past decade is the rise of social networking. Facebook, Twitter, Tumblr, YouTube and Reddit enable a single person to tell their story and have that story visible to the entire worldAmazonYelp, and eBay and other sites enable a person to write a review that can influence many future purchases.

Publicity works because of human psychology. We are a social, tribal species. We are identify with people who seem like us. We love the underdog. Sympathy is powerful.

I've never met a company or PR agent that is as sympathetic as a normal person. Companies in general, and PR agents in particular, aren't authentic. Much of that is cultural; the language that agents use is framed by their worldview (profit, marketing, damage control, shareholder value, etc). I'm glad; it makes fighting the principal-agent problem with publicity really easy.

Publicity works for a second reason: unequal risk. Bad publicity costs far, far more than good customer service does. This is doubly true if an agent lies, argues with their customers, and denies things they have done.

Let's look at two examples.

Law Enforcement Corruption

Let's look at when agents have all of the power. Cops can use "asset forfeiture" to seize whatever they want, with low risk of punishment.

Individuals (the principals) in this situation have minimal power. The average person can't afford lawyers to fight police corruption. However, individuals collectively have massive power; they can vote out the politicians running these towns and states. Even the threat to do so (opinion polls) has tremendous clout.

Lost Baggage

British Airways (BA) lost somebody's luggage (newsworthy, I know). Unlike an email to the company (easily ignored), that man's son, @HVSVN bought $1,000 in Twitter ads to publicize his complaint.

Let's look at what happened:

Primary Effects

Let's guess that 1% of the 70K Twitter users were also BA customers, and the average ticket costs around $1000.

  • Lost revenue: $700,000

Secondary Effects

Let's guess that each person knows ~150 people. We can guess the impact:

700 People * 
150 People in Network *
50% Chance of Mentioning *
$1000 ticket price
20% are BA customers =
$10.5 million
  • Secondary lost revenue: $10.5 million
  • Total Lost Revenue: $11.2 million
  • Advertising ROI: 11200:1 (1,120,000%)
  • British Airways annual revenue: ~$17.3 billion
  • Agent:principal income ratio: 302,000:1
  • Agent:principal income ratio with advertising ROI: ~1545:1

It's still not an even playing field. But the odds are better.

The Lesson

Businesses care about money. People care about their jobs. Publicity gives a principal leverage against an agent's primary weak spot: their wallets.

While researching this post, I found that most successful publicity campaigns had common elements:

  1. There was evidence. Document everything; use cameras, smartphones, copies of documents, screenshots, Google Glasses, scene re-creationseverything
  2. An honest, compelling story
  3. A catchy headline
  4. Social media. Twitter is great. So are Facebook and Reddit.

3. Fight In Groups

"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has." - Margaret Mead

The third way to fight the principal-agent problem is with other people. Other people have skills, ideas and connections you don't. A small team is far more capable than an individual.

The Internet makes organizing far easier than ever before. Anyone can make a site, and anyone else can find it. It's relatively easy to identify competent professionals and bypass layers of middlemen / bureaucracy.

There are already sites that act as online watering holes and gathering places:

Online gathering places are still in their infancy. Here are some examples of what I can't find:

  • Collaborative analysis of legal cases
  • A gathering place for medical professionals.
  • A gathering place for professors of various disciplines.
  • Collaborative development and feedback of political data.

The tools for Internet-based gathering places are largely mature:

  • The ability to create and edit information in groups (Google Docs, wikis)
  • The ability for people to provide feedback in a productive way (Discourse, reddit's karma points, StackOverflow's voting system)
  • The ability for people looking for specialized information and collaborators to find them (Google Search)

Let's look at another area with a bad principal-agent problem: finance.

In the US, the financial system is huge and corrupt. Here's a partial list of what financial institutions have done over the last 10 years:

The power and influence that exists in finance comes from the collective money of the average person (i.e. principals). It's a horrific example of the principle-agent problem.

It's the Incentives, Stupid

If you can't fight agents, pick different ones. I've found a great alternative to banks: credit unions (which have higher satisfaction ratings than banks).

Financial institutions are vulnerable to customers moving their money somewhere else. Their influence comes from our checking accounts, credit cards, loans, and 401Ks. Move it, and you end up with better service and less organizational corruption. Win win. One recent effort has been the Move Your Money project.

Let's look at why credit unions are better for customers than banks.

People behave according to incentives. Moral courage is admirable, and rare. Incentives encourage what they measure, and care must be taken to avoid unintended consequences.

Here are my questions to identify incentives when choosing an agent:

  • Can the company (agent) make more money by providing bad/no customer service?
  • Does the company/agent pay their employees well? If they're not willing to pay their employees well, how will they treat their customers?
  • Are the executives given pay raises & bonuses when the company loses money or market share?
  • Do customers influence the hiring/firing of executives? How is bad behavior prevented?
  • Is there effective oversight? Does the industry try to capture regulators?
  • Is there an incentive for an agent to sell me something I don't want or need? (e.g. real estate agents, the entire wedding industry)
  • What are the industry's success metrics? How can they be twisted?
  • Does the company try and produce more profit each quarter as a percentage of revenue? How do they do that and maintain quality?
  • Have other people had a bad experience? How bad? What has been done to provide reparations?
  • How easy is it to move to a different agent? How many other options are there?
  • How many of the company's press releases mention "focus on the bottom line", "maximizing shareholder value", "profit", "delighting customers" and other misleading phrases?
  • Can I create custom incentives? Some great examples are deferred compensation, flat fees, and profit-sharing arrangements.

When looking at banks and credit unions, incentives are the big difference. Banks are designed to take money from their customers, as profit. The customers aren't owners. In contrast, credit unions have aligned incentives. They're nonprofits. The customers are owners. Plus, customers can vote to fire the the board of directors and top executives.

Here are types of organizations with good incentives:

Now What?

We are all principals and agents. As principals, we want the best service for the price. As agents, we should provide the best service we can.

Since we are more often principals than agents, we should remember tactics to fight the principal agent problem:

  1. Know the context
  2. Fight with data
  3. Fight with publicity
  4. Fight with others
  5. Look at the incentives

Good luck!


SQL Saturday Redmond - Machine Learning for Mere Mortals

10 April 2013

I am privileged to say that I have been chosen to give a presentation at SQL Saturday 212, in Redmond WA, on May 18, 2013.

The topic? Machine Learning for Mere Mortals.

This will be a gentle introduction to the world of machine learning. I will be covering topics including supervised vs. unsupervised learning, clustering, decision trees, association rules, and learning at scale. I shall also provide examples and demos of real-world machine learning applications.

Finally, I'll provide resources to help people get started.

I hope to see you there!