The Drunkard's Search

28 December 2016

There are many ways to use data to answer questions. I love these questions; they reveal the breadth of interest and depths of curiosity that people have.

However, the most common scenario is cringe-inducing.

Friend: So, our [Insert VIP Title] announced that we're doing a Big Data project.

Me: Oh, interesting. What big questions/headaches are you thinking about?

Friend: We're not there yet. First we want to know what we can do with all this data (web logs / network traffic / payroll / customer surveys ). 

Me: ...

Cart before horse

The purpose of a data science project is to change something. The most important things you can change are usually not in the biggest data set. Looking at the most convenient data set(or the largest) is a drunkard’s search.

Drunkard's search

Let’s look at an example. Several universities are collecting and analyzing network traffic to identify students who will fail their classes. The idea is to identify when a student’s phone/laptop is still at their dorm and not in class.

Sure, you can do that. A consultant will cheerfully sell you a giant ‘Big Data’ stack when you can use a small data set you already have (student transcripts) to achieve similar results faster, and at 1/1000th the price.

Your Brain Lies to You

Where you look for information (which streetlight to use) dramatically influences the decisions you can make.

Some other common biases are to focus on what you think of first, over-focus on recent details, avoid emotionally negative information, react differently depending on the ‘frame’ of conversation, and to follow paths familiar to us.

It turns out cognitive bias is everywhere. It is the work of a lifetime to ‘see’ these blind spots, and to compensate for them.

Good Luck!


Air Travel

11 January 2016

Plane flights are uncomfortable. However, they are supremely useful; they enable me to go to a completely different culture in hours. The main costs are money, invasions of privacy, and being forced to LARP as a sardine for a day.

Since I can only afford to travel economy cramped class. I crowd sourced advice on how to make long flights more bearable.


I’m intrigued by the lack of comments about hidden factors, such as:

It’s part of human nature to ignore what is working, and air travel is very safe. I find it bemusing when important factors (safety, reliability, pilot resilience) are mentioned less than “Bring Ambien! Books! Nachos!”

Advice, Distilled

  1. Bring things to pass the time. Books, movies and games are popular choices
  2. Sleep on the flight. Bring things to help with that (pills, pillows, earplugs)
  3. Make yourself more comfortable physically (snacks and slightly larger seats)

Happy Flying!