TBM 19/52: The Analytics ROI Paradox

Contrived thought experiment.

You have a choice. You can add two engineers and one designer to your team of one-hundred. Yearly payroll will increase from $35MM to $36MM (+$1MM). Or you can invest $1M/year in analytics software. The analytics software will help one-hundred people make better (and faster) decisions. What do you do?

A common response to the question goes something like this:

Well, look, talented people will always get my vote. They can ship actual product. How will I explain spending one million dollars on tools when we have so much work to do? Leadership needs us to execute. We've committed to a roadmap. We aren't flying completely blind either. It is messy, and takes a while, but we can dig up some data if we need it.

At the day-job (Amplitude), I end up talking to lots of teams about how they view the ROI of analytics. I'm fascinated by these conversations. One theme stands out. Many teams have very little sense of their current impact. So the idea of improving decision quality doesn't resonate.

It's a paradox of sorts. Without situational awareness, the idea of improving situational awareness feels distant and abstract. The tendency to focus on output (and other proxy measures) increases. Success theater reigns. Output—building more stuff—becomes the currency of progress. It all gets baked into the culture.

What I find so interesting is that teams habituate to this type of thinking. They seem to forget they are making decisions in the first place. A one-hundred person team will make tens of thousands of decisions in a year. Even a small improvement could mean a huge difference in outcomes (compounding improved decision quality FTW). The $12.5MM spent in pre-scheduled meetings could be made more effective too! Yet somehow the very act of entertaining that thought causes a certain amount of dissonance. The prospect of improvement implies we have room for improvement…which is a scary thought.

Consider two teams (I have two real teams from two similar companies in mind here):

  • Team A works in fairly large batches. Their work is decent (but they don’t really know that, because they aren’t reflecting on impact). Once they are done with one thing, they jump to the next “project”

  • Team B works in smaller batches, pivots off of losing bets early, and uses that information to achieve better-than-decent outcomes. They reflect on decision quality and impact.

It all adds up! In the real world, Team B isn’t more skilled or talented. They do, however, work differently and have better tooling. My rough guess is that they produce ~2x better outcomes. This type of difference between teams is common (and sometimes much more pronounced).

Nothing ground-breaking this week, but I think what inspired me to write this was the paradox discussed above. Namely…to see the value of something that improves X, we need to have a sense of X. And this is where many orgs struggle.

If you are a change agent struggling to persuade your org to invest in better tooling (be it related to quantitative OR qualitative insights), one tip is to start with the basic argument that the team makes thousands of decisions, and that marginal gains could have an outsized impact. Even basic reflections—we expected this might happen, but this happened instead—could go a long way.