What are You Doing that is Causing Unexpected Harm?

Do you measure the outcomes of your actions and decisions? This week’s fascinating Freakonomics podcast was loaded with lessons on doing so. But not the lessons you might expect.

If you have an hour available, it’s well worth a listen: http://freakonomics.com/podcast/when-helping-hurts/

Without being too much of a spoiler, the episode tells the story of a well-intentioned mentorship program for at-risk boys. Not only did the program not help, it actually hurt. But no one knew that for decades, because it seemed to be obvious, common sense that the program must help.

Several important lessons for mission-driven organizations came to mind as I was listening:

1) Don’t assume – measure. “It ain’t so much the things we don’t know that get us into trouble, it’s the things we do know that just ain’t so." No, that is not a Mark Twain quote. Nor Will Rogers. Nor Abraham Lincoln. The delicious irony of that quote is that it is so often misattributed.*

Unfortunately, it’s not so “delicious” when real people suffer because well-intentioned benefactors assume that what they are doing must be good. Assumptions are not knowledge. Frequently, our assumptions are just flat wrong. And even when they are right on one aspect, there are almost always unintended consequences to our actions that we never consider. It is important to any organization, business or nonprofit, to measure the actual impact of what they do, no matter how “obvious” the benefit may be.

2) Be careful of what you measure – and of what you do with the measurements. How could measuring outcomes be bad? Well, as Father Greg Boyle points out in the podcast, “If you are driven by outcomes, then you’re going to only work with a population that will give you good ones.” The hardest cases, arguably the kids who need the most help, would be ignored.

There’s an old saying that what gets measured, gets done. You have to be careful about choosing what gets measured (make sure it’s something related to what you want to achieve). And you have to be careful about what gets done to achieve the desired measures. People will take action to achieve the measurement targets, but those actions may not be what you actually want them to do. Design your measures, and any incentives tied to them, in such a way as to deliver the decisions, actions, and behaviors that fit your mission.

3) Open your mind to the data. I can only imagine the nauseated feeling that seeing the data would have produced in the researchers’ stomachs. How can it be true?

It must have been tempting to ignore the findings, or force-fit the data to say something that it didn’t. It takes courage to see data that contradicts everything you believe and expect, and acknowledge its validity. It takes guts to admit that your beliefs were wrong. It takes humility to accept the new information and adjust your mindset. But doing so is what will drive genuinely better outcomes.

*If you are interested in research on that quote, check out “The Quote Investigator” and his research. Thank you, Garson O’Toole!