We wholeheartedly support rigor in assessment, but were nervous about the author’s suggestion that organizations focus their efforts purely on monitoring implementation. Here is Executive Director Liana Downey’s response:
Thanks Caroline for raising such an important issue and bringing attention to the importance of really doing one’s research before rolling out large-scale initiatives. Your article raises a number of incredibly important points. I wish the example you shared were the only ones, but of course there are many examples of well-intentioned groups “jumping on the bandwagon” of one approach or another, at enormous cost and investment, to only find, years later, that the outcome they are seeking to drive is not being realized.
Having said that – I am very nervous about the impact of your message on many organizations. I have heard time and time again from nonprofit leaders that measuring is “all too hard”. While at the behest of funders there has been some shift towards better accountability for outcomes, the reality is that there are hundreds of thousands of non-profits in America alone who do not measure their impact in any meaningful way.
My experience is that these organizations are doing so NOT because they are dismissive of the scientific method. In fact, it is precisely because they are so convinced that the ONLY meaningful research is a peer-review control study, that they do nothing, as Megan Golden and I wrote about (in Just Do It). Of course you are right, such research is the gold standard, and well it should be. But suggesting that nonprofits wait around for someone else to do the research and challenge themselves only to monitor implementation (as per your previous articles) has me breaking out in sweats.
I’ve witnessed the enormous power of encouraging organizations to take on responsibility for asking the question – is what we are doing working for our clients?
When organizations start to monitor outcomes (not process), they start to ask much smarter questions. They start to ask not just “how many attendees did you have at your after school program,” but “what was the change in (obesity), (vocabulary), (school attendance) across different program areas?” When they do that, they notice differences. These observed differences then lead to meaningful conversations – which drives critical innovation and improvement. While of course in the day-to-day running of a program it is difficult to account for a range of control variables with a great degree of rigor, powerful insights are still generated. It also shifts the dynamic of staff on the ground to actually drive innovation and accountability.
Another example is how this can play out in the medical world. Of course hospitals implement only clinically proven and safe interventions. They don’t feel obliged to run control studies at every turn. But even so, if they do not hold themselves accountable for monitoring outcomes, they put people’s lives at risk.
While at McKinsey, I worked with hospital management teams to collate and review comparative data on outcomes across a range of standardized operations. What we saw were meaningful variations in life/death outcomes. This information of course drove much deeper investigation, uncovering both positive (and negative) innovations by surgeons and nurses. By encouraging management to track and understand these differences, all kinds of innovations and improvements can be made which literally save people’s lives. Sometimes it is implementation (hand washing), sometimes it is innovation (checklists a la Atul Gawande). But if an organization does as you suggest and holds itself accountable only for monitoring implementation, there is little incentive to innovate and improve.
You are right on so many fronts. Funders should think about getting behind testing interventions in a rigorous way. Information should be made open and shared. But I believe organizations should absolutely be encouraged and supported to check whether their work is making a difference in people’s lives.
Liana Downey is an experienced management consultant, who has consistently delivered results for clients on critical topics around the world. Liana is an expert advisor to the Australian Department of Prime Minister and Cabinet. Prior to establishing Liana Downey & Associates, she led McKinsey & Company’s Australian government and social-sector practices, and holds an MBA (Public Management) from Stanford University.