The limits of  Return on Investment analysis

Not that long ago the world was clamouring for good evidence the return on investment for public health interventions. 
There was a great response and the world is now full of ROI tools, models and papers.

I won’t rehash those here. You can do your own googling.

One great example

A particularly good ROI tool was published recently by the SW London Academic Health and Social Care System. It builds on work previously done by the Kings Fund and others. 

Making the case for public health interventions

It’s really good, well worth having in your toolbox.
Covers a lot of territory, often in space that’s more social policy than “public health” (depending if you have a contemporary understanding of those two words).  In quite a nuanced way. 

The caveats on using ROI models.
Nobody likes pages and pages of caveats. The small print IS important. 

We all want the answer. It doesn’t exist. Answers come with caveats!

As David Buck often points out we need to be careful that we use these wisely and don’t make the playing field even more uneven than it already is by focusing attention on the need for ROI for “prevention” whilst ignoring other areas of resource commitment. The ROI in this area is as good as it’s ever going to be – and is considerably better evidentially speaking than many other things we spend considerably larger sums of money on!

I’ve commented on this before –

And this paper is well worth a read – 

Evidence limitations

 Also it’s important to understand the nature of the primary evidence of effectiveness that goes into ROI models – often there’s a dearth of good quality effectiveness evidence and intervention evaluation, and / or there’s inappropriate attempt to ram what is in effect a social policy paradigm into a medical model of evidence mindset. Imperfect marriage. Evidence is good to make an argument with, it shouldn’t be the totality of the argument unless it is grade A crystal clear evidence in a no brainier situation.

Chapter 6 of the SW London tool sets out a whole host of important considerations. 

Most of you will simply gloss over these in your hunt for the silver bullet. I will spell some of them out for you here, but you should really read for yourself.

  • Many residual questions about approach and methodologies – there is no ‘correct’ way to identify and report on return on investment, many different methodologies and many choices are possible. It is important is that there is explicitness about what is included, and what is not. 

  • The approaches are non standardised with often wildly different parameter values used to cost and value specific items in a model – i.e. Lack of consistency.
  • Timeframes of cost and return measurement and valuation are important, check this.
  • Often social care return is missing from models obviously tilted towards NHS. If excluded – mostly for evidential reasons – social care will be a beneficiary of interventions that reduce demand in the NHS as social care is often used downstream of the NHS service. So it’s regrettable this is mostly not factored into models. Good example – sugar tax to prevent diabetes. Yes of course it’s a fiscal policy with an NHS and personal return. In the long run there’s a benefit in terms of social care for those who may otherwise have suffered long term complications of diabetes – strokes, heart attacks, amputations etc… return to social care is long term and quite intangible, mostly undercounted or not counted, but real.

It’s important you understand the methodologies (and limitations) around valuation of return. Especially Heath related return. It’s an ‘imprecise science’.  

Lastly, we confuse and conflate SROI, ROI, CBA and other forms of cost effectiveness types of analysis.  

So tread careful.

Finally   The ROI tool doesn’t shift investment. Why not.

Even when there’s good ROI for interventions or suites of interventions often the evidence doesn’t shift the gravity of resource commitments. 

Even when there’s amazing evidence re ROI we still largely ignore it when we are making investments / disinvestments (at the margin) or reviewing the value for money across a whole area of spend.

So, sadly, sparkly ROI evidence isn’t by itself good enough to sort the problem….. the money is locked up in the cath lab and other places…..(vested interests, the challenge of immediate vs long term important, got to change the status quo, 
It’s important to consider why and address those issues directly. Reasons include

  • Lack of belief in the evidence. Different mindsets, different ways of viewing the world. Take time to present your ROI in the perspectives of how other people think
  • The issue might be that it’s not about ‘evidence’ but it’s about belief….. beliefs are far harder to change 
  • Shifting the gravity of resources upsets vested interests and the balance of power – NO easy answers to that one. Political will, small and big p.
  • Long term economic thinking about investments, vs short term financial (and political) cycles. Cash in hand now is king, etc.
  • Cross sector stuff – agency X invests but agency Y benefits from the return

There are many others.
Asymmetric approach to us locks in low value resource commitments

 However our asymmetrical approach to selectively requiring ROI only for new stuff coming in effectively  locks in low value current spend. 

If we must apply ROI analysis (ensuring we do read and understand the small print  then it should be equally applied to all (cancer drugs, cath labs, liaison psychiatry, expansion of x or y service to meet ever growing demand) and NOT only applied to “prevention” projects.
NOT doing this essentially locks in low value investments of the past and doesn’t allow higher value (but not cost saving) to crowd out lower value.
Maybe reflect on the reasons why

  • Lack of belief in the evidence
  • Sounds too good to be true
  • Payback in future, but Rome is burning now.
  • Vested interest in maintaining status quo
  • Lack of skill to change to future alternative model of delivery…. Lack of will to
  • Financial, performance, other systems don’t enable change.
  •  cross sector investment and return. Agency X invests, agency y gets payback financially speaking. Maybe not an issue if whole government / whole society perspectives are taken

This excellent blog is also well worth a read, from Rethink Health

The Sense–and Nonsense–of Using ROI in Population Health –







Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s