The limits of  Return on Investment analysis

Not that long ago the world was clamouring for good evidence the return on investment for public health interventions.
There was a great response and the world is now full of ROI tools, models and papers.

PHE are producing excellent stuff in this space

I won’t rehash those here. You can do your own googling.

One great example

A particularly good ROI tool was published recently by the SW London Academic Health and Social Care System. It builds on work previously done by the Kings Fund and others.

Making the case for public health interventions, Optimity.

It’s really good, well worth having in your toolbox. Covers a lot of territory, often in space that’s more social policy than “public health” (depending if you have a contemporary understanding of those two words).  In quite a nuanced way.

It seems to have disappeared from the web. Email me

The caveats on using ROI models.
Nobody likes pages and pages of caveats. The small print IS important.

We all want the answer. It doesn’t exist. Answers come with caveats!

As David Buck often points out we need to be careful that we use these wisely and don’t make the playing field even more uneven than it already is by focusing attention on the need for ROI for “prevention” whilst ignoring other areas of resource commitment.

We don’t expect kidney transplants to return cashable savings, we do expect kidney transplants to be objectively cost effective against a threshold of willingness to pay. We definitely expect kidney transplants to improve outcomes. And so it should be with investments in the space of public health.

The ROI in this area is as good as it’s ever going to be – and is considerably better evidentially speaking than many other things we spend considerably larger sums of money on!

I’ve commented on this before on the asymmetry re our approach in vol sector decisions

And this NEJM paper is well worth a read –

Evidence limitations

 Also it’s important to understand the nature of the primary evidence of effectiveness that goes into ROI models – often there’s a dearth of good quality effectiveness evidence and intervention evaluation, and / or there’s inappropriate attempt to ram what is in effect a social policy paradigm into a medical model of evidence mindset. Imperfect marriage.

Evidence is good to make an argument with, it shouldn’t be the totality of the argument unless it is grade A crystal clear evidence in a no brainier situation.

Chapter 6 of the SW London tool sets out a whole host of important considerations.

Most of you will simply gloss over these in your hunt for the silver bullet.

I will spell some of them out for you here, but you should really read for yourself.

  • Many residual questions about approach and methodologies – there is no ‘correct’ way to identify and report on return on investment, many different methodologies and many choices are possible. It is important is that there is explicitness about what is included, and what is not.

  • The approaches are non standardised with often wildly different parameter values used to cost and value specific items in a model – i.e. Lack of consistency.
  • Timeframes of cost and return measurement and valuation are important, check this.
  • Often social care return is missing from models obviously tilted towards NHS. If excluded – mostly for evidential reasons – social care will be a beneficiary of interventions that reduce demand in the NHS as social care is often used downstream of the NHS service. So it’s regrettable this is mostly not factored into models. Good example – sugar tax to prevent diabetes. Yes of course it’s a fiscal policy with an NHS and personal return. In the long run there’s a benefit in terms of social care for those who may otherwise have suffered long term complications of diabetes – strokes, heart attacks, amputations etc…..so return to social care is long term and quite intangible, mostly undercounted or not counted, but real.

Other issues to pick up.

It’s important you understand the methodologies (and limitations) around valuation of return. Especially Heath related return. It’s an ‘imprecise science’.

Lastly, we confuse and conflate SROI, ROI, CBA and other forms of cost effectiveness types of analysis.

So tread careful.

Finally   The ROI tool doesn’t shift investment. Why not.

Even when there’s good ROI for interventions or suites of interventions often the evidence doesn’t shift the gravity of resource commitments.

Even when there’s amazing evidence re ROI we still largely ignore it when we are making investments / disinvestments (at the margin) or reviewing the value for money across a whole area of spend.

So, sadly, sparkly ROI evidence isn’t by itself good enough to sort the problem….. the money is locked up in the cath lab and other places…..(vested interests, the challenge of immediate vs long term important, got to change the status quo,

It’s important to consider why and address those issues directly. Reasons include

  • Lack of belief in the evidence. Different mindsets, different ways of viewing the world. Take time to present your ROI in the perspectives of how other people think
  • The issue might be that it’s not about ‘evidence’ but it’s about belief….. beliefs are far harder to change
  • Shifting the gravity of resources upsets vested interests and the balance of power – NO easy answers to that one. Political will, small and big p.
  • Long term economic thinking about investments, vs short term financial (and political) cycles. Cash in hand now is king, etc.
  • Cross sector stuff – agency X invests but agency Y benefits from the return
  • Sounds too good to be true
  • Payback in future, but Rome is burning now.
  • Vested interest in maintaining status quo
  • Lack of skill to change to future alternative model of delivery…. Lack of will to
  • Financial, performance, other systems don’t enable change.
  •  cross sector investment and return. Agency X invests, agency y gets payback financially speaking. Maybe not an issue if whole government / whole society perspectives are taken

There are many others.

See here from Sally Burlington (LGA) at the 2017 PHE conference.

This excellent blog is also well worth a read, from Rethink Health – The Sense–and Nonsense–of Using ROI in Population Health

Why all this geeky stuff matters

Asymmetric approach to us locks in low value resource commitments our asymmetrical approach to selectively requiring ROI only for new stuff coming in effectively  locks in low value current spend. If we must apply ROI analysis (ensuring we do read and understand the small print  then it should be equally applied to all (cancer drugs, cath labs, liaison psychiatry, expansion of x or y service to meet ever growing demand) and NOT only applied to “prevention” projects. NOT doing this essentially locks in low value investments of the past and doesn’t allow higher value (but not cost saving) to crowd out lower value.

If we make the point of the exercise only about ROI, and “saving” then we may miss the point. In the health and well being space, prevention is about:

  • achieving better value from current resource envelope i.e. Better outcomes
  • Better health for its own sake.
  • The right to health as a fundamental human right
  • Social justice – esp if approaching with inequality lens
  • Downstream care costs
  • Healther folk as building block of a vibrant society/economy

I.e. It’s about more than saving the NHS cash by next week. There’s an excellent well evidence based way to achieve that – it’s called stopping providing health care.

Till we have a more nuanced approach to all this we will obviously completely fail in our mission to shift the gravity of spending to a more preventive model.

Maybe we should reframe it complete – and Frame prevention as the ONLY choic, or framed it in the language of what happens if we DONT DO something as well as the language of cost effectiveness. We should also relook at the level playing field argument set out above and here.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s