How do I compare thee?

4 Dec 2024

A review of benchmarking in healthcare

Professor Danny Keenan, HQIP Medical Director and Associate Medical Director to the Manchester University NHS Foundation Trust

Benchmarking in healthcare is far more than mere comparison. It is a powerful tool that can support healthcare providers to identify opportunities for improvement and improve patient care. But, in a minefield of data and information, what are the key resources and developments in relation to benchmarking in healthcare, and what steps are HQIP taking to support effective measurement of performance? Our Medical Director, Professor Danny Keenan, provides a helpful overview… Your first port-of-call should be audit and similar programme reports and outputs. Be familiar with what is available in your field, and understand how often – and when – data is shared. For HQIP commissioned audits and programmes, the reports and other outputs can be found on the HQIP website, while our publication schedule (which is updated monthly) is here. Importantly, our outputs were changed after the COVID-19 pandemic, following a series of webinars with the national audit providers to explore if the programme was ‘digital ready’. As a result, changes – such as shorter reports, less metrics and a move towards near real-time dynamic reporting – were introduced to reduce the burden on Trusts and healthcare service providers. It is also worth noting that, in addition to these commissioned summary reports, many clinical audit and outcome review programmes have websites where further background data can still be accessed. A benchmarking chart based on data from the National Hip Fracture Database (NHFD) showing ‘prompt surgery’ (surgery by the day following presentation with hip fracture) with confidence intervals. Each hospital is denoted on the x axis (not shown here):

Another starting point is the National Clinical Audit Benchmarking website (NCAB), which provides a visual snapshot of individual Trust audit data set against individual national benchmarks. There is no barrier to use, such as login or an NHS email address, so it’s easy to use and available to all – from healthcare professionals through to policy makers and patients. Created by HQIP in collaboration with the Care Quality Commission (CQC), this resource contains datasets on a variety of clinical disciplines, with additional data being added on an on-going basis (to receive notifications of new datasets as they are added, subscribe to HQIP’s mailing list). It provides a snapshot view of each healthcare provider, stating whether, for example, they are above, in line or below expectations for each measure. NCAB also enables Trusts to determine if there are any metrics for which they are a (positive or negative) outlier. This is important for highlighting when patient outcomes fall significantly outside of the norm of what is expected. In light of its significance and in response to the COVID-19 pandemic, HQIP has undertaken extensive consultation with stakeholders, including patients, to revise its guidance relating to outliers. The result is a ‘softer approach’ which retains the principles of benchmarking, and includes:

The introduction of a ‘nonparticipation category’ so that Trusts that should be contributing data towards national audits but are not, will be regarded as an outlier.
Changes to the notification of significant outliers. For key predetermined audit metrics, such as mortality, ‘alert’ level results will be notified directly to the CQC and NHS England. Other less significant metrics with alert outlier results would be available for review when annual reports are published.

In addition to these resources, HQIP is involved in a number of developments to support Trusts and other health and care providers to measure performance, starting with talking to clinicians and analysts to identify better ways of visualising HQIP and NHS outputs (to ensure maximum impact). Work in this area includes making more timely data available on NCAB, and looking at how we could standardise coding systems that are already in use. We are also talking to patient and service user groups as well as the independent sector, to include the latter in the National Clinical Audit programme. The Paterson review made it clear that all patients’ data should be included, no matter where their operation has taken place or how their care is commissioned. This will ensure that patient care is equally assured, and that their data is available for quality improvement initiatives irrespective of geography. Of course, performance measurement and benchmarking are of utmost importance to the NHS too. In 2023, I chaired a series of ‘National Clinical Audit for Improvement Implementation Group’ webinars run by NHS England. These explored the development of clinical effectiveness across the service and, in particular, looked at ways to support clinical audit colleagues who are very much on the frontline of this work, with initiatives such as NHS IMPACT, the Futures NHS platform and the Model Health System. To further support the sharing of innovation, they also coordinate the NHS Benchmarking Network, which helps members to improve patient outcomes, raise health standards and deliver quality health and care services through data excellence, benchmarking and the sharing of innovation.

The Paterson review made it clear that all patients’ data should be included, no matter where their operation has taken place or how their care is commissioned

One area that everyone is interested in (quite rightly), is how to address health inequalities; and audit data and performance measurement is proving to be a powerful tool for this, shining a light on where inequalities exist. The COVID-19 pandemic highlighted inequalities in health outcomes due to ethnicity and deprivation. As a result, HQIP is investigating how the National Clinical Audit programme can track patients’ outcomes better using markers such as ethnicity and deprivation. In particular, we sponsor a National Medical Director’s Faculty of Medical Leadership and Management (FMLM) Fellow each year, and they have led a series of reviews on this topic. The 2023 review, due for publication in late 2023, is a survey concerning the obstructions encountered in relation to health inequalities, and includes a number of strong recommendations concerning basic issues such as coding, use of postcode and how to manage small numbers in the audit programme (all of which could make a big difference). We are now working with the Health Inequalities team at NHS England, regarding the plans for implementation of these important recommendations.

Audit data and performance measurement is proving to be a powerful tool for… shining a light on where inequalities exist

All the tools and resources I have mentioned so far are available to support benchmarking here and now (and I do hope you will take a look at them, if you are not already doing so). But I will end on more of a nod to the future. Firstly, we must improve how we celebrate excellence. Currently HQIP works with audit providers to produce “scenarios” concerning units that appear at the ‘excellent end’ of benchmarking charts. But, we need to raise excellence across the board. One of the downsides of benchmarking is that units sitting in the middle of the chart can become complacent, whereas we all need to keep moving towards the excellent end. HQIP will work with audit providers on ways of celebrating excellence better, so as to promote a ‘move to the right’. Secondly, to support strategic- and forward-thinking around audit data and performance review in healthcare, HQIP runs a Methodology Advisory Group (MAG), comprised of a broad spectrum of stakeholders including policy makers, healthcare professionals, and patients. In 2023, we hosted a MAG webinar dedicated to Artificial Intelligence (AI) and machine learning. We reviewed the use of these technologies in relation to National Clinical Audit, and discussed what work was already taking place and how we could foster best practice in this area. As a result, we are now exploring how best to share current and proposed best practice using AI. Furthermore, we have also committed to investigate the use of ChatGPT-4, or alternatives, to explore current anonymous datasets to search for disease and outcomes linkages. I’m sure that you, as do I, await news on how these developments can support us in measuring performance and improving outcomes for patients with eager anticipation. Watch this space…

Further information and resources

This article was originally featured in HQIP’s quality improvement magazine, CORNERSTONE – to see more articles on topics such as healthcare inequalities and sustainability in healthcare, go to: www.hqip.org.uk/wp-content/uploads/2023/11/HQIP_Cornerstone_2024.pdf.