NEW: Outliers guidance 2024

5 Jan 2024

Updated guidance is now available here. HQIP is pleased to share our revised guidance, Outliers 2024. The result of extensive consultation with NHS England, the CQC and other stakeholders, this is a revision of a policy that has been in existence for well over a decade and is an important part of benchmarking. It is the core policy from which our audit and other providers develop their own policies, bespoke to their particular circumstances. “Benchmarking in healthcare is far more than mere comparison. It is a powerful tool that can support healthcare providers to identify opportunities for improvement and improve patient care.” Excerpt from an article on benchmarking from HQIP’s Quality Improvement publication, CORNERSTONE (pages 9-11) Outlier analysis is a valuable approach to benchmarking, where assessment of the performance of healthcare providers can identify organisations with unexpectedly extreme values for particular aspects of care. This is important for deepening understanding when patient outcomes fall significantly outside of the norm of what is expected, either positively or negatively; and this is why it is a key component of the analysis carried out during many national clinical audits. “Outliers was necessarily set aside during the height of the COVID pandemic. This gave us time to reflect on the policy and work on it with others across the sector, to deliver robust guidance that will lead to even greater insights and improved care,” Professor Danny Keenan, HQIP Medical Director. Measurement of outliers has traditionally been considered primarily a quality assurance activity. However, an outlier policy also provides opportunities for national clinical audits to support quality improvement. As such, HQIP has recently updated its guidance around outlier management for national clinical audits and programmes, which is produced to support NCAPOP provider organisations to support the development of their own outlier strategies. The result is a ‘softer approach’ which retains the principles of benchmarking, and includes:

The introduction of a ‘non-participation category’ so that Trusts that should be contributing data towards national audits but are not, will be regarded as outliers.
Changes to the notification of significant outliers. For key predetermined audit metrics, the highest level outliers will be notified directly to the CQC and NHS England. Other metrics with outlier results will be available for review when annual reports are published as part of the national audit cycles.
NHS England, as part of their approach to quality management, also receiving first-hand notification of such outliers, as they also have an important role, as commissioners, to assist in the management of quality improvement falling out of the outlier process.

It is expected that this revision to our guidance will help audit colleagues, through benchmarking, to continue their important work to assure our services but also to drive the quality improvement agenda. The revised guidance can be read in full here.

NEW Benchmarking data available: National Joint Registry

4 Jan 2024

The National Joint Registry (NJR) is the latest dataset to be published on the National Clinical Audit Benchmarking (NCAB) website. This data was updated on NCAB on 19 December 2023 from the NJR Surgeon and Hospital Profile website, published on 17/01/2023, covering April 2021 – March 2022 and August 2017 – August 2022.

NCAB is an online portal, hosted by HQIP, which provides access to national audit performance data. Users do not need to register, and can access audit benchmarked data searchable by speciality, Trust, hospital or unit. For all datasets currently published, go to the NCAB site.

NEW benchmarking data available: National Neonatal Audit Programme

4 Jan 2024

The National Neonatal Audit Programme (NNAP) is the latest dataset to be published on the National Clinical Audit Benchmarking (NCAB) website. This data was updated on NCAB on 15 December 2023 from the NNAP report 2021 published on 10/11/2022 covering data Jan 2019 to Dec 2021.

The role of Artificial Intelligence in healthcare: potential use in national clinical audit

21 Nov 2023

Dr Claudia Snudden, Clinical Fellow and Drew Smith, Associate Director at HQIP.

In the ever-evolving realm of technological advancements, Artificial Intelligence (AI) stands out as a beacon of transformative change. From smart home systems to predictive algorithms in finance, AI is reaching into more and more corners of our modern existence. The face of this revolution is generative AI, that is: learning models that can generate text, images, and other content based on the data they were trained on. Perhaps the most well known generative AI tool is OpenAI’s ChatGPT, but others such as Bing and Bard work on the same principle. This rapid assimilation of data and generation of information is not just transforming industries on a macro scale; it’s redefining tasks as granular as administrative chores and as expansive as biomedical research.

For those of us at The Healthcare Quality Improvement Partnership (HQIP) and our audit providers, the rise of AI offers both challenges and opportunities. We stand at the cusp of a new era where we must consider the nuances of integrating this technology into our daily workflows. How can we harness it for efficiency, while also navigating its complexities to ensure we remain compliant, especially concerning data protection? In this exploration, we’ll delve into two ways that generative AI can add value to national clinical audits: data analysis and content generation.

Your virtual data analyst

The processing and interpretation of data can often be a bottleneck for many organisations, especially in the realm of healthcare where accuracy is paramount. Generative AI can alleviate some of these challenges. For programmers and analysts, AI can act like an on-demand mentor, providing insights on programming languages like Python, Java, or R, offering solutions to coding conundrums, and even guiding on the nitty-gritty of debugging. It can also advise on data cleaning techniques, suggest appropriate statistical tests, and recommend variables or relationships worthy of investigation.

By feeding or uploading aggregate non-personalised data to the model, you can receive not just analyses but also guidance on presentation. Should you opt for a bar chart or a scatter plot? Which patterns should be highlighted for your target audience? Even more excitingly, as the functionality of AI tools grows these visualisations can be crafted in real-time, serving as a dynamic aid in data interpretation and communication.

Content creation and synthesis

In an age of information overload, the ability to effectively communicate findings is as critical as the discoveries themselves. Beyond numbers and codes, generative AI boasts impressive capabilities in the domain of text. The myriad of possibilities here are genuinely extensive: from distilling the core findings of a lengthy research paper into a concise summary, to translating content across languages, or to converting raw bullet points into coherent prose.

And it’s not just about simplification; it’s about tailoring. An audit report, for instance, could be transformed into an accessible blog post or an informative pamphlet, depending on the target audience. Stuck with where to start when writing a report? Upload your outline to a generative AI tool and let it kick off the creative process for you.

Walking with caution: the limitations

As with any technological advancement, generative AI isn’t without its limitations. A primary concern resides in the realm of data protection and privacy. It’s vital to approach online tools as if you’re releasing information into the public domain. Personalised or even pseudonymised details should never be entered into such a tool. Similarly, one should refrain from uploading unpublished content such as national clinical audit report drafts.

Moreover, whilst the reservoirs of knowledge that AI tools are built on are vast they are not infinite, nor are they always up-to-date. Tools may sometimes stumble with deeply technical jargon or misinterpret the context behind a query.

With all this in mind our most important caveat is that artificial intelligence is… well… artificial. As exciting, insightful, and revolutionary as AI can be, it doesn’t (yet) replace the 2 million years of human evolution that’s gone into us. Generative AI tools are known to ‘hallucinate’, that is: to create incorrect information and present it as fact. Even, in some cases, fabricating its own citations. AI hallucinations can be difficult to spot and arise from a model being trained on a limited, out-of-date, or low-quality data set. Or they can occur because the tool doesn’t possess the contextual understanding of the prompt.

It follows therefore that as users the onus is on us to cross-check citations provided by generative AI, and sense-check the information it produces. As Gartner points out: “Generative AI creates artifacts that can be inaccurate or biased, making human validation essential and potentially limiting the time it saves workers.”

Looking forward

The incorporation of generative AI within both the infrastructure at HQIP and our audit providers promises a slew of benefits. From data analysts to communication specialists, the potential for AI models to elevate our workflows, bolster efficiency, and sharpen our outputs is undeniable. As we look ahead, there’s also the exhilarating prospect of using tools to develop bespoke AI analytical models tailored for audit data extraction and analysis across clinical audit themes. This endeavour, though filled with promise, will require concerted effort, continual learning, and strategic investments.

Current models, whilst impressive, largely result in users being able to perform existing tasks more quickly. But in the next few years we anticipate that continued growth in investment and computational power will see a blossoming of AI capabilities and an increasing democratisation in the use of tools.

The advent of generative AI offers a glimpse into a future brimming with possibilities. As we embark on this journey, let’s tread with curiosity, caution, and the unwavering aim to better our practices for the greater good. The future of HQIP, in many ways, is intertwined with the possibilities this technology brings, and it’s a future we eagerly anticipate.

When is data the ‘right data’?

17 Jan 2023

Using the right data to support robust healthcare services.

Mirek Skrypak, previously Associate Director for Quality and Development, HQIP

“As I was gearing myself up to write this article about using data to develop robust healthcare services, I took receipt of a rather unusual delivery: A golden microphone with Bluetooth and a voice synthesiser. It proved momentarily inspirational, though how is that relevant, you might ask? Well, it prompted me to think about change and, in particular, how using the right data can support the right change.

On seeing my delivery (which was a present for someone else, by the way), I was struck by how much microphones have changed. I researched early microphones. Although it still performs the same essential function as the original from 1878, my new microphone differs significantly. This golden one with buttons, Bluetooth and charging cable etc shows many visible signs of change, most of which are in response to modern-day needs. Which led me to think about change in healthcare services.

Practice won’t change if you don’t have the right data

The NHS is about 70 years younger than the microphone. Just like the microphone, you could say that it still performs the same function (to improve outcomes for those needing medical care). It also needs to change in order to respond to current need (not least in response to an increasingly ageing population). But how do we ensure that any change implemented is relevant, targeted where it’s needed most, and meaningful to service users? In response, I will share some learning that I think is invaluable in relation to data-informed service design within healthcare…

First, practice won’t change if you don’t have the right data. Whatever tools or techniques you use – whether process improvement, error reduction, waste trimming, agile scrum meetings etc – all of it needs to be done within context. In addition, you need to be curious; you need to ask questions and have ideas. Assuming you have all that, knowing what and how to measure is crucial. Simple right? Well, no! There are a number of variables, models, systems, frameworks and contexts etc that you also need to think about. However, I suggest that there is one other consideration that that is more important than all others: There should be no data without stories and no stories without data. The percentages or figures in charts, tables, recommendations and p values etc are all people. Think about it in terms of this fictional example… Let’s say that there is a national target of 80% for a particular process measure which evidence suggests will improve outcomes. There are circa 170 NHS sites, and let’s assume that 75% are achieving this 80% target for a particular clinical area and pathway. Then, let’s say that, in this pathway, there are 100,000 people who receive this care per year. That means 60,000 patients will achieve the target. But, what about the other 40,000? Suddenly this is not so positive. I know which side – or cohort – I would like to be on as a patient.

There should be no data without stories and no stories without data

So how can we improve these numbers? Here at HQIP, we help service providers to find the right data, to take an effective approach to measuring, and to use that knowledge to build consensus for change. We strongly advocate the synchronisation of Quality Improvement (QI) with national clinical audit and confidential enquiries to enable improvement in outcomes at a national level. Find below what I hope is a useful summary of the key questions that need to be addressed by healthcare leaders and their teams when doing this…

As a manager or clinician:

Have you collected the data in the right way to be able to identify trends?
Do you have the right roles reviewing the data? Who owns it?
Do you know your audience? What is it they need to know?
Are you at risk of decision makers reacting unnecessarily?
How are you presenting your data? Have you presented the data to show a true picture over time?
Is there variation, and is this normal or the result of an exceptional circumstance?
Do you need to consider if it’s actually the system that needs reviewing, or does the target need to change?
Do you need to measure for a longer duration?

As a team or service:

Do we know how good we are?
Do we know where we stand relative to the best?
Over time, where are the gaps in our practice that indicate a need for change ie improvement?
In our efforts to improve, what’s working (and what isn’t)?
Do we know/understand where variation exists in our organisation?
Why are we measuring all this and what difference is this actually going to make to the quality of services?

That’s a lot of food for thought. And I don’t suggest, for one moment, that it’s an easy task. But it is necessary. While I’m not sure that we need a golden, ‘Bluetooth-enabled’ (or the equivalent) healthcare service, we do need to instigate change. I’m convinced – and I wish I had my gold microphone to hand to make this point – that the right data is the way forward in ensuring that those changes are relevant, meaningful and, above all, deliver maximum impact. However, I’m going to give a final word of caution to the renowned statistician David Spiegelhalter, who says: “Signals always come with noise: It is trying to separate out the two that makes the subject interesting.”

This article is one of a number of articles written for CORNERSTONE, HQIP’s free publication, designed to support Quality Improvement. It was featured in the 2023 edition, which features other articles on patient engagement, patient safety, and the importance of health data (among other topics). For more information, go to Healthcare improvement magazine – HQIP.

BLOG Taking the long view: Future-proofing healthcare

15 Dec 2022

Sustainable Respiratory Care Audit Team.
Alice Fitzpatrick, Marsden Rd. Health Centre,
Sustainable Respiratory Care Audit Team.

In June 2022, the Sustainable Respiratory Care project I work with was named winner of the Future-proofing Healthcare category of the 2022 Clinical Audit Heroes awards, as part of HQIP’s Clinical Audit Awareness Week. Of course we were delighted that our work was recognised in this way, but perhaps what was more important was that we would get to raise the profile of the need for sustainable healthcare projects (and share details of our work, into the bargain). Many months have passed since we collected our award but the value of our work continues, and so I am sharing some key highlights in the hope that other professionals will be inspired to ‘take the long view’ and support sustainability in healthcare…

Savings of £8.2 million annually and 58 kilotonnes of carbon dioxide equivalent emissions could be made

In summary, the Sustainable Respiratory Care Audit project provides a structure for the audit of patients’ technique, preferences and knowledge about inhalers as well as the need for clinical review. Selecting appropriate inhaler devices and improving user technique are interventions which can reduce the carbon footprint of healthcare, while improving the quality of care. To improve the control of respiratory conditions and reduce both exacerbations and medicines wastage, it is important to review patients’ inhalers and technique and consider switching a patient’s inhalers where appropriate. It can also provide financial savings.

Commonly used inhalers include Metered Dose Inhalers (MDIs), which contain hydrofluoroalkane (HFA), and Dry Powder Inhalers (DPI) which do not contain HFA. DPIs have a much lower carbon footprint than MDIs; the carbon footprints of MDIs are 10-37 times higher than those of DPIs. While not all patients have sufficient lung function to use a DPI effectively, many patients will receive more effective dosing of inhaled medications using a DPI than using an MDI and prefer not to carry a spacer (MDIs should always be used with a spacer). However researchers have calculated (based on 2017 data) that savings of £8.2 million annually and 58 kilotonnes of carbon dioxide equivalent emissions could be made by replacing just one in ten MDIs in England with the cheapest equivalent DPIs.

More specifically, the audit had a number of key aims. Firstly, to assess whether inhaler technique has been checked regularly and effectively in patients with respiratory disease (as per BTS Asthma Bundle and NICE COPD guidance). Then, it was designed to allow patients to express attitudes and preferences towards inhalers in line with the approach outlined by NICE in their Asthma patient decision aid and the NICE Shared decision-making guidance. Finally, it established a baseline from which to improve the quality and environmental sustainability of respiratory care in the NHS.

So, what did we do in practice? An audit proforma, in the form of a survey that could be carried out with a patient within 10 minutes, was designed by the project team. A data collection spreadsheet was also designed. The proforma was reviewed by respiratory nurses and senior respiratory physicians, piloted on the respiratory ward at Newcastle hospitals, and improved based on patient and health professional feedback. Edits were made – for example, to the information about how to implement the audit, providing exact wording for questions and detailing the need to check with nursing staff about a patient’s cognition and any reasons not to invite a patient to take part before approaching a patient.

The creation of a network of clinicians across the country enabled the collection of national results to which local results could be compared

Next, clinicians from multiple hospitals were recruited using emails sent to professional networks, and introduced to the audit via an introductory webinar. They were provided with the audit proforma and given time and a further webinar in which to propose and discuss any revisions. As a result, minor revisions were made to the wording of questions about patient preferences and inhaler disposal. Clinicians then applied for Caldicott approval and implemented the audit locally.

We are particularly proud of the level of engagement achieved around this project. The creation of a network of clinicians across the country enabled not only collaborative development of an audit proforma, but also the collection of national results to which local results could be compared. The engagement of local teams (which were required to include one senior respiratory clinician and one junior clinician) meant that local solutions to challenges and opportunities for improvement could be identified. Furthermore, the audit engaged patients to give their preferences about their respiratory care and describe their knowledge, to demonstrate how they use their inhaler (thereby auditing patients’ practices, not simply proxies of practice or documentation of others’ practice). This provided information that could inform not only service-wide improvement, but also improvements for individual patients where the need for review or education was identified.

At the time of writing this article, this project is still very much on-going. However, we are already starting to gather some key insights. For example, clinicians found that reviewing patients’ prescriptions highlighted discrepancies between the inhalers that patients were taking and the inhalers prescribed. Importantly, we were able to engage pharmacists in remedying this issue for individual patients, and in considering how to improve practice going forward. The results of the audit will be published once available from all groups, to engage stakeholders in healthcare providers as well as national bodies influencing NHS care. In the meantime, we hope that you agree with the Clinical Audit Heroes award judges that our project demonstrates how incorporating a sustainability perspective in audit can identify opportunities to improve care for individual patients and reduce environmental impacts of healthcare.

Further information: Sustainable Respiratory Care

This project was a collaboration between Dr Sarah Walpole (Newcastle NHS Trust), Dr Lewis Standing (Newcastle NHS Trust), Dr Maria van Hove (University of Exeter), Dr Joseph McElvaney (previously Newcastle NHS Trust, now Greater Glasgow and Clyde) and Dr Anya Gopfert (Northumbria NHS Trust).

BMJ Blog: Collecting data is just the start – Josie O’Heney

2 Jul 2022

“…one day we won’t be reporting that the colour of your skin impacts your risk of dying at what should be one of the happiest times of your life.”

Clinical fellow Dr Josie O’Heney explores health inequalities in maternity in her blog: collecting data is just the start. The blog, written for the BMJ Leader series, is one of five penned by clinical fellows on secondments at NHS England and NHS Improvement and Healthcare Quality Improvement Partnership (HQIP).

Each one explores the ‘5’ focus areas of the CORE20PLUS5 approach, the links between health inequalities and the invaluable contribution of leadership to narrowing the life expectancy inequality gap.

Read the full blog on the BMJ website.