Healthgrades Patient Safety Methodology

TABLE OF CONTENTS

Healthgrades Patient Safety Ratings & Patient Safety Excellence Award™

Data Source and Applications

Impact of Coronavirus (COVID-19)

Table 1. Patient Safety Indicators and Translation

Eligibility for the 2023 Patient Safety Excellence Award

Determining Patient Safety Indicator Ratings

Designating 2023 Patient Safety Excellence Award Recipients

Limitations of the Data Analysis

Patient safety ratings evaluate the safety and quality of care at a hospital by measuring the rate at which serious, potentially preventable complications and adverse events occurred in the hospital.

To evaluate hospital patient safety, Healthgrades uses inpatient data from the Medicare Provider Analysis and Review (MedPAR) file and the Agency for Healthcare Research and Quality’s (AHRQ) Patient Safety Indicators Technical Specifications and AHRQ WinQI software (version 2022.0.1 ICD-10-CM/PCS January 2022). We analyze 14 AHRQ-defined patient safety indicators (PSIs), each of which represents a serious, potentially preventable complication.

This methodology includes descriptions of how Healthgrades:

• Determines patient safety ratings for each of 14 PSIs
• Determines which hospitals are eligible for the Patient Safety Excellence Award™
• Designates Patient Safety Excellence Award™ recipients

Healthgrades uses Medicare inpatient data from the Medicare Provider Analysis and Review (MedPAR) file purchased from the Centers for Medicare and Medicaid Services (CMS). We evaluate all short-term acute care hospitals in the MedPAR file for three years (2019 through 2021 fiscal years). To process patient safety ratings, we utilize all available positions in the medical claim for diagnosis and procedure codes.

For 13 of the 14 indicators, Healthgrades utilized the AHRQ WinQI software, version 2022.0.1 to predict the number of patient safety incidents that are likely to occur at a hospital based on the types of patients treated at that hospital. More information about the AHRQ patient safety indicators (PSIs) and risk adjustment can be found at https://www.qualityindicators.ahrq.gov/Modules/psi_resources.aspx#techspecs

Table 1 provides the AHRQ definition for each of the 14 AHRQ-defined patient safety indicators (PSIs) Healthgrades analyzes, as well as its respective description used in Healthgrades reports.

In 2020, the global pandemic changed healthcare, including how and when healthcare was delivered and, in many cases, not delivered or delayed. This changed the MedPAR data set with the data showing volume drops in many areas and volume spikes in others (e.g., COVID-19 admissions). After examining the effects to the data set caused by the pandemic, Healthgrades made the decision to exclude COVID-19 patients from the AHRQ analysis for patient safety. Healthgrades does not believe that the presence of COVID-19 on patient records can be adequately risk-adjusted in the 2020 data year. We will continue to analyze the impact of COVID-19 on an ongoing basis, and for additional data years in the future. The exclusions currently encompass the following ICD-10 codes in any position in the data set for the 2020 and 2021 data years only: U07.1, B97.29, and B34.2.

Patient Safety Indicator	Translated in Healthgrades Reports as…
Death Rate Among Surgical Inpatients With Serious Treatable Complications	Death following a serious complication after surgery
Death Rate in Low-Mortality Diagnosis Related Groups (DRGs)	Death in procedures where mortality is usually very low
Pressure Ulcer Rate	Pressure sores or bed sores acquired in the hospital
Iatrogenic Pneumothorax Rate	Collapsed lung resulting from a procedure or surgery in or around the chest
Central Venous Catheter-Related Bloodstream Infection Rate	Catheter-related bloodstream infections acquired in the hospital
In-Hospital Fall With Hip Fracture Rate	In-hospital fall resulting in hip fracture
Postoperative Hemorrhage or Hematoma Rate	Excessive bruising or bleeding as a consequence of a procedure or surgery
Postoperative Acute Kidney Injury Requiring Dialysis	Acute kidney dysfunction (requiring dialysis) following surgery
Postoperative Respiratory Failure Rate	Respiratory failure following surgery
Perioperative Pulmonary Embolism or Deep Vein Thrombosis Rate	Deep blood clots in the lungs or legs during the surgical stay
Postoperative Sepsis Rate	Bloodstream infection following surgery
Postoperative Wound Dehiscence Rate	Breakdown of abdominal incision site
Abdominopelvic Accidental Puncture or Laceration Rate	Accidental cut or puncture resulting from a procedure of the abdominopelvic region
Retained Surgical Item or Unretrieved Device Fragment Count	Foreign objects left in body during a surgery or procedure (reported as number of events)

For more details on the criteria for each individual PSI, please refer to the technical specifications documents available directly from AHRQ at https://www.qualityindicators.ahrq.gov/Modules/psi_resources.aspx#techspecs

To be eligible for the Healthgrades Patient Safety Excellence Award, a hospital must meet clinical quality thresholds, have zero occurrences of PSI Foreign Object Left During Surgery or Procedure, and have data on at least seven out of eight core PSIs.

• Clinical Quality Threshold – To be eligible, a hospital must have been rated by Healthgrades and be in the top 80% of hospitals for clinical quality, as measured by volume weighted z-score across conditions and procedures evaluated using Medicare data. The volume weighted z-score is calculated from all rated cohorts for each facility. There is no minimum number of ratings for this evaluation. Other hospitals that have not been evaluated by Healthgrades that have patient safety data but no Healthgrades ratings are also eligible.
• Zero Occurrences of PSI Foreign Object Left During Surgery or Procedure – Eligible hospitals must not have any occurrences of the PSI Foreign Object Left During Surgery or Procedure. A hospital is ineligible for this award if even one patient has experienced this adverse event. 
• Data for Seven of Eight Core PSIs – Healthgrades identifies a core set of eight PSIs that are serious, but potentially preventable, complications related to medical or surgical inpatient hospital care. Eligible hospitals may have data for all 14 PSIs, but they must have data for at least seven of the eight core PSIs (Table 2).  
• Adequate sample size for statistical evaluation – Eligible hospitals must have at least 15 patients evaluated across three years and five patients in the most recent year for seven of the eight core PSIs.

Table 2. Healthgrades Core Set of Eight PSIs and Translation

Patient Safety Indicator	Translated in Healthgrades Reports as…
Pressure Ulcer Rate	Pressure sores or bed sores acquired in the hospital
latrogenic Pneumothorax Rate	Collapsed lung resulting from a procedure or surgery in or around the chest
Central Venous Catheter-Related Bloodstream Infection Rate	Catheter-related bloodstream infections acquired in the hospital
In-Hospital Fall With Hip Fracture Rate	In-hospital fall resulting in hip fracture
Postoperative Sepsis Rate	Blood stream infection following surgery
Perioperative Pulmonary Embolism or Deep Vein Thrombosis Rate	Deep blood clots in the lungs or legs during the surgical stay
Postoperative Wound Dehiscence Rate	Breakdown of abdominal incision site
Abdominopelvic Accidental Puncture or Laceration Rate	Accidental cut or puncture resulting from a procedure of the abdominopelvic region

To determine ratings for 13 of the 14 AHRQ-defined patient safety indicators (PSIs) for each hospital, Healthgrades performs the following process:

1. Healthgrades uses the MedPAR data set for the most recent three fiscal years (2019 through 2021). 
2. This data set is risk-adjusted and patient-level actual and predicted rates for each of 13 PSIs are calculated. The remaining PSI—Foreign Object Left During Surgery—does not have a predicted rate*. 
3. Healthgrades then uses the Medicare Case Mix Index (CMI), a hospital-level indicator provided by CMS annually that represents the seriousness of cases seen at each hospital.  Higher CMI values indicate more seriously ill patients. With the often-higher risk of Medicare patients, the AHRQ WinQI software may under-adjust for large hospitals and hospitals with high CMIs. This CMI adjustment step accounts for differences in the AHRQ WinQI software model and Medicare patient data.
4. For each year of data, Healthgrades stratifies hospitals into eight CMI groups according to their CMS supplied case mix index for that year. The groups are displayed in Table 3. 
5. For each CMI group, Healthgrades adjusts the AHRQ-QI supplied predicted values so the actual-to-predicted ratio for each PSI is equal to 1, for each CMI group.
6. The patient-level actual and CMI-predicted rates are then combined across the three years of data for each facility. A patient-level Hosmer-Lemeshow variance is calculated and aggregated across all patients for each hospital and PSI. 
7. A z-score is calculated for each hospital and PSI.
8. Healthgrades then translates the z-score to a patient safety rating based on an evaluated 90% confidence interval for that PSI. 
9. Performance for each PSI rating is grouped into one of three Healthgrades performance categories:

• Better Than Expected – Actual performance was better than predicted and the difference was statistically significant at alpha = 0.1. 
• As Expected – Actual performance was not statistically significantly different from what was predicted at alpha = 0.1. 
• Worse Than Expected – Actual performance was worse than predicted and the difference was statistically significant at alpha = 0.1. 

When a hospital is not rated by Healthgrades in an individual PSI, it means the hospital did not have at least 15 patients in three years and/or five patients in the most recent year that were eligible to be evaluated for that PSI.

*Note: One additional PSI (Foreign Object Left During Surgery or Procedure) differs from the other 13 PSIs because it is an event that should never happen—referred to by AHRQ as a “never event.” For this indicator, we report the number of events from 2019 through 2021 where a foreign object was left in a patient during a surgery or procedure.

To recognize hospitals that provide excellent patient safety, Healthgrades uses the following process:

1. Healthgrades creates a composite patient safety z-score by calculating a weighted average z-score using the sum of the 13 individual PSI z-scores and the total number of patients evaluated for each PSI as the weight. (See above for details on how Healthgrades produces a z-score for each individual PSI.) 
2. Healthgrades identifies those hospitals in the top 10% based on volume weighted z-score. Those hospitals are designated as Patient Safety Excellence Award recipients.

While these analyses may be valuable in identifying hospitals that perform better than others, one should not use this information alone to determine the quality of care provided at each hospital. Also, note that if more than one hospital reported to CMS under a single provider ID, Healthgrades analyzed patient safety data for those hospitals as a single unit. Throughout this document, therefore, "hospital" refers to one hospital or a group of hospitals reporting under a single provider ID.

The analyses are limited by the following factors:

• The AHRQ WinQI software contains risk-adjustment models that are not part of Healthgrades methodology. Risk factors for patient safety, therefore, are weighted and accounted for through the AHRQ WinQI software.
• Cases may have been coded incorrectly or incompletely by the hospital. 
• Healthgrades creation of patient safety ratings can only account for risk factors that are coded into the billing data. Therefore, if a particular risk factor was not coded into the billing data (such as a patient's socioeconomic status and health behavior), then it was not accounted for. 
• Although Healthgrades has taken steps to carefully compile these data, no techniques are infallible; therefore, some information may be missing, outdated or incorrect.
• For the Patient Safety Indicator Accidental Puncture or Laceration Rate, the restriction of two procedures being at least a day apart could not be included given the lack of procedure dates in the Medicare inpatient data from the Medicare Provider Analysis and Review (MedPAR) file.

Click here to download a PDF version of the complete 2023 Patient Safety Methodology.