News & Events

From Heartwire

Procedure Volume Is a Close Proxy for Popular Hospital Quality Rankings

Reed Miller

February 4, 2010 (Ann Arbor, Michigan) — The highest-rated hospitals in the US News and World Report and HealthGrades "best-hospital" rankings are also likely to be high-volume institutions, supporting research showing that the best outcomes correlate with high procedure volumes, a new Medicare analysis suggests [1]. But the same hospitals are not necessarily found on both lists, and patients seeking the best care may also find it at hospitals not listed by these popular ranking tools, authors say. "Both the US News and World Report's and HealthGrades' ratings identify high-quality hospitals for cardiovascular operations, [but] patients can experience equivalent outcomes by seeking care at other high-volume hospitals," study authors Dr Nicholas Osborne (University of Michigan, Ann Arbor) and colleagues conclude. Their analysis is published in the January 2010 Journal of the American College of Surgeons. "This study has important implications for patients trying to choose safe hospitals for cardiovascular procedures," the authors state.

The authors cite previous research by Dr Harlan Krumholz (Yale University School of Medicine, New Haven, CT) and colleagues, reported by heartwire, that showed that these popular ratings systems cannot distinguish differences in quality between individual hospitals but only identify a list of centers that perform better than average in aggregate. Therefore, Osborne and colleagues focused on comparisons of aggregate outcomes of the top 50 centers in both rankings.

The authors studied 2005 and 2006 Medicare outcomes data for all patients undergoing abdominal aortic aneurysm repair, coronary artery bypass, aortic-valve repair, and mitral-valve repair, a total of 312 813 patients. The primary outcome measured was 30-day mortality. The researchers compared mortality rates at the 50 top-rated hospitals in the US News and World Report rankings and the 50 best hospitals in the HealthGrades rankings with all other hospitals and then adjusted the results for hospital volume to determine whether the differences in mortality correlate to differences in a hospital staff's experience with that particular procedure.

What Makes the Grade?

US News and World Report bases its rankings on a combination of three equally weighted measures: hospital infrastructure, hospital reputation among subspecialists, and 30-day mortality. To be considered for the rankings, a hospital must be a member of the Council of Teaching Hospitals, be affiliated with a medical school, or have a minimum number of advanced technologies. The hospital must also meet minimum thresholds for surgical volume and discharge for 12 subspecialties.

HealthGrades' "Best Hospitals" list is created with a proprietary method for calculating predicted 30-day mortality rates based on Medicare Part A billing data. Predicted mortality rates are compared with the observed mortality rates at each hospital for 27 procedures and diagnoses. The hospitals with the best observed-to-expected mortality ratio make the top 50 list.

Importantly, Osborne et al found little concordance in hospital rankings between the report cards. Only eight hospitals rank in the top 50 of both the US News and World Report "America's Best Cardiovascular Hospitals" and the HealthGrades "Best Hospitals."

The characteristics of the patients in the best-hospitals lists were "considerably different" from the patients in all other hospitals in the Medicare data. The top-ranked centers in the US News and World Report list treated a higher proportion of African American patients and were less likely to operate emergently or urgently, but patient comorbidities were similar to those of all other hospitals in the Medicare database. The centers in both the US News and World Report and HealthGrades lists of top cardiovascular hospitals were much less likely to have low volumes of the procedures evaluated in the study and were much more likely to be teaching hospitals compared with all of the centers in the Medicare data.

The study compared hospitals, controlling for available patient risk factors such as age, gender, race, comorbidities, and acuity of admission and operation. After adjustment for these characteristics, the 30-day mortality rates were lower in US News and World Report's top-ranked cardiovascular hospitals for all four procedures, but the difference was statistically significant only for abdominal aortic aneurysm repair. The centers in the HealthGrades list of top hospitals had considerably better-than-average adjusted 30-day mortality rates for coronary artery bypass, aortic-valve repair, mitral-valve repair, and abdominal aortic aneurysm repair.

Volume=Quality?

"We demonstrated that although highly rated hospitals have lower risk-adjusted mortality for some cardiovascular procedures, a substantial portion of these differences can be explained by hospital volume," Osborne et al explain. As previously described in research led by Dr John Birkmeyer (University of Michigan) and reported by heartwire, the correlation between a surgeon's procedure volume and outcomes for these procedures is well established, but the volume-to-outcome relationship varies across operations.

Neither the HealthGrades nor US News and World Report rating systems explicitly account for differences in procedural volume in their quality ratings, but Osborne et al calculate that procedure volume explains between 14% and 79% of the differences in the mortality rates between the highly rated hospitals and all the hospitals in the database.

After adjustment for hospital volume, US News and World Report's best cardiovascular hospitals did not beat the 30-day mortality rates of other hospitals for any of the four procedures, but HealthGrades' best hospitals had lower volume-adjusted mortality rates for aortic-valve repair and coronary bypass. The superior mortality for aortic-valve repair and coronary bypass at the centers on the HealthGrades' "Best Hospitals" list could be a result of other unmeasured differences in hospital quality, such as hospital infrastructure or unmeasured surgical processes of care, the authors conclude.

Given that procedure volume is the best predictor of outcomes, the authors conclude that while the hospitals that make these lists of best hospitals certainly provide high-quality care, many hospitals that also provide high-quality care are not on the lists.

"While we did find a mortality benefit to choosing these highly rated hospitals, patients can benefit equally from going to a similarly high-volume hospital closer to their home." The authors cite previous research showing that over 75% of patients in the US live within 30 minutes of a high-volume hospital. Unfortunately, "these ratings fail to identify equally performing high-volume hospitals closer to the patients."

Transparency Needed

Commenting on the Osborne study, Krumholz told heartwire, "These systems that are judging others should make their methodology completely transparent. They should submit their methods to external evaluation, ideally through the National Quality Forum. We need to have high standards applied to systems that are seeking to steer patients to high-quality institutions. As they judge others, they need to be willing to be judged.

"Making performance visible, through the use of tested and credible measurement that focuses on outcomes of importance to patients, is sorely needed in our healthcare system. Without these measures, performance is invisible, and it is difficult to muster efforts to improve or even know what is being achieved by current efforts," he explained. "Certainly I expect measurement to grow, and our challenge will be to ensure that the efforts strengthen our ability to improve and do not beget unintended consequences that diminish the quality of care."