Advertisement

Meaningful measure of performance: A foundation built on valid, reproducible findings from surveillance of health care-associated infections

  • Scott K. Fridkin
    Correspondence
    Address correspondence to Scott K. Fridkin, MD, Centers for Disease Control and Prevention, Surveillance Branch, Division of Healthcare Quality Promotion, 1600 Clifton Rd, Mailstop A24, Atlanta, GA 30333.
    Affiliations
    Surveillance Branch, Division of Healthcare Quality Promotion, Centers for Disease Control and Prevention, Atlanta, GA
    Search for articles by this author
  • Russell N. Olmsted
    Affiliations
    Infection Prevention & Control Services, Saint Joseph Mercy Health System, Ann Arbor, MI
    Search for articles by this author
      A tenet of public health practice is that surveillance systems must be able to evolve in response to ever changing needs of the communities and society they serve. In the case of health care-associated infection (HAI) surveillance, the needs of patients, providers, other consumers, and payers of health care have become drivers of recent evolution, although sometimes they appear to generate movement in different directions. A leading example is the apparent tension between comprehensive public reporting of facility-specific HAI data intended to meet mounting consumer and payer demands for a broad set of comparative quality measures and the more focused, traditional surveillance of select HAIs developed to satisfy the needs of providers for data to identify internal HAI prevention priorities and measure the impact of interventions within their institutions. In the past, if infections that met HAI surveillance criteria were deemed nonpreventable by clinicians, the consequences were limited to debates within the facility and internal decisions about how best to use the data. However, the advent of state-based mandates for HAI reporting and public release of facility-specific HAI data, coupled with a new federal HAI reporting requirement, have cast a spotlight on certain operational methods and definitional criteria used. Methods and criteria considered sufficient for HAI surveillance within facilities have elicited criticism when proposed as a means to measure performance.
      The prime focus of these critiques is public reporting of central line-associated bloodstream infection (CLABSI) data collected in accordance with the methods and criteria specified by the Centers for Disease Control and Prevention’s (CDC) National Healthcare Safety Network (NHSN).
      Niedner MF and the 2008 National Association of Children’s Hospitals and Related Institutions Pediatric Intensive Care Unit Patient Care FOCUS Group
      The harder you look, the more you find: catheter-associated bloodstream infection surveillance variability.
      • Lin M.Y.
      • Hota B.
      • Khan Y.M.
      • Woeltje K.W.
      • Borlawsky T.B.
      • Doherty J.A.
      • et al.
      Quality of traditional surveillance for public reporting of noscomial bloodstream infection rates.
      • Backman L.A.
      • Melchreit R.
      • Rogriguez R.
      Validation of the surveillance and reporting of central-line associated bloodstream infection data to a state health department.
      • Sexton D.J.
      • Chen L.F.
      • Anderson D.J.
      Current definitions of central line-associated bloodstreem infection: is the emperor wearing clothes?.
      In 22 states and the District of Columbia, the NHSN provides the reporting requirements and technical infrastructure for mandatory HAI reporting by health care facilities in the state or District’s jurisdiction. In all 23 jurisdictions, CLABSI reporting is mandated, making CLABSIs the most frequently mandated HAI where use of the NHSN is required. Beginning in January 2011, hospitals participating in the Centers for Medicare and Medicaid Services (CMS) Hospital Inpatient Quality Reporting Program are using the NHSN to report CLABSIs among adult, pediatric, and neonatal intensive care unit patients. The CLABSI data reported via the NHSN to the CMS will be used to qualify hospitals for their annual payment update and for public reporting at the Department of Health and Human Services Hospital Compare Web site.
      This rapidly expanding use of NHSN for mandatory public reporting of CLABSIs has highlighted several issues. First, some clinicians have taken issue with whether particular instances of CLABSI that meet surveillance criteria are preventable and hence fairly included in publicly reported data used as a comparative quality measure or even accurately attributed to an indwelling central line.
      • Sexton D.J.
      • Chen L.F.
      • Anderson D.J.
      Current definitions of central line-associated bloodstreem infection: is the emperor wearing clothes?.
      Under these circumstances, infection preventionists (IPs) may be thrust into the unenviable position of adhering to traditional surveillance criteria, as stipulated by a mandatory reporting program, whereas clinical colleagues, with whom IPs seek to collaborate on HAI prevention, challenge the application of traditional criteria in particular instances on the grounds that the infections were not preventable or were not central line related. An additional concern is whether facilities joining the NHSN to meet reporting requirements, many of them smaller and with few or no dedicated infection prevention staff than facilities previously reporting to the NHSN, are able to assume the burden of identifying and reporting CLABSIs. The patient populations served and characteristics of central venous access in these facilities likely differ compared with larger facilities. Will the variation in surveillance capacity and intensity across the spectrum of facilities, rather than true differences in infection frequency among patients, account for publicly reported differences in CLABSI frequency? In other words, will NHSN users report CLABSIs reliably across the spectrum of health care facilities currently and soon to be reporting to the NHSN?
      These issues and concerns are not new, but they are newly highlighted by the rapid transition of the NHSN from a purely voluntary, confidential epidemiologic surveillance system when it was launched in 2005 to a predominantly mandatory and public reporting system 5 years later. As this transition has accelerated, the different uses of the NHSN have prompted criticisms that warrant resolute attention and carefully calibrated changes that preserve the NHSN’s foundation in sound epidemiologic principles while meeting needs of multiple new users and uses.
      Towards this end, several challenges become clear. One major challenge is making changes that assure simplicity of reporting specifications and their use in surveillance operations. Every decision regarding changes in criteria or methodology in the NHSN needs to be made considering the impact the changes will have on the 4,000 facilities enrolled in the system and over 9,000 unique users. There is substantial variability among members of this “cohort” in terms of infrastructure and staffing of infection prevention and control programs. Others have described the increasing scope of infection prevention and control programs that create a tension between desire for more data with little if any change in resources for these and other important activities.
      • Stricof R.L.
      • Schabses K.A.
      • Tserenpuntsag B.
      Infection control resources in New York State hospitals, 2007.
      • Stone P.W.
      • Dick A.
      • Pogorzelska M.
      • Horan T.C.
      • Furuya E.Y.
      • Larson E.
      Staffing and structure of infection prevention and control programs.
      Mitigating this variation and addressing limits on resources will continue to necessitate training, practice, and networking among IPs; this support has been developed by the NHSN and facilitated by professional organizations such as the Association for Professionals in Infection Control and Epidemiology, Inc, and Society for Healthcare Epidemiology of America, which provide training of new members and ongoing education.

      The Society for Healthcare Epidemiologist of America. Available from: http://www.shea-online.org/. Accessed January 3, 2011.

      The Association for Professionals in Infection Control. Available from: http://www.apic.org. Accessed January 3, 2011.

      Thanks in large part to funding provided through the American Recovery and Reinvestment Act, state health departments are now increasingly taking a leadership role in providing the NHSN training and user support to health care facilities in their jurisdictions.

      CDC. Healthcare-associated infections: Recovery act. Resources for state partners in HAI elimination. Available from: http://www.cdc.gov/HAI/recoveryact/stateResources/stateResources.html. Accessed January 28, 2011.

      NHSN leadership is working to reduce data collection burden by evaluating streamlined operational methods, considering their ability to maintain utility and value of the data. Towards this end, the NHSN has created the NHSN Steering Working Group, charged with reviewing proposed changes in methodology and providing perspectives on variation in infrastructure and practical realities of NHSN users. This group is composed of 18 people representing external stakeholder organizations (Appendix) and NHSN users.
      A second challenge is minimizing variability in the surveillance process to ensure case finding can be reliably applied; this is essential for providing a valid comparison between institutions.
      • Terris D.D.
      • Litikar D.G.
      Data quality bias: an underrecognized source of misclassification in pay-for-performance reporting?.
      Current methods allow for some subjectivity in applying surveillance case definitions and may explain some differences in reported CLABSI rates. This subjectivity involves evaluating the presence of symptoms or more often to subjectivity in the classification of secondary bloodstream infections as CLABSIs. Related to this is the fact that some of the NHSN primary HAI infection types have definitions with more subjective components (eg, symptoms, radiographic findings) than CLABSI; the IP determination of defined HAIs at these infection types are also vulnerable to misclassification if documentation in the medical records is suboptimal or the reviewer is unfamiliar with the definitions. The use of “maximal information” through a consensus approach, eg, multidisciplinary review of CLABSI events, is attractive because it could foster a team approach to patient care, promote patient safety, and potentially identify some problems needing attention. However, this approach requires an increase in resources and could inject even greater subjectivity into the application of surveillance criteria. The result would exacerbate the variability in the application of surveillance case definitions between institutions because all reporting facilities will not have equivalent access to staffing, expertise, clinical laboratory methods, and information sources. In addition, an increased use of clinical data and multidisciplinary review of possible CLABSI events tips the balance toward increased burden of data collection, injecting more variability, and diminishing the accuracy of interfacility comparisons or comparison with an external benchmark. On the other hand, precluding this additional data or input from local experts can lead to a loss of credibility with the health care provider community.
      At the heart of this particular tension between reliability and credibility is the application of surveillance definitions to measurement of clinical performance.
      • Sexton D.J.
      • Chen L.F.
      • Anderson D.J.
      Current definitions of central line-associated bloodstreem infection: is the emperor wearing clothes?.
      To help resolve this tension, an immediate priority of the NHSN is assessment of best means to improve the definitions and to incorporate increasingly encountered scenarios in which mucositis or gastrointestinal-related fungemia or enterococcal bacteremia are more likely related to something other than CLABSI. Whereas substantial data supporting routine classification of coagulase-negative staphylococci as a common skin commensal justify treating these organisms differently than recognized pathogens, the same is not true for Enterococcus spp or Candida spp; recent and older studies that incorporated detailed medical record reviews to categorize the clinical significance of blood cultures documented that enterococcal bacteremia represents bloodstream infection in most cases (63%-70%).
      • Pien B.C.
      • Sundaram P.
      • Raoof N.
      • Costa S.F.
      • Mirrett S.
      • Woods C.W.
      • et al.
      The clinical and prognostic importance of positive blood cultures in adults.
      • Weinstein M.P.
      • Towns M.L.
      • Quartey S.M.
      • Mirrett S.
      • Reimer L.G.
      • Parmigiani G.
      • et al.
      The clinical significance of positive blood cultures in the 1990s: a prospective comprehensive evaluation of the microbiology, epidemiology, and outcome of bacteremia and fungemia in adults.
      One specialized study using enhanced methods to maximize specificity of catheter-related bloodstream infection found that vancomycin-resistant enterococci were as likely to be associated with infection as contamination.
      • Shuman E.K.
      • Washer L.L.
      • Arndt J.L.
      • Zalewski C.A.
      • Hyzy R.C.
      • Napolitano L.M.
      • et al.
      Analysis of central line-associated bloodstream infections in the intensive care unit after implementation of central line bundles.
      Therefore, more data demonstrating a substantial subset of enterococci or other possible candidate microorganisms can be routinely dismissed as contaminant is needed before these should be reclassified as common skin commensals. Such changes, when supported by consistent, reproducible evidence, can improve the ability with which infection prevention staff are able to identify and classify a primary type of infection using simple, standard criteria and improve credibility among both clinical providers and consumers. However, it remains an institution’s responsibility to evaluate the potential preventability of all CLABSIs reported to the NHSN to determine whether local practice, as well as changes in the patient mix, laboratory methods (eg, blood culturing practices contributing to frequent contamination), and other factors may be influencing the occurrence of CLABSIs.
      An additional strategy to ease this tension relates to automated capture of electronic data elements to aid the detection and reporting of CLABSI; this approach has the potential to both minimize variability (increase reliability) and decrease data collection burden.
      • Edwards J.R.
      • Pollock D.A.
      • Kupronis B.A.
      • Li W.
      • Tolson J.S.
      • Peterson K.D.
      • et al.
      Making use of electronic data: the National Healthcare Safety Network eSurveillance Initiative.
      Much scientific progress has occurred in evaluating the use of computer algorithms not only to improve the workflow of the IP but to improve detection of possible CLABSI for reporting as part of a surveillance program. This progress is highlighted by recent research funded through the CDC’s Prevention Epicenters Program and published by Dr. Mike Lin et al in the Journal of the American Medical Association.
      • Lin M.Y.
      • Hota B.
      • Khan Y.M.
      • Woeltje K.W.
      • Borlawsky T.B.
      • Doherty J.A.
      • et al.
      Quality of traditional surveillance for public reporting of noscomial bloodstream infection rates.
      This study demonstrated that some interinstitutional variability between CLABSI rates is due to variability in IP implementing standard methods, either because of subjectivity or inconsistency in applying methodology. These findings have led to ongoing collaborations with large numbers of NHSN facilities to demonstrate the use of proxy measures obtained through an automated process to create an objective CLABSI measure that could eliminate subjectivity at a defined loss of specificity. Although the infection prevention community will unlikely rely completely on proxy measures, if some automated HAI measures could be reliably applied across all facilities and remain credible to the provider community, consumers, and payers, many more objectives of HAI surveillance could be fulfilled. Toward this end, the NHSN is committed to exploring the utility of algorithmically detected CLABSI and moving to full-scale implementation once this approach is interpreted as credible in the health care epidemiology community and once the technology is widely available in health care.
      A third challenge is assuring that data reported to the NHSN for public reporting and other purposes are systematically validated. The CDC has established strong partnerships with state departments of health to develop and extend processes for systematic validation of CLABSI data. Since 2009, the CDC has funded 51 states and territories to develop an HAI program; validation of HAI data (including CLABSI data reported to the NHSN) is underway or planned in many of these states or territories. These efforts have provided considerable insight into challenges in assuring consistent, valid application of NHSN infection type definitions. Although the validation approach has varied among states, and health department HAI coordinators have indicated variable sensitivity of CLABSI reporting in their states,
      • Backman L.A.
      • Melchreit R.
      • Rogriguez R.
      Validation of the surveillance and reporting of central-line associated bloodstream infection data to a state health department.

      Kainer MA, Mitchell J, Frost BA, Soe MM. Validation of central line associated blood stream infection [CLABSI] data submitted to the National Healthcare Safety Network [NHSN]: a pilot study by the Tennessee Department of Health [TDH]. (abstract). Presented at 5th Decennial International Conference on Healthcare-Associated Infections. March 20, 2010. Atlanta, GA. Available from: http://shea.confex.com/shea/2010/webprogram/Paper2238.html. Accessed January 3, 2011.

      there is evidence of improved accuracy as state-based validation continues. For example, during 2007-2009, the New York State Department of Health found that sensitivity of CLABSI reporting remained at around 74% and specificity increased from 90% to 98.5% (personal communication, Carole Van Anwerpen, New York State Department of Health, December 2010). However, these efforts serve a greater purpose than just ensuring improved accuracy of the data: they also raise awareness of actionable data and improve implementation strategies.
      The CDC and partners are committed to making changes to the NHSN so that the system is better able to meet the needs of an ever growing number of users and uses. The need to balance the burden of data collection and the benefit of having local data for action have never been more critical. Considering the importance of ensuring that a level playing field exists for all facilities to implement and operationalize NHSN surveillance methodologies for reporting to the CMS, we believe maximizing reliable case finding may at times take priority to clinical relevance of case reporting. We encourage facilities to evaluate and determine the preventability (eg, blood culturing contamination problems, mucositis-related bacteremia) of cases reported for internal purposes. However, the preventability assessment is not part of routine surveillance reporting because redefining cases based on subjective criteria is not acceptable in the current reporting paradigm. The initiatives outlined above, including state-based validation efforts, enhancing reliability through simplification of methods, review and modification of operations and definitions, and exploring the use of electronically derived data elements, demonstrate the commitment of the NHSN leadership to refine the system’s methodology. Such modifications should maximize reliable case identification, be responsive to new scientific findings, justify changes, and simplify implementation through utilization of advances in health information technology, while maintaining credibility through partnerships with state health departments on data validation.

      Appendix

      National Healthcare Safety Network (NHSN) Steering Working Group includes external stakeholder representatives from state health departments, NHSN users, CDC prevention Epi-Centers, Society for Healthcare Epidemiology, Association for Professionals in Infection Control and Epidemiology (APIC), the American Hospital Association (AHA), Council of State and Territorial Epidemiologists (CSTE), the Association of State and Territorial Health Officers (ASTHO), and the US Department of Health and Human Services: Health Resources and Services Administration (HRSA), Centers for Medicare and Medicare Services (CMS), Agency for Healthcare Research and Quality (AHRQ).

      References

        • Niedner MF and the 2008 National Association of Children’s Hospitals and Related Institutions Pediatric Intensive Care Unit Patient Care FOCUS Group
        The harder you look, the more you find: catheter-associated bloodstream infection surveillance variability.
        Am J Infect Control. 2010; 38: 585-595
        • Lin M.Y.
        • Hota B.
        • Khan Y.M.
        • Woeltje K.W.
        • Borlawsky T.B.
        • Doherty J.A.
        • et al.
        Quality of traditional surveillance for public reporting of noscomial bloodstream infection rates.
        JAMA. 2010; 304: 2035-2041
        • Backman L.A.
        • Melchreit R.
        • Rogriguez R.
        Validation of the surveillance and reporting of central-line associated bloodstream infection data to a state health department.
        Am J Infect Control. 2010; 38: 832-838
        • Sexton D.J.
        • Chen L.F.
        • Anderson D.J.
        Current definitions of central line-associated bloodstreem infection: is the emperor wearing clothes?.
        Infect Control Hosp Epidemol. 2010; 31: 1286-1289
        • Stricof R.L.
        • Schabses K.A.
        • Tserenpuntsag B.
        Infection control resources in New York State hospitals, 2007.
        Am J Infect Control. 2008; 36: 702-705
        • Stone P.W.
        • Dick A.
        • Pogorzelska M.
        • Horan T.C.
        • Furuya E.Y.
        • Larson E.
        Staffing and structure of infection prevention and control programs.
        Am J Infect Control. 2009; 37: 351-357
      1. The Society for Healthcare Epidemiologist of America. Available from: http://www.shea-online.org/. Accessed January 3, 2011.

      2. The Association for Professionals in Infection Control. Available from: http://www.apic.org. Accessed January 3, 2011.

      3. CDC. Healthcare-associated infections: Recovery act. Resources for state partners in HAI elimination. Available from: http://www.cdc.gov/HAI/recoveryact/stateResources/stateResources.html. Accessed January 28, 2011.

        • Terris D.D.
        • Litikar D.G.
        Data quality bias: an underrecognized source of misclassification in pay-for-performance reporting?.
        Q Manage Health Care. 2008; 28: 19-26
        • Pien B.C.
        • Sundaram P.
        • Raoof N.
        • Costa S.F.
        • Mirrett S.
        • Woods C.W.
        • et al.
        The clinical and prognostic importance of positive blood cultures in adults.
        Am J Med. 2010; 123: 819-828
        • Weinstein M.P.
        • Towns M.L.
        • Quartey S.M.
        • Mirrett S.
        • Reimer L.G.
        • Parmigiani G.
        • et al.
        The clinical significance of positive blood cultures in the 1990s: a prospective comprehensive evaluation of the microbiology, epidemiology, and outcome of bacteremia and fungemia in adults.
        Clin Infect Diseases. 1997; 24: 584-602
        • Shuman E.K.
        • Washer L.L.
        • Arndt J.L.
        • Zalewski C.A.
        • Hyzy R.C.
        • Napolitano L.M.
        • et al.
        Analysis of central line-associated bloodstream infections in the intensive care unit after implementation of central line bundles.
        Infect Control Hosp Epidemiol. 2010; 31: 551-553
        • Edwards J.R.
        • Pollock D.A.
        • Kupronis B.A.
        • Li W.
        • Tolson J.S.
        • Peterson K.D.
        • et al.
        Making use of electronic data: the National Healthcare Safety Network eSurveillance Initiative.
        Am J Infect Control. 2008; 36: S21-S26
      4. Kainer MA, Mitchell J, Frost BA, Soe MM. Validation of central line associated blood stream infection [CLABSI] data submitted to the National Healthcare Safety Network [NHSN]: a pilot study by the Tennessee Department of Health [TDH]. (abstract). Presented at 5th Decennial International Conference on Healthcare-Associated Infections. March 20, 2010. Atlanta, GA. Available from: http://shea.confex.com/shea/2010/webprogram/Paper2238.html. Accessed January 3, 2011.