If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to Chatura Prematunge, MSc, Infection Prevention and Control Research, 2380 St Laurent Blvd, Ottawa, ON K1G 6C4, Canada. (C. Prematunge).
Public Health Ontario, Toronto, Ontario, CanadaSt Joseph's Health Center, Toronto, Ontario, CanadaDepartment of Medicine, University of Toronto, Toronto, Ontario, CanadaDalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada
Public Health Ontario, Toronto, Ontario, CanadaDepartment of Medicine, University of Toronto, Toronto, Ontario, CanadaDepartment of Medicine, University of Ottawa, Ottawa, Ontario, Canada
Over 25% of VRE bacteremias were incorrectly reported in public reporting.
•
Quarterly audits were linked to reduced reporting errors among Ontario's hospitals.
•
Independent and routine audits of VRE bacteremia are associated with improved data quality.
Background
All Ontario hospitals are mandated to self-report vancomycin-resistant enterocococi (VRE) bacteremias to Ontario's Ministry of Health and Long-term Care for public reporting purposes. Independent quarterly audits of publicly reported VRE bacteremias between September 2013 and June 2015 were carried out by Public Health Ontario. VRE bacteremia case-reporting errors between January 2009 and August 2013 were identified by a single retrospective audit.
Methods
Employing a quasiexperimental pre–post study design, the relative risk of VRE bacteremia reporting errors before and after quarterly audits were modeled using Poisson regression adjusting for hospital type, case counts reported to the Ministry of Health and Long-term Care, and autocorrelation via generalized estimating equation.
Results
Overall, 24.5% (126 out of 514) of VRE bacteremias were reported in error; 114 out of 367 (31%) VRE bacteremias reported before quarterly audits and 12 out of 147 (8.1%) reported after audits were found to be incorrect. In adjusted analysis, quarterly audits of VRE bacteremias were associated with significant reductions in reporting errors when compared with before quarterly auditing (relative risk, 0.17; 95% confidence interval, 0.05-0.63). Risk of reporting errors among community hospitals were greater than acute teaching hospitals of the region (relative risk, 4.39; 95% CI, 3.07-5.70).
Conclusions
This study found independent quarterly audits of publicly reported VRE bacteremias to be associated with significant reductions in reporting errors. Public reporting systems should consider adopting routine data audits and hospital-targeted training to improve data accuracy.
Many jurisdictions across North America have implemented mandatory HAI reporting systems that require hospitals to routinely self-report HAIs for public disclosure.
Public reporting of HAIs is intended to communicate health care quality and patient safety, drive health care performance, and provide evidence to support future health care programs and policies.
Although HAI public reporting guidelines recommend regular data validations and reviews of reported HAI data, most HAI reporting systems do not disclose their quality assurance processes nor the findings of these processes.
Maryland Health Care Commission Health Care-Associated Infection Technical Advisory Committee Public reporting of health care-associated infections (HAIs): approach to choosing HAI measures.
Bloodstream infections caused by vancomycin-resistant enterococci (VRE) bacteremia are associated with significant morbidity and mortality, and considered to be a patient safety priority in many jurisdictions, including Ontario, Canada.
To investigate VRE bacteremia rate changes, Public Health Ontario (PHO), an arm's-length scientific and technical service agency of the Ontario government, began a quarterly audit of Ontario's publicly reported VRE bacteremia cases with reporting hospitals.
The quarterly audits were conducted through a separate and independent process from public reporting. The objective of this study was to examine the influence of PHO's independent quarterly audits on VRE bacteremia case reporting errors within Ontario's publicly available HAI data.
Methods
Setting
Since January 2009, under the Canada Public Hospitals Act, all Ontario hospitals (n = 219) have the mandate to self-report VRE bacteremias that meet reporting definitions to Ontario's Ministry of Health and Long-term Care (MOHLTC). MOHLTC's definition for a reportable VRE bacteremia is a single positive laboratory-confirmed blood culture with VRE strains of Enterococcus faecium or Enterococcus faecalis, with a minimum inhibitory concentration ≥32 µg/mL to vancomycin and containing the resistance genes vanA or vanB.
VRE bacteremias are reported by hospital facilities to MOHLTC as a key hospital patient safety performance indicator, and is publicly available on the Health Quality Ontario Web site at 3-month reporting quarter intervals.
Quarterly audits of publicly reported VRE bacteremias (intervention period)
Hospitals publicly reporting VRE bacteremias between September 2013 and June 2015 were subjected to quarterly audits matching public reporting quarter intervals. Quarterly aggregated VRE bacteremia data reported by hospitals to MOHLTC and publicly reported were used by PHO as the source data to create standardized case reports. At the end of each reporting quarter, a study team at PHO e-mailed standardized case report forms to each reporting hospital's Infection Prevention and Control (IPC) department contact(s) (Appendix 1). Hospitals participated in the quarterly audits by reviewing patient records to confirm whether each VRE bacteremia identified within the case report forms were correctly reported and met MOHLTC reporting case definitions.
When a VRE bacteremia reporting error was identified, the reporting hospital provided reason(s) for the error and corrected the error with MOHLTC public reporting. The study team followed-up with hospital IPC contacts via e-mail messages and telephone calls until all case report forms were completed and returned to PHO.
Retrospective audit of publicly reported VRE bacteremias (preintervention period)
Quarterly aggregated VRE bacteremia data reported by hospitals to MOHLTC and publicly reported between January 2009 and September 2013 were confirmed by PHO with each reporting hospital's IPC contact(s).
VRE bacteremia reporting error reasons
After completion of the audits, all reasons for reporting errors provided by the hospitals were categorized by the study team into the following reporting error categories: VRE-positive screening cultures, reported to MOHTLC in error, Enterococcus species not included within MOHLTC VRE bacteremia case definitions, or reason unknown.
Research ethics approval
The study received research ethics board approval from PHO before its commencement.
Statistical analysis
Descriptive statistics were generated to summarize VRE bacteremia reporting errors and reasons for reporting errors. Applying a quasiexperimental, pre–post intervention study design, we defined the quarterly audit period as the intervention of interest and compared VRE bacteremia reporting error proportions before and after quarterly auditing. The relative risk (RR) of VRE bacteremia reporting errors for reporting quarters before and after quarterly auditing was modeled using Poisson regression fitted using the generalized estimating equation to account for autocorrelation due to repeated measures within hospital sites.
Extending the simple linear regression model to account for correlated responses: an introduction to generalized estimating equations and multi-level mixed modelling.
Among-hospital variability in reporting errors were adjusted for in all models using source MOHLTC data (ie, quarterly VRE bacteremia case counts initially reported by hospitals to MOHLTC). To better compare VRE bacteremia reporting errors based on hospital characteristics, including similar patient acuity and patient case-mix, data were stratified based on reporting hospital type (acute teaching vs small and large community hospitals) using Ontario Hospital Association classifications, and the final model adjusted for reporting hospital type.
Extending the simple linear regression model to account for correlated responses: an introduction to generalized estimating equations and multi-level mixed modelling.
RR of VRE bacteremia reporting errors and associated 95% confidence intervals (95% CIs) are reported for all model based analyses, and a 2-tailed P value < .05 was deemed statistically significant. SAS (version 9.3, SAS Institute Inc, Cary, NC) was used in the data analysis.
Results
Based on publicly reported HAI data collected by MOHLTC, 1 or more VRE bacteremias were identified by 39.7% of Ontario hospitals (87 out of 219 hospital sites) during the study period, and 98.8% of these hospital sites (86 hospital sites) participated in the data audits. In our hospital sample, 75.6% of sites reporting VRE bacteremia cases (65 out of 86 hospital sites) were small and large community hospitals, whereas the remaining 24.4% of reporting hospitals (21 out of 86 hospital sites) were acute teaching hospitals. In total, there were 514 VRE bacteremias reported during the study period and 24.5% of the reported VRE bacteremia cases (126 out of 514 cases) were determined to be incorrectly reported cases by the data audits. When reporting errors were examined based on audit period, 31.1% of VRE bacteremia reporting errors (114 out of 367 cases) were found to have occurred before the implementation of quarterly audits, whereas 8.2% of reporting errors (12 out of 147 cases) occurred during quarterly auditing. VRE bacteremia reporting errors, case counts, and error proportions before, during, and after quarterly auditing are presented in Table 1. The majority of identified reporting errors were reported by community hospital sites 78.6% (99 out of 126 cases). VRE bacteremia reporting error proportions by reporting quarter and hospital type are presented in Figure 1.
Table 1Vancomycin-resistant Enterococcus bacteremia case counts publicly reported (ie, Ontario Ministry of Health and Long-term Care source data) in Ontario Canada, includes identified case reporting errors and proportions of reporting errors by audit period and hospital type
Fig 1Proportions of vancomycin-resistant Enterococcus (VRE) bacteremia case-reporting errors within publicly reported health care-associated infection data in Ontario, Canada, by reporting quarter and reporting year (n = 592). The grey area represents the VRE bacteremia quarterly audit period.
Incorrect reporting of VRE-positive screening rectal swabs (ie, VRE colonizations) as VRE bacteremias was the most frequently provided reason for reporting errors during the overall study period at 69% (87 out of 126 cases), as well as before quarterly auditing at 69.3% (79 out of 114 cases) and after quarterly auditing at 66.7% (8 out of 12 cases). Alternate reasons for the identified VRE bacteremia reporting errors included missing or inaccessible health records. All reasons for reporting errors are presented in Figure 2.
Fig 2Flow diagram of vancomycin-resistant Enterococcus (VRE) bacteremia reporting error reasons before and after quarterly auditing of publicly reported VRE bacteremia in Ontario, Canada. MOHLTC, Ontario Ministry of Health and Long-term Care.
The unadjusted and adjusted analyses of all reporting hospitals demonstrated reporting errors were significantly reduced following the initiation of quarterly audits (unadjusted RR, 0.17; 95% CI, 0.06-0.52; adjusted RR, 0.17; 95% CI, 0.05-0.63). Stratified analysis revealed the magnitude of VRE bacteremia reporting error risk reductions associated with quarterly auditing to be similar among community hospitals (RR, 0.24; 95% CI, 0.07-0.80) and acute teaching hospitals (RR, 0.30; 95% CI, 0.06-1.40]). The risk of VRE bacteremia reporting errors among community hospitals was found to be 4.39 (95% CI, 3.07-5.70) times greater than among acute care hospitals within the final adjusted analysis.
Discussion
In this study, we demonstrated that independent quarterly audits of publicly reported VRE bacteremia case data are associated with significant reductions in case reporting error risk. The reductions in reporting errors are especially noteworthy because more than 30% of VRE bacteremias reported to HAI public reporting before quarterly auditing (114 out of 367 cases) were incorrectly reported cases.
Although some hospitals continued to report incorrect VRE bacteremias to public reporting throughout the study period, we found quarterly auditing to be associated with a substantial reduction in reporting errors across all hospitals. Based on these findings, we hypothesize that quarterly auditing of publicly reported HAI data provide hospital staff with a better knowledge of HAI data collection criteria and reporting case definitions. This regular feedback on reporting errors may lead to an associated learning effect.
We identified a number of additional studies within the existing literature to have assessed the accuracy of publicly reported HAI data reported by hospitals.
Validity of the agency for health care research and quality patient safety indicators and the centers for Medicare and Medicaid hospital-acquired conditions: a systematic review and meta-analysis.
Although these studies report on the validity of central line-associated bloodstream infections (CLABSIs) via patient chart reviews, these studies also find 30%-50% of CLABSI cases within public reporting systems to be incorrectly reported.
Validity of the agency for health care research and quality patient safety indicators and the centers for Medicare and Medicaid hospital-acquired conditions: a systematic review and meta-analysis.
However, in contrast to our own findings, underreporting of cases appears to be the greatest source of data inconsistencies between hospital records and publicly reported CLABSI data for most instances.
Validity of the agency for health care research and quality patient safety indicators and the centers for Medicare and Medicaid hospital-acquired conditions: a systematic review and meta-analysis.
Differences in overreporting versus underreporting of cases in HAI public reporting may be due to differences in case identification methods or surveillance definitions across HAI reporting systems, as well as between CLABSI versus VRE bacteremia cases. Despite these differences, we note similar data accuracy and data validity issues among a number of unrelated publicly reported HAI datasets from different jurisdictions.
Validity of the agency for health care research and quality patient safety indicators and the centers for Medicare and Medicaid hospital-acquired conditions: a systematic review and meta-analysis.
The reported limited validity within these publicly reported HAI datasets may be due to these reporting systems relying on hospitals to passively audit self-reported data in the absence of independent and routine data reviews or other quality assessment processes.
Another key finding of our analysis is the variability in VRE bacteremia reporting errors based on reporting hospital type. Our main analysis revealed VRE bacteremia reporting error risk among community hospitals to be 4.39 fold (95% CI, 3.07-5.70) higher when compared with acute care hospitals. VRE bacteremias are a hospital patient safety performance indicator in Ontario and are typically reported by IPC personnel, either directly to public reporting or indirectly through the hospital's decision support. Therefore, the variability in reporting errors by hospital type may be due to more frequent identification and treatment of VRE bacteremias among Ontario's acute teaching hospital patient populations, a consequence of acute care hospitals more often providing highly complex patient care for severe illness and diseases associated with increased burden of VRE.
The frequent identification and treatment of VRE bacteremias among acute teaching hospitals may have contributed to the observed hospital type variability in VRE bacteremia reporting errors. Nevertheless, the marked reductions in VRE bacteremia reporting errors among all hospital types following the initiation of quarterly auditing highlight the need for additional support and training on HAI reporting to target all hospitals of the region. The need for HAI public reporting support and training among hospitals is further emphasized by hospital-provided reasons for VRE bacteremia reporting errors. The most frequently cited reasons for reporting errors before and during quarterly auditing were related to misreporting of VRE-positive rectal swabs, which are a positive diagnosis of VRE colonization but not a VRE bacteremia (ie, VRE-positive bloodstream infection). The reporting of VRE colonizations as VRE bacteremias point to a need for better knowledge and application of reporting case definitions, case identification, and record management practices by hospital personnel submitting data for HAI public reporting.
A major strength of this study is that 99% of VRE bacteremia-reporting hospitals in Ontario participated in the confirmation of publicly reported HAI data reported over a 6-year reporting period via the review of hospital patient records. This study also had several limitations. First, this study used a quasiexperimental design that lacked a control group, and as such the influences of this intervention may have been explained by changes in reporting that were not explicitly due to the intervention. Second, this study was dependent on self-review of previously reported VRE bacteremia cases by reporting hospitals. Third, quarterly audits were limited to the confirmation of VRE bacteremias reported by hospitals to public reporting, and as such underreporting was not captured. Although the literature reports underreporting to be the greatest source of data inconsistencies in publicly reported HAI datasets, underreporting was not a serious issue within the data reviewed by this study; VRE bacteremia counts confirmed to be accurately reported to public reporting by hospitals through the data audits were comparable to VRE bacteremia counts reported by hospital laboratories to the Institute for Quality Management in Healthcare, in a separate dataset collection.
In this study, we found more than one-quarter of publicly reported VRE bacteremia cases to be incorrectly reported by hospitals. Yet, an independent and quarterly audit of these data was associated with reductions in reporting errors, signaling a learning effect among reporting hospitals. Our findings suggest HAI reporting systems should consider implementing routine and independent data audits and offer hospital-targeted training and support on HAI reporting case definitions, case identifications, and case reporting processes to reduce HAI reporting errors.
Conclusions
Our independent and routine audits of VRE bacteremias within a regional publicly reported HAI data source were associated with reductions in reporting errors, and as a result help improve the quality of publicly available HAI data.
Appendix 1. Sample view of standardized case report forms used in data audits
Extending the simple linear regression model to account for correlated responses: an introduction to generalized estimating equations and multi-level mixed modelling.
Validity of the agency for health care research and quality patient safety indicators and the centers for Medicare and Medicaid hospital-acquired conditions: a systematic review and meta-analysis.
30138-X&id=ymic4749-fig-0001.jpg){kind=link}
30138-X&id=ymic4749-fig-0002.jpg){kind=link}
30138-X&id=ymic4749-fig-0003.jpg){kind=link}