Advertisement

The impact of automatic video auditing with real-time feedback on the quality and quantity of handwash events in a hospital setting

Open AccessPublished:July 27, 2019DOI:https://doi.org/10.1016/j.ajic.2019.06.015

      Background

      Poor quality handwashing contributes to the spread of nosocomial infections. We investigate the impact of automatic video auditing (AVA) with feedback on the quality and quantity of handwashing in a hospital setting.

      Methods

      AVA systems were mounted over all handwash sinks in a surgical unit. Phase 1 established baseline handwashing quality and quantity. Phase 2 examined the impact of real-time performance feedback, and phase 3 examined the incremental impact of weekly team performance reports. Phase 4 remeasured the baseline without feedback.

      Results

      A total of 3,606 handwash events were audited. During phase 2 and 3, compliance with the World Health Organization technique improved from 15.7%-46% (P < .0001), and the average number of handwash events per patient per day increased from 0.91-2.25 (P < .0001). Performance returned to baseline in phase 4.

      Conclusions

      AVA with real-time feedback significantly improved the quality and quantity of handwashing. The combination of AVA with electronic monitoring will allow simultaneous auditing of hand hygiene quantity and quality. The impact of cognitive offloading onto the technology may have contributed to the return to baseline at the end of the study, and suggests further research is required in this area.

      Key Words

      The hands of health care workers (HCW) are identified as the primary vector of transmission for health care–acquired infections, and as a consequence hand hygiene is critical to patient safety.
      • Allegranzi B
      • Pittet D.
      Role of hand hygiene in healthcare-associated infection prevention.
      The World Health Organization (WHO) guidelines
      World Health Organization
      WHO guidelines on hand hygiene in healthcare: a summary.
      recommend the quantity of hand hygiene be determined by opportunities in the “5 Moments” of patient care,
      • Salmon S
      • Pittet D
      • Sax H
      • McLaws ML
      The 'My Five Moments for Hand Hygiene' concept for the overcrowded setting in resource-limited healthcare systems.
      and that quality hand hygiene uses the “6 Steps of Hand Hygiene” to ensure a significant reduction in the microbial load on hands.
      • Reilly JS
      • Price L
      • Lang S
      • Robertson C
      • Cheater F
      • Skinner K
      • et al.
      A pragmatic randomized controlled trial of 6-step vs 3-step hand hygiene technique in acute hospital care in the United Kingdom.
      Many studies investigate the quantity of hand hygiene in health care settings.
      • Pittet D
      • Hugonnet S
      • Harbarth S
      • Mourouga P
      • Sauvan V
      • Touveneau S
      • et al.
      Effectiveness of a hospital-wide programme to improve compliance with hand hygiene. Infection Control Programme.
      • Lam BC
      • Lee J
      • Lau YL
      Hand hygiene practices in a neonatal intensive care unit: a multimodal intervention and impact on nosocomial infection.
      • Ling ML
      • How KB.
      Impact of a hospital-wide hand hygiene promotion strategy on healthcare-associated infections.
      Boyce
      • Boyce JM
      Measuring healthcare worker hand hygiene activity: current practices and emerging technologies.
      • Boyce JM.
      Current issues in hand hygiene.
      reviewed a wide range of electronic devices to measure the quantity of hand hygiene. Gould et al
      • Gould DJ
      • Moralejo D
      • Drey N
      • Chudleigh JH
      • Taljaard M
      Interventions to improve hand hygiene compliance in patient care.
      reviewed a range of interventions to improve compliance with the 5 Moments, and notes the potential reductions in hospital-acquired infections.
      Relatively few studies investigate the quality of the hand hygiene in health care settings.
      • Tschudin-Sutter S
      • Sepulcri D
      • Dangel M
      • Schuhmacher H
      • Widmer AF
      Compliance with the World Health Organization hand hygiene technique: a prospective observational study.
      • Widmer AF
      • Conzelmann M
      • Tomic M
      • Frei R
      • Stranden AM
      Introducing alcohol-based hand rub for hand hygiene: the critical need for training.
      • Arias AV
      • Garcell HG
      • Ochoa YR
      • Arias KF
      • Miranda FR
      Assessment of hand hygiene techniques using the World Health Organization's six steps.
      These studies show low adherence, 8%-15%, with the WHO 6-step technique for hand hygiene. Various strategies have been used to improve hand hygiene quality, and frequent short training sessions have reduced methicillin-resistant Staphylococcus aureus infection rates.
      • Conrad A
      • Kaier K
      • Frank U
      • Dettenkofer M
      Are short training sessions on hand hygiene effective in preventing hospital-acquired MRSA? A time-series analysis.
      Individualized regular training was more effective than lectures at creating a “culture of prevention,” and resulted in a reduction in bloodstream infections.
      • Lobo RD
      • Levin AS
      • Oliveira MS
      • Gomes LM
      • Gobara S
      • Park M
      • et al.
      Evaluation of interventions to reduce catheter-associated bloodstream infection: continuous tailored education versus one basic lecture.
      In the video auditing of hand hygiene practices, videos are manually coded by researchers to classify and count different behaviors. Video auditing has been used to record activates at handwashing sinks
      • Grabowski M
      • Lobo JM
      • Gunnell B
      • Enfield K
      • Carpenter R
      • Barnes L
      • et al.
      Characterizations of handwashing sink activities in a single hospital medical intensive care unit.
      • Gautschi N
      • Marschall J
      • Candinas D
      • Banz VM
      Effect of music on surgical hand disinfection: a video-based intervention study.
      and in patient rooms,
      • Brouqui P
      • Boudjema S
      • Reynier P
      • Dufour JC
      • Florea O
      • et al.
      Hand hygiene analyzed by video recording.
      but real-time feedback was not provided. Remote video auditing is where remote human operators examine samples of live video footage from the clinical setting and deliver near real-time feedback to a cohort of staff. This approach has been used to improve the quantity of hand hygiene from 10% opportunities to 85%.
      • Armellino D
      • Hussain E
      • Schilling ME
      • Senicola W
      • Eichorn A
      • Dlugacz Y
      • et al.
      Using high-technology to enforce low-technology safety measures: the use of third-party remote video auditing and real-time feedback in healthcare.
      Remote video auditing does not directly address hand hygiene quality, and concerns have been raised regarding its impact on privacy.
      Automatic video auditing (AVA) is where a computer automatically analyzes the video and provides real-time feedback to each individual user. Privacy is ensured as no videos are stored on the device and no videos leave the device. AVA has been shown to improve training outcomes.
      • Ghosh A
      • Ameling S
      • Zhou J
      • Lacey G
      • Creamer E
      • Dolan A
      • et al.
      Pilot evaluation of a ward-based automated hand hygiene training system.
      • Stewardson AJ
      • Iten A
      • Camus V
      • Gayet-Ageron A
      • Caulfield D
      • Lacey G
      • et al.
      Efficacy of a new educational tool to improve handrubbing technique amongst healthcare workers: a controlled, before-after study.
      • Lacey G
      • Corr M
      • Morrow H
      • McQueen A
      • Cameron F
      • Connolly C
      The impact of structured incentives on the adoption of a serious game for hand hygiene training in a hospital setting.
      To our knowledge, this study represents the first use of AVA to measure handwashing quality at sinks and to provide individualized real-time feedback on technique.
      Our study measured the quality and quantity of handwashing by HCWs using AVA with real-time feedback at handwash sinks. The AVA can also be used for measuring the use of alcohol-based handrub (ABHR). However, we chose to focus on handwashing because the AVA does not interrupt clinical workflow at sinks and ensures that all handwashes, no matter the quality, are measured. Unlike handwashing at sinks, ABHR usage is not tied to a specific location and is often performed “on the move,” therefore to use AVA for auditing, HCWs would have to perform handrubs standing in view of the camera. This would change clinical practice and would have allowed HCWs to self-select for auditing, which would have skewed the data.
      The auditing of handwashing in this study was carried out as part of the normal hand hygiene compliance auditing procedure for the hospital. As the study captured less personal information than the normal observational audit procedures, the Department of Health determined that the study did not need additional ethical approval.

      Methods

      A custom-designed AVA system with a computer screen for feedback (SureWash, Dublin, Ireland) was used in this study. An AVA unit was placed over all the handwash sink within an active surgical unit, as shown in Figure 1. In total, 8 AVA units were used to ensure all handwash events were captured. Toilets, bathrooms, and the dirty utility were excluded from the study.
      Fig 1
      Fig 1Map of ward B7 in Broomfield Hospital showing the numbered locations of each of the units.
      The AVA systems provided real-time training feedback to each HCW on their hand hygiene technique. The screen showed each step of the WHO protocol, each with a red/green traffic light symbol. As the HCW completed each step, its associated traffic light changed from red to green. The aim was to get all the traffic lights to turn from red to green. To ensure anonymity, the AVA camera faced straight down so that only the sink could be seen and the HCW's face or any other identifying information was excluded from view.
      The AVA systems operated from 6 AM to 8 PM each day for approximately 4 months. The study was divided into 4 phases: phase 1 (10 days) measured the baseline of handwash quality and quantity without providing any feedback. Phase 2 (32 days) and phase 3 (32 days) aimed to assess the longer-term impact of the AVA system and the associated training feedback on handwashing quality and quantity. In phase 3, incremental feedback was provided by means of a weekly performance report for staff meetings. The aim of this report was to determine if group targets had any impact on HCW hand hygiene performance. During phase 4 (6 days) feedback was turned off and the baseline of handwash performance was measured again.
      As noted in the article by Grabowski et al,
      • Grabowski M
      • Lobo JM
      • Gunnell B
      • Enfield K
      • Carpenter R
      • Barnes L
      • et al.
      Characterizations of handwashing sink activities in a single hospital medical intensive care unit.
      up to 32 different types of events happen at a hand hygiene sinks. To avoid counting these other events as handwash events, we only included handwash events taking more than 5 seconds long. In total, 3,606 handwash events were audited for quality. In the surgical unit, the number of patients and staffing varied daily with a peak on Tuesdays and a minimum over the weekend. To address this issue, the data were normalized using the number of hand hygiene events per patient per day, that is the total number of hand hygiene events per day divided by the number of patients that day.
      We measured quality using the percentage of hand hygiene events that met the full WHO protocol, that is the total number of hand hygiene events that turned all the “traffic lights” green, divided by the total number of hand hygiene events. The automatic analysis of the images was validated using 2 human observers. The inter-rater reliability between the observers and the computer was 0.85. As expected, there was some variance between the 6 different human reviewers in the study, but in each pairing the computer agreed with 1 human reviewer to a greater degree than the human reviewers agreed with each other, demonstrating that computer analysis produced classifications as consistent as a human reviewer.
      The data on hand hygiene quantity and quality were tested for normality and for the presence of daily or weekly patterns using an autoregressive integrated moving average model. The data were also tested to verify that there was no correlation between the number of staff or patients on the ward. To verify the impact of the intervention, a homogeneity test was applied to each of the phase transitions. The baseline values for quality and quantity was calculated in phases 1 and 4. The study effect size was large (Hedges g = 1.54), resulting in a power of 0.8 after just 16 hand hygiene events. The study had ample power as baseline values were calculated using 240 and 160 hand hygiene events, respectively. Each phase was modeled using a segmented linear regression model and significance was measured using the Mann–Whitney–Wilcoxon 2-tailed test. Statistical calculations were performed using XLstat (Addinsoft, Paris, France).

      Results

      Analysis of both hand hygiene quality and quantity showed that the addition of a weekly dashboard report in phase 3 had no impact, and therefore data from phases 2 and 3 have been combined for the purposes of further analysis. The difference between the baseline measured at phase 4 and in phase 1 were not statistically significant.
      After the introduction of feedback in phase 2, the average number of handwash events per patient day increased from the baseline of 0.91-2.25, and the average percentage of handwash events meeting the quality standard increased from a baseline of 15.7%-46%. Both of these increases were highly statistically significant (P < .0001). Table 1 shows the data for the study along with the 95% confidence intervals. The segmented linear regression in Figure 2 shows that this positive impact on pass rates was sustained for the intervention.
      Table 1Study data showing the duration of each phase and the high-level data produced by the system
      Handwash events per patient dayPercentage of handwash events that comply with the WHO protocol
      DaysNumber of handwash eventsAverage95% CIAverage95% CI
      Phase 1102400.910.75-1.0715.7%10.6%-20.9%
      Phases 2 and 3643,2062.252.10-2.3946.0%43.8%-48.1%
      Phase 461601.150.73-1.5813.7%5.9%-21.5%
      CI, confidence interval, WHO, World Health Organization.
      Fig 2
      Fig 2A graph showing the segmented linear regression of the study data over the entire study.
      After the provision of feedback at the beginning of phase 2, the average quantity handwash events increased by 147% (2.25/0.91), and the average quality of the handwashes improved by 193% (46/15.7). The provision of real-time feedback did change the behavior of a significant cohort of the HCWs, but behavior returned to baseline when feedback was removed.

      Discussion

      The limitations of the study are that it was not possible to track ABHR usage for 14 hours per day during the 88 days of the study, nor was it possible to count all of the hand hygiene opportunities during this period. Because the study was anonymous, it was not possible to associate improved performance to a specific subset of the staff, or to a more general change in behavior across all the staff.
      The main purpose of the study was to assess the impact of using AVA with feedback on the quality and quantity of handwash events. The study did show a very significant increase in both the quality and quantity of hand hygiene over baseline. The baseline quality of 15% is very similar to that reported in the studies by Tschudin-Sutter et al
      • Tschudin-Sutter S
      • Sepulcri D
      • Dangel M
      • Schuhmacher H
      • Widmer AF
      Compliance with the World Health Organization hand hygiene technique: a prospective observational study.
      and Widmer.
      • Widmer AF
      Infection Control Team
      Andreas F Widmer for the Basel infection control team.
      Because it was not possible to measure the overall rate of hand hygiene, any displacement in the use of ABHR by extra handwashing could not be determined. The combination of AVA and electronic measurement of ABHR usage would deliver a study that can simultaneously measure the quantity and quality of hand hygiene.
      The study design compared hand hygiene events on a binary pass–fail basis, but work by Reilly et al
      • Reilly JS
      • Price L
      • Lang S
      • Robertson C
      • Cheater F
      • Skinner K
      • et al.
      A pragmatic randomized controlled trial of 6-step vs 3-step hand hygiene technique in acute hospital care in the United Kingdom.
      showed that even partial completion of the WHO protocol delivers a significant reduction in the microbial load. Our future evaluations will also consider gradual improvements in technique.
      There were no performance incentives used in the study, nor were there any punishments if HCWs achieved a low score. Peer-to-peer learning was reported as HCWs challenged each other to achieve a perfect score. Interaction with the AVA system promoted personal reflection on hand hygiene, as evidenced by comments such as “it made us think about what we are doing.” However, some HCWs simply ignored the system. This anecdotal evidence is in line with studies that report the level of compliance is strongly influenced by cultural factors
      • Kwok YL
      • Harris P
      • McLaws ML
      Social cohesion: the missing factor required for a successful hand hygiene program.
      and incentives.
      • Lacey G
      • Corr M
      • Morrow H
      • McQueen A
      • Cameron F
      • Connolly C
      The impact of structured incentives on the adoption of a serious game for hand hygiene training in a hospital setting.
      After the removal of feedback, handwashing performance returned to baseline. This result is similar to studies by Kwok et al
      • Kwok YLA
      • Juergens CP
      • McLaws ML
      Automated hand hygiene auditing with and without an intervention.
      and Staats et al.
      • Staats BR
      • Dai H
      • Hofmann D
      • Milkman KL
      Motivating process compliance through individual electronic monitoring: an empirical examination of hand hygiene in healthcare.
      In the article by Staats et al,
      • Staats BR
      • Dai H
      • Hofmann D
      • Milkman KL
      Motivating process compliance through individual electronic monitoring: an empirical examination of hand hygiene in healthcare.
      the authors suggest a cultural explanation, that the removal of technology is interpreted by HCWs as a signal from management that hand hygiene is no longer a priority. Another explanation, based on cognitive science, is that the constant availability of feedback acts as transitive memory,
      • Sparrow B
      • Liu J
      • Wegner DM
      Google effects on memory: cognitive consequences of having information at our fingertips.
      and that HCWs cognitively offload
      • Risko EF
      • Gilbert SJ
      Cognitive offloading.
      the task of memory onto the technology. We see this effect in smartphone use, in which we no longer need to remember things because we have easy access to the information. Whatever the underlying mechanism, this effect should be considered when interpreting the results of technology studies that use an interrupted time series design. Our future research will examine ways of varying the feedback so as to minimize cognitive offloading and encourage the building of muscle memory.
      Although the difference between phases 2 and 3 of the study are not statistically significant, there was a small downward trend. Staats et al
      • Staats BR
      • Dai H
      • Hofmann D
      • Milkman KL
      Motivating process compliance through individual electronic monitoring: an empirical examination of hand hygiene in healthcare.
      points out that engagement with all initiatives reduces over time, and this points to a need to vary the feedback to maintain engagement. In learning technology, adaptive gamification
      • Lavoue E
      • Monterrat B
      • Desmarais M
      • George S
      Adaptive gamification for learning environments.
      has been shown to improve learning by tailoring the reward to individual learners’ preferences. Our future research will focus on personalizing the feedback based on individual performance so as to maintain engagement.

      Conclusions

      The AVA with real-time feedback supports the WHO multimodal strategy for hand hygiene by providing “frequent easy access to training, evaluation, and feedback,” a “reminder in workplace,” and provides a visible “demonstration of the hospital's safety climate.” In line with other studies,
      • Armellino D
      • Hussain E
      • Schilling ME
      • Senicola W
      • Eichorn A
      • Dlugacz Y
      • et al.
      Using high-technology to enforce low-technology safety measures: the use of third-party remote video auditing and real-time feedback in healthcare.
      • Kwok YLA
      • Juergens CP
      • McLaws ML
      Automated hand hygiene auditing with and without an intervention.
      • Staats BR
      • Dai H
      • Hofmann D
      • Milkman KL
      Motivating process compliance through individual electronic monitoring: an empirical examination of hand hygiene in healthcare.
      feedback on handwash performance delivered a substantial improvement in the average quantity of handwashes by 147% (2.25/0.91), and the average quality of the handwashes by 193% (46/15.7). The provision of a weekly performance report on hand hygiene at staff meetings had no measurable impact on hand hygiene practice.
      Future studies that combine AVA systems with electronic hand hygiene monitoring technology will allow, for the first time, the assessment of both hand hygiene quality and quantity. When the feedback was removed, performance returned to baseline, this was also seen in other studies
      • Kwok YLA
      • Juergens CP
      • McLaws ML
      Automated hand hygiene auditing with and without an intervention.
      • Staats BR
      • Dai H
      • Hofmann D
      • Milkman KL
      Motivating process compliance through individual electronic monitoring: an empirical examination of hand hygiene in healthcare.
      and suggests that care should be taken interpreting the results of interrupted times series designs using technology interventions. Further research is required into the impact of cognitive offloading when using technology with real-time feedback to support hand hygiene.

      Acknowledgements

      The authors would like to thank the patients and staff of Mid Essex Hospital Services NHS Trust for their support during this study.

      References

        • Allegranzi B
        • Pittet D.
        Role of hand hygiene in healthcare-associated infection prevention.
        J Hosp Infect. 2009; 73: 305-315
        • World Health Organization
        WHO guidelines on hand hygiene in healthcare: a summary.
        World Health Organization, Geneva, Switzerland2009
        • Salmon S
        • Pittet D
        • Sax H
        • McLaws ML
        The 'My Five Moments for Hand Hygiene' concept for the overcrowded setting in resource-limited healthcare systems.
        J Hosp Infect. 2015; 91: 95-99
        • Reilly JS
        • Price L
        • Lang S
        • Robertson C
        • Cheater F
        • Skinner K
        • et al.
        A pragmatic randomized controlled trial of 6-step vs 3-step hand hygiene technique in acute hospital care in the United Kingdom.
        Infect Control Hosp Epidemiol. 2016; 37: 661-666
        • Pittet D
        • Hugonnet S
        • Harbarth S
        • Mourouga P
        • Sauvan V
        • Touveneau S
        • et al.
        Effectiveness of a hospital-wide programme to improve compliance with hand hygiene. Infection Control Programme.
        Lancet. 2000; 356: 1307-1312
        • Lam BC
        • Lee J
        • Lau YL
        Hand hygiene practices in a neonatal intensive care unit: a multimodal intervention and impact on nosocomial infection.
        Pediatrics. 2004; 114: e565-e571
        • Ling ML
        • How KB.
        Impact of a hospital-wide hand hygiene promotion strategy on healthcare-associated infections.
        Antimicrob Resist Infect Control. 2012; 1: 13
        • Boyce JM
        Measuring healthcare worker hand hygiene activity: current practices and emerging technologies.
        Infect Control Hosp Epidemiol. 2011; 32: 1016-1028
        • Boyce JM.
        Current issues in hand hygiene.
        Am J Infect Control. 2019; 47S: A46-A52
        • Gould DJ
        • Moralejo D
        • Drey N
        • Chudleigh JH
        • Taljaard M
        Interventions to improve hand hygiene compliance in patient care.
        Cochrane Database Syst Rev. 2017; 9CD005186
        • Tschudin-Sutter S
        • Sepulcri D
        • Dangel M
        • Schuhmacher H
        • Widmer AF
        Compliance with the World Health Organization hand hygiene technique: a prospective observational study.
        Infect Control Hosp Epidemiol. 2015; 36: 482-483
        • Widmer AF
        • Conzelmann M
        • Tomic M
        • Frei R
        • Stranden AM
        Introducing alcohol-based hand rub for hand hygiene: the critical need for training.
        Infect Control Hosp Epidemiol. 2007; 28: 50-54
        • Arias AV
        • Garcell HG
        • Ochoa YR
        • Arias KF
        • Miranda FR
        Assessment of hand hygiene techniques using the World Health Organization's six steps.
        J Infect Public Health. 2016; 9: 366-369
        • Conrad A
        • Kaier K
        • Frank U
        • Dettenkofer M
        Are short training sessions on hand hygiene effective in preventing hospital-acquired MRSA? A time-series analysis.
        Am J Infect Control. 2010; 38: 559-561
        • Lobo RD
        • Levin AS
        • Oliveira MS
        • Gomes LM
        • Gobara S
        • Park M
        • et al.
        Evaluation of interventions to reduce catheter-associated bloodstream infection: continuous tailored education versus one basic lecture.
        Am J Infect Control. 2010; 38: 440-448
        • Grabowski M
        • Lobo JM
        • Gunnell B
        • Enfield K
        • Carpenter R
        • Barnes L
        • et al.
        Characterizations of handwashing sink activities in a single hospital medical intensive care unit.
        J Hosp Infect. 2018; 100: e115-e122
        • Gautschi N
        • Marschall J
        • Candinas D
        • Banz VM
        Effect of music on surgical hand disinfection: a video-based intervention study.
        J Hosp Infect. 2017; 95: 352-354
        • Brouqui P
        • Boudjema S
        • Reynier P
        • Dufour JC
        • Florea O
        • et al.
        Hand hygiene analyzed by video recording.
        J Nurs Care. 2016; 5: 338.
        • Armellino D
        • Hussain E
        • Schilling ME
        • Senicola W
        • Eichorn A
        • Dlugacz Y
        • et al.
        Using high-technology to enforce low-technology safety measures: the use of third-party remote video auditing and real-time feedback in healthcare.
        Clin Infect Dis. 2012; 54: 1-7
        • Ghosh A
        • Ameling S
        • Zhou J
        • Lacey G
        • Creamer E
        • Dolan A
        • et al.
        Pilot evaluation of a ward-based automated hand hygiene training system.
        Am J Infect Control. 2013; 41: 368-370
        • Stewardson AJ
        • Iten A
        • Camus V
        • Gayet-Ageron A
        • Caulfield D
        • Lacey G
        • et al.
        Efficacy of a new educational tool to improve handrubbing technique amongst healthcare workers: a controlled, before-after study.
        PLoS One. 2014; 9e105866
        • Lacey G
        • Corr M
        • Morrow H
        • McQueen A
        • Cameron F
        • Connolly C
        The impact of structured incentives on the adoption of a serious game for hand hygiene training in a hospital setting.
        Stud Health Technol Inform. 2016; 220: 179-184
        • Widmer AF
        • Infection Control Team
        Andreas F Widmer for the Basel infection control team.
        BMC Proceedings. 2011; 5: P123
        • Kwok YL
        • Harris P
        • McLaws ML
        Social cohesion: the missing factor required for a successful hand hygiene program.
        Am J Infect Control. 2017; 45: 222-227
        • Kwok YLA
        • Juergens CP
        • McLaws ML
        Automated hand hygiene auditing with and without an intervention.
        Am J Infect Control. 2016; 44: 1475-1480
        • Staats BR
        • Dai H
        • Hofmann D
        • Milkman KL
        Motivating process compliance through individual electronic monitoring: an empirical examination of hand hygiene in healthcare.
        Manag Sci. 2017; 63: 1563-1585
        • Sparrow B
        • Liu J
        • Wegner DM
        Google effects on memory: cognitive consequences of having information at our fingertips.
        Science. 2011; 333: 776-778
        • Risko EF
        • Gilbert SJ
        Cognitive offloading.
        Trends Cogn Sci. 2016; 20: 676-688
        • Lavoue E
        • Monterrat B
        • Desmarais M
        • George S
        Adaptive gamification for learning environments.
        IEEE Transactions on Learning Technologies. 2019; 12: 16-28