Advertisement

Using item analysis to evaluate hand hygiene self-assessments at Alberta health services

  • Vinita Kumar
    Correspondence
    Address correspondence to Vinita Kumar, Department of Infection Prevention and Control, Alberta Health Services, 1403 29 Street NW, Calgary, Alberta, T2N 2T9, Canada.
    Affiliations
    Department of Infection Prevention and Control, Alberta Health Services, Calgary, Alberta, Canada

    Faculty of Applied Health Sciences, Department of Health Sciences, Brock University, St. Catharines, Ontario, Canada
    Search for articles by this author
Published:September 27, 2022DOI:https://doi.org/10.1016/j.ajic.2022.08.030

      Highlights

      • Lack of research on assessing hand hygiene assessments for reviewer training.
      • Item analysis highlighted need to improve hand hygiene assessments.
      • This framework for evaluating validity and reliability is more robust.
      • This framework can be adapted for assessing other educational assessments.

      Abstract

      Background

      It is important to evaluate the quality of assessments used in education as the evaluation can identify poorly worded questions, or questions which are not contributing to the validity or reliability of the test. The objective of this study was to perform an item analysis on the hand hygiene self-assessments used at Alberta Health Services to determine the assessments’ validity, reliability, and to highlight questions that required rewording or removal.

      Methods

      The sample included all first completed attempts between April 2018 and December 2020. Components of the analysis included item difficulty, item discrimination, B-index, point biserial correlation, overall discrimination score, distractor efficiency, and Cronbach's alpha.

      Results

      A total of 650 health care workers completed the assessment. The average item difficulty was 0.76 (0.32-0.99), average discrimination index was 0.23 (0.02-0.65), average B-index was 0.15 (0.02-0.45), and average point biserial correlation was 0.26 (0.07-0.54). Seventeen of 45 items had an overall discrimination score greater than 7 of 9 and 20 of 21 (95%) multiple choice questions had at least 1 non-functioning distractor. The self-assessment had an overall Cronbach's alpha value of 0.68.

      Conclusion

      The item analysis provided robust evidence for improving the validity and reliability of the self-assessment. The steps performed in this item analysis can be useful for educators interested in improving assessments that include different question types.

      Key Words

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to American Journal of Infection Control
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Garcell HG
        • Arias AV
        • Garcés IB
        • et al.
        Assessment of trained hand hygiene observers using parallel observations.
        Oman Med J. 2018; 33: 76-79https://doi.org/10.5001/omj.2018.15
        • De Champlain AF.
        A primer on classical test theory and item response theory for assessments in medical education.
        Med Educ. 2010; 44: 109-117https://doi.org/10.1111/j.1365-2923.2009.03425.x
        • Sullivan GM.
        A primer on the validity of assessment instruments.
        J Grad Med Educ. 2011; 3: 119-120https://doi.org/10.4300/JGME-D-11-00075.1
        • Matlock-Hetzel S.
        Basic concepts in item and test analysis.
        in: Paper presented at: Annual Meeting of the Southwest Educational Research Association. ERIC, January 23-25, 1997 (Accessed October 26, 2022)
      1. Alberta Health Services. Hand Hygiene Reviewer Training Modules. Alberta Health Services. Accessed October 26, 2022. https://www.albertahealthservices.ca/ipc/Page15620.aspx.

        • Namdeo SK
        • Sahoo B.
        Item analysis of multiple choice questions from an assessment of medical students in Bhubaneswar, India.
        Int J Res Med Sci. 2016; 4: 1716-1719https://doi.org/10.18203/2320-6012.ijrms20161256
        • McGahee TW
        • Ball J.
        How to read and really use an item analysis.
        Nurse Educ. 2009; 34: 166-171https://doi.org/10.1097/NNE.0b013e3181aaba94
        • Tarrant M
        • Ware J
        • Mohammed AM.
        An assessment of functioning and non-functioning distractors in multiple-choice questions: a descriptive analysis.
        BMC Med Educ. 2009; 9https://doi.org/10.1186/1472-6920-9-40
        • Kaur M
        • Singla S
        • Mahajan R.
        Item analysis of in use multiple choice questions in pharmacology.
        Int J Appl Basic Med Res. 2016; 6: 170-173https://doi.org/10.4103/2229-516X.186965
        • Streiner DL.
        Starting at the beginning: an introduction to coefficient alpha and internal consistency.
        J Pers Assess. 2003; 80: 93-103https://doi.org/10.1207/S15327752JPA8001_18
        • Kumar D
        • Jaipurkar R
        • Shekhar A
        • Sikri G
        • Srinivas V.
        Item analysis of multiple choice questions: a quality assurance test for an assessment tool.
        Med J Armed Forces India. 2021; 77: S85-S89
        • Mehta G
        • Mokhasi V.
        Item analysis of multiple choice questions – An assessment of the assessment tool.
        Int J Health Sci Res. 2014; 4: 197-202
        • Rao C
        • Prasad K
        • Sajitha K
        • Permi H
        • Shetty J.
        Item analysis of multiple choice questions: assessing an assessment tool in medical students.
        Int J Edu Psych Res. 2016; 2: 201-204https://doi.org/10.4103/2395-2296.189670
        • Gould D
        • Lindström H
        • Purssell E
        • Wigglesworth N.
        Electronic hand hygiene monitoring: accuracy, impact on the Hawthorne effect and efficiency.
        J Infect Prev. 2020; 21: 136-143https://doi.org/10.1177/1757177420907999
        • Iversen AM
        • Hansen MB
        • Kristensen B
        • Eriksen SE.
        Clinical evaluation of an electronic hand hygiene monitoring system.
        Am J Infect Control. 2022; 00: 1-4https://doi.org/10.1016/j.ajic.2022.06.017
        • Pineles LL
        • Morgan DJ
        • Limper HM
        • et al.
        Accuracy of a radiofrequency identification (RFID) badge system to monitor hand hygiene behavior during routine clinical activities.
        AM J Infect Control. 2014; 42: 144-147https://doi.org/10.1016/j.ajic.2013.07.014
      2. Brown JD. Criterion-Referenced Item Analysis. (The difference index and B index). SHIKEN: JALT Testing & Evaluation SIG Newsletter. Accessed October 26, 2022. https://hosted.jalt.org/test/PDF/Brown18.pdf.

      3. Varma S. Preliminary Item Statistics Using Point-Biserial Correlation and p-values. Educational Data Systems. Accessed October 26, 2022. https://jcesom.marshall.edu/media/24104/Item-Stats-Point-Biserial.pdf.

        • Tavakol M
        • Dennick R.
        Making sense of Cronbach's alpha.
        Int J Med Educ. 2011; 2: 53-55https://doi.org/10.5116/ijme.4dfb.8dfd