Tools For Direct Observation And Assessment Of Psychomotor

Published Date: 02 Nov 2017

Funding: None

Conflicts of Interest: None

Abstract

Context: The Accreditation Council for Graduate Medical Education (ACGME) Milestone Project mandates programs assess attainment of training outcomes including psychomotor (surgical or procedural) skills of medical trainees. The objectives were to determine what tools exist to directly assess psychomotor skills of medical trainees on patients and what are the data in support of their psychometric and edumetric properties.

Methods: Electronic search was conducted from January 1948 to May 2011 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL), Web of Science electronic databases and review of references from article bibliographies. Studies were included that described a tool or instrument designed for direct observation of psychomotor skills in patient care settings by supervisors. Tools were excluded if they assessed only clinical or non-technical skills, involved non-medical health professions, or only when skills were performed on a simulator. Overall, 4,114 citations were screened, 168 (4.1%) articles were reviewed for eligibility and 51 (1.2%) manuscripts met inclusion criteria. Three authors abstracted and reviewed studies using a standardized form for the presence of key psychometric and edumetric elements per ACGME and American Psychological Association recommendations as well as assigned an overall grade based on the ACGME Committee on Educational Outcome Assessment.

Results: A total of 30 tools were identified. Construct validity based on associations between scores and training level was identified in 24 tools, internal consistency in 14, test-retest reliability in 5 and inter-rater reliability in 20. A modification of attitudes, knowledge or skills was reported using 5 tools. The 7-item Global Rating Scale and Procedure-Based Assessment received an overall Class 1 ACGME grade and are recommended based on Level A ACGME evidence.

Conclusions: Numerous tools are available for the assessment of psychomotor skills of medical trainees but evidence supporting psychometric and edumetric properties are limited.

Introduction

Surgical or procedural (psychomotor) skills are learned skills. Dexterity (motor) alone does not equate with skill; skill is intelligently (psycho) applied manipulation.(1) Assessment of these skills involves measuring two tangible things: the information that the trainee is using to learn and, the most common method in the literature, performance through observation of the trainee. If a trainee successfully ‘shows how’ or ‘does’ an operation or procedure (2), this at least proves that the knowledge to perform the procedure was in store. There is no need to separately test for knowledge.(3) There is only the need to decide whether the score of the items within the tool provide an accurate and precise estimate of performance and whether it does or does not meet predetermined criteria of that performance. Program directors can use scores from assessment tools to monitor trainee and program performance; and accrediting bodies can use them to better gauge program performance across institutions. Thus, tool reliability and validity are of utmost importance, especially when used in ‘high-stakes’ decision-making settings.

Assessment tools also have a role in being a facilitator of learning; particularly, when they are judiciously integrated into the educational culture of a program and patient care. This characteristic has been termed an edumetric property or "usefulness" of an instrument.(4-6) Additionally, implementation of assessment itself may drive learning behavior; thus, becoming part of an instructional design process. Therefore, it becomes important to be aware and if possible, measure these edumetric properties when evaluating assessment instruments.

These important aspects of assessment tools are summarized by the contemporary framework proposed by van der Vleuten and Shuwirth’s utility equation (utility of an assessment tool = validity x reliability x acceptability x educational impact x cost effectiveness). This formula was recently incorporated into a set of guidelines put forth by the Accreditation Council for Graduate Medical Education (ACGME) to grade assessment tools that includes a set of standards, a methodology for applying the standards, and grading rules for their review of assessment method quality.(4-6) The objectives of this review were to determine to what degree existing assessment tools of psychomotor skills and procedures incorporate this conceptual framework and to develop a simple report card for displaying grades on each standard as well as an overall grade for each tool reviewed. By summarizing and reporting the presence or absence of these important properties, we hope developers will be able to incorporate the important properties into future tool design and users might be able to take advantage of the edumetric properties or psychometric properties depending on the specific educational need.

Methods:

A systematic review of the literature was performed to determine what tools exist to directly assess psychomotor (surgical or procedural) skills of medical trainees on live patients and to determine the data in support of their psychometric and edumetric properties. We attempted to capture all tools available rather than focus on a restricted group of specialties. Specifically, the following objectives were identified for the conduct of the literature review:

To identify observation tools used to assess medical trainees’ psychomotor skills with actual patients;

To summarize the evidence of their psychometric and edumetric properties;

Make recommendations for further research and practice.

A systematic electronic and hand literature search was conducted using specific eligibility criteria to minimize bias.(7) The review was conducted using methods set forth from Best Evidence in Medical Education Collaborative (8) and reporting are in accordance with guidelines for reporting systematic reviews and meta-analyses using the PRISMA statement.(9)

Definitions

For purposes of this review, the following definitions were determined:

Psychomotor skills: Psychomotor and technical skills are rarely defined in medical literature but most likely refer to cognitive, motor, and dexterity skills associated with physical examinations, clinical procedures, surgery, specialized medical equipment, and medicines.(10) The definition of psychomotor skills for the purposes of this review includes cognitive, psychomotor, and dexterity skills associated with performing clinical procedures and surgery. It excludes skills associated with physical examinations, specialized non-surgical equipment and medicines.

Psychometric properties: are summarized in terms of reliability and validity and are defined according to definitions laid out in Overview of Standards for Evaluating the Quality of Assessment Methods.(6) Since these definitions, many of which incorporate recommendations made by the American Psychological Association (APA), attempt to capture only the most important properties but are not comprehensive in nature, additional key elements of validity and reliability were identified using the attributes and criteria for reviewing patient-centered quality of life instruments published by the Scientific Advisory Committee of the Medical Outcomes Trust.(11)

Edumetric properties or Usefulness: are summarized in terms of ease of use, resources required, ease of interpretation, and educational impact and are also defined according to definitions laid out in Overview of Standards for Evaluating the Quality of Assessment Methods.(6)

Outcomes

Outcomes measured included:

Available observation tools used to assess medical trainees’ psychomotor skills with actual patients.

Psychometric and edumetric properties of each tool.

Grades on each property and an overall summary grade for each tool reviewed.(6)

Identification of areas which need further research.

Search Strategy

Three authors independently reviewed citation titles and abstracts to assess eligibility for review with each title/abstract reviewed by at least two authors. Review articles, letters, and case reports were excluded. When reviewers disagreed or when an abstract was insufficient to determine study eligibility, the full article was retrieved. Eligible English-language studies were selected through an electronic literature search from January 1948 to May 2011 using the PubMed, Education Resource Information Center (ERIC), Cumulative Index to Nursing and Allied Health Literature (CINAHL) and Web of Knowledge electronic databases. The search strategy was formulated and conducted with the assistance of a professional medical research librarian. A combination of search terms were used related to: motor skills, assessment tools, medical trainee, and procedures. A detailed search strategy is described in the Appendix 1. Reference lists of all included articles were also examined when necessary. This resulted in a list of assessment tools.

Studies were included if they: 1) at least described a tool or instrument designed for direct observation of technical task or procedural skill in patient care settings with actual patients (observer in the setting or by camera) and 2) for use by educational supervisors (interns, residents, fellows, faculty, nurses, nurse practitioners, other trained observers) with any medical trainee defined as medical students, interns, residents, or fellows. Studies were excluded that 1) describe tools intended to assess clinical or non-technical skills (physical examination, communication, etc.); 2) those without a full article available for review; 3) non-English language publications; 4) studies involving other non-medical health professions such as dental or veterinarian practitioners; 5) articles describing the construction of a tool, but not testing and validation in an educational context; and 6) articles describing tools only studied using a model or simulator rather than in live patient care. A standard form was created and a pilot process whereby a sample of representative articles that passed the screening were used to pre-test the coding form and instructions to reviewers. If changes to the overall structure of the form were needed, wording of the items were edited for clarification, and new items added or deleted. Three authors independently abstracted and reviewed studies using a standardized coding form informing judgment regarding the presence of key elements of reliability, validity, ease of use, resources required, ease of interpretation, and educational impact per Accreditation Council for Graduate Medical Education (ACGME) recommendations.

All three reviewers were clinically active, fellowship-trained surgeons in an academic setting, experienced at performing systematic review methodology and reviewing or publishing articles measuring psychometric and edumetric properties. At least two authors reviewed each abstract and manuscript. Several of the grading rules within the ACGME evidence-based guidelines are derived from important properties of validity and reliability in the American Psychological Association guidelines.(6) We also identified the presence of additional important elements of validity and reliability such as criterion validity and internal consistency using the attributes and criteria for reviewing quality of life instruments published by the Scientific Advisory Committee of the Medical Outcomes Trust.(11) Together, the evidence from each framework was summarized and each reviewer assigned an overall evidence grade of each tool based on standardized grading rules proposed by the ACGME Committee on Educational Outcome Assessment with differences in grading reconciled by consensus.(6) The ACGME grades determined using Grading Standards and Grading Rules are outlined by Swing et al. and presented in Appendix 2.(6)

Results of Literature Review

Study Characteristics:

The PRISMA diagram (Figure 1) summarizes the overall review process. Overall, 4,114 citations were screened and 168 (4.1%) articles met criteria after title and abstract review. Fifty-one (1.2%) manuscripts met inclusion criteria and the full manuscripts were reviewed. The characteristics of these manuscripts are presented in Table 1. Since 1990, there has been a steep increase in the number of manuscripts studying various tools for assessing psychomotor performance in the live setting. The majority of manuscripts studied postgraduate (resident) trainees in the United States or Europe who were enrolled in surgical or obstetrics and gynecology training programs. One third of studies were performed in a multi-center setting and study designs were predominately prospective cohort studies without a control group. Randomized trials were performed in only four studies including two in obstetrics and gynecology (12, 13), one in general surgery (14), and one in anesthesia (15). Thirty-two studies reported institutional review board approval and no studies assessed cost of implementing the assessment tool.

Description of Tools

Details are summarized about each manuscript in Table 2 and actual evidence are provided in Appendix 3. A total of 30 tools were identified out of 51 manuscripts with some tools used in more than one specialty including: 25 tools in surgical specialties: 4 tools in obstetrics and gynecology and 2 tools in anesthesia. The global rating scale, originally developed for assessing basic general surgical procedures in a simulated setting, (16) was the most commonly studied (12 studies) tool in different settings by different specialties using live performance.(12-15, 17-24) Many of the remaining instruments used some modification of this tool in respective specialties. Most tools used either a Likert-type response consisting of a behavioral anchor or a checklist format (usually the presence or absence of a particular task) with the assessor marking whether they did or did not observe the trainee performing a particular psychomotor skill (Table 2). Several tools including the Procedure-Based Assessment (PBA) tool, introduced by the Orthopedic Competence Assessment Project, (25) combined Likert-type responses of a global rating scale with checklist type responses. This combination of response types made it difficult to draw summative conclusions about broad categories of either global rating or checklist methods.(26-29) Procedure-Based Assessment tools can be accessed at https://www.iscp.ac.uk/surgical/assessment_pba.aspx. Table 2 summarizes tools that used a global rating-type method in their approach. These approaches often applied the method to rate broad categories of surgical technique (i.e. flow of the operation and suturing) rather than an overall global rating of performance of the procedure as a whole. Overall, studies were heterogeneous in the number of subjects studied with ranges of 3 to 81 subjects and in the number of assessments performed ranging from 0 to 3,849 assessments with multiple assessments being performed on the same trainees. A summary of the psychometric and edumetric properties of each tool are presented in Table 3 and actual evidence from each manuscript in presented in Appendix 3.

Validity evidence:

Construct-related validity, defined as evidence that supports a proposed interpretation of scores based on theoretical implications associated with the constructs being measured,(11) was identified in 38 manuscripts including 24 tools (12-14, 17-19, 21, 23, 24, 27, 29-56) and was largely based on associations between scores on the assessment tools and training level. Content-related validity, defined as evidence that the domain of an instrument is appropriate relative to its intended use, was demonstrated in 29 articles including 22 tools (15, 18, 19, 21, 22, 24, 27, 29, 30, 34, 36, 37, 39, 41-46, 49, 51, 54-61) and criterion-related validity, defined as evidence that shows the extent to which scores of the instrument are related to a criterion measure,(11) was demonstrated in four manuscripts and four tools (17, 24, 34, 48). Specifically, a rationale for each interpretation and use of evaluation results along with evidence and theory (APA Standard 1.1) was presented in five studies and five tools.(26, 30, 43, 57, 60) Processes and procedures used for selection of the content of assessment and for any criteria used to sample content was described and justified (APA Standard 1.6) in 21 studies and 17 tools.(19, 21, 26, 27, 30, 37, 41-46, 49, 51, 54-60) Theoretical or empirical evidence supporting the interpretation of the tool was provided when the rationale for the use and interpretation of an assessment depends on the psychological processes or cognitive operations of the learner or the processes of the evaluator (APA Standard 1.8) in 16 studies and 12 tools.(13, 19, 20, 24, 27, 30, 31, 33-35, 37, 38, 41, 57, 60, 62) No studies attempted to measure unintended consequences resulting from use of a specific assessment tool (APA Standard 1.24). Only four manuscripts and tools reported the degree of agreement between a single expert rater and ''gold standard'' or consensus ratings for the same performance when a single rater using subjective judgments was the basis of the assessment (ACGME Standard 1).(17, 20, 43, 48) When a single rater using subjective judgment was the basis of the assessment, the degree to which known strengths and weaknesses of the learner are detected (ACGME Standard 2) was met in 21 studies and 14 tools.(12-14, 17-20, 24, 27, 30, 31, 33-35, 37, 38, 41, 59, 60, 63, 64)

Reliability evidence:

Internal consistency (11) was reported in 16 manuscripts and 14 tools (14, 19, 33, 34, 37-39, 41, 42, 48, 49, 51, 53, 59, 60, 65), test-retest reliability (11) in five manuscripts and tools,(17, 19, 27, 42, 43) and inter-rater reliability (11) in 33 manuscripts including 20 tools.(12, 13, 15, 17, 19-21, 24, 27, 29-31, 37, 38, 41-47, 49-56, 60-62, 64) Reliability indicators were available for any total score or subscore (APA Standard 2.1) in 19 manuscripts and 14 tools.(13, 14, 19, 20, 24, 27, 29, 31, 33, 37, 38, 41-43, 46, 48, 59, 60, 62) Interrater and intrarater reliability for multiple ratings of the same learner were provided when scoring or rating entailed subjective judgment (APA Information Standard 2.10) in 9 manuscripts and 6 tools.(13, 14, 17, 19, 24, 27, 42, 43, 62) Interrater reliability testing was blinded in 21 manuscripts and 13 tools (14, 15, 17, 19, 21, 24, 27, 30, 31, 41, 44, 45, 47, 50, 52-56, 62, 64) and intrarater reliability testing was blinded in four manuscripts and tools.(19, 27, 52, 53) Generalizability statistics were not studied for any of the tools reviewed. For high-stakes decisions, an estimate was provided of the percentage of learners who would be classified the same on two applications of the same method or rating process (APA Standard 2.15) in only 2 manuscripts and tools.(20, 42)

Usefulness:

Ease of use

The assessment tool was easily carried out or accessed in the course of daily clinical or teaching activity in 37 manuscripts including 24 tools.(12-15, 18, 19, 21, 22, 24, 28, 29, 33-35, 37, 39, 41-51, 53-56, 58, 59, 62, 63, 65, 66) The tool appeared to require little special setup in 42 manuscripts and 28 tools. (12-15, 18, 19, 21, 22, 24, 28-31, 33-37, 39, 41-51, 53-60, 62, 63, 65, 66) and required less than 20 minutes for the assessor to complete in 11 manuscripts and 7 tools.(13, 15, 18, 19, 21, 22, 29, 33, 37, 43, 65)

Resources required

Faculty training was reported in 12 manuscripts and 9 tools (12, 21, 22, 29, 37, 46, 49-52, 63, 65) and no additional resources were required beyond the documentation tools for 41 manuscripts and 27 tools.(12-15, 18, 19, 22, 24, 27-31, 33-37, 39, 41-51, 53-59, 62, 63, 65, 66) Training requirements for assessors were reported to not exceed an hour for two manuscripts and tools (15, 34) and no additional persons other than an individual assessor were required to complete the assessment in 45 manuscripts and 28 tools.(12-15, 17-19, 21, 22, 24, 27-31, 33-39, 41-51, 53-60, 62, 63, 65, 66)

Ease of interpretation

Individual scores were interpretable on an easily understood scale, such as percent correct or against behavioral or other descriptive criteria, and were accompanied by interpretation guidelines for 32 studies and 22 tools.(12-15, 17-21, 24, 26, 27, 29, 30, 34-37, 39, 41-43, 45, 57-61, 63-65) Normative data were available consisting of: (1) a standard of care; (2) performance of other residents at the same level of training and/or experience; (3) performance of other residents with more or less experience; and (4) the resident's performance level at an earlier stage of education and experience for 7 studies and 6 tools.(14, 20, 26, 38, 41, 59, 60) Preprogrammed, easy-to-read reports and graphs made it simple to compare individual to group performance in 7 studies and four tools.(12, 13, 17, 18, 41, 59, 65)

Educational impact

Educational impact was assessed using the learners' or observers' views on the tool or its implementation in one study and tool (19), the learner or observer self-assessed modification of attitudes, knowledge or skills in 6 studies and five tools (19, 22, 28, 29, 33, 37), and transfer of learning (objectively measured change in learner or observer knowledge or skills) was reported in two manuscripts and tools.(43, 67) No studies reported a change in organizational delivery or quality of patient care as a result of using any of the tools. Use of the tool was shown to positively affect individual learner performance by resulting in a change in knowledge, skills, or attitudes in five studies and four tools (28, 42, 43, 66, 67), shown to positively affect or change a program curriculum in three studies (19, 20, 33) and shown to provide specific actionable results that are regarded as useful by the learners in two studies and tools.(29, 67)

Overall ACGME Grade and Summary of Evidence for Tools:

The ACGME grading scheme for evidence-based evaluation of assessment approaches against the standards for assessment methods was applied to all studies. These were combined to generate an overall grade for each tool and are presented in Table 3. Grading rules include the most critical aspects of the standard, plus other supportive evidence and are required for the highest grade.(6) Two tools had evidence to justify the highest ACGME Grade for validity: the 10-item Operative Performance Rating System (38, 60) and the Structured Assessment of Microsurgery Skills (SAMS) (30); one tool for reliability: the 70-item tool reported by Paisley et al.(42); seven tools for ease of use: the global rating scale (12-15, 17-24), modified global rating scale (GRITS) (33), Procedure-Based Assessment (PBA) (26-29), Vaginal Surgical Skills Index (VSSI) (19, 20), Rigid Bronchoscopy Global Rating Checklist (37) and a scale to assess performance of tonsillectomy.(43); one tool for resources required: the Mastoidectomy Global Rating Scale and Checklist (34); two tools for ease of interpretation including the Endoscopic Sinus Surgery (ESS) Assessment Tool (49, 59) and the Objective Structured Assessment of Laparoscopic Salpingectomy (OSA-LS) (41). No tool received the highest grade for educational impact.

Two tools received an overall Class 1 ACGME grade (the highest grade) and are recommended as a core component of a training program’s evaluation based on Level A ACGME evidence from methodologically sound studies. These included the 7-item global rating scale (16) as reported in 12 studies (12-15, 17-24) and Procedure-Based Assessment as reported in four studies. (26-29)

Discussion

This systematic review summarizes the tools that have been studied to directly assess psychomotor (surgical or procedural) skills of medical trainees on live patients and data in support of their psychometric and edumetric properties. Past reviews have been descriptive in nature, focus on listing available tools, and unsystematic in methods.(68-70) Tools identified in this study are similar to a recent systematic review of available tools for assessing general surgery or obstetrics and gynecology procedures inside or outside of the operating room using grading from the Oxford Centre for Evidence-based Medicine levels of evidence for diagnostic studies reported by van Hove et al.(71) In contrast, this review focused on only those tools used for live observation of trainees by supervisors and applied a grade to existing tools using recommended grading criteria provided by the ACGME. The application of ACGME standards and grading rules to these existing tools provides users a useful summary of psychometric and edumetric properties of each tool. The graded "report card" (Table 3) is intended as a user-friendly source of evidence to validate use of methods and guide selection of additional assessment approaches.(6) The report card could be used in a number of ways. For example, if a particular tool is used for assessment, or as part of an educational standard, the report card summarizes the quality of the evidence in support of important psychometric properties. This "graded" summary is useful to the program as well as trainees in whom it will be applied in establishing credibility and consensus among decision makers. Tools identified by the report card that have poor or unknown psychometric properties should be cautiously interpreted prior to using them in these contexts. The report can also be used by researchers as a list of important elements that should be considered at the beginning of instrument building. The findings of this study demonstrated that most developers measured psychometric properties of the tools but often failed to measure important edumetric properties. We recommend that study designers incorporate both of these broad outcome measures in their studies. Finally, the edumetric properties summary of the report card allows users to know which tools have been used increase learning or to motivate learners to learn a procedure. This was often simultaneously performed when investigators used the tool for quantitative or qualitative feedback after performance of the procedure. Used in this way, the tools become teaching tools and the report card helps users gain a comprehensive understanding of how and which assessment tools can provide benefits such as these when incorporated into curricula.

This review comes at a time when there is increased pressure from within and outside of the medical profession for physicians to demonstrate competence in procedural and surgical skills. Various stakeholders including accreditation and licensure organizations, as well as the public who funds and receives care from medical trainees, expect to see more objective outcomes related to competence of psychomotor performance prior to performing unsupervised procedures in practice. This study identified a number of different tools that can be used to assess psychomotor performance. Available evidence was most robust for two tools: the 7-item global rating scale and Procedure-Based Assessment. Both of these tools received the highest recommendation using ACGME evidence grading by reviewers in this study.(Table 3) Depending on the procedure, Procedure-Based Assessment has greater than 30 items that need to be answered and it takes slightly longer to fill out than the 7 items in the global rating scale. This increase in items allows for assessment of a range of issues that occur during the perioperative period such as informed consent, preoperative planning, preoperative preparation, communication, and postoperative management. These items are in addition to a checklist of technical skills that are generic enough to be used during assessment of different procedures. The global rating scale only has the technical skills noted under each heading which limits the assessment to psychomotor skills. Overall, there appear to be advantages and disadvantages to both tools and future studies should investigate assessment properties and educational impact of using the two tools together or developing tools that incorporate both response types.

Similar to a recent review on clinical assessment tools (7), this review identified few studies investigating the educational impact of using each tool. The education community understands the almost lawful relationship that ‘assessment drives learning.’ However, this strong relationship is often overlooked by the medical education community. Future studies should measure associations between learning or teaching and the ‘process’ of using an assessment tool. It may be that certain tools motivate different learning or teaching behaviors which are just as important if not more than their ability to assess. Additionally, we identified few studies that described assessment tools for procedures that occur outside the operating room. Two tools: the Global Rating Scale for Epidural Anesthesia and Checklist (24) and Rigid bronchoscopy global rating checklist (37) were designed for non-operative procedures and could be used at the bedside. The surgical direct observation of procedural skills (Surgical DOPS) is a commonly used instrument in the United Kingdom used to assess "interventionalists" procedures and can be accessed at https://www.iscp.ac.uk/home/assessment_sdops.aspx. Similar to the review performed by Memon et al., we failed to identify any studies assessing its validity and reliability in the live setting.(70)

Strengths of this review are that it included over 4,000 articles, reviewed studies using accepted educational frameworks and summarized them using a user friendly ‘report card’ format recommended by the ACGME. However, several limitations should be considered. Although the standardized grading rules proposed by the ACGME Committee on Educational Outcome appear to be useful in the absence of current methods of summarizing assessment tools, these rules were not easy to apply to existing studies. For instance, it was often difficult to determine whether original tool developers were using explicit criteria, other than implied clinical judgment, to make decisions regarding which items should be included in a particular tool. For example, when our reviewers had to determine whether one important element of validity (APA Standard 1.6 - Processes and procedures used for selection of the content of assessment and for any criteria (eg, importance, frequency, and criticality) used to sample content should be described and justified when validation rests in part on the assessment content) was presented the decision was based on judgment of the reviewer. Fortunately for this review, all reviewers were operating surgeons and could use surgical and procedure-based principles in medical judgment. However, caution should be exercised when decisions regarding tool validity are made by reviwers who are not familiar with relevant procedure-based and anatomic principles. Additionally, the method of making an overall recommendation of a particular tool is not explicitly described in the original description of the ACGME’s assessment framework. Future work should be performed to establish the reliability of reviwers in making these overall decisions using the grading rules for each element as well as the grading rules for evidence supporting the tool.

Despite these limitations, we chose to use these guidelines in the absence of identifying other useful evaluation frameworks to summarize psychometric and edumetric properties of tools. However, we recommend the medical community continue to refine the criteria into more user friendly language. We included only English-language studies and did not include unpublished or non-indexed tools. Additionally, we did not include assessment tools that have only been studied in a simulated or laboratory setting. The increase in simulation-based training has allowed and required many investigators to use various assessment instruments to assess psychomotor performance on models. These instruments may have value when used in a live observational setting and we urge investigators to study and report such tools to determine their utility in the workplace setting. We chose to review only tools that had been used and studied in the live environment since evaluation in the patient care environment is still considered to be the "gold standard" and thus establishes user acceptability. Acceptability by the user is a critical characteristic of any assessment tool. The major value when used in the live setting is that the tool must allow the teacher to exploit their expertise and utilize their professional judgment. Additionally, the live setting may introduce significant variability in the construct being assessed and we wanted to report the psychometric properties of these tools under these variable conditions as it is not uncommon for measures of validity and reliability to change under unpredictable clinical situations.

In conclusion, numerous tools are available for the assessment of psychomotor skills of medical trainees in the live setting but evidence supporting psychometric and edumetric properties are limited to a few of these tools. Guiding frameworks to assess appropriate psychometric and edumetric properties of assessment tools should be the subject of future research. By combining several methods for assessing important properties using one framework and summarizing these into a simple report card, we hope future researchers will similarly use it to guide important tool design and curricula designers will use it make decision on which tools may be used to increase procedure learning. Users may also select different tools with different psychometric and edumetric properties to provide a more holistic evaluation of learning a specific procedure. It is also our hope that future studies continue to build on the report card given the rapid increase in tool design and their use in different clinical and educational settings.

Figure Legend

Figure 1 - PRISMA Flow Diagram