DEVELOPMENT OF WORKPLACE-BASED ASSESSMENTS FOR SURGICAL AND ENDOVASCULAR PROCEDURES

Aims:
Workplace based assessments (WBAs) are methods used in medical training to assess those aspects of real day-to-day performance that a remote-controlled assessment of competence cannot assess.
The aims of the project were
1.to develop the existing Surgical- Direct Observation of Procedural Skills (S-DOPS) tool.
2.to develop WBAs for endovascular procedures from existing Procedure-based assessments (PBAs) for surgical procedures.

Methods:
1: Data on S-DOPS for one year (2007-08) were extracted and analysed. A new version based on existing PBAs was designed and trialled. Analysis of data on S-DOPS for 2010-11 was undertaken to assess if the changes made had resulted in better utilisation of the tool.
2: Endovascular procedures performed by consultants were observed and analysed by a group of vascular radiologists and surgeons to identify and agree the task-specific and global domains. Procedure-based assessments are WBAs which rate a combination of task-based and global competencies, together with a summary judgement about the competence of the trainees to perform that procedure. The PBAs were created and piloted by the consultants. Assessments of trainees performing endovascular procedures were undertaken.
Results:
1: There was increased uptake of the new version of S-DOPS (2010-11), with more number of trainees completing a higher number of S-DOPS. There was good correlation between the level of training and the S-DOPS score (higher level trainees achieved higher mean scores).
2: Limited sample numbers made it difficult to draw any statistically significant conclusions on validity and reliability. However, qualitative analysis suggests that endovascular PBAs appear to be acceptable and feasible, and trainers and trainees find them educationally useful.

Conclusions:
1: The absence of ratings below 4 and the low number per trainee (2007-08) suggest that DOPS was used only when trainees felt able to meet expectations and that assessors might be confused by the norm-based scale. The new S-DOPS form (2010-11) has criterion-based ratings and more space for recording feedback. Both assessors and trainees seem more satisfied with the new form. Criterion-based ratings seem more acceptable and confers improved construct validity to the S-DOPS tool.
2: Endovascular PBAs have been incorporated into the RCR e-portfolio and are used by IR trainees. Similar PBAs have been designed for non-vascular IR procedures, and together with endovascular PBAs are collectively called IR-PBAs.