Assessment of General Practice Consulting Skills
using Standardised Patient Interviewsby Dr Michael Yelland MB BS, FRACGP, Grad. Dip. Musculoskeletal Medicine
Senior Lecturer in General Practice
Centre for General Practice, The University of Queensland
Introduction
This paper describes the use of standardised patient interviews as a means of assessing the consulting skills of Fifth Year Medical students studying general practice. Volunteers play standardised roles of common clinical problems. These interviews allow a more valid assessment of consulting skills than written assessments. They offer greater control of the choice of clinical problems and better reliability in assessment than real patient interviews. A detailed marksheet has been developed to increase inter-examiner reliability, discourage global marking and incorporate a patient assessment of the student. Problems with this form of assessment are discussed and solutions suggested.
What Are Standardised Patients?
The term 'standardised patient' (SP) encompasses both normal people who have been coached to portray a specific patient (otherwise called a 'simulated patient' ) and actual patients who have been coached to present their own illnesses in a standardised way (1). In the medical setting they are used at an undergraduate and postgraduate level in the teaching and assessment of consultation and physical examination skills. They have been used in the United States for over 30 years and have been the subject of a significant body of educational research. SP interviews offer a more valid and direct measure of clinical performance than indirect methods, such as modified essay questions and clinical vivas. They offer selection of clinical problems which are consistent with course objectives. They also offer uniformity of assessment across a group of students .
The Application of SP's in the Assessment of General Practice Consulting Skills
SP's at the University of Queensland's Centre for General Practice were introduced into examinations for Fifth Year Medical students in 1990 in response to a perception that the course objective of improving consultation skills was inadequately assessed. SP interviews currently occupy two stations in an objective structured clinical exam (OSCE), along with a series of short answer questions on problems encountered in general practice, and a station where physical examination or procedural skills are assessed.
The task of organising the OSCE is a substantial one. Eight groups of 20 to 30 students are examined each year, four of these groups being split between the Medical School in Brisbane and the North Queensland Clinical School in Townsville. For reasons related to the structure of the OSCE, interviews are limited to 10 minute stations (nine minutes for interviewing and a minute for completion of marking). As this time-frame is too short for students to complete a full consultation, interviews focus on either diagnostic or management tasks. In diagnostic interviews, students are asked to take a history from a patient who is new to them, and then to explain to them a problem list derived from this history. In management interviews, students are given a summary of history and examination findings and a working diagnosis prior to the interview and then asked to discuss management issues with the patient. Students are given one diagnostic interview and one management interview in their exam.
Interviews are drawn from a pool of nine cases based on common general practice problems selected so as to be well within the experience of Fifth Year students. Uncommon problems and problems requiring a high level of medical knowledge are avoided so that the students can focus on processes without the distraction of difficult content. Cases usually include an unspoken concern which will only be revealed if the student actively elicits it. This emphasises the importance of addressing the agendas of both the doctor and the patient during consultation, something specifically taught in preceding coursework.
SPs are volunteers from the patients of University General Practice who do not have the conditions they are asked to depict. They are given roleplay instructions to learn prior to a training session. They are trained in pairs to encourage standardisation of roleplays. In the training session, the trainer plays the part of students of varying competencies with the SP's and then gives them feedback on their roleplays to aid with their standardisation. The SP's are also asked to practise marking the performance of the roleplaying trainer in preparation for marking real students. Training sessions take one to one and a half hours.
Rating Performance in SP Interviews
In rating performance in SP interviews a standard form is used for each type of interview which assesses seven competencies which are allocated 10 marks each (Table 1.). On the evaluation forms, specific behaviours or qualities within each competency are described and are weighted according to their importance. Examiners are provided with case-specific guidelines for marking these behaviours or qualities where appropriate. This three tiered marking scheme aims to discourage global marking and improve interexaminer reliability. Students are provided with copies of these evaluation forms at the beginning of term as a guide to desirable consultation skills.
One component is a global assessment of the student's performance by the SP who is encouraged to assess the student's performance independently of the examiner based largely on their overall satisfaction with the interview. This input is valuable as it offers a different perspective to that of the examiner.
Table 1. Competencies assessed on rating forms for diagnostic and management interviews
COMPETENCY ASSESSED MARK ALLOCATED Diagnostic Interview Management Interview (Total mark = 70) Rapport, empathy and attitude Rapport, empathy and attitude 10 Verbal communication Verbal communication 10 History taking style Problem solving approach 10 Patient centredness Management style 10 Patient education Patient centredness 10 Problem list formulation Patient education, and
prevention10 Assessment by SP Assessment by SP 10 Potential Problems with SP's
Potential problems lie in variation in portrayal of case details by SPs The problem we have experienced is not so much in SP's remembering the details of the case but in achieving consistency in the amount of information that they give to the student in response to certain questions, especially about their concerns. This can be improved by discussing any observed inconsistencies in roleplays with the SP's and the provision of a number of stock phrases for use in response to certain questions.
Fatigue is another potential threat to the performance of SPs. Our students are examined in two groups of 12 to 16 patients in an afternoon. Each SP sees half of these students and often has one or two breaks within each group exam. There is also a generous break with a cup of tea in between groups.
Interexaminer reliability is monitored by comparing the means and standard deviations of marks granted by different examiners marking the same role play. Marks are moderated if the means differ by more than 7%. This occurs in less than 30% of exams. Currently examiners are provided with details of their case and marking guidelines well before the exam. Formal examiner training sessions where examiners observe and mark roleplays together would be likely to improve interexaminer reliability but have not been held to date.
The reproducibility of student performances across a range of SP interviews has been shown to improve with the number of interviews performed however even two interviews is still far from what is needed to achieve what is regarded as a satisfactory level of reproducibility for total exam scores (a generalisability coefficient of 0.8). For this an exam length of between 3 and 12 hours has been estimated (2). Such an exam is well beyond our current resources, but the high face validity of SP interviews more than justifies their continued use in our exam.
Test security is a concern in our exam which has to be held eight times each year. This is more of a concern in the content based parts of the exam, i.e. the short answer questions, than in the more process oriented interview segments. It is improved by increasing the pool of cases, a significant task considering that it takes approximately 10 hours of academic time to prepare roleplays and their marking guidelines and to train volunteers to play them accurately.
Conclusion
General practice is just one of many areas where SP's can be of benefit in broadening and deepening the assessment base. SP's have application not only in most health-related areas but in other professional areas where interview skills are important. They offer a reliable and valid form of assessment with a high degree of control of problem selection. The cost of these advantages is the considerable time and effort required to develop, stage and assess the interviews. In our experience this cost seems to be justified.
References
1.Barrows HS. An overview of the uses of standardised patients for teaching and evaluating clinical skills. Academic Medicine 1993, 68;6:443-453.
2. Colliver JA, Williams RG. Technical issues: test application. Academic Medicine 1993, 68;6:454-460.
Address for Correspondence: Dr Michael Yelland University General Practice PO Box 52 Inala 4077 Ph (07) 32755444 Fax (07) 32789987
Footnote:
A more extensive account of this work was published in Medical Education, 1998, 32:8-13
Conference Entry Page About the Conference Abstracts Conference Program
TEDI Home About Assessment UQ Home
This site built on a Macintosh using Dreamweaver.. Last modified 20/1/99; 10:14:29 AM.