TEDI conferences: Planning Integrated Assessment across 12 Departments in a PBL Course

Planning Integrated Assessment across 12 Departments in a PBL Course
by
HG Alexander
JR Jass and
CB Campbell
Graduate School of Medicine,
The University of Queensland

Introduction
The Graduate School of Medicine at the University of Queensland introduced a new graduate entry, four-year medical course in 1997. The design of the GMC is based on the principles of problem-based learning (PBL) in a clinical context and emphasis is placed upon self directed, life-long learning, critical evaluation of the scientific basis of medicine, integration of basic and clinical sciences, a biopsychosocial approach to medicine, and early clinical experience with the development of communication skills from the outset (Graduate Medical Course Curriculum 1995).
The GMC is structured not on departments, but on four domains of learning: Basic and Clinical Sciences (BCS); Interpersonal Skills, Clinical Reasoning and Practice (ICSCRP); Doctor & Community (D&C); and Ethics, Personal and Professional Development (EPPD). Twelve departments (some of which represent more than one discipline) in two Faculties of the University contribute to the GMC.
Major summative assessments, which determine students' progression through the course, occur at the end of Years 1 and 2 and during Year 4. To progress through each stage of the course, and to graduate, students must pass each domain. The weighting of the domains varies according to the stage of the course. Criterion-referenced assessment is used in the GMC, in contrast to the norm-referenced assessment in the undergraduate course. Formative assessment is provided regularly in all years of the course, with instruments used matching those in the summative assessment.
The first summative assessment for Year I of the GMC took place in October 1997. This assessment introduced new academic and administrative issues. It included three written papers and two Multi-Station Assessment Tasks (MSATs). Most of the questions in the written papers are Problem Solving Exercises (PSEs) which are similar to Modified Essay Questions (Knox 1989; Feletti & Smith 1986) but do not progress to all clinical aspects of the case in Years 1 and 2 of the GMC. PSEs are integrated, vary in length from 20 to 45 minutes, and can be used to test a range of knowledge, skills and attitudes. Although some questions aimed at fact recall may be a part of a PSE, it primarily tests the students' ability to formulate hypotheses, explain underlying mechanisms, order investigations, interpret data, and in later years of the GMC, to make diagnoses and develop management plans.
The papers also included Extended Matched Questions (EMQs), Structured Short Answer Questions (SSAs) and essays. The PSEs and EMQs were new assessment instruments in the Graduate School of Medicine. The MSATs combine features of a laboratory practical examination with those of an Objective Structured Clinical Examination (OSCE) into one examination. The Departments contributing to the GMC have considerable experience in both of these latter instruments.
The nature of PBL may be debated (Norman 1991; Finucane, Johnson & Prideaux 1998), but a central feature of all descriptions is the acquisition of knowledge, through learning in clinical contexts. The PBL weeks present the student with an integrated program where resources from many disciplines are available for students to use in self-directed learning, relevant to the week's clinical case. This is reinforced by the assessments, integrated at the level of individual questions, that is, each question is planned to cover the basic sciences and clinical aspects of the given scenario. Thus, many domains and disciplines may contribute to any one summative assessment question. This form of assessment is more complex than when integration is viewed merely as a paper containing a range of questions set by different departments.
The problems encountered are of three types:
1. Inertia to change
The faculty has had to adapt to radical change with the introduction of the new medical curriculum. Departments have also had the responsibility and workload of simultaneously running the remaining years of the previous undergraduate course. Assessment in that course was departmentally organised, norm-referenced and frequently based on testing of fact by multiple choice and multiple true/false questions. Changes to the new assessment format presented the faculty with additional challenges.
In the first year of the GMC, an overall matrix was outlined and the development of questions was started at a multi-disciplinary workshop. Attendance at this workshop varied between disciplines and this inhibited some question development. After the workshop departmental representatives developed further material. The process proved to be cumbersome.
For trial papers and formative assessment, assessment material was requested directly from departments and domains, but this did not achieve the overall integration required.
It became obvious that the development of the integrated questions needed to be driven centrally in a structured manner, while maintaining the academic input and ownership of the assessment within departments and domains.
To address this a small Assessment team was formed. This team meets on a weekly basis and consists of the Chair of the Assessment Committee, the Assessment Officer at the Queensland Medical Education Centre, together with a small group of clinicians and basic scientists. The team uses the overall matrix provided by the Assessment Officer, the year's learning objectives and each Case's learning objectives to draft outlines of PSEs indicating where input from academic staff can be inserted to test that students have mastered the range of learning objectives. The cases upon which questions are based do not necessarily match those on which the PBL cases are based. The outlines provided by the team are discussed at the Assessment Committee, which has broad representation across Faculties, Departments and Domains. When the outline is approved for development, Departmental representatives and Domain Chairs are responsible for ensuring that the individual questions are written, together with answer guides, mark and time allocation. The Assessment team reviews the PSE produced. This process has resulted in drafts of PSEs being sent for development earlier, allowing more time for the critical review of questions at many stages. The team has developed expertise in question design, particularly in identifying ambiguous wording and mismatches between the question's instructions and suggested answer.
2. Differing standards and expectations
The discussion of questions at all levels has highlighted the differing expectations that various staff have of the students. This has occurred between Scientists and Clinicians, and between Clinicians from different disciplines. This process has shown that the opinions of staff about the correct answer or technique may vary from that given in prescribed textbooks.
The inclusion of all answers with questions, at each stage of the question development and review process, has drawn attention to this problem. The question material needs to be reviewed by multi-disciplinary groups as occurs in the Assessment Team and Assessment Committee, and in other informal working groups.
3. Marking integrated assessments
A broad spread of disciplines/domains and departments in an exam that emphasised free text responses in an integrated format, required changes to previous marking practices.
Marking of each of the three written papers occurs on the afternoon following each exam. The structure of marking groups is planned for each question, depending on the nature and content of the question. The marking groups discuss the answer guide and standardise the approach to difficulties that may arise during the marking.

Conclusion
The format of the end-of-year assessments supports the learning process of the GMC. The strategies used to drive the development of integrated assessment material have facilitated the change to a new style of assessment. The question design and wording has improved over time and the interest and expertise of faculty members in assessment is increasing. The integrated nature of the assessment has brought faculty together in the exchange of ideas, standards and resources. Considerable benefits flow from this to students, staff and the Graduate School of Medicine, encouraging integration in other areas such as research.

References
Feletti, G.I., Smith, E.K.M. (1986) Modified Essay Questions: are they worth the effort? Medical Education, 20,126-132.
Finucane, P.M., Johnson, S.M., Prideaux, D.J. (1998) Problem-based learning: its rationale and efficacy. Medical Journal of Australia, 168, 445-448.
Graduate Medical Course Curriculum (1995) Faculty of Medicine, The University of Queensland.
Knox, J.D.E. (1989) What is ... a Modified Essay Question? Medical Teacher, 11,51-55.
Norman, G.R. (1991) ' What Should be Assessed?', in The Challenge of Problem Based Learning, eds D. Boud & G. Feletti, Kogan Page Limited , London.

Conference Entry Page About the Conference Abstracts Conference Program

TEDI Home About Assessment UQ Home

This site built on a Macintosh using Dreamweaver.. Last modified 20/1/99; 10:14:12 AM.