Introduction of criterion referenced assessment in biological science laboratory subjects

by

Dianne Stomfay

Department of Microbiology

and

Hock Ng

Department of Botany

The University of Queensland

 

Introduction

In general, Science students at The University of Queensland rely heavily on three years of traditional lecture and practical instructional formats to gain their discipline-related knowledge and practical skills. The "hands-on" experience in a laboratory setting is considered essential for undergraduate students majoring in a biological science discipline. Some of these students continue in Science by enrolling in a research-based Honours year, and during this time, they become bench "tutors" in undergraduate practical classes. Each department offers a "Tutor Training Program" at the start of each semester, to help prepare tutors for their varied teaching duties, including assessment. Most tutors assess a small group of students during the semester. Traditionally the subject co-ordinator prepares a brief written description of each assessment item (as part of the subject profile), and additional information is usually given to tutors for marking purposes (as part of a mark sheet).

Currently, most of the bench tutors have experienced assessment the "old" way - norm referenced assessment for grading, which makes a summative judgement of a class of students, and limits the number of students allocated to each grade. The University has recently made changes to its assessment policy, to enhance the quality of student learning in terms of how and what students learn, and the types of guidelines students need. The "new" way is criterion-referenced assessment, which provides students with written guidelines (criteria and standards) prior to starting the assessment task. It is imperative that the guidelines are attainable, and that they are clear for both the students and tutors. This provides a focus on the requirements for each task, and maximises the learning experiences. This paper describes criterion-referenced assessment evaluated in two subjects in 1998.

 

Data collection

Two large laboratory class subjects were chosen for evaluation of criterion-based assessment guidelines: a first year botany subject and a second year microbiology subject. Each subject has four assessment items, and one item from each subject was chosen for analysis by the students and tutors. There were two evaluations done in this study. (i) The botany class evaluated during class time the assessment guidelines for the writing of a scientific essay. This item was assessed once and comprises 20% of the total mark. (ii) The microbiology class evaluated the assessment guidelines for the writing of a scientific notebook. This item was assessed twice and comprises 10% of the total mark.

An evaluation was done using questionnaires given to tutors and students. The questionnaires consisted of a series of statements and space for written comments. Two questionnaires were designed in consultation with Mr Geoff Isaacs (TEDI). The first seven statements focused on the transparency of the assessment information, as perceived by students and tutors (Appendix 1). The students then rated three more statements which focused on when they used the assessment guidelines (Appendix 2). The tutors rated four different statements which focused on their use of the student guidelines and the tutor marksheets during marking (Appendix 3). There was a choice of five responses for each statement: agree, agree with reservations, unsure, disagree with reservations, and disagree.

 

Results

The results for the evaluation of the two guidelines are presented: i) the scientific essay, and ii) the scientific notebook.

1. Scientific essay

   Respondents: There were 14 tutors (postgraduate Botany students) in this subject, and 11 tutors completed the questionnaire. There were 221 students in this subject, and 172 students completed the questionnaire.

   Tutor responses:

   1. The tutors answered the seven statements of Q1 (Appendix 1). Three of the seven statements were answered by all tutors as either "agree" or "agree with reservations". The other four statements were answered as either "agree" or "agree with reservations" by the majority (81-91%) of tutors.

   2. The tutors then answered the four statements for tutors only (Q2 - Appendix 3). All of the tutors agreed that the guidelines enabled them to "mark consistently"; however only 36% perceived that there was consistency in marking between tutors. The tutors were expected to use both the tutor marksheet and the student guidelines when marking. The other two statements required the tutors to indicate whether they used both sets of guidelines for marking. The majority of tutors (64%) indicated that they used only used the marking guide for tutors, and did not refer to the information given to the students. One tutor marked "hard" and this work had to be re-marked by the subject co-ordinator.

   3. All of the tutors wrote comments in the space provided. Examples included:
      • criteria were very vague
      • marksheet needed more structure
      • requirements were well set out and explained, but marks differ significantly between tutors. Marks need to be set by standards more.
      • some tutors too picky and demanding
      • comments students had made to them during class

   Student responses:

   1. The students answered the seven statements of Q1 (Appendix 1). Students mainly agreed with all of these statements (shown below). The brackets at the end of each statement indicate the percentage of students who answered "agree" or "agree with reservations":
      • "assessment information was presented at the start of the semester" (94%)
      • "there were opportunities to ask questions about assessment" (89%)
      • "assessment requirements were clear" (78%)
      • "assessment criteria gave sufficient information and guidance" (68%)
      • "assessment standards gave sufficient information and guidance" (65%)
      • "it was clear what was needed to get a good mark for this" (65%)
      • "grading system was clearly explained" (56%).

   2. The students then answered the three statements of Q2 for students only (Appendix 2). These statements focused on when the students used the guidelines. A summary of the student responses to these statements is shown below. The brackets at the end of each statement indicate the percentage of students who answered "agree" or "agree with reservations":
      • "assessment guidelines were useful while I was doing the assessment" (78%).
      • "I reviewed the assessment guidelines to understand how the mark was derived" (55%).
      • "I used the assessment guidelines to check my work before handing it in" (67%).

   3. There was a range of comments from the students. Only 52 students (30%) wrote comments about the subject and assessment items. The majority of comments indicated that the workload was too great for the number of credit points allocated to this subject. Other comments about assessment included:
      • Good and clear what is needed. Very helpful.
      • Information in the lab book was very helpful and easy to understand.
      • Could be more clear - it is hard to read "between the lines" to see exactly what is required. However it is extremely helpful.
      • Suggest that examples of good past assignments are made available to guide with appropriate format.
      • Assessment information in the manual is great. I used it as a guide and got good results for all my assignment.
      • Not specific enough with regards to requirements of what we had to write. Some more definition would be nice.
      • Unclear and not very directional
      • Needed a bit more guidance with format, etc.
      • I found the suggestions and criteria to be useful but complex, and not really easy to understand.

2. Scientific notebook

   Respondents: There were seven tutors (postgraduate Microbiology students) in this subject, and three of the tutors completed the questionnaire. There were over 80 students enrolled in this subject in first semester, and 25 students completed the questionnaire in the following semester (after the subject was completed and the item had been marked twice). The survey form and the assessment guidelines were given out to students.

   Tutor responses:

   1. The tutors answered the seven statements of Q1 (Appendix 1). Six of the seven statements were answered by all tutors as either "agree" or "agree with reservations". Tutors answered the other statement "...standards gave sufficient information and guidance" differently. Two tutors answered "agree", and one tutor answered "disagree with reservations".

   2. The tutors then answered the four statements for tutors only (Q2 - Appendix 3). Two of the four statements referred to using the guidelines to enable "consistent marking" within one group and between tutors. These two statements were answered as either "agree" or "agree with reservations" by all tutors. Tutors were expected to use both sets of guidelines when marking. The other two statements required tutors to indicate whether they used both sets of guidelines for marking. All tutors indicated that they used both sets of guidelines.

   3. All of the tutors wrote comments in the space provided. The tutors indicated at the start of the semester that the guidelines given to the students were clear. The tutors' written comments mainly referred to the difficulty in the interpretation of assessment standards while marking the first time. They also indicated that it was difficult to allocate one standard to the work, as they felt that most books fell "between" standards. The tutors then allocated additional part-marks at their discretion (1/4, 1/2, 3/4). Consequently, the tutors had marked inconsistently. The books were re-marked by one tutor who helped design the assessment guidelines. The tutors compared their mark to the new mark and then made comments which clarified issues arising from the marksheet. The second time books were marked, the tutors had learned how to interpret the assessment guidelines, and marked consistently by NOT allocating the additional part marks.

   Student responses:

   1. The students answered the seven statements of Q1 (Appendix 1). Students mainly agreed with all of these statements (shown below). The brackets at the end of each statement indicate the percentage of students who answered "agree" or "agree with reservations":
      • "assessment information was presented at the start of the semester" (96%)
      • "there were opportunities to ask questions about assessment" (84%)
      • "assessment requirements were clear" (76%)
      • "grading system was clearly explained" (72%)
      • "assessment criteria gave sufficient information and guidance" (64%)
      • "assessment standards gave sufficient information and guidance" (56%).
      • "it was clear what was needed to get a good mark for this" (52%).

   2. The students then answered the three statements for students only (Q2 - Appendix 2). These statements focused on when the students (individually) used the guidelines. A summary of the student responses to these statements is shown below. The brackets at the end of each statement indicate the percentage of students who answered "agree" or "agree with reservations":
      • "assessment guidelines were useful while I was doing the assessment" (52%).
      • "I reviewed the assessment guidelines to understand how the mark was derived" (48%).
      • "I used the assessment guidelines to check my work before handing it in" (40%).

   3. There was a range of comments from the students. Only 16 students (64%) wrote comments about the subject and assessment items.
      • Everything was clear and well set out; quite informative
      • Names of good tutors
      • Liked knowing the details of the assessment e.g. allocation of marks (with each standard), grading system for whole subject
      • Marking between tutors was not consistent (each interpreted guidelines differently); level of help given by tutors was uneven
      • Too much time spent on prac book; too much work for 10%
      • Requirements for books were unclear and harsh; need clear explanation from tutors at start of semester; need more detail with criteria
      • Unclear what information required for some sections
      • Mark books more often (twice during semester is not enough)
      • Tutor provided more help and information than what was printed in the book
      • Comments on other assessment items in this subject.

 

Discussion

The assessment this year in the two subjects was similar to that of the previous year. However, the format of the assessment guidelines for each assessment item was different this year in accordance with the new assessment policy. Students were given information on the correlation of final marks and grades, and the criteria and standards for each assessment item. Another consequence of the new assessment policy was that we critically examined the subject goals, and rewrote these to ensure that they linked directly to the four assessment items.

Unexpected feedback

• A range of written comments was received from the students. However, comments on a single issue were varied, and often they expressed opinions at the opposite ends of the spectrum. Students then have a mixed perception of whether the assessment information was clear or not.
• A large number of students answered "unsure" to questions, which we expected would only be answered by "agree" or "disagree".
• Students did not understand the grading system (they were provided with the total % correlating with a rating). They felt it was good to know this information, but we do not know what additional information they require to understand this better.
• Overall, the students are not continuing to learn after their work is returned with a mark. This should be useful feedback for their learning, but they are not doing it, probably due to a lack of time and other priorities.
• Some students now perceive that criterion referenced assessment means equity in marking between tutors; however they "know" this did not happen with the essay marking.

The positives

• Tutors' perceptions of criterion referenced assessment are usually higher than the students (if the teachers don't believe in the new assessment process, then there is little hope in offering something new to the students).
• Tutors perceive that they were able to mark consistently when assessing a group of students.
• Tutors need to mark student work by applying the criterion referenced assessment guidelines (rather than just perusing them). Microbiology tutors had the chance to apply the same guidelines twice, and this gave them the opportunity to "learn by doing".

The negatives

• An inconsistent teaching approach by tutors within a class was identified as a problem by both tutors and students. This is a difficult issue to address because large classes require a large number of tutors, who vary in their teaching experience.
• Tutors and students had difficulty in interpreting the assessment standards the first time they attempted to use them. Practice in interpretation of guidelines is essential to ensure that tutors mark consistently.
• Tutors who marked the essay did not perceive the student guidelines were important, and did not refer to them when marking. They are important, and this issue needs to be addressed. One written comment from a tutor noted that some tutors were "too picky". There is a difference in tutor's expectations of themselves and their students' work.

The future

This year we provided more assessment information and linked the subject goals directly to assessment. Overall, it was a positive exercise to introduce criterion referenced assessment into laboratory based subjects, but the details still need refining - both in terms of training the tutors to be experienced in using assessment guidelines and how to apply them, and helping the students interpret what is expected. This trial provided a starting point for identifying a range of issues not previously encountered in norm based assessment.

There is a need to find out more detailed information from our tutors and students about the problems they identified. Focus groups which interview a sample of students may be useful. However, currently, undergraduate students like being given the extra information, but they cannot give us useful cues and clues about rewording the guidelines to make them more specific. It would be ideal to involve both the tutors and students in the design of criteria and standards, but with large laboratory classes, this goal seems difficult to achieve at the moment. Large laboratory classes are prepared well in advance, including the printing of the student manual. However, we do have another useful resource within our departments: we can gain the student perspective from our postgraduate students who are most frequently our tutors. They provide useful insights into problems which undergraduate students encounter, as they were most recently in that situation themselves. Currently we are on a cycle of tutors who have not experienced criterion referenced assessment as part of their undergraduate course. However, within the next year or so that situation will change, and we will have a pool of tutors who have experienced (and some helped design) criterion referenced assessment.

 

References

Chalmers, D and Fuller, R (1996) Teaching for learning at university, Kogan Page.

Ramsden, P (1992) Learning to teach in higher education, London: Routledge.

Rowntree, D (1987) Assessing students: How shall we know them? Kogan Page.

 

Appendix 1

This is Question 1 of the questionnaire issued to students and tutors. It was designed to evaluate criterion referenced assessment guidelines.

1. The assessment information was presented at the start of the semester.

2. The assessment requirements were clear.

3. It was clear what was needed to get a good mark for this assessment item.

4. There were opportunities to ask questions about the assessment.

5. The assessment criteria gave sufficient information and guidance.

6. The assessment standards gave sufficient information and guidance.

7. The grading system was clearly explained.

 

Appendix 2

This is Question 2 of the questionnaire issued to students. It was designed to evaluate criterion referenced assessment guidelines prepared for students.

1. The assessment guidelines were useful to me while I was doing the assessment item.

2. After I had completed the assessment item, I used the assessment guidelines to check my work before handing it in.

3. After work was returned, I reviewed the assessment guidelines to understand how the mark was derived.

 

Appendix 3

This is Question 2 of the questionnaire issued to tutors. It was designed to evaluate the student guidelines and the tutor's marksheet.

1. Assessment criteria and standards enabled me to mark all of the books/essays consistently.

2. I only used the TUTOR'S marksheet to assess the students' work.

3. I only used the STUDENT'S assessment criteria and standards information to assess the students' work.

4. The wording of the assessment criteria and standards ensured consistent marking between tutors.

 

Conference Entry Page

About the Conference

Abstracts

Conference Program

TEDI Home

About Assessment

UQ Home

---

This site built on a Macintosh using Dreamweaver.. Last modified 20/1/99; 10:14:28 AM.