BrainZone use in Physiology and Pharmacology

by

Alison Bailey

Department of Physiology and Pharmacology,
The University of Queensland

 

Introduction

The Web based assessment program BrainZone was used in many departments during the university-wide trial period of Semester 2, 1997, by 60 lecturers and over 3,800 students. This widespread use throughout the University reflects the ease with which tests can be compiled by academics without advanced computer skills and contrasts with the previous finding that "early adopters of summative assessment through the World Wide Web appear to be predominantly in the fields of computers and education" 1

BrainZone was ready for use during the last three weeks of first semester 1997. It was used in one first year science subject with about 1300 enrolled students and in second year Physiology subjects which had lectures in common (about 500 students). Over 1000 students were then offered tests in first and second year subjects during second semester. Tests were formative, "learning" tests consisting of multiple choice questions, each question selected randomly from a bin of comparable questions. Feedback to students was provided both as a score and comments which explained the correct responses.

These trials encouraged us to offer more formative tests in Semester 1 1998, to cover more lecture topics in more subjects. In addition, summative "examinable" tests were introduced in the first year subject and one second year subject. First year students had two tests which contributed 15% of the final mark for the subject; they had only one attempt at these tests but up to five attempts at related formative tests. The second year students had three tests which contributed 10% of the final mark if at least 8/10 was scored in each of the tests; no limit was placed on use of related formative tests but only two attempts at each summative test were permitted unless a student undertook remedial study.

 

Influence of formative tests on students' performance

Students who used BrainZone earned higher marks in their end of semester examination than those who did not. This has been observed in each of the subjects for which we have compared students' marks with their use of formative tests and some measure of their ability on entry to the subject. The higher mark largely reflects the better motivation and study patterns of students who chose to use the tests, but the difference is clear enough to promote test-use to students.

Even when we exclude the motivational factor, there is evidence that the tests significantly improved final examination marks: the mean score of students who did the tests was significantly higher in topics covered by tests than in other topics whereas there was no significant difference between topics for students who did not use the tests.

 

Student use of formative tests

The number of students who used the formative tests in 1997 was disappointing, though this was not unexpected: student reluctance to use aids to study has been noted in many fields 2 .

As anticipated, the most successful students used the tests most. In 1997, tests were used in the first year subject by 35 -43 % of students with the best university entry scores, (OP 3 or less) whereas only 10 - 20% with poorer scores used them. Less than a third of second year students used their tests and only half the dental and veterinary students -who are recognised as high achievers with strong motivation - used a test covering lecture material which they were finding difficult.

The effect of introducing summative tests this year was dramatic: in the first year subject, 85% of enrolled students did the first formative test, 78% the second and 92% did the summative test; in the second year science subject with summative tests, 98% of students did formative tests. BrainZone use was again disappointing in subjects with formative tests only. Even when class time was allocated to BrainZone, only 61% of the veterinary students logged on to tests.

A few students used only one test once but most made from five to fifteen attempts with their final attempts completed in two to three minutes.

 

Influence on students' learning

Summative tests were introduced to encourage students to use BrainZone and gain feedback on their understanding of lecture material. Setting the time-limit for a summative test can promote early revision and reward students who study consistently throughout semester. Some students commented that getting a poor score in their first summative test would lead to better use of formative tests in future - we will look for evidence of this by comparing use of first with subsequent formative tests.

Although all our tests have used multiple choice questions, even these can test synthesis and interpretation rather than mere rote recall if written carefully. BrainZone questions are easily edited and refined in response to comments from students.

The test report already contains all the information needed for factor analysis to assess the difficulty of questions and their ability to discriminate between students; downloading of reports in a format which facilitates this analysis is planned. The "drag and drop" question type under development can test deeper learning, enabling students to construct flow charts and concept maps.

 

Summative assessment

Many departments have found that tests which contribute to final assessment are taken more seriously than tests which are offered solely for learning feedback. There is also general acceptance that up to 20% of final marks can be awarded without a high level of security without compromising the final grades achieved.

Some first year students cheated by doing summative tests in a group but, because students had to get over 50% in the final exam to pass the subject, a BrainZone derived benefit was more likely to arise from knowledge gained than from the mark scored. Summative tests in the second year subject helped to increase the proportion of students with a grade of 5 and reduced the number who failed compared with previous years.

Time limits can control cheating if they are adequate for thought but too short to permit reference to written material and still answer all questions; collaboration to accumulate all questions in the master bank also requires surplus time for copying questions.

The nature of questions can discourage cheating: simple rote recall questions could be answered quickly by using a textbook index but questions which test interpretation and synthesis demand reference to knowledge in memory. Summative tests can be compiled from questions that are similar but different to those used in formative tests.

BrainZone could contribute a greater proportion of final assessment if students prove their identity when they log on to supervised computers; some universities use this system for intranet based tests 3. Students benefit from being able to choose the timing of their test but the demand on computer facilities is much greater.

This is, however, a very new field. Very soon, owning a computer with modem will be as essential for University education as owning textbooks is now. Relatively simple and inexpensive desktop videoconferencing equipment would provide an easy increase to security if students accept remote observation while doing a summative test.

 

Student acceptance

Students clearly like BrainZone, based on evidence from personal demonstrations in the CAL laboratories and surveys. In 1997, 90% of the first year students who responded to a questionnaire rated BrainZone 4 or 5 on a scale where 5 indicated 'extremely helpful' and 1 ' no use'. This year, when there was more compulsion to use the tests, 79% of respondents rated it 4 or 5, 16% rated it 3 or 'helpful'; the handful of students who rated it 1 or 2 indicated that they had experienced problems using the tests outside the Department or were not happy using computers. Employing student helpers in our CAL laboratories for 2 hours each day was a successful method for assisting computer illiterate students.

 

Resource implications

Pressure on university computer facilities is greatly reduced if students can use their own computers. Our surveys over the last three years show computer and modem ownership steadily increasing, 75% of respondents in 1997 having access to a computer at home, 36% with a modem. BrainZone, being Web based, is platform-independent. During the weekend immediately before a summative test deadline, 215 students did tests from computers at home or in a University library or college, easing pressure on our computer laboratories, removing the need to extend opening hours.

Using a computer assessment system saves paper and printing costs for both students and the department. On the other hand, demand for dial-in access, gateway and BrainZone servers is increased.

A very large commitment in time is necessary to compile questions. Various approaches were tried:

Speedy troubleshooting from a responsive support team in Educational Multimedia Services was vital for success with our relatively ambitious use of BrainZone. Proposed modifications to make the program browser and plug-in independent would be very worthwhile.

Other useful developments have been suggested but the program has already proved to be a most worthwhile aid to student learning if lecturers can find the time to generate good questions.

 

References

  1. O'Reilly M. and Patterson K., Assessing learners through the WWW, Computer Networks and ISDN Systems 30, 727-729 (1998)

  2. Elliott J., Voluntary study skills seminars: why they miss the target, Inaugural Pacific Rim First Year Experience Conference (1995) (proceedings on disk).

  3. Fyfe S., Using computer managed learning: the Curtin experience, Proceedings of the Computer Assessment Workshop (1997)

 

Conference Entry Page About the Conference Abstracts Conference Program Copyright information

TEDI Home About Assessment UQ Home

This site built on a Macintosh using Dreamweaver.. Last modified 20/1/99; 10:14:13 AM.