Three ways to pursue academic quality? by Dr. Duncan D. Nulty
Teaching and Learning Development Unit,
Queensland University of Technology
Determining the dimensions
A questionnaire survey sent to 1500 full time academic staff in the three Brisbane universities resulted in 375 replies. The survey asked questions about physical and cognitive procedures associated with marking students written work. Three questions in particular (at different points in the questionnaire) related to the dimensions used when marking.
The first of these questions asked respondents to rate the degree of influence each of 37 characteristics of students' written work had on their judgments of that work. The second question sought short open-ended descriptions of the principal characteristics respondents associated with work of different standards. The third question asked respondents to indicate the principal quality looked for in each of several different kinds of work and then sought a verbal description of the evidence that would be used to determine the presence and/or mastery of the principal quality looked for in examination essays.
The dimensions that were elicited from analysis of responses to these questions were then discussed in interviews with a structured sample of respondents drawn from different disciplines and institutions. In the discussion, the dimensions derived from the questionnaire were compared to the personal construct systems of the interviewees (which were determined using the Repertory Grid Technique (Kelly, 1955)). In all cases the interviewees stated that the personal construct system was a good representation of the way they thought they marked students' work, and in all cases interviewees were able to reconcile their own personal construct system against the dimensions derived from the questionnaire analysis. Several interviewees stated that the wording of the dimensions needed some minor fine tuning. In spite of this, several interviewees also stated that the dimensions were a better summary of what they thought they gave marks to than anything they had prepared themselves. Most found the experience enlightening and thought the analysis was useful.
Without the time to go into the same depth as in each interview, I want to try to capture some of that benefit for you. I'll do this by sketching out the application of these dimensions to three areas. But first I'll say what the dimensions are.
What the dimensions are
Six dimensions of academic quality (most influential first)
The dimensions are not mutually exclusive - they relate to each other to some extent. Where particular overlap exists those characteristics have been italicised. The dimensions are in order of decreasing influence on the judgements made. Within each dimension the most influential characteristics are listed first. Interviews suggested that the sixth dimension was rarely used at all while dimensions 4 and 5 tend to be prerequisite to dimensions 1, 2 and 3. Furthermore, interviewees often commented on the inter-dependence of dimensions 1, 2 and 3. Finally, there were some particular differences between disciplines, for example, in Marketing dimension 5 was of primary importance.
The students' own intellectual content
Does the student have an argument? Does the student answer the question?
Characteristics of students' work that relate to this dimension:
Reasoning; Analysis, Logic, Interpretation, Ideas, Application, Creativity, Originality, Abstraction, Conjecture.
Dimension 2. Strength of analysis and argument
Is the argument a strong one (well argued)?
Characteristics of students work that relate to this dimension:
Analysis, Ideas, Logic, // Organisation, Structure, Fluency
Dimension 3. The Focus of the factual content on the student's idea
Do the facts cited support the argument? Is the argument valid? Is it right?
Characteristics of students work that relate to this dimension:
Content, Idea, Accuracy, Factual Errors, Evidence, Focus, Detail, Proof
Dimension 4. Clarity of expression
Is the argument well expressed, well organised, well structured, fluent?
Characteristics of students work that relate to this dimension:
Clarity, Organisation, Structure, Conciseness, Fluency, Style, Form, Mechanics, Rhetoric
Dimension 5. Presentation
Is the work free of writing errors?
Characteristics of students work that relate to this dimension:
Presentation, Grammar, Spelling, Neatness, Length, Punctuation
Dimension 6. Special considerations (which are not usually very influential).
Anything else?
Characteristics of students work that relate to this dimension:
e.g.: Format, number and recency of references, attendance and participation in tutorials.
Application to curriculum design
Curriculum design, in my opinion should start with a rationale. This informs consideration of what it is that the curriculum is to achieve. Thus, why a course or unit is offered is followed by aims and objectives. Next there will likely be some consideration given to content and the method of teaching it. Last - usually it is last - thought is given to how to assess whether the students learned what it was intended for them to learn. Also at this time, there might be some consideration about whether the assessment methods chosen can contribute to the learning process.
In theory there is a circularity about all this: the assessment seeks to assess what you set out to achieve. This is where the six dimensions come in: if these dimensions represent in generic terms what we value then it should be possible to use these headings to inform the construction of aims and objectives, choice of content and teaching method and selection of appropriate assessment techniques.
Application to marking practices
When we mark students' written work we will read that work and make judgements about whether it demonstrates certain things about the authors. We will do this with or without the aid of a formal marking guide. If we use a formal marking guide, it will often be created as the marking proceeds, or, in those cases where the guide was pre-prepared, it may be modified or interpreted differently as marking proceeds (see for example Gilliat and Hayward, 1996). In higher education, the range of circumstances in which one form of this evolution or other does not occur are, in my view, rare - although I accept that it is more common in a few particular disciplines. As an aside, I wonder what is "higher" about the education we are providing if we are able to pre-specify performance criteria in such detail that no adaptation or modification of judgement is required in the light of actual observed performances. But I digress ...
Despite this evolution - which I argue is inevitable and necessary - there are common dimensions to the judgements we make of students written work. What varies then, is not so much the dimensions, but rather the way in which they are manifested by students' performances. In other words, there is some variability in what we consider to be evidence of a particular level of performance because this is, in part, determined by context as well as experience. But, in broad terms, what the students' performances are evidence of is pretty stable.
It seems sensible to make use of this awareness by first, considering whether the dimensions I've specified do validly relate to judgements about the quality of students' learning in the unit or course; secondly, if so, to develop a marking guide that accords with that reality in a systematic rather than serendipitous or ad hoc way and third to seek to illustrate this by use of what Saddler (1987) has called exemplars.
Application to students' learning
The flip side of us working out what we value in students work, and therefore what we will give marks for, is the students' trying to work out what is valued and what they will get marks for. This is not an entirely cynical process: students need to learn what is valued and how to recognise it for themselves if they are to be successful as professionals. Thus, it is obvious that they will be motivated to direct their study efforts in ways that yield the best results. To the extent that short term gains like this are correlated with long term benefits this is a good thing; it is only when the pursuit of short term gains leads to superficial learning that we should be concerned.
One way to direct the students attention directly onto what we value is to present them with the six dimensions, again with exemplars, and to show the students a range of different ways performance on the dimensions may be manifested within the context of the unit or course in question.
Limitations
Among the limitations that I could mention two are worthy of special note.
First, the dimensions were derived from consideration of the marking of written work. Artistic works or performances, clinical work and so on were not included. Whilst there is good reason to think the six dimensions specified will apply to these situations as well as to written work, it is possible that there may be another dimension (or two?) which could and should be added to make the set of dimensions more comprehensive.
Second, the study focussed on undergraduate work. Accordingly, whilst the dimensions may (arguably) be the same for postgraduate work, the relative weight given to each dimension may be different.
Conclusion
This is a paper designed to promote thought and discussion rather than to provide answers. It is, however, also designed to provide a good starting point and a direction which will help readers to find their own answers. It does this by providing a framework and by leaving the detail of application for readers to determine in their own context.
References
Gilliat, S. E.., & Hayward, N. F. (1996). A testing time: The role of subjective practices in making sense of student performance. Assessment and Evaluation in Higher Education., 21(2), 161-171.
Kelly, G. A. (1955). The psychology of personal constructs. (Vol. 1 and 2). New York: Norton.
Nulty, D. D. (1997). The constructs of examiners in higher education: What do academic standards mean? Doctoral Thesis. Queensland University of Technology.
Sadler, D. R. (1987). Specifying and promulgating achievement standards. Oxford Review of Higher Education, 13(2), 191-209.
Conference Entry Page About the Conference Abstracts Conference Program
TEDI Home About Assessment UQ Home
This site built on a Macintosh using Dreamweaver.. Last modified 20/1/99; 10:14:24 AM.