Rating Interpretation Guides (RIGs)
The Rating Interpretation Guides (RIGS) offer a normative approach for interpreting teaching and course evaluations results. They allow contextual interpretation of item means based on reference to previous results for similar disciplines, class size, and year level.
Rationale for the RIGs
Teaching and course evaluation results typically report the mean score and percent agreement for each item. This creates a problem of interpretation. It is a reasonable thing for a lecturer to wonder: "What does my score mean?" One would like to be able to compare one's scores with those of others, but what should they be compared against?
There are systematic relationships between ratings and class size, year level and discipline groupings. These relationships are well documented in research literature that shows that larger classes are rated lower than smaller classes, that later years rate higher than earlier years, and that different disciplines are rated differently with Humanities and Social Sciences generally being rated higher than Natural Sciences. These findings are consistent throughout the literature, albeit, the discipline-related findings are a little less consistent that those reporting for class size and year level.
To control for these trends in the evaluation data, subsets of previous evaluation results are used to produce the RIGs normative database.
Categories
The RIGs categories fall under three types;
-
Class size
- >0 – 50
- >51 – 100
- 101+
- Year level
- 1st year
- 2nd year
- 3rd year and above
- Disciplines (see Appendix 1)
- Natural sciences and maths
- Humanities and social sciences
- Science-based professions
- Social science-based professions
This results in a total 36 combinations of class size, year level and disciplinary grouping.
Interpreting your results
The distribution of the previous results for each of the 36 combinations of class size, year level, and disciplines is summarised using the 25th and 75th percentile for each item. The 25th and 75th percentiles give the average range for classes of a similar size, year level, and discipline. That is, the range which the middle 50% of previous results fell within. In this way a lecturer can situate their own mean for any question either within the average range, below the average range, or above the average range. They can answer the question "How am I doing compared with colleagues teaching similar classes?".
Normative database
The normative databases used to create the RIGs are updated as evaluation results become available. To date there have been four databases created for this purpose.
2009 RIGs Database
In 2009 TEVAL and iCEVAL reports will use the 2009 RIGs database which was based on results from 18,000 evaluations that were conducted between 2004 and 2008. In each category RIGs were only produced for those items which had results from more than 30 referenced evaluations.
2007 RIGs Database
In 2007 & 2008 TEVAL and iCEVAL reports used the 2007 RIGs database which was based on results from 16,000 evaluations that were conducted between 2002 and 2006. In each category RIGs were only produced for those items which had results from more than 30 referenced evaluations.
In 2006 TEVAL and iCEVAL reports used the 2006 RIGs database which was based on results from 19,000 evaluations that were conducted between 2000 and 2004. In each category RIGs were only produced for those items which had results from more than 30 referenced evaluations.
2005 RIGs Database
Prior to 2006 TEVAL and iCEVAL reports used the 2005 RIGs database. The 2005 RIGs database was based on results from 7,000 evaluations that were conducted in 1996 and 1997. Like the 2006 database, RIGs were only produced for those items within a category that had results from more than 30 of the referenced evaluations.
The full list of RIGS used for comparison in 2009 can be accessed here. (PDF 310KB). This document will also allow you to compare your previous TEVAL reports against the new 2009 database.
Cautionary tales: Caveats and notes on the use of the RIGS
The RIGS are designed to provide contextual data to aid the interpretation of evaluation results. They are designed to allow lecturers to interpret their mean ratings in terms of the scores of their colleagues, whilst controlling for the systematic variation associated with class size, year level and discipline. In this sense the adoption of the RIGs is commensurate with a policy of best practice in the provision of student ratings information to UQ academic staff. However, it is essential that the following caveats are heeded to prevent the misuse or misinterpretation of the RIGs. When comparing an individual mean with the middle 50% of previous results care should be taken in the following matters:
- You can not infer the median value from the 25th and 75th percentiles.
The median value is not necessarily the mathematical midpoint between
the 25th and 75th percentile values.
- You can not infer any evaluative proposition from the proximity of an
individual's score to either the 25th or 75th percentile. For instance,
you can not infer that being near to the 25th percentile is necessarily
a poor result relative to the distribution. The scores are not necessarily
evenly distributed within the 50% band. Similarly, you cannot say whether
it is meaningful that an individual score is near the upper end of the
band. This is especially pertinent when comparing two individuals' scores.
- The RIGs tell you whether your score is in within the average range,
above the average range, or below the average range. The correct interpretive
approach is simply to focus on the question whether or not your score is
within or above the average range. Use this information about your scores
to decide whether to change any of your teaching practices. You should
feel confident if you are within the average range because you are in good
company with at least half of your colleagues. Of course, congratulate
yourself if your score is above the average range.
- Remember that evaluation results are only one of several evidential components
of any teaching portfolio and should not be interpreted as the be-all-and-end-all
of your teaching profile.
- The RIGs are simply one interpretation tool, based on a normative approach. We recommend using the RIGs in conjunction with a criterion-based referencing approach. The RIGs are based on the mean which is simply one measure of central tendency. An alternative indicator of the distribution of responses is percent agreement, the proportion of students that either Agreed or Strongly agreed with an item. We recommend 70% agreement as a minimum standard.
Appendix 1: Disciplines falling into each RIGs discipline grouping
- Natural sciences and maths
- Humanities and social sciences
- Science-based professions
- Social science-based professions
| AG Agriculture |
| AN Anatomical Sciences |
| BC Biochemistry |
| BL Science (Biological Sciences) |
| BT Botany |
| CH Chemistry |
| ET Entomology |
| GM Earth Sciences |
| IC Psychology (Cognitive Science) |
| MA Mathematics (Applied Mathematics) |
| ME Mathematics (Engineering Mathematics) |
| MN Mathematics (Computational Mathematics) |
| MP Mathematics (Pure Mathematics) |
| MS Mathematics (Statistics) |
| MT Mathematics |
| PA Parasitology |
| PH Physics |
| PK Physiology & Pharmacology (Pharmacology) |
| PL Physiology & Pharmacology (Physiology) |
| SC Science (Non-Biological Sciences interdisciplinary subjects) |
| ZL Zoology |
| AS Anthropology & Sociology |
| AY Anthropology & Sociology (Anthropology) |
| AY/SO Anthropology & Sociology |
| CL Classics & Ancient History (Classics) |
| CN Classics & Ancient History (Ancient) |
| DM English (Drama) |
| EL English Linguistics) |
| EN English (Literature & Communication & Culture Studies) |
| FH/FR Romance Languages (French) |
| GO Government (International Relations & Asian Politics) |
| GR German & Russian Studies (German) |
| GT Government |
| HA Art History |
| HS History |
| HT History |
| JA Asian Languages & Studies (Japanese) |
| JB Asian Languages & Studies (Jap/Chin/Indonesian/Korean Interdisc.) |
| JC Asian Languages & Studies (Chinese) |
| JF Asian Languages & Studies (Korean) |
| JN Asian Languages & Studies (Indoneasian) |
| MU Music |
| PD Philosophy |
| PY Psychology |
| RE Studies in Religion |
| RM Romance Languages |
| RU Russian Language Studies |
| SH Romances Languages (Spanish) |
| SO Anthropology & sociology (Sociology) |
| AT Architecture |
| CJ Paediatrics & Child Health |
| CS Computer Science & Electrical Engineering |
| DN Dentistry |
| E Engineering |
| EM (Biothechnology) |
| GMC Medicine |
| HM Human Movement |
| IH Indigenous Health |
| MD Medicine |
| MY Microbiology |
| NU Nutrition Program |
| OT Occupational Therapy |
| PB Pathology |
| PC Pharmacy |
| PT Physiotherapy |
| PX Psychiatry |
| QA Animal Production |
| QF Food Science & Technology |
| QM Natural & Rural Systems Management |
| QP Plant Production |
| SL Speech Pathology & Aduiology |
| SM Social & Preventive Medicine |
| ST Speech Pathology & Aduiology |
| SU Surgery |
| TH Tropical Health Program |
| VE Veterinary Science |
| VJ Veterinary Science |
| VP Veterinary Science |
| BF Management (Graduate School subjects) |
| CO Commerce |
| EC Economics |
| ED Education (Graduate School subjects) |
| EU Education (Graduate School subjects) |
| GN Geographical Sciences & Planning |
| JR Journalism |
| LA Law |
| LG Centre for Language Teaching & Research (Language Teaching) |
| QB Hospitality, Tourism & Property Management |
| SW Social Work & Social Policy (School subjects) |
