Interpreting SETC Reports
Reports provided for standard surveys such as SETC present information in ways that allow users to identify strengths and areas in need of development. The statistical information contained in these reports is often clarified by reading the student comments on the survey forms which are returned after student results have been formally posted.
Interpretation
We recommend two approaches to understanding the results of your SETC reports.
- Criterion-referenced approach: Percent Agreement
The percent agreement is the proportion of respondents who either agreed or strongly agreed with the item. Reports indicate when agreement levels are high (above 70%) or low (below 30%). We recommend a criterion of 70% agreement.
- Normative approach: Rating Interpretation Guides (RIGs)
The RIGs compare your results for each item against a set of norms and lets you know if your results fall in the average range, below the average range, or above the average range. Your results will be compared against lecturers in similar disciplines, with similar class sizes and course levels. These are referred to as Ratings Interpretation Guides (RIGs). The RIGs are only available for items which have an adequate amount of comparison data available.
Rating Interpretation Guides (RIGs)
Teaching and course evaluation results typically report the mean score and percent agreement for each item. This creates a problem of interpretation. It is a reasonable thing for a lecturer to wonder: "What does my score mean?" One would like to be able to compare one's scores with those of others, but what should they be compared against?
There are systematic relationships between ratings and class size, year level and discipline groupings. These relationships are well documented in research literature that shows that larger classes are rated lower than smaller classes, that later years rate higher than earlier years, and that different disciplines are rated differently with Humanities and Social Sciences generally being rated higher than Natural Sciences. These findings are consistent throughout the literature, albeit, the discipline-related findings are a little less consistent than those reporting for class size and year level.
To control for these trends in the evaluation data, subsets of previous evaluation results are used to produce the RIGs normative database.
Categories
The RIGs categories fall under three types:
- Class size
- >0 – 50
- >51 – 100
- 101+
- Year level
- 1st year
- 2nd year
- 3rd year and above
- Disciplines: see Appendix 1 - Disciplines falling into each RIGs discipline grouping (PDF; 50KB)
- Natural sciences and maths
- Humanities and social sciences
- Science-based professions
- Social science-based professions
This results in a total 36 combinations of class size, year level and disciplinary grouping.
Interpreting your results
The distribution of the previous results for each of the 36 combinations of class size, year level, and disciplines is summarised using the 25th and 75th percentile for each item. The 25th and 75th percentiles give the average range for classes of a similar size, year level, and discipline. That is, the range which the middle 50% of previous results fell within. In this way a lecturer can situate their own mean for any question either within the average range, below the average range, or above the average range. They can answer the question "How am I doing compared with colleagues teaching similar classes?"
Normative databases
The normative databases used to create the RIGs are updated as evaluation results become available.
2010 RIGs Database
Currently there are no available normative databases available for the SET-C or SET-Tutor. The databases will be generated as a sufficient volume of comparative data becomes available. Until this is possible RIGs information will not be available on reports.
2009 RIGs Database
In 2009 TEVAL and iCEVAL reports will use the 2009 RIGs database which was based on results from 18,000 evaluations that were conducted between 2004 and 2008. In each category RIGs were only produced for those items which had results from more than 30 referenced evaluations. To date there have been four databases created for this purpose.
2007 RIGs Database
In 2007 & 2008 TEVAL and iCEVAL reports used the 2007 RIGs database which was based on results from 16,000 evaluations that were conducted between 2002 and 2006. In each category RIGs were only produced for those items which had results from more than 30 referenced evaluations.
2006 RIGs Database
In 2006 TEVAL and iCEVAL reports used the 2006 RIGs database which was based on results from 19,000 evaluations that were conducted between 2000 and 2004. In each category RIGs were only produced for those items which had results from more than 30 referenced evaluations.
2005 RIGs Database
Prior to 2006 TEVAL and iCEVAL reports used the 2005 RIGs database. The 2005 RIGs database was based on results from 7,000 evaluations that were conducted in 1996 and 1997. Like the 2006 database, RIGs were only produced for those items within a category that had results from more than 30 of the referenced evaluations.
The full list of RIGS used for comparison in 2009 can be accessed here (PDF; 310KB). This document will also allow you to compare your previous TEVAL reports against the new 2009 database.
Cautionary tales: Caveats and notes on the use of the RIGS
The RIGS are designed to provide contextual data to aid the interpretation of evaluation results. They are designed to allow lecturers to interpret their mean ratings in terms of the scores of their colleagues, whilst controlling for the systematic variation associated with class size, year level and discipline. In this sense the adoption of the RIGs is commensurate with a policy of best practice in the provision of student ratings information to UQ academic staff. However, it is essential that the following caveats are heeded to prevent the misuse or misinterpretation of the RIGs. When comparing an individual mean with the middle 50% of previous results care should be taken in the following matters:
- You can not infer the median value from the 25th and 75th percentiles. The median value is not necessarily the mathematical midpoint between the 25th and 75th percentile values.
- You can not infer any evaluative proposition from the proximity of an individual's score to either the 25th or 75th percentile. For instance, you can not infer that being near to the 25th percentile is necessarily a poor result relative to the distribution. The scores are not necessarily evenly distributed within the 50% band. Similarly, you cannot say whether it is meaningful that an individual score is near the upper end of the band. This is especially pertinent when comparing two individuals' scores.
- The RIGs tell you whether your score is in within the average range, above the average range, or below the average range. The correct interpretive approach is simply to focus on the question whether or not your score is within or above the average range. Use this information about your scores to decide whether to change any of your teaching practices. You should feel confident if you are within the average range because you are in good company with at least half of your colleagues. Of course, congratulate yourself if your score is above the average range.
- Remember that evaluation results are only one of several evidential components of any teaching portfolio and should not be interpreted as the be-all-and-end-all of your teaching profile.
- The RIGs are simply one interpretation tool, based on a normative approach. We recommend using the RIGs in conjunction with a criterion-based referencing approach. The RIGs are based on the mean which is simply one measure of central tendency. An alternative indicator of the distribution of responses is percent agreement, the proportion of students that either Agreed or Strongly agreed with an item. We recommend 70% agreement as a minimum standard.
