Review principles
Evaluation
Types of evaluation
Evaluation
instruments
Foci of evaluation
Trialling
Monitoring
Evaluation
Evaluation is a vital and integral aspect of all sound course design and development. In a sense, it is the bottom line. Valid evaluation will tell you how well something has worked - whether or not the course has accomplished what you intended it to. Educationally, it is important to try to gauge the effectiveness of your practice.
Evaluation is a means of being accountable to your students, your profession and your employing institution. It has benefits for both individual professional growth and for the improvement of courses. To find out more about the evaluation services TEDI offers go to the Evaluation section of our website.
Benefits
Participating in a culture of evaluation is an illuminating process. Designed and implemented validly, teaching evaluation will inform you about the value of what you do. Everybody wins from well designed and well executed evaluation - the students, the academics, the wider community which employs graduates and the field of education generally.
The outcomes of evaluation can feed into research and publication
opportunities.
Types of evaluation
Evaluation can be classified in different ways - formative or summative; quantitative or qualitative. All these categories of evaluation have a legitimate place in the educational process.
Quantitative evaluation makes use of (more or less) objective measures such as the TEVAL form which you'll be familiar with. The advantage of such measures is that they are easily quantifiable and can yield numbers. The data can be analysed quite quickly.
Qualitative evaluation, as the term suggests, provides data which consists of something other than numbers and may legitimately include opinion. This data results from open questions where respondents are free to comment in their own words. Though somewhat more difficult to analyse, such data often provides rich sources of information to the evaluator.
A combination of both quantitative and qualitative analysis is often desirable, depending on the particular focus of the evaluation.
Formative evaluation
Depending on the purpose and focus of evaluation, it can be carried out at different times. Evaluation which takes place during the teaching of a course is known as formative evaluation. Formative evaluation can be conducted at any point during the course - at the conclusion of a module or set of modules, mid-way through the course, or at any other point at which you consider the gathering of evaluative data to be necessary. The benefit of formative evaluation is that academics are able to respond promptly to emerging difficulties rather than allowing them to continue and perhaps cause ongoing problems for students for the remainder of the course.
Summative evaluation
Summative evaluation occurs at the completion of a course. Summative evaluation has the advantage of providing a broad view of the whole course and of informing future developments.
Both formative and summative evaluation yield useful data.
It is possible
for an instance of evaluation to be both formative and summative at
the same time.
For
example,
if students
are asked to evaluate
Module 1 of a course, then this evaluation
can be seen as summative (for that particular module)
but
formative
(in
terms of the
overall course).
Evaluation instruments
Evaluation can be carried out using a variety of instrumentation. A common instrument is the questionnaire, for example, the TEVAL form. However, several other options are also available, including focus groups and interviews.
Focus groups and interviews
Both these methods are qualitative and open-ended, and can be used as a confirmation of written responses by the course cohort. These approaches permit more far ranging responses than written surveys. While interviews are one-to-one, focus groups are larger, containing a sample of students from the whole cohort.
In relation to both interviews and focus groups, the evaluator/interviewer will have some prepared questions to put to the students. The answers to these questions can operate as stimuli for further discussion and clarification.
The group discussion
within focus groups, which results from the pre-set questions,
can be
quite illuminating
and even
surprising in raising
issues which the evaluator had not
anticipated.
Foci of evaluation
It is not necessary to evaluate all aspects of a course every time an evaluation occurs. At different times and for different reasons, you may wish to gather information about a combination of the following:
- achievement of learning objectives
- student satisfaction
- student achievement
- student response to an innovation.
Trialling
It is sound practice to trial all new (or adapted) courses prior to implementation. Despite your best efforts (and those of your reference group and educational designer) during the course development process, it is not unusual to find one or two glitches when the course is actually implemented. Trialling provides the opportunity to identify and remedy at least some of these problems.
What to trial
Trialling can occur in different ways. If constrained for time, you might decide that, instead of trialling the whole course, you might trial only certain parts of it - perhaps those sections which you anticipate might be particularly challenging or which you plan to deliver in different ways.
When to trial
Ideally, trialling should occur before the course is implemented. This allows time to remedy any emerging flaws. However, if (for some unforeseen reason) this is not possible, the first-time users then, in fact, become the trial group. Special care needs to be taken in these circumstances and good communication mechanisms put in place so that students are able to provide prompt feedback, especially in relation to any difficulties which may be occurring.
Who to use
Small groups or larger
ones can be used. The
trialling group
should replicate the
target group as closely
as possible. If a group
with
different characteristics
is used, confounding
effects may occur, some of which
may relate
to
the presence
of prior
knowledge.
For
instance, if trialling
a first-year
course, it is not appropriate
to use second-year
students whose prior knowledge
would inform
their responses
to learning
activities
and
quite possibly disguise
inherent problems which
might occur
with novices.
Monitoring
Essentially, monitoring means checking that the learning process is progressing successfully and that the objectives of the subject which you have designed are being met. It is important to monitor the delivery of all subjects which are delivered in flexible mode. This is especially vital the first time round. Monitoring is not the equivalent of formative evaluation but the two may be related in that formative evaluation can provide valuable information which contributes to the monitoring process.
Administration
Be faithful to the feedback and communication mechanisms which you designed into the course. For instance, if a course requires students to submit assignments via email, you need to build in time to deal with this aspect of your workload.
