Effective Flexible Delivery: Some Lessons Learnt
by
Peter Green,
and
James Lamb,
Department of Commerce, Univeristy of Queensland, Ipsiwch
ABSTRACT
In 1999, the Bachelor of Electronic Commerce degree started at the Ipswich campus of the University of Queensland with an initial intake of approximately 50 students. Subjects were offered to students using technology and flexible delivery methods. This paper details the authors' experiences in building and presenting a capstone subject in this new degree, Introduction to computer-based Information Systems, using flexible delivery techniques. This paper discusses the question, "What is flexible delivery?" Then, it proceeds to reflect on how the concept was operationalised using a combination of email, web-based technologies, and face-to-face contact classes. The proposition is put forward that flexible delivery does not mean contactless delivery, nor does it mean structureless delivery. Finally, the results of a preliminary survey of the students involved on the relative effectiveness of the presentation methods employed are presented. These results, in conjunction with data from the subject TEVAL, provide preliminary support for several of the propositions in the paper.
INTRODUCTION
Flexible Delivery (FD) and its use for the production and delivery of subjects and whole courses at Universities is a topical area currently. Not least at the forefront of such discussions are the questions, "What is Flexible Delivery? What does it mean?" Indeed, such discussions can go on at length. Real progress in these discussions can be made however when academics who have "grasped the nettle" and operationalised their concept of FD put their experiences and findings up for examination and further discussion.
Accordingly, the aim of this paper is to present our experiences and findings in operationalising and presenting subjects in FD mode for the Bachelor of Electronic Commerce degree. This degree began in February, 1999, at the new Ipswich campus of the University of Queensland with an initial student intake of 47. In particular, we will focus on the presentation of one subject during the first semester, 1999, in that degree - CO261 Introduction to Computer-based Information Systems. This subject provides an introduction to knowledge about information systems and basic "hands-on" skills required in dealing with information. This subject assumes no prior knowledge of computers and it is a compulsory subject for students doing the Electronic Commerce degree.
We were motivated to do the work reported in this paper for a number of reasons. First, we wanted to get feedback from the students on what they thought of the subject and how it was presented. As this initial offering of the subject was our first attempt at operationalising FD, we of course wanted some basis on which we could improve the subject and the operationalisation of FD. Second, we wanted to progress the discussions on FD by presenting some initial empirical data from one operationalisation of the concept. Certainly this operationalisation has many shortcomings and there are limitations in the research methodology used. However, we now have some concrete basis on which we can improve the operationalisation of our concept of FD. Finally, drawing our experiences, findings, and thoughts together into a paper and presenting it to a scholarly forum provides us with an opportunity to obtain a richer set of suggestions for improvements.
This paper progresses in the following manner. First, the concept of Flexible Delivery (FD) is pondered and various dimensions of flexibility are presented. Next, we present our concept of FD and how it was operationalised. This discussion will involve a brief comparison of our operationalisation against the dimensions of flexibility presented earlier. Third, measures of the effectiveness of the subject presentation were gathered from three sources: a small survey instrument completed by the students, the standard teaching evaluation form (TEVAL) for the subject completed by the students, and a focus group discussion conducted by the Department of Management (Ipswich) staff involving a small subgroup of students who did the subject. The results of these measures are presented and discussed. Finally, the paper is concluded by summarising the important implications from the data and adding to this knowledge with some anecdotal lessons learnt from the second presentation of the subject thusfar in semester two, 1999.
WHAT IS FLEXIBLE DELIVERY (FD)?
In tertiary institutions in Australia today, academics are being encouraged by University management to embrace flexible delivery presentation methods in the design and delivery of their subjects. University managers are encouraging such moves because, in their view in the long run, they will lead to higher quality courses being presented at a lower overall cost. Moreover, many beneficial by-products will derive from the process. For example, skilled researchers and educators who participate in the construction of FD subjects will perhaps formalise and document as part of the materials prepared intellectual property and techniques about the area that has taken these specialists years to acquire and refine. Such a process will provide the benefits of preparing and maintaining a quality course but also providing some assurance of the quality level irrespective of the personnel who may be involved in presenting the material from time to time into the future
However, in being encouraged to move towards FD methods of subject preparation and presentation, there remains much confusion over what constitutes flexible delivery. Distance education through the use of comprehensive written materials, short intensive residential schools, and, more recently, computer-based education (CBE/CBT) modules has been in place and operational for many years (e.g., Birchall & Bennett 1986, Hawes 1987). Over the intervening time, much research has been done on the evaluation of such educational techniques and many improvements have been made to our understanding of the effectiveness of such methods (e.g., Wagner 1993, Knott 1993).
More recently, open learning initiatives involving such techniques as audiographics conferencing and presentation of materials via broadcasts using the Australian Broadcasting Commission (ABC) television channel have received increasing popularity and evaluation (e.g., Lawson 1993, Cooper 1995, Atkinson et al. 1995). The current popular conception of flexible delivery appears to be "putting everything from individual courses (subjects) to entire degree programs in cyberspace" (Grossman 1999 p. 25). By contrast, our concept of flexible delivery lies along the continuum between traditional lecture/tutorial in-class methods and distance/open learning methods. More specifically, our conceptualisation of flexible delivery involves a mix of learning methods that combines the benefits of distance/open learning - flexibility in access, in time, and in place (Atkinson et al. 1995) - with the benefits of traditional methods - regular opportunity for face-to-face contact and discussion/resolution of problems. In this way, it was similar to the approach to flexible learning examined by Baron (1995) - the 'learners' were to be made more responsible for their own learning and have more control over it. In addition, the learning materials provided would be supplemented by other resources including 'facilitators' who were to be regarded by the 'learners' as another resource rather than an imparter of knowledge (teacher).
To conclude this discussion, Table 1 provides the dimensions of flexibility (of learning) suggested by Brown (1999). This table presents a description of the various dimensions that can be used to comprehensively define Flexible Delivery (FD).
Of course, there is no implication in Table 1 that "more flexible" is better in all cases. Rather, Table 1 provides a comprehensive framework by which subject developers can review the various dimensions of flexibility and then position their development at the point on each dimension appropriate to their circumstances. These circumstances will be unique to each subject development and they will be determined by such factors as the characteristics of the developer, the content domain, and the audience to whom the subject will be presented.
THE SUBJECT IN QUESTION
CO261 - Introduction to Computer-based Information Systems - is a first-level introduction to hardware, software, data communications, networks, the development and different types of information systems in business, and the Internet. By way of "hands-on" component, the subject introduced students to the use of email, browsing and searching the Internet, and the use of spreadsheets (Excel) and databases (Access) in business.
A subject development team developed the subject in second semester, 1998, for presentation in first semester, 1999, at the new Ipswich campus of the University of Queensland. The team consisted of two content experts, an instructional designer, and support staff from the Learning Resources and Development Unit (LRDU) of the University of Queensland.
The team developed a study guide that comprehensively covered the topic areas of the subject. Each section of the study guide led the student through a series of readings, preliminary activities, written activities, practical activities, and Internet exercises. The study guide referred students to reading and exercise material contained in three textbooks and various online resources. The online resources consisted of a general website (the Companion website) and a WebCT set of materials to support one of the textbooks. Both these online resources contained such materials as lecture notes and quizzes for each chapter of the textbook, self-assessment tests, and real-world short cases. Also, a CD-ROM prepared by CBT Systems covering the various packages of Microsoft's Office 97 was referred to in the Study Guide particularly when the spreadsheet and database topics were being covered. This CD-ROM was part of the text package obtained by students.
The study guide and WebCT textbook-based materials (purchased from the textbook suppliers) were loaded onto the subject site (CO261) on the WebCT WWW server (flex) at the Ipswich campus. (WebCT is name of the course technology product used by the University of Queensland to present materials to students over the World Wide Web. It has many additional features including facilities to accumulate and record students' marks on various assessment items, bulletin board, email, homepage for each student, chat rooms, and secured access to materials through userid and password protection.) Because this semester represented the first offering of the subject in this new FD format and there were some concerns with regard to the readiness of the facilities at Ipswich, students were also provided with a copy of the study guide in printed form.
To assist students in structuring their progress through the materials during the semester, a weekly two-hour seminar/tutorial/laboratory class was offered to the students. Numbers in these face-to-face classes were limited to 15-17 students. To support these classes, a series of weekly task sheets was devised. Each task sheet suggested to students what section of the reading they should cover this week, what were the important topics to be covered, various of the activities from the study guide that might attempted prior to, and in preparation for, the next class, and activities that were going to be performed during the class session. The weekly face-to-face classes were supported by a series of five (5) two-hour lectures presented throughout the semester. These lectures were presented at critical points throughout the semester. At each of these points, they were able to introduce, and provide a summary overview, of major topics that were going to reviewed by the students over the following weeks. Moreover, the first lecture introduced students to the objectives, the structure, and the assessment of the subject. The last lecture summarised the work throughout the semester and it gave students information regarding the structure and format of the final examination. Finally, staff were available in their offices at set times each week for student consultation. (Interestingly, staff found that virtually no use was made of this consultation facility by students throughout the semester given the other contact opportunities (both face-to-face and electronic) available to them.)
Intensive communication was maintained with the students outside of face-to-face contact opportunities using general student email and the bulletin board facilities within WebCT
The assessment of the subject was planned to attempt to gauge students' progress in acquiring the skills and knowledge presented throughout the semester. Moreover, incentive was included in the assessment structure for students to work progressively throughout the semester rather than leaving everything to the end of semester. Accordingly, Table 2 summarises the assessment structure used in the subject.
A novel approach was trialled with the assignments. Students were given the specification of the assignment tasks and asked to present their attempts at these tasks to their seminar leader/facilitator by their class time in the due week. Students could submit their attempts either on paper or by email. (Interestingly, as printing was provided "free" to students in computer laboratories at Ipswich in semester one, virtually every student chose to submit their assignment attempt in hardcopy format.)
At the normal lecture time in the due week also, students were asked to sit a test based on the material and skills covered in the assignment. The test was paper-based and it consisted of a series of multiple-choice and short-answer questions. If students submitted a complete assignment (i.e., they submitted a conscientious attempt at each required task), then the mark they achieved on the test was their percentage out of 20 for the assignment. However, for each assignment task not conscientiously attempted, the student's mark on the test was discounted proportionately.
The progressive class preparation and participation was assessed by the individual seminar leader/facilitator. Students could obtain one percent per session, up to a maximum of ten percent. There was a maximum of thirteen (13) opportunities throughout the semester for students to obtain their ten percent.
The final examination was a two-hour centrally timetabled written paper. It consisted of 60 multiple-choice questions and four short-answer questions. There was no examination of the practical materials that had been assessed throughout the semester. The material in the final examination consisted of the "theory" covered by the study guide, textbook, and various online resource sources, and which was reviewed during the seminar classes throughout the semester.
In an attempt at self-appraisal, Table 3 provides an assessment of this particular operationalisation of FD against the dimensions of flexibility introduced earlier. For each dimension, we use the ratings less flexible, flexible, and more flexible to indicate our assessment.
ASSESSMENT OF THE EFFECTIVENESS OF THE OPERATIONALISATION OF FD IN THIS SUBJECT
Measures of the effectiveness of the FD methods used in this subject were obtained from three sources:
- A short, specifically designed survey administered to students.
- The standard subject teaching evaluation (TEVAL) survey form.
- A focus group interview conducted using some of the students who did the subject.
Source 1
A survey instrument was developed to gather, inter alia, demographic information on students doing the Bachelor of Electronic Commerce and their perceptions of the effectiveness of the various FD component methods employed in the subject. (A copy of the instrument is provided in the Appendix to this paper.) Students were asked to assess different methods of delivery/learning under two criteria essentially. The first criterion was whether the method "had a large, positive impact on my effectiveness in successfully completing this subject". The second criterion was whether the method was "an important and valuable aid to me in the performance of my study in this subject". These two measures of effectiveness were derived from a validated survey instrument developed and used by Goodhue and Thompson (1995) in the Information Systems research literature. These two researchers were principally concerned with measuring the effectiveness of computer-based systems in helping users in organisations accomplish tasks.
The component FD methods assessed were:
- E-mail; * On-line study guide;
- On-line bulletin board;
- Prentice-Hall cis.edu WebCT materials; and the
- Prentice-Hall companion web site.
- Face-to-face weekly seminars.
The students assessed the impact of these methods on a five-point scale, where:
1 = strongly disagree;
2 = disagree;
3 = neutral;
4 = agree; and
5 = strongly agree.
Table 4 presents the average scores (and their standard deviations) for each of the effectiveness measures for each of the FD methods surveyed. The methods are categorised into two groups - electronic/online and face-to-face.
The individual scores show clearly that in the students' perceptions each method was effective in helping them complete the subject successfully. However, for the companion website and bulletin board methods, there was enough variation to indicate uncertainty on those responses. Clearly, students were in strongest agreement on the effectiveness of the face-to-face classes.
Two sets of comparisons were performed. First, the average of the responses for the electronic/online methods was compared with that for the face-to-face seminars. Despite there being only 33 useable responses, students exhibited a stronger belief that the face-to-face method had a large, positive impact on their effectiveness in completing the subject, compared to the electronic/online methods. The average score for electronic/online methods on this criterion was 3.7, compared to 4.5 for face-to-face seminars
. In addition, students showed a stronger belief that face-to-face seminars were an important and valuable aid in the performance of their study, compared to electronic/online methods. The average score for electronic/online methods on this criterion was 3.8, compared to 4.5 for face-to-face seminars. These results are statistically significant at the 0.001 level.
Second, each individual electronic/online method was compared with the face-to-face method on both effectiveness criteria. The same results were obtained. For example, the average score for e-mail having a large, positive impact on students successfully completing the subject was 3.9, which was a significant difference away from the score of 4.5 for face-to-face seminars.
Overall Result - Students have a stronger belief in face-to-face seminars than electronic/online FD methods.
Source 2
Measures 2 and 3 did not look at individual components of the FD operationalisation. Rather, they simply attempted to obtain a measure of the students' perceptions of the effectiveness of the combination of FD components used.
Measure 2 consisted of targeted questions that were administered to students through the standard teaching feedback (TEVAL) form for the subject at the end of the semester. Table 5 summarises these items. The items are measured on a six-point scale.
0 = No answer or N/A
1 = Strongly Disagree
2 = Disagree
3 = Uncertain
4 = Agree
5 = Strongly Agree
Source 3
As part of their evaluative procedures in semester one, the Department of Management (Ipswich) conducted a series of focus group sessions with their students to obtain their perceptions of the subjects that the students did during the semester. A number of the students had participated in CO261 as an elective subject. The focus group methodology provided a nice contrast and qualitative alternative to the quantitative effectiveness measures employed on Sources 1 and 2. A summary of the results of the focus group discussions regarding CO261 is extracted below.
CO261 Intro to Computer-Based Information Systems: (four students enrolled in this subject) Positives: one of the best presented subjects overall; enjoyable; step-by-step - basic; variety (tutes, lectures, Internet access, hard copies). Even someone who had been at uni for years said its one of the best subjects because of the way the lecturer presents it.
WHAT HAVE WE LEARNED?
This paper has explored the question, "What is Flexible Delivery?". It has provided a review of various learning systems ranging from distance learning, to open learning, and finally arriving at flexible delivery. It has provided a range of dimensions of flexibility that could prove useful in determining what mix of components might be appropriate for the design and implementation of a particular subject in so-called flexible delivery mode.
The paper went on to describe the design and implementation of one such subject in the new Bachelor of Electronic Commerce program - CO261 - Introduction to Computer-based Information Systems. It showed how the design and delivery of the subject mapped into the various dimensions of flexibility. "More flexible" on every dimension is not necessarily the goal. Rather, a mix of flexibilities across the components would appear to be more appropriate. However, such a mix decision is driven by the characteristics of the developer/facilitator, the material being presented, and the audience to which it is being presented.
Using various quantitative and qualitative sources, data on the effectiveness of this operationalisation of the flexible delivery/learning concept was gathered and presented.
From the evidence obtained then, it would appear that the mix of components used in the design and presentation of CO261 was appropriate for the type of material and the characteristics of the audience to which it was presented. Moreover, it would appear that there is reasonable support for our conceptualisation and operationalisation of the FD concept. Furthermore, it is apparent from the data analysis and feedback received that structure and face-to-face contact remain important components of any successful FD mix. This last result is reinforced by the results of previous studies that have looked at the comparison of learning strategies between traditional and open learning systems (e.g., Koymen 1992) and more recently, research into the efficacy of different types of distance-learning technology (Institute of Higher Education Policy report, quoted in Grossman 1999).
Anecdotally, from presenting this subject in flexible delivery mode in second semester 1999 to a larger number of students (nearly 200) across three campuses of the University of Queensland (Gatton, Ipswich, and St Lucia), an additional insight has been gained. In this subsequent presentation of the material, a larger number of facilitators (seven) has been required. The range in quality of the facilitation skills of the facilitators appears to have reinforced the finding from the first semester of the importance of the face-to-face sessions in the effectiveness of the flexible delivery mix used.
REFERENCES
Atkinson, E., Conboy, I., Dodds, A., McInnis, C., & Atkinson, A. 1995 "Evaluation of the Open Learning Initiative: Interim Report March 1995", Centre for the Study of Higher Education, University of Melbourne, Parkville.
Baron, J. 1995 "A Personal Reflection of an Evaluation of a Flexible Learning System" Masters thesis, University of South Australia, Adelaide.
Birchall, D.W. and Bennett, R. 1986 "Through the Open Door - Today's Revolution in Open Access and Distance Learning: Distance Learning in Management Education: A Case Example", Journal of European Industrial Training, 10(6), 38-47.
Brown, A. 1999 "Dimensions of Flexibility", presented at the University of Queensland (Ipswich) Program Director's Retreat, 2 July 1999.
Cooper, M.J. 1995 "Guidelines to Producers of Text-based Open Learning Materials", Training and Management Development Methods, 9(1), 7.01-7.11.
Goodhue, D.L. and Thompson, R.L. 1995 "Task-Technology Fit and Individual Performance", MIS Quarterly, 19(2), 213-236.
Grossman, W. 1999 "Cyber View: On-Line U", Scientific American, July, p. 25.
Hawes, M. 1987 "Distance Learning - Getting Started", Journal of European Industrial Training, 11(3), 21-25.
Knott, T.D. 1993 "Evaluating Distance Learning in Public Education", Journal od Instruction Delivery Systems, 7(3), 31-35.
Koymen, U.S. 1992 "Comparison of learning and study strategies of traditional and open-learning system students in Turkey", Distance Education, 13(1), 108-117
Lawson, C. 1993 "Managing Change by Promoting Open Learning", In Access through Open Learning with a Major Focus on Managing the Change to Open Learning, ed. A. Ellis and B. Hansen, University of New England:Ballina, 112-120.
Wagner, E.D. 1993 "Evaluating Distance learning Projects: An Approach for Cross-Project Comparisons", Proceedings of 15th Annual Meeting of the Association for Educational Communications and Technology, New Orleans, Jan. 13-17.