Reflecting on CALL Effectiveness: Conditions & Implications
by
Debra Hoven
Department of Asian Languages and Studies, University of Queensland
Abstract
This paper outlines the preliminary findings of an evaluation of learner use of software tools in a multimedia software package for teaching Indonesian listening comprehension and culture. This software package was introduced into the Introductory (1st year) programme to help address the need of total beginners for exposure to native speakers using daily language in ordinary ways, on common, everyday topics. The package is designed to introduce, develop, and reinforce appropriate learning strategies at the same time.
Data on learner perceptions of the ease of use, and appropriateness of the software interface are being investigated, in order to improve our understandings of the perceptions and reactions of novices compared to experienced users of multimedia software packages. This investigation includes an analysis of the differences in strategy use between high, medium and low proficiency learners, and their use of the help tools such as grammar reference notes, language laboratory simulation, replay facility, and mid-task answer checking.
Introduction
Many different kinds of CALL (Computer-Assisted Language Learning) software packages and approaches have been developed over the past 15-20 years but is has only really been in the past 10 years that researchers and teachers have begun to look at how effective these packages might be in the learning process. One of the most important issues that needs to be addressed is to what extent the stated aims and objectives of a CALL package meet the needs of the learners using it. Unfortunately, neither the methodology on which a given package is based nor the aims and objectives are often made very clear. The software package used as the basis for the evaluation reported in this presentation was designed by the author on the basis of sociocultural methodology, focussing particularly on embedded learning strategies and the incorporation of activities to cater for a range of learning styles and cognitive processes (see Figure 1 below).
The findings of investigations such as this are crucial to our understandings of how students learn using computers, particularly as computers play an increasing range of roles in our lives. With the proliferation of multimedia software both commercially available, and produced within institutions, we need to be able to assess and evaluate the effectiveness of this software as a language teaching and learning resource, better to determine the role or roles it should play in the learning environment. The software which forms the basis of the investigation has been thoroughly researched and purpose-designed, students are trained in how to use it, and the use of the software plays an integral role in the total learning environment for the learners.
As Chapelle comments:
[...] to begin to address the comparative questions about CALL, it is necessary to specify the other relevant features of CALL contexts, as well as to document and analyze the texts resulting from those contexts so elements of the texts believed to be significant for language learning can be identified.
(Chapelle, 1994: 43)
So what are the features of CALL contexts that can be specified? These typically include the learning environment in which the package is to be used, the curriculum (including objectives, assessment criteria, skills, competencies), the teaching and learning approach and methodology employed in the learning environment, the roles of the participants, the nature of interactions, the resources used, and the task-types (Hoven, 1992; 1997a).
CALL 'texts' here can include interactions, either between or among learners working on CALL tasks, between learners and computers, or between learners and their teacher, as well as the texts that form the content on which the CALL tasks are based (Hoven, 1992; Hubbard, 1992). In order, then, to make an effective evaluation of a CALL software package, the different aspects involved in the design of the package need to known and specified.
(Hoven, 1999: 95)
About the software
The software used in this project was initially designed and developed by the author as part of an earlier National Priority Reserve Fund project. Further refinements were made as a result of the author's PhD study on improving learners' control over their interactions with multimedia software, to produce a package that is capable of collecting data electronically, as well as providing a medium for listening comprehension development (Hoven, 1997b). The data analysed includes, among other things, the number of times the learner requests various features of the help system, the length of time spent on any task, how many attempts made on any task, and how often the learner requests answer checking during a task. Features of the help system include help on how the program works and how to navigate through it, complete Indonesian grammar reference notes searchable by topic, information on language learning strategies and where they are useful, and a language laboratory simulator. This simulator allows learners to record and compare their attempts at utterances from the listening and viewing texts, and to vary the rate and order of utterances. The package is designed for learners in their first 2 semesters of Introductory Level Indonesian as a second or foreign language.
This project has four main aims:
- to evaluate learners' use of the software tools available to them in the multimedia package, including consideration of learners' different learning styles and strategies;
- to evaluate learner perceptions of the usefulness, ease of use, and appropriateness of the software interface;
- to compare the perceptions of novice and experienced computer users on their use of the software package; and
- to compare high and low proficiency learners on their use of the help features and on their perceptions of the usefulness and effectiveness of the software package.
This project therefore represents a first step towards improving our understanding of how learners use multimedia packages for language learning, and the specific features of such packages that may prove more useful or effective for learners at different levels of proficiency. This has led to a close examination of the changes in learners' strategy awareness after using the package for one semester. Subsequent investigations can now be initiated to evaluate the difference that using multimedia packages can make to the speed, appropriateness, or effectiveness of learners' second language learning.
About the project
This project aims to begin a comprehensive evaluation of the use of multimedia software packages for the teaching of second or foreign language by firstly investigating learner perceptions and comparing high and low proficiency learners for their performances using a purpose-designed package. It represents the first stage of a longer-term project to evaluate more fully the impact of multimedia in the language learning process. Increasingly, multimedia language learning software packages are being developed and marketed, while very few attempts have been made to evaluate the effectiveness of such software in general, or of more specific features within the packages which may or may not contribute to improved learning (Brandl, 1995; Liou, 1995; Bradin, 1996). Many administrators, researchers and users of computers in language learning have asked questions about the effectiveness of computers in the language learning process (Pederson, 1987; Luff et al. 1990; Dunkel, 1991; Johnson, 1991; Chapelle, 1994).
Johnson, in fact came up with a 'usual benefits' list which includes:
motivation
sense of control over the learning process
individualisation (of pace, level, language assignment)
privacy of interaction
non-judgmental feedback
In addition the following should also be mentioned:
authentic materials can be incorporated as required
visual, audiovisual & text available simultaneously and on one delivery platform
and, for self-access modes
frees up class time for more face-to-face interaction.
However, as Papert, (1987) and Chapelle (1994) have pointed out, it is not so much the computer but the kinds of tasks and activities that learners do on the computer that can make the difference. As Jonassen also comments, in order for learners to make use of technologies to learn, the tasks which they complete must be designed to encourage the kind of thinking (mental processes) that allows learning to take place (Jonassen, 1992: 2).
It is therefore learner interactions with these tasks and how they use them that need to be investigated. Some researchers have investigated the effectiveness or usefulness of various feedback mechanisms (Burston, 1990; Brandl, 1995), while others have examined learners' awareness of the learning strategies they use during computer-based language learning tasks (Liou, 1995). However, there has not to date been a multimedia software package designed specifically to collect and analyse data on language learner interactions with the features of the package, particularly in relation to the interaction between learning strategies and multimedia characteristics.
Research Methods and Techniques
To achieve the aims of this project, a combination of data collection and analysis techniques has been used. As a result of a Research Infrastructure Grant which the Department of Asian Languages and Studies received jointly with the Centre for Language Teaching and Research, a small branch computer language laboratory has been established within the department. This laboratory area and computing equipment is where the students have been working with the software package. First year (Introductory Level) students come in and use the software on the computers on a self-access basis, as part of their regular out-of-class practice and revision. As students only need to access the software for one hour per week, on average, access has not been a problem
. Data for analysis which is stored electronically in the software include:
- Student name, Student Number and Subject Code
- which tasks students work on
- records of student oral practice
- the number of times the student requests various features of the help system
- how long the student spends on any task
- how many attempts the student makes on any task
- how often the student requests answer checking during a task
- how often the student requests different features of the help system, and
- which features of the help system the student requests
. In addition to this electronic collection of data, information on learner perceptions of the usefulness, ease of use, and effectiveness has been collected by means of two questionnaires ( one administered prior to the beginning of use (early Semester 1), and again at the end of the period of use (end of Semester 1). The first of these questionnaires included questions on learners' prior experience with computers, their preferred learning styles and strategies, and their impressions of their level of proficiency in the Indonesian language across the four skills. Together with information collected from formal classroom assessment during semester 1, this data determined whether learners were assigned to high, medium, or low proficiency categories, in order to address the third and fourth aims of this project: the difference in perception between novice and expert computer users, and the difference between high and low proficiency learners in their use of the various features of the help system. This information, in combination with data collected from the second questionnaire administered, has provided the basis for investigating the second aim of this project: to evaluate learner perceptions of the usefulness, ease of use, and appropriateness of the software interface.
The second questionnaire was administered at the end of Semester 1, after students had used the package for a semester. This questionnaire elicited information on learners' perceptions of the usefulness of the software in developing:
- certain specific skills of listening and viewing comprehension;
- the use of specific learning strategies incorporated into the tasks;
as well as
- which features of the help system they found most useful;
and
the same self-evaluation questions on learning styles and strategies that were used on the first questionnaire.
These questions used a Likert scale to elicit learners' frequency of use of a range of language learning strategies and their preferred learning styles for language learning. This questionnaire also included questions relating to the design of the software package and interface, such as:
- the appropriateness and arrangement of interface elements,
- intuitiveness of the shape and location of icons, and
- any improvements the learners could suggest in any of the above areas.
In summary, the methods used for data collection in this project included the a combination of pre- and post-course questionnaires using both Likert scales and open-ended comment questions, data collection in software of learner actions, and face-to-face discussions of the value or otherwise of the software:
I Pre-Course Questionnaire
Prior language experience
Computing skills/experience
Learning styles/strategies
II Post-Course Questionnaire
Changes in level of computing expertise/confidence
Learning styles/strategies (same questions/options as pre-course questionnaire
Perceptions of software Learner open-ended comments
III Focus groups
IV Software data collection
Data on learner use collected in software
Learners were classified according to whether they had reached Low, Medium or High proficiency. This was determined on the basis of 3 difference test instruments used during the 14-week semester: dictation tests, dialogue completion tests, and several oral roleplay tests. There were two dictation tests administered, one in the 5th and one in the 12th weeks of the semester to determine auditory discrimination and familiarity with Indonesian lexis presented using an oral/aural medium. In each of weeks 8 and 13, a written test was administered requiring learners to fill in missing parts of dialogues modelled on dialogues or sentence patterns used in the activities in the software. In week 14 of the semester, students participated in a variety of roleplay activities in varied pairs and in small groups. These roleplay activities also varied in the range of freedom or task prescription presented to learners. A sample of 3 learners representative of each category was chosen for the study of strategy changes on the pre- and post-questionnaires. For the High proficiency category, the learners with the three highest scores were chosen, for the Medium category, the three on or closest to the mean (within 3.5%), and for the Low proficiency category, the three learners who scored lowest.
Findings so far & discussion
In response to the open-ended questions inviting general comments on the their use of the software, students commented with some useful suggestions as well as mostly positive comments.
These open-ended questions are as follows:
Question 4.1 Please comment on what you think are the best features
of the software package.
Question 4.2 Please comment on what you think are the worst features
of the package.
Question 4.3 In what areas/aspects do you suggest improvements?
In the table below (Table 1) are listed all comments made by the students in the Low (LP), Medium (MP) and High proficiency (HP) categories. Comments made by student #6 (LP) in response to Q4.2, for example, suggest that more concentrated training in the use of the software needs to be provided at the beginning of subsequent semesters, particularly for those learners who evaluate themselves as novice computer users. On the other hand, student #12 (HP) provided some useful suggestions for modifying the software to make it more user-friendly. These modifications are currently being integrated into the package.
| Q# | Low Proficiency | Medium Proficiency | High Proficiency |
| 4.1 | 6 I think it's helpful for learning languages
14 Easy to use. Valuable learning tool (as it helped with sent-ence structure or word order) |
21 The audio function and the check answers
button. 22 The audio features. Listening exercises. |
12 Listening practice, the rearranging of the
sent-ences practice grammar well. 16 I like being able to hear Indonesian people speak the language. 19 tasks - well structured, relevant |
| 4.2 | 6 If a person doesn't know how to use the computer
it's not a good way to learn a language 11 I would need to compare it against some others for any mean-ingful comment. |
21 The long and unre-cognisable 'reconstruct-ing
sentence' activities. 22 Its length. Far too small, There isn't 1/2 hour per week for each topic. |
12 Where did I finish last week? Instructions
at top too small, feel I want to skip them and just do the page. 16 I found it hard to get a definition of a word when I wanted it. |
| 4.3 | 11 I thought the package was well set out and quite logical in design. It used a variety of screen features which were simple to process information. | 21 A dictionary would be most useful. | 12 Page numbers, simplified instructions, pictures
or video of Indo-nesian ads or interviews. 16 I would like to hear a question asked in Ind-nesian before doing the exercise. 19 more activities |
Table 1: Learner comments
When we look at the timing of learners' use of the software, some interesting, though predictable trends emerge (see Figure 2 below). For all learners, use of the software peaked before each of the assessment items. However, before the mid-semester test, the HP group started intensive practice use of the software earlier, peaking again at the time of the test, while at the end-of-semester test time, they began their revision earlier than the other groups, tailoring off towards the end. The MP and LP learners, on the other hand, seemed more prone to panic revision at the actual time of the tests, with less regular use overall.
This pattern is again duplicated in Figure 3 (below) showing the number of lessons worked by learners of different proficiency over the course of the semester.
Another aspect to the picture emerging of learners' use of the software is shown in Figure 4 below. 'Lessons left incomplete' refers to lesson screens that learners opened and tried, but left after checking the correct answer, without working through it and correcting their mistakes. From this graph we can see that by the end of the semester, mid-proficiency learners were working more for accuracy than either the high or low proficiency groups.
Changes in strategy use
Learners were asked to respond to the same questions on their use of learning strategies and their preferred learning styles on both the pre- and post-semester questionnaires. In general, the evidence from this study reinforces the findings of other studies in the area of good language learning strategies (Naiman et al. 1978; Green & Oxford 1995). The high proficiency learners are active learners, taking risks, Perhaps most importantly, these learners are also thinking about and planning their learning and as a result their capacity to continue to expand their learning has also increased. In addition, these learners are both looking at language and the learning process more holistically and at the same time adding strategies of language analysis (bottom-up strategies). Also in keeping with other contemporary language learning strategy studies, the learners in the mid-range for proficiency are just beginning to use strategies in clusters and to become clearer about the metalinguistic side of language learning. Findings for the learners in the low proficiency range indicate the beginnings of strategy awareness with a decrease in the number of strategies used, and an increase in the number of effective comprehension-, though not production-focused strategies (see Table 2 below).
| High Proficiency | Medium proficiency | Low proficiency |
| - increase in active approach to learning | - decrease in risk-taking | - decrease in number of strategies used overall |
| -increase in risk-taking | - decrease in active approach to whole language | - strategy types still spread across the range |
| - increase in the number and importance of metalinguistic strategies | - high use of memory/association strategies | - few risk-taking tendencies (especially in speaking) |
| - increase in interactional strategies, particularly guessing from context & using alternatives | - low frequency of metalinguistic strategies, BUT starting to form clusters | - little whole-language focus |
| - increase in cognitive processing strategies | - decrease in number of processing strategies (possibly because of increased awareness) | - some evidence of strategy clustering starting to emerge (perhaps with longer time period) |
| - increased use of pre-existing top-down strategies as well as addition of bottom-up strategies | - slight increase in number & importance of interactional strategies | - decrease in processing & metacognitive strategies (though still the most important) |
| - increase in affective (metacognitive) strategies | ||
| - decrease in cognitive processing strategies | ||
| - increase in comprehension-related strategies BUT decrease in production-related ones |
Table 2: Post-semester changes in language learning strategy use
Summary and Future Directions
Information and understandings emerging from this investigation can best be seen as a form of action research in the area of computer-assisted language learning environments. The process of progressive trialing, feedback, modification will allow modifications to be made to the software package, in keeping with learner perceptions. These modifications are essential, not only to enhance the learning of students using the package in class and in self-access situations, but also to strengthen plans to adapt this software package for the delivery, by means of the World Wide Web, of Indonesian language listening and viewing comprehension in the near future.
As mentioned earlier, this project represents the first step in a broader investigation of how learners interact with computers mediated by multimedia software: how they use the various tools and media available; the cognitive and other processes activated during this interaction; the kinds of learning strategies developed or inhibited in a computer-mediated language learning environment; the effectiveness of computer-based multimedia as a resource in language learning relative to media presented in other ways; and the differences in learning improvement that may emerge between high, medium, and low proficiency learners, and between novice and experienced users of computers. These larger questions are the material for a long term and wider-ranging project.
References
Bradin, C. 1996. How second language learners utilize input in CALL. In Borchardt, F. L., C. L. Bradin, E. M. T. Johnson & L. Rhodes (eds.) 1996. Proceedings of the Computer Assisted Language Instruction Consortium 1996 Annual Symposium: "Distance Learning". Durham, Nth. Carolina: Duke University.
Brandl, K. K. 1995. Strong and Weak Students' Preferences for Feedback Options and Responses. The Modern Language Journal 79: 2: 194-211.
Burston, J. L. 1990. Maximising Intelligent Use of Unintelligent Response-Handling Devices. ON-CALL 4: 4: 7-12.
Chapelle, C. A. 1994. CALL activities: Are they all the same? System 22: 1: 33-45.
Dunkel, P. 1991. The Effectiveness Research on Computer-Assisted Instruction and Computer-Assisted Language Learning. In Dunkel, P. (Ed.) Computer-Assisted Language Learning and Testing. New York: Newbury House, 5-36.
Green, J. M. & R. Oxford. 1995. A closer look at learning strategies, L2 proficiency, and gender. TESOL Quarterly 29: 2: 261-297.
Hubbard, P. 1992. A methodological framework for CALL courseware develpoment. In Pennington, M. C. & V. Stevens (eds.): 39-65.
Hoven, D. 1992. CALL in a Language Learning Environment. CLL Journal 3: 2: 19-27.
Hoven, D. 1997a. Instructional design for multimedia: Towards a learner-centered CELL (Computer Enhanced Language Learning) Model. In Murphy-Judy, K. A. (ed.) NEXUS - The convergence of language teaching and research using technology. Nth. Carolina: CALICO: 98 - 111.
Hoven, D. 1997b. Improving the management of flow of control in computer-assisted listening comprehension tasks for second and foreign language learners. Unpublished doctoral dissertation, University of Queensland, Brisbane, Australia.
Hoven, D. 1999. A model for listening and viewing comprehension in multimedia environments. Language Learning and Technology, 3: 1, July: 88-103.
Johnson, D. 1991. Second Language and Content Learning with Computers: Research in the Role of Social Factors. In Dunkel, P. (Ed.) Computer-Assisted Language Learning and Testing. New York: Newbury House, 61-84.
Jonassen, D. 1992. What are Cognitive Tools? In Kommers, P. A. M. et al (Eds.) Cognitive Tools for Learning. Berlin: Springer-Verlag, 1-6.
Liou, H-C. 1995. Evaluation of Interactive Videodisc Courseware: Effects of Strategy Training and Collaborative Learning. In Borchardt, F. L. & Johnson, E. M. T. (Eds.) Proceedings of the Computer assisted Language Instruction Consortium 1995 Annual Symposium "Computers and Collaborative Learning. Durham, Nth. Carolina: Heinle & Heinle Publishers, 112-115.
Luff, P., Gilbert, N. & Frohlich, D. (Eds.) 1990. Computers and Conversation. London: Academic Press.
Naiman, N., M. Frohlich, H. H. Stern & A. Todesco. 1978. The good language learner. Toronto: The Ontario Institute for Studies in Education, Research in Education Series/7.
Papert, S. 1987. Computer criticism vs. Technocentric thinking. Educational Researcher January-February: 22-27.
Pederson, K. M. 1987. Research on CALL. In Smith, W. F. (Ed.) Modern Media in Foreign Language Education: Theory and Implementation. Lincolnwood, Ill.: National Textbook Company, 99-131.