Studies of Effectiveness of Learning Networks

Starr Roxanne Hiltz, (Hiltz@adm.njit.edu).

Yi Zhang (yxz1847@njit.edu), and

Murray Turoff (Turoff@njit.edu)

Department of Information Systems,

College of Computing Sciences,

New Jersey Institute of Technology,

Newark NJ 07102

Abstract

The WebCenter for Learning Networks Effectiveness Research includes entries for empirical studies of Learning Networks published in refereed journals and conference proceedings. Nineteen of these studies were identified which both measure learning effectiveness for students, and compare ALN to traditional face-to-face courses on the same campus. These studies employ objective measures of student learning (e.g., grades) about as frequently as subjective measures (survey responses by students). The evidence is overwhelming that ALN tends to be as effective or more effective than traditional modes of course delivery, at the university level.

Key Words: ALN, Learning Effectiveness Measures, E-Learning, CSCL

1. Creating a knowledge base of ALN research results

The first "alpha" version of the WebCenter appeared about the end of January, 2010; and a "beta" version is coming up in October 2010. We have completed the coding of studies and creation of a database for approximately 50 empirical research articles related to the effectiveness of learning networks. In this paper, we will summarize those studies that explicitly compare ALN effectiveness to that of traditional courses.

Learning networks are defined as groups of people who use computer networks (the Internet and World Wide Web) to communicate and collaborate in order to build and share knowledge. The emphasis for studies in the database of empirical research is on asynchronous (anytime, anyplace) use of networks, but the project includes studies of courses that emphasize use of synchronous (same time) technology or which compare face-to-face, synchronous and asynchronous learning processes. Secondly, the emphasis will be on post-secondary, for-credit courses, but information will also be collected about studies of the use of ALN in pre-college courses and in continuing professional education (not for academic credit) courses or learning communities. Effectiveness is defined in this project to focus on both learning outcomes for students, and positive or negative impacts on faculty. To the extent that other measures of effectiveness are reported in empirical studies (e.g., fiscal impacts on educational institutions, cost-benefit analysis, or societal level impacts in terms of educational access and equity), they will be included in a separate database of "other papers" to be created in the future.

1.1 . Criteria for Inclusion in the Published Research List of Papers

1. Papers included in the ALNResearch database must be empirical studies of the effectiveness of learning networks and have been published in a refereed journal or conference proceedings, in the English language. They must be full papers, not just extended abstracts. This is operationally defined as at least five pages in length.

2. "Learning networks" technology and pedagogy refers to the use of computer-mediated communication among students as well as between instructor and students, for a substantial part of the course work. They may be used asynchronously and/or synchronously, though we are mainly concerned with courses that include substantial use of asynchronous (anytime) media. They may be used alone or in combination with other media, such as face-to-face lectures, videotapes, Web postings of lecture or reading or tutorial material, etc. Not all web based courses use learning networks; some just post lecture type materials or exams for downloading, do not involve extensive interaction among students in a class, and therefore do not qualify as "learning networks" courses. One synonym for learning networks is "computer-supported cooperative learning" (CSCL).

3. "Effectiveness" is defined as primarily concerned with learning outcomes for students, but also includes effectiveness from the instructor's point of view. It thus includes studies that look at student perceptions, student performance, or faculty perceptions, satisfaction, or performance in this mode of course delivery.

4. To be considered as an empirical research study, the paper must include research questions or hypotheses (at least implicitly), describe some data collection methods, and report some empirical results. In order to be considered an adequate empirical study, it must have a reasonable number of subjects on which conclusions are based. We have operationally defined this for the time being as a minimum of at least 20 subjects included in the study.

1.2. Coding of the studies for the Database

Most of the coding of studies is done by Ph.D. students working under the direction of the project director, who checks them over and ascertains that the study "qualifies" according to our criteria. One objective is to make this WebCenter the "first stop" for the literature review of ALN researchers who are planning a new study or article. This should save researchers time and assure them a more complete overview of related prior research than they are liable to obtain on their own.

For this analysis, we decided to focus on analysis of a key subset of the papers: those that compare the effectiveness of ALN courses in terms of student outcomes, to that of traditional "face to face" courses (Appendix 3). We identified 19 of the studies that clearly meet this criterion. We have included only the most important study characteristics in the charts and in the analysis presented here: the research methods, the way effectiveness was measured, and the results. As for the remaining studies in the database, they tend to be case studies of ALN rather than comparative studies measuring comparative effectiveness, or to be concerned primarily with variables that are correlated with good outcomes in ALN, or to be focussed on different outcomes, such as faculty satisfaction. We will probably be adding some more studies to the 19 included here, in the future.

1.3. Procedure: Coding of the Studies for this Paper

Two of the authors categorized the types of individual measures used, and whether each individual finding reported showed ALN to be better than traditional courses, no different, or worse. The second step was that all of the results for each study were categorized in terms of whether they, in total, showed ALN to better, worse, or no different on the whole.

2. Research Models, Methods and Measures for Assessing the Effectiveness of ALNs

Asynchronous Learning Networks may be considered to be one type of information system: a computer-based system designed to support the work of teachers and learners. There are two dominant research models in Information Systems and in other fields using social science methodologies to study human subjects: the "positivists" and the "interpretivists." The positivists strive to follow the model of scientific inquiry developed in the natural sciences, with the objective being to specify quantitative measures of all variables, state hypotheses, collect data using random sampling and other procedures that will enable the testing of hypotheses using inferential as well as descriptive statistics, and then analyze the data, report the results and their limitations. The "interpretivists" strive for an "in depth" understanding of the processes that are occurring in human social systems; they generally start with research questions, use qualitative methods such as participant observation, unstructured or semi-structured interviews, and content analysis to obtain a rich description of the phenomenon and arrive at an interpretation of why and how things work-- or do not work (Ngwenyama and Lee, 1997). The most sophisticated research projects combine both quantitative measures (describing quantitatively "what" is happening") and qualitative measures (describing "why" the results are occurring, in terms of the details of behavior and interaction that transfer an "input" of course materials provided in various modes to an "output" of what the student does or does not learn.)

To qualify as an empirical study using generally accepted methods, a research project should start with specific hypotheses and/or research questions, which guide the selection of methods and measures. Unfortunately, the majority of publications related to ALN's do not have any explicit research questions, let alone specific hypotheses. They tend to be accounts by instructors of courses they designed and of their experiences and impressions, whose value in building a scientific body of knowledge is questionable.

The paradigm of positivist research has enshrined experimental design as the most valid method for determining "cause and effect," specifically, the "pre-test, post-test, control group design" using random assignment of individual subjects to conditions. It is basically impossible to randomly assign students to take a traditional section or an ALN section of a course; they may be unable to travel to campus if they live 2009 miles away, or unable to take an ALN section if they have no PC and Internet provider. One is therefore left with "quasi-experimental" designs at best, in most cases, in which student’s self-select mode of course delivery, but the study designer and instructors try to "hold constant" everything else, such as the syllabus, assignments and exams. The "pre-test, post-test" design means that ideally one measures the dependent variable ( such as knowledge about accounting or data bases or English Literature) before the course, then measures again after the course, to determine "amount learned" as the difference between scores. Most ALN studies do not do this either, since it is not usual to give students the equivalent of a final exam on the first day of the course, and in many project-based courses, it is not appropriate, since there will be no "final exam" either. (One exception is the Worrell et. al study of a graduate accounting course [#], for which standardized professional exams are readily available to test "knowledge" of accounting). Thus, we often do not know whether differences in grades at the end of the course are caused by differences in mode of course delivery, or differences among the students who self-selected the different modes. Positivists would thus tend to say that most ALN research to date is not very rigorous.

There are several different types of measures of effectiveness of ALN's for students that are commonly used. Objective measures of performance and subjective assessments by students have been used about equally (See Table 1). Objective measures include the following; the number of studies shown in Appendix 3 using each of these measures is noted in parentheses:

Grades, for specific projects or exams or for the entire course, compared to sections or students using other delivery modes (16).

Measures of the quality of work (e.g., group projects may be judged on creativity, completeness, length, etc.) (9). Such judgements may be biased if they are made by the instructor who designed the online course, and who knows who did a particular piece of work and the mode that student was in. Thus, procedures need to be designed to make the quality of work measure as reliable as possible by using multiple judges, who are "blind" to the identity and course delivery condition of the student (e.g., see Fall, 2009).

· Course completion rates (3 studies)

· Counts or measures of activity levels or patterns (5)

"Subjective" measures are frequently used in the current body of ALN research on effectiveness, though they are not usually considered as valid as objective measures. These include student self-assessments (through questionnaires or interviews) of course learning outcomes (absolute or compared to traditional courses; 8 of the studies included in this paper use such a measure); the effectiveness of the mode or system used for delivery (including convenience, motivation, usability, time required, access to professor; 18 studies); or of the quality of the instruction or materials (3 studies). Note that the total number of studies using each type of measure adds up to more than 100% of the studies, since many used more than one measure of effectiveness. Generally, multiple measures enhance the reliability of the conclusions about effectiveness.

Table 1: Summary of Measures and Results for All Nineteen Studies

Measures	Positive for ALN	No Difference	Negative for ALN
Objective Measures Course grade	2	6
Final exam grade	2	1
Midterm/quiz grades	2	3
Quality of work rated by instructor	3	2
Assignment Measures Length of Report	1
Rated by Judges Forgetting		1
Content Quality	1
Completeness	1
Amount of Collaboration		1	1
Amount of Activity / Participation	2
Female participation	1
Course completion	1		2
Use of instructor who does not prepare materials	1		1
Subjective measures via students Learning more		5
Skill development
Quality of work		2	1
Quality of course materials		1
Quality of discussion			2
Motivation/Interest	3	1	1
Progress to degree	2
Access to degree	2
Access to instructor	1
Access to educational resources	1
Usability of technology	2	1	2
Participation			1
Social Presence			1
Totals	28	24	12

3. Results: ALN vs. Traditional Face-to-Face Course Delivery

A. Summary Tables of Results

Most of the studies either measure effectiveness in more than one way (e.g., grade distributions plus subjective student assessments) and/or study different courses, resulting in many "mixed results." Looking at the results, we have classified them as falling into one of two categories, those generally show ALN to have better outcomes than traditional courses (Table 2), and those that tend to show no difference, overall (Table 3).

Table 2: Summary for eight studies with largely positive ALN findings

Measures	Positive for ALN	No Difference	Negative for ALN
Objective Measures Course grade	2	2
Final exam grade	2
Midterm/quiz grades	2
Quality of work rated by instructor	3
Assignment Measures Length of Report	1
Rated by Judges Forgetting		1
Content Quality	1
Completeness	1
Amount of Collaboration
Amount of Activity / Participation	1
Female participation
Course completion	1
Use of instructor who does not prepare materials
Subjective measures via students Learning more	6
Skill development	1
Quality of work			1
Quality of course materials
Quality of Discussion
Motivation/Interest	3
Progress to degree	2
Access to degree	2
Access to instructor	1
Access to educational resources
Usability of technology
Participation
Social Presence

For these eight studies in Table 2, the preponderance of the evidence is that ALN is more effective than traditional courses.. All of the results indicate ALN to be better, or else some results show ALN as better and others show "no difference." The studies that are judged to be in this category include those by Alavi; Andriole; Benbunan-Fich et.al. (2010), Hiltz; Hsu et.al.; Hiltz & Wellman; Thoennenssen et. al; and Turoff and Hiltz.

For the remaining 12 studies, the preponderance of the evidence shows "no significant difference" between ALN and the traditional courses or sections or experiences that are compared. Either all the results show "no significant difference," or there are mixed results with results better for ALN for some courses or measures, and worse on others (e.g., for the SCALE projects at Illinois [Arvan, 1998], there are some conflicting results related to course and/or experience of the instructor.).

There are no qualifying empirical studies for which ALN is clearly shown to be less effective than traditional modes of course delivery, on the whole.

The first important point to note is that the "No Difference" cases really indicate that ALN is just as effective as face to face and when this is added to the positive results for ALN there is a four to one ratio of positive results to negative results in these 19 studies. Furthermore, if we realize there are only four instances of objective measures that are negative and none of them are direct measures of learning, we find the results rather overwhelming support the hypothesis that ALN is a meaningful alternative to the classical face to face class, which tends to be as effective or more effective, depending on the circumstances of the particular implementation and the measure used.

In terms of some of the negative results, the course completion or drop out rate is probably higher than it should be because of mistaken expectations by students, either because they are new to the use of ALN or because they do not have the student network to warn them away from particular offerings where the course might not live up to the description. In fact, many studies of ALN show that the role of the instructor and his or her ability to deal with this new mode of learning is a principal factor in ALN success. Teaching ability is important, as well as experience in the mode of delivery. A lot of face-to-face courses may be taught by less than perfect instructors but the face to face environment can tolerate a wider range of instructor abilities. We must evolve a mechanism to specify in evaluation work the competence of instructors or it will be quite clear that the results can easily be confounded.

Most universities have a standardized survey for measuring student reactions to an instructor at the end of the course. It is only this year that the union at NJIT agreed to a slightly modified version of a teaching evaluation course to be put online for all ALN courses, so all on line courses will receive the same survey and those teaching online sections can be directly compared with respect to their face to face sections and with others teaching the given course. It is this sort of data that should be assessed longitudinally to determine how the performance of the instructor evolves over time, and which instructors are the important ones to carefully debrief in order to determine relative success factors in the teaching of a given course.

Table 3: Summary for eleven studies with largely mixed or No Difference" ALN findings

Measures	Positive for ALN	No Difference	Negative for ALN
Objective Measures Course grade		4
Final exam grade		1
Midterm/quiz grades		3
Quality of work rated by Instructor		2
Assignment Measures Length of Report
Rated by Judges Forgetting
Content Quality
Completeness
Amount of Collaboration		1	1
Amount of Activity / Participation	1
Female participation	1
Course completion			2
Use of instructor who does not prepare materials	1		1
Subjective measures via students Learning more		5
Skill development
Quality of work		2
Quality of course materials		1
Quality of discussion			2
Motivation/Interest		1	1
Progress to degree
Access to degree
Access to instructor
Access to educational resources	1
Usability of technology	2	1	2
Participation			1
Social Presence			1

Some of the variables that are related to the degree of effectiveness of a particular course implementation are suggested by the various correlations with learning effectiveness reported in the studies included in this analysis. These are shown in Table 4. There are not enough replications testing specific relationships to reach any firm conclusions on these relationships at this point, but they do suggest hypotheses to be tested in future studies.

_____________________________________________________________________________

Table 4: Correlations or Interactions Reported for the positive studies.

· Perceived collaboration correlates with perceived learning

· Activity by individuals correlates with Perceived learning

· Perceived collaboration correlates with motivation

· Higher objective Grades in Computer Science correlates with the use of ALN

· SAT Scores and/or GPA are the dominant correlation with course grades (independent of ALN or face-to-face)

· Initial subjective expectations correlates with later actual participation

· Course type and mode of delivery interacts.

· Degree of collaboration correlates with satisfaction with course.

Correlations or Interactions Reports (unique to one study) for the Mixed or No Difference Studies

· Use of quizzes (for feedback only) improves learning perceptions by instructors and students

· Collaboration falters when there is not sufficient critical mass.

· Laboratory use and collaboration are correlated in a Chemistry course.

· Collaborative groups online have higher satisfaction than individuals online.

· Group Communications with Learning Game led to higher learning levels than with either alone.

· No benefit found for use of telephone communications

· Dense Hyper-links interfere with ease of learning

· More efficient management of distance students possible

· Larger courses in distance possible.

Technology problems correlate with dissatisfaction by students.

____________________________________________________________________________

With respect to student feedback it is not surprising to find from a number of studies that face to face students often think that the quality of their work or the quality of discussion is better face to face, when in fact the opposite may be true when expert judges are used. This is similar to when an employee changes from using a manual approach to using a computer approach to handle their tasks. The way one is used to doing work is usually felt to be a better way than having to learn a new one. It often takes a long interval of time for the employee or anyone else to get used to a new way of solving problem or learning and to begin to actually realize they are doing well.

It is also time to capitalize on the records that exist for students to begin to conduct some longitudinal analyses on students. We need to relate a student's subjective opinions about an ALN course to how many such courses they have taken. We must view their grades in relation to how they have preformed in the prerequisite course and their overall grade average, in order to help to control for possible effects of self-selection of mode of delivery. In this manner we can begin to get a handle on the true long-term impacts of ALN. One would suspect that a longitudinal analysis will begin to show that those who actually desire to use the ALN alternative (as compared to those who are basically "forced" to take an ALN section because there are no available alternatives) do better than those in comparable face to face classes

IV. Conclusions and Discussion

The evidence is overwhelming that ALN tends to be as effective or more effective than traditional modes of course delivery, at the university level. There really is no need for more studies to explore this question. What we need is more research that will enable us to make it even more effective, especially as new technologies proliferate. For example, most early ALN's were "text-only" discussion; many now include multi-media web based tutorials instead or in addition to a textbook, and possibly audio and/or video interaction with the instructor. How important is multi-media in student teacher and student-student interaction, and how may it best be used? With very small and pervasive devices coming into use (e.g., wireless and pocket PCs), how can these devices best be used to further improve convenience of access to ALN's? What forms of collaborative learning vs. individual assignments are most appropriate for different kinds of courses or class sizes? As Alavi and Leidner(2010) point out in their call for "greater depth and breadth of research" on technology-mediated learning, research in this area must look at how technology influences learning, which involves an "explicit consideration of relationships among technology capabilities, instructional strategy, psychological processes, and contextual factors involved in learning" (p. 1). We need to develop both more sophisticated and more comprehensive theoretical frameworks, and also more valid methods and instruments that those which have characterized a majority of studies to date.

In conclusion, it is time to start looking at the evaluation of ALN beyond the single course and consider the full range of evaluation variables. Consider the following top down structure of evaluation variables that might form an overall model for the study of ALN effectiveness and impacts:

Evaluation Measures

Student Performance Measures

· Amount of learning

· Time to Degree

· Satisfaction

· Motivation

· Enjoyment

· Participation

Resource Measures

· Effort to Learn

· Time to learn

· Time to educate

· Cost of resources

· Effort to access learning

· Cost of learning resources

· Availability of course as needed

· Convenience of courses

Opportunity Measures

Do new things not possible before

· Accessibility for new populations

· Reduce or eliminate distinction between distance and regular students

· Make direct use of human resources not normally available in a course

· Introduce new programs

Do things differently than before (organizational impacts)

· Change the nature of courses

· Change the nature of degree programs

· Change the nature of educational institutions

· Change the nature of departments

· Change the nature of teaching

· Change the evaluation criteria of faculty

Long Term Impacts

· Impact on individual (job, earnings, advanced study. etc.)

· Reputation of Program

· Reputation of Institution

· Accreditation of Program

Significant Intervening Variables

· Type of Students (objectives, part time, learning abilities, gender, age, experience, and other demographics)

· Type of Course (Laboratories, skill, subject area, etc.)

· Teaching Methodologies employed (e.g., is discussion graded, are individual or group assignments and activities used)

· Size of course

· Information on courses available to potential students

· Technology employed (media mix, usability, functionality)

From the above and the preliminary correlations based upon courses as well as our understanding of higher education, it is time to begin formulating testable models that involve many of the key variables, and to begin to use meaningful historical data to verify or improve our models.

For all the talk of university research, as with most commercial organizations, universities are reluctant to expose the critical historical data that they have, for fear of negative publicity. The few times that this has succeeded in studying industry it has tended to be the result of an independent and non competitive organization established that would guarantee the anonymity of detailed data supplied to the organization when it came to the publication of the resulting analysis reports. The incentive for the organizations participating was to be able to compare their performance with others and in so doing, have a better idea of how to make decisions to improve their situation.

It might be that the formulation of such an independent depository of historical information is one of the next important steps in furthering the evaluation of ALN. While we have done a reasonable job, as an academic community, in proving the viability of ALN, our form of distance learning is probably still a minority in terms of the methods whereby distance learning is being offered. ALN researchers tend to believe that those approaches that do not include extensive communication and interaction among students and faculty are inferior to ALN approaches as well as face to face education, but this message has not "gotten through" to a lot of distance course designers. We need to raise our evaluation and research dissemination efforts to a new plateau and use the opportunity presented to provide a deeper understanding of higher education effectiveness than exists to date.

Acknowledgements

This work is partially supported by a grant from the Alfred P. Sloan Foundation, by NJIT, the New Jersey Center for Multimedia Research, and the New Jersey Center for Pervasive Information Systems. David Spencer and Jayalamathy Sadagopan contributed many of the article reviews for the database. Razvan Bot has primary responsibility for the database of papers and other aspects of the web site construction and maintenance.

References

1. Alavi, Maryam (1994), Computer-Mediated Collaborative Learning: An Empirical Evaluation, MIS Quarterly, June, pp.159-174.

2. Alavi, M. and Leidner, D.E. (1991). Research commentary: Technology-mediated learning-- A call for grater depth and breadth of research. Information Systems Research, 12, 1 (March), 1-10.

3. Alavi, Maryam, Youngjin Yoo and Douglas R. Vogel (1997), Using Information Technology to Add Value to Management Education, Academy of Management Journal, Vol 40, No 6, pp.1310-1333.

4. Andriole, Stephen J. (1997), Requirement-Driven ALN Course Design, Development, Delivery & Evaluation, JALN, Vol 1, No 2 - August, 10p.

5. Arbaugh, J.B. (2009), Virtual classroom versus physical classroom: An exploratory study of class discussion patterns and student learning in an asynchronous internet-based MBA course, Journal of Management Education, Vol.24 Issue 2 Apr., pp.213-233.

6. Arvan, Lanny, John Ory, Cheryl D. Bullock, Kristine K. Burnaska and Matthew Hanson (1998), The SCALE Efficiency Projects, JALN, Vol 2, No 2 - September, 27p.

7. Benbunan-Fich, Raquel and Starr Roxanne Hiltz (1999), Impacts of Asynchronous Learning Networks on Individual and Group Problem Solving: A Field Experiment, Group Decision and Negotiation, Vol.8, pp.409-426.

8. Benbunan-Fich, Raquel, Starr Roxanne Hiltz and Murray Turoff (2010), A comparative content analysis of face-to-face vs. ALN mediated teamwork. Proceedings of the 34^th Hawaii International Conference on System Science, IEEE Computer Society.

9. Fallah, M.H., W. J. How and R. Ubell (2009), Blind Scores in a Graduate Test: Conventional Compared with Web-based Outcomes. ALN Magazine, Vol. 4, Issue2 – December.

10. Hadidi, R and C. Sung (2009), Pedagogy of Online Instruction – Can it be as Good as Face-to-face? Proceedings of the American Conference on Information Systems, Long Beach, California, Aug 2009, pp. 2061-2065.

11. Hiltz, S. R. (1993), Correlates of learning in a virtual classroom, Int’l Journal Man-Machine Studies, Vol.39, pp.71-98.

12. Hiltz,S.R. The Virtual Classroom: Learning Without Limits Via Computer Networks. Norwood, NJ, Ablex Publishing (Human-Computer Interaction Series), 1994.

13. Hiltz, S.R., Benbunan-Fich, R., Coppola, N., Rotter, N., and Turoff, M. Measuring the Importance of Collaborative Learning for the Effectiveness of ALN: A Multi-Measure, Multi-Method Approach. JALN 4, 2 , 2009.

14. Hsu, Enrico Y. P., Starr Roxanne Hiltz and Murray Turoff (1992), Computer-Mediated Conferencing System as Applied to a Business Curriculum: A Research Update, Int’l Business School Computer Users Group, 20th Annual North American Conference, pp.214-227.

15. O. Ngwenyama and A. Lee, Communication Richness in Electronic Mail: Critical Social Theory and the Contextuality of Meaning. MISQ June 1997. Available at

16. Ocker, Rosalie J. and Gayle J. Yaverbaum (1999), Asynchronous Computer-Mediated Communication versus Face-toFace Collaboration: Results on Student Learning, Quality and Satisfaction, Group Decision and Negotiation, Vol.8, pp.427-440.

17. Sandercock, G.R.H. and G. Shaw (1999), Learners’ Performance and Evaluation of Attitudes Towards Web Course Tools in the Delivery of an Applied Sports Science Module, ALN Magazine Vol.3 Issue 2 Dec., 9p.

18. Sener, John and Mary Liana Stover (2009), Integrating ALN into an Independent Study Distance Education Program: NVCC Case Studies, JALN, Vol. 4 Issue2.

19. Thoennessen, M., E. Kashy, Y. Tsai and N.E. Davis (1999), Impact of Asynchronous Learning Networks in Large Lecture Classes, Group Decision and Negotiation, Vol.8, pp.371-384.

20. Turoff, Murray and Starr Roxanne Hiltz (2009), Effectively Managing Large Enrollment Courses: A Case Study. In Bourne, J. and Moore, J.C., eds, Online Education: Volume 2. Learning Effectiveness, Faculty Satisfaction, and Cost Effectiveness, pp. 55-80. Needham, MA, Sloan Center for Online Education.

21. Wade, Vincent P. and Conor Power (1998), Evaluating the Design and Delivery of WWW Based Educational Environments and Courseware, ACM, ITICSE Proceedings, July, pp.243-248.

22. Worrell, Malcolm, Starr Roxanne Hiltz, Murray Turoff and Jerry Fjermestad (1995), An Experiment in Collaborative Learning using a Game and a Computer Mediated Conference in Accounting Courses, Proceedings, Hawaii Int’l Conference on System Sciences, 28th Annual, pp.63-71.