Studies of Effectiveness of Learning Networks
Copyright, 2010
Starr
Roxanne Hiltz, (Hiltz@adm.njit.edu).
Yi
Zhang (yxz1847@njit.edu), and
Murray
Turoff (Turoff@njit.edu)
Department
of Information Systems,
College
of Computing Sciences,
New
Jersey Institute of Technology,
Newark
NJ 07102
Abstract
The WebCenter for
Learning Networks Effectiveness Research includes entries for empirical studies
of Learning Networks published in refereed journals and conference proceedings.
Nineteen of these studies were identified which both measure learning
effectiveness for students, and compare ALN to traditional face-to-face courses
on the same campus. These studies
employ objective measures of student learning (e.g., grades) about as
frequently as subjective measures (survey responses by students). The evidence
is overwhelming that ALN tends to be as effective or more effective than
traditional modes of course delivery, at the university level.
Key Words: ALN, Learning
Effectiveness Measures, E-Learning, CSCL
Learning networks are defined as groups of people who use computer networks (the Internet and World Wide Web) to communicate and collaborate in order to build and share knowledge. The emphasis for studies in the database of empirical research is on asynchronous (anytime, anyplace) use of networks, but the project includes studies of courses that emphasize use of synchronous (same time) technology or which compare face-to-face, synchronous and asynchronous learning processes. Secondly, the emphasis will be on post-secondary, for-credit courses, but information will also be collected about studies of the use of ALN in pre-college courses and in continuing professional education (not for academic credit) courses or learning communities. Effectiveness is defined in this project to focus on both learning outcomes for students, and positive or negative impacts on faculty. To the extent that other measures of effectiveness are reported in empirical studies (e.g., fiscal impacts on educational institutions, cost-benefit analysis, or societal level impacts in terms of educational access and equity), they will be included in a separate database of "other papers" to be created in the future.
1. Papers included in the ALNResearch database
must be empirical studies of the effectiveness of learning networks and have
been published in a refereed journal or conference proceedings, in the English
language. They must be full papers, not
just extended abstracts. This is operationally defined as at least five pages
in length.
2. "Learning
networks" technology and pedagogy refers to the use of computer-mediated
communication among students as well as between instructor and students, for a
substantial part of the course work.
They may be used asynchronously and/or synchronously, though we are
mainly concerned with courses that include substantial use of asynchronous
(anytime) media. They may be used alone
or in combination with other media, such as face-to-face lectures, videotapes,
Web postings of lecture or reading or tutorial material, etc. Not all web based courses use learning
networks; some just post lecture type materials or exams for downloading, do not
involve extensive interaction among students in a class, and therefore do not
qualify as "learning networks" courses. One synonym for learning
networks is "computer-supported cooperative learning" (CSCL).
3. "Effectiveness"
is defined as primarily concerned with learning outcomes for students, but also
includes effectiveness from the instructor's point of view. It thus includes studies that look at
student perceptions, student performance, or faculty perceptions, satisfaction,
or performance in this mode of course delivery.
4. To
be considered as an empirical research study, the paper must include research
questions or hypotheses (at least implicitly), describe some data collection
methods, and report some empirical results.
In order to be considered an adequate empirical study, it must have a reasonable
number of subjects on which conclusions are based. We have operationally
defined this for the time being as a minimum of at least 20 subjects included
in the study.
Most of the coding of
studies is done by Ph.D. students working under the direction of the project
director, who checks them over and ascertains that the study
"qualifies" according to our criteria. One objective is to make this
WebCenter the "first stop" for the literature review of ALN researchers
who are planning a new study or article.
This should save researchers time and assure them a more complete
overview of related prior research than they are liable to obtain on their own.
For this analysis, we
decided to focus on analysis of a key subset of the papers: those that compare
the effectiveness of ALN courses in terms of student outcomes, to that of
traditional "face to face" courses (Appendix 3). We identified 19 of the studies that clearly
meet this criterion. We have included
only the most important study characteristics in the charts and in the analysis
presented here: the research methods, the way effectiveness was measured, and
the results. As for the remaining studies in the database, they tend to be case
studies of ALN rather than comparative studies measuring comparative
effectiveness, or to be concerned primarily with variables that are correlated
with good outcomes in ALN, or to be focussed on different outcomes, such as
faculty satisfaction. We will probably
be adding some more studies to the 19 included here, in the future.
Two of the authors categorized the types of individual measures used, and whether each individual finding reported showed ALN to be better than traditional courses, no different, or worse. The second step was that all of the results for each study were categorized in terms of whether they, in total, showed ALN to better, worse, or no different on the whole.
2. Research
Models, Methods and Measures for Assessing the Effectiveness of ALNs
Asynchronous Learning Networks may be considered to be one type of information system: a computer-based system designed to support the work of teachers and learners. There are two dominant research models in Information Systems and in other fields using social science methodologies to study human subjects: the "positivists" and the "interpretivists." The positivists strive to follow the model of scientific inquiry developed in the natural sciences, with the objective being to specify quantitative measures of all variables, state hypotheses, collect data using random sampling and other procedures that will enable the testing of hypotheses using inferential as well as descriptive statistics, and then analyze the data, report the results and their limitations. The "interpretivists" strive for an "in depth" understanding of the processes that are occurring in human social systems; they generally start with research questions, use qualitative methods such as participant observation, unstructured or semi-structured interviews, and content analysis to obtain a rich description of the phenomenon and arrive at an interpretation of why and how things work-- or do not work (Ngwenyama and Lee, 1997). The most sophisticated research projects combine both quantitative measures (describing quantitatively "what" is happening") and qualitative measures (describing "why" the results are occurring, in terms of the details of behavior and interaction that transfer an "input" of course materials provided in various modes to an "output" of what the student does or does not learn.)
To qualify as an empirical study using generally accepted methods, a research project should start with specific hypotheses and/or research questions, which guide the selection of methods and measures. Unfortunately, the majority of publications related to ALN's do not have any explicit research questions, let alone specific hypotheses. They tend to be accounts by instructors of courses they designed and of their experiences and impressions, whose value in building a scientific body of knowledge is questionable.
The paradigm of positivist research has enshrined experimental design as the most valid method for determining "cause and effect," specifically, the "pre-test, post-test, control group design" using random assignment of individual subjects to conditions. It is basically impossible to randomly assign students to take a traditional section or an ALN section of a course; they may be unable to travel to campus if they live 2009 miles away, or unable to take an ALN section if they have no PC and Internet provider. One is therefore left with "quasi-experimental" designs at best, in most cases, in which student’s self-select mode of course delivery, but the study designer and instructors try to "hold constant" everything else, such as the syllabus, assignments and exams. The "pre-test, post-test" design means that ideally one measures the dependent variable ( such as knowledge about accounting or data bases or English Literature) before the course, then measures again after the course, to determine "amount learned" as the difference between scores. Most ALN studies do not do this either, since it is not usual to give students the equivalent of a final exam on the first day of the course, and in many project-based courses, it is not appropriate, since there will be no "final exam" either. (One exception is the Worrell et. al study of a graduate accounting course [#], for which standardized professional exams are readily available to test "knowledge" of accounting). Thus, we often do not know whether differences in grades at the end of the course are caused by differences in mode of course delivery, or differences among the students who self-selected the different modes. Positivists would thus tend to say that most ALN research to date is not very rigorous.
There are several
different types of measures of effectiveness of ALN's for students that are
commonly used. Objective measures of
performance and subjective assessments by students have been used about equally
(See Table 1). Objective measures
include the following; the number of studies shown in Appendix 3 using each of
these measures is noted in parentheses:
Grades, for specific projects or exams or for the entire course, compared to sections or students using other delivery modes (16).
Measures of the
quality of work (e.g., group projects may be judged on creativity,
completeness, length, etc.) (9). Such
judgements may be biased if they are made by the instructor who designed the
online course, and who knows who did a particular piece of work and the mode
that student was in. Thus, procedures
need to be designed to make the quality of work measure as reliable as possible
by using multiple judges, who are "blind" to the identity and course
delivery condition of the student (e.g., see Fall, 2009).
· Course completion rates (3 studies)
· Counts or measures of activity levels or patterns (5)
"Subjective" measures are frequently used in the current body of ALN research on effectiveness, though they are not usually considered as valid as objective measures. These include student self-assessments (through questionnaires or interviews) of course learning outcomes (absolute or compared to traditional courses; 8 of the studies included in this paper use such a measure); the effectiveness of the mode or system used for delivery (including convenience, motivation, usability, time required, access to professor; 18 studies); or of the quality of the instruction or materials (3 studies). Note that the total number of studies using each type of measure adds up to more than 100% of the studies, since many used more than one measure of effectiveness. Generally, multiple measures enhance the reliability of the conclusions about effectiveness.
Table 1: Summary of Measures and Results for All Nineteen
Studies
|
Measures |
Positive for ALN |
No Difference |
Negative for ALN |
|
Objective Measures Course grade |
2 |
6 |
|
|
Final exam grade |
2 |
1 |
|
|
Midterm/quiz grades |
2 |
3 |
|
|
Quality of work rated by instructor |
3 |
2 |
|
|
Assignment Measures Length of Report |
1 |
|
|
|
Rated by Judges Forgetting |
|
1 |
|
|
Content Quality |
1 |
|
|
|
Completeness |
1 |
|
|
|
Amount of Collaboration |
|
1 |
1 |
|
Amount of Activity / Participation |
2 |
|
|
|
Female participation |
1 |
|
|
|
Course completion |
1 |
|
2 |
|
Use of instructor who does not prepare materials |
1 |
|
1 |
|
Subjective measures via students Learning more |
|
5 |
|
|
Skill development |
|
|
|
|
Quality of work |
|
2 |
1 |
|
Quality of course materials |
|
1 |
|
|
Quality of discussion |
|
|
2 |
|
Motivation/Interest |
3 |
1 |
1 |
|
Progress to degree |
2 |
|
|
|
Access to degree |
2 |
|
|
|
Access to instructor |
1 |
|
|
|
Access to educational resources |
1 |
|
|
|
Usability of technology |
2 |
1 |
2 |
|
Participation |
|
|
1 |
|
Social Presence |
|
|
1 |
|
Totals |
28 |
24 |
12 |
3. Results: ALN vs. Traditional Face-to-Face Course
Delivery
A. Summary
Tables of Results
Most of the studies either measure effectiveness in more than one way (e.g., grade distributions plus subjective student assessments) and/or study different courses, resulting in many "mixed results." Looking at the results, we have classified them as falling into one of two categories, those generally show ALN to have better outcomes than traditional courses (Table 2), and those that tend to show no difference, overall (Table 3).
:
Table 2: Summary for
eight studies with largely positive ALN findings
|
Measures |
Positive for ALN |
No Difference |
Negative for ALN |
|
Objective Measures Course grade |
2 |
2 |
|
|
Final exam grade |
2 |
|
|
|
Midterm/quiz grades |
2 |
|
|
|
Quality of work rated by instructor |
3 |
|
|
|
Assignment Measures Length of Report |
1 |
|
|
|
Rated by Judges Forgetting |
|
1 |
|
|
Content Quality |
1 |
|
|
|
Completeness |
1 |
|
|
|
Amount of Collaboration |
|
|
|
|
Amount of Activity / Participation |
1 |
|
|
|
Female participation |
|
|
|
|
Course completion |
1 |
|
|
|
Use of instructor who does not prepare materials |
|
|
|
|
Subjective measures via students Learning more |
6 |
|
|
|
Skill development |
1 |
|
|
|
Quality of work |
|
|
1 |
|
Quality of course materials |
|
|
|
|
Quality of Discussion |
|
|
|
|
Motivation/Interest |
3 |
|
|
|
Progress to degree |
2 |
|
|
|
Access to degree |
2 |
|
|
|
Access to instructor |
1 |
|
|
|
Access to educational resources |
|
|
|
|
Usability of technology |
|
|
|
|
Participation |
|
|
|
|
Social Presence |
|
|
|
For these eight
studies in Table 2, the preponderance of the evidence is that ALN is more
effective than traditional courses..
All of the results indicate ALN to be better, or else some results show
ALN as better and others show "no difference." The studies that are judged to be in this
category include those by Alavi; Andriole; Benbunan-Fich et.al. (2010), Hiltz;
Hsu et.al.; Hiltz & Wellman; Thoennenssen et. al; and Turoff and Hiltz.
For the remaining 12
studies, the preponderance of the evidence shows "no significant
difference" between ALN and the traditional courses or sections or
experiences that are compared. Either
all the results show "no significant difference," or there are mixed
results with results better for ALN for some courses or measures, and worse on
others (e.g., for the SCALE projects at Illinois [Arvan, 1998], there are some
conflicting results related to course and/or experience of the instructor.).
There are no
qualifying empirical studies for which ALN is clearly shown to be less
effective than traditional modes of course delivery, on the whole.
The first important
point to note is that the "No Difference" cases really indicate that
ALN is just as effective as face to face and when this is added to the positive
results for ALN there is a four to one ratio of positive results to negative
results in these 19 studies.
Furthermore, if we realize there are only four instances of objective
measures that are negative and none of them are direct measures of learning, we
find the results rather overwhelming support the hypothesis that ALN is a
meaningful alternative to the classical face to face class, which tends to be
as effective or more effective, depending on the circumstances of the
particular implementation and the measure used.
In terms of some of
the negative results, the course completion or drop out rate is probably higher
than it should be because of mistaken expectations by students, either because
they are new to the use of ALN or because they do not have the student network
to warn them away from particular offerings where the course might not live up
to the description. In fact, many
studies of ALN show that the role of the instructor and his or her ability to
deal with this new mode of learning is a principal factor in ALN success. Teaching ability is important, as well as
experience in the mode of delivery. A
lot of face-to-face courses may be taught by less than perfect instructors but
the face to face environment can tolerate a wider range of instructor
abilities. We must evolve a mechanism
to specify in evaluation work the competence of instructors or it will be quite
clear that the results can easily be confounded.
Most universities have
a standardized survey for measuring student reactions to an instructor at the
end of the course. It is only this year
that the union at NJIT agreed to a slightly modified version of a teaching
evaluation course to be put online for all ALN courses, so all on line courses
will receive the same survey and those teaching online sections can be directly
compared with respect to their face to face sections and with others teaching
the given course. It is this sort of
data that should be assessed longitudinally to determine how the performance of
the instructor evolves over time, and which instructors are the important ones
to carefully debrief in order to determine relative success factors in the
teaching of a given course.
Table 3: Summary for eleven studies with largely mixed or No
Difference" ALN findings
|
Measures |
Positive for ALN |
No Difference |
Negative for ALN |
|
Objective Measures Course grade |
|
4 |
|
|
Final exam grade |
|
1 |
|
|
Midterm/quiz grades |
|
3 |
|
|
Quality of work rated by Instructor |
|
2 |
|
|
Assignment Measures Length of Report |
|
|
|
|
Rated by Judges Forgetting |
|
|
|
|
Content Quality |
|
|
|
|
Completeness |
|
|
|
|
Amount of Collaboration |
|
1 |
1 |
|
Amount of Activity / Participation |
1 |
|
|
|
Female participation |
1 |
|
|
|
Course completion |
|
|
2 |
|
Use of instructor who does not prepare materials |
1 |
|
1 |
|
Subjective measures via students Learning more |
|
5 |
|
|
Skill development |
|
|
|
|
Quality of work |
|
2 |
|
|
Quality of course materials |
|
1 |
|
|
Quality of discussion |
|
|
2 |
|
Motivation/Interest |
|
1 |
1 |
|
Progress to degree |
|
|
|
|
Access to degree |
|
|
|
|
Access to instructor |
|
|
|
|
Access to educational resources |
1 |
|
|
|
Usability of technology |
2 |
1 |
2 |
|
Participation |
|
|
1 |
|
Social Presence |
|
|
1 |
Some of the variables
that are related to the degree of effectiveness of a particular course
implementation are suggested by the various correlations with learning
effectiveness reported in the studies included in this analysis. These are shown in Table 4. There are not
enough replications testing specific relationships to reach any firm conclusions
on these relationships at this point, but they do suggest hypotheses to be
tested in future studies.
_____________________________________________________________________________
Table 4: Correlations or Interactions Reported for
the positive studies.
·
Perceived collaboration correlates with
perceived learning
·
Activity by individuals correlates with
Perceived learning
·
Perceived collaboration correlates with
motivation
·
Higher objective Grades in Computer Science
correlates with the use of ALN
·
SAT Scores and/or GPA are the dominant correlation
with course grades (independent of ALN or face-to-face)
·
Initial subjective expectations correlates with
later actual participation
·
Course type and mode of delivery interacts.
·
Degree of collaboration correlates with
satisfaction with course.
Correlations or Interactions Reports (unique to one study)
for the Mixed or No Difference Studies
·
Use of quizzes (for feedback only) improves
learning perceptions by instructors and students
·
Collaboration falters when there is not
sufficient critical mass.
·
Laboratory use and collaboration are correlated
in a Chemistry course.
·
Collaborative groups online have higher
satisfaction than individuals online.
·
Group Communications with Learning Game led to
higher learning levels than with either alone.
·
No benefit found for use of telephone
communications
·
Dense Hyper-links interfere with ease of
learning
·
More efficient management of distance students
possible
·
Larger courses in distance possible.
Technology
problems correlate with dissatisfaction by students.
____________________________________________________________________________
With respect to
student feedback it is not surprising to find from a number of studies that
face to face students often think that the quality of their work or the quality
of discussion is better face to face, when in fact the opposite may be true
when expert judges are used. This is
similar to when an employee changes from using a manual approach to using a
computer approach to handle their tasks.
The way one is used to doing work is usually felt to be a better way
than having to learn a new one. It
often takes a long interval of time for the employee or anyone else to get used
to a new way of solving problem or learning and to begin to actually realize
they are doing well.
It is also time to
capitalize on the records that exist for students to begin to conduct some
longitudinal analyses on students. We
need to relate a student's subjective opinions about an ALN course to how many
such courses they have taken. We must
view their grades in relation to how they have preformed in the prerequisite
course and their overall grade average, in order to help to control for
possible effects of self-selection of mode of delivery. In this manner we can begin to get a handle
on the true long-term impacts of ALN.
One would suspect that a longitudinal analysis will begin to show that
those who actually desire to use the ALN alternative (as compared to those who
are basically "forced" to take an ALN section because there are no
available alternatives) do better than those in comparable face to face classes
The evidence is
overwhelming that ALN tends to be as effective or more effective than
traditional modes of course delivery, at the university level. There really is no need for more studies to
explore this question. What we need is
more research that will enable us to make it even more effective, especially as
new technologies proliferate. For
example, most early ALN's were "text-only" discussion; many now
include multi-media web based tutorials instead or in addition to a textbook,
and possibly audio and/or video interaction with the instructor. How important is multi-media in student
teacher and student-student interaction, and how may it best be used? With very small and pervasive devices coming
into use (e.g., wireless and pocket PCs), how can these devices best be used to
further improve convenience of access to ALN's? What forms of collaborative learning vs. individual assignments
are most appropriate for different kinds of courses or class sizes? As Alavi and Leidner(2010) point out in their call for "greater
depth and breadth of research" on technology-mediated learning, research in this area must look at how
technology influences learning, which involves an "explicit consideration
of relationships among technology capabilities, instructional strategy,
psychological processes, and contextual factors involved in learning" (p.
1). We need to develop both more
sophisticated and more comprehensive theoretical frameworks, and also more
valid methods and instruments that those which have characterized a majority of
studies to date.
In conclusion, it is
time to start looking at the evaluation of ALN beyond the single course and
consider the full range of evaluation variables. Consider the following top down structure of evaluation variables
that might form an overall model for the study of ALN effectiveness and
impacts:
Evaluation Measures
Student Performance Measures
·
Amount of learning
·
Time to Degree
·
Satisfaction
·
Motivation
·
Enjoyment
·
Participation
Resource Measures
·
Effort to Learn
·
Time to learn
·
Time to educate
·
Cost of resources
·
Effort to access learning
·
Cost of learning resources
·
Availability of course as needed
·
Convenience of courses
Opportunity Measures
Do new things not possible before
·
Accessibility for new populations
·
Reduce or eliminate distinction between distance
and regular students
·
Make direct use of human resources not normally
available in a course
·
Introduce new programs
Do things differently than before
(organizational impacts)
·
Change the nature of courses
·
Change the nature of degree programs
·
Change the nature of educational institutions
·
Change the nature of departments
·
Change the nature of teaching
·
Change the evaluation criteria of faculty
·
Impact on individual (job, earnings, advanced
study. etc.)
·
Reputation of Program
·
Reputation of Institution
·
Accreditation of Program
Significant Intervening Variables
·
Type of Students (objectives, part time,
learning abilities, gender, age, experience, and other demographics)
·
Type of Course (Laboratories, skill, subject
area, etc.)
·
Teaching Methodologies employed (e.g., is
discussion graded, are individual or group assignments and activities used)
·
Size of course
·
Information on courses available to potential
students
·
Technology employed (media mix, usability,
functionality)
From the above and the
preliminary correlations based upon courses as well as our understanding of
higher education, it is time to begin formulating testable models that involve
many of the key variables, and to begin to use meaningful historical data to
verify or improve our models.
For all the talk of
university research, as with most commercial organizations, universities are
reluctant to expose the critical historical data that they have, for fear of
negative publicity. The few times that
this has succeeded in studying industry it has tended to be the result of an
independent and non competitive organization established that would guarantee
the anonymity of detailed data supplied to the organization when it came to the
publication of the resulting analysis reports.
The incentive for the organizations participating was to be able to
compare their performance with others and in so doing, have a better idea of
how to make decisions to improve their situation.
It might be that the
formulation of such an independent depository of historical information is one
of the next important steps in furthering the evaluation of ALN. While we have done a reasonable job, as an
academic community, in proving the viability of ALN, our form of distance
learning is probably still a minority in terms of the methods whereby distance
learning is being offered. ALN
researchers tend to believe that those approaches that do not include extensive
communication and interaction among students and faculty are inferior to ALN
approaches as well as face to face education, but this message has not
"gotten through" to a lot of distance course designers. We need to raise our evaluation and research
dissemination efforts to a new plateau and use the opportunity presented to
provide a deeper understanding of higher education effectiveness than exists to
date.
Acknowledgements
This work is partially
supported by a grant from the Alfred P. Sloan Foundation, by NJIT, the New
Jersey Center for Multimedia Research, and the New Jersey Center for Pervasive
Information Systems. David Spencer and
Jayalamathy Sadagopan contributed many of the article reviews for the database. Razvan Bot has primary responsibility for
the database of papers and other aspects of the web site construction and
maintenance.
1. Alavi, Maryam (1994), Computer-Mediated
Collaborative Learning: An Empirical Evaluation, MIS Quarterly, June,
pp.159-174.
2. Alavi, M. and Leidner, D.E. (1991). Research commentary: Technology-mediated learning-- A call for grater depth and breadth of research. Information Systems Research, 12, 1 (March), 1-10.
3. Alavi, Maryam, Youngjin Yoo and Douglas R.
Vogel (1997), Using Information Technology to Add Value to Management
Education, Academy of Management Journal, Vol 40, No 6, pp.1310-1333.
4. Andriole, Stephen J. (1997), Requirement-Driven ALN Course Design, Development, Delivery & Evaluation, JALN, Vol 1, No 2 - August, 10p.
5. Arbaugh, J.B. (2009), Virtual classroom versus physical classroom:
An exploratory study of class discussion patterns and student learning in an asynchronous
internet-based MBA course, Journal of Management Education, Vol.24 Issue 2
Apr., pp.213-233.
6. Arvan, Lanny, John Ory, Cheryl D. Bullock, Kristine K. Burnaska and Matthew Hanson (1998), The SCALE Efficiency Projects, JALN, Vol 2, No 2 - September, 27p.
7. Benbunan-Fich, Raquel and Starr Roxanne Hiltz (1999), Impacts of
Asynchronous Learning Networks on Individual and Group Problem Solving: A Field
Experiment, Group Decision and Negotiation, Vol.8, pp.409-426.
8. Benbunan-Fich, Raquel, Starr Roxanne Hiltz and Murray Turoff
(2010), A comparative content analysis of face-to-face vs. ALN mediated
teamwork. Proceedings of the 34th Hawaii International Conference on
System Science, IEEE Computer Society.
9. Fallah, M.H., W. J. How and R. Ubell (2009),
Blind Scores in a Graduate Test: Conventional Compared with Web-based Outcomes.
ALN Magazine, Vol. 4, Issue2 – December.
10. Hadidi, R and C. Sung (2009), Pedagogy of Online Instruction – Can
it be as Good as Face-to-face? Proceedings of the American Conference on
Information Systems, Long Beach, California, Aug 2009, pp. 2061-2065.
11. Hiltz, S. R. (1993), Correlates of learning in a virtual classroom,
Int’l Journal Man-Machine Studies, Vol.39, pp.71-98.
12. Hiltz,S.R. The Virtual Classroom: Learning Without Limits Via Computer Networks. Norwood, NJ, Ablex Publishing (Human-Computer Interaction Series), 1994.
13. Hiltz, S.R., Benbunan-Fich, R., Coppola, N., Rotter, N., and Turoff, M. Measuring the Importance of Collaborative Learning for the Effectiveness of ALN: A Multi-Measure, Multi-Method Approach. JALN 4, 2 , 2009.
14. Hsu, Enrico Y. P., Starr Roxanne Hiltz and Murray Turoff (1992),
Computer-Mediated Conferencing System as Applied to a Business Curriculum: A Research
Update, Int’l Business School Computer Users Group, 20th Annual North American
Conference, pp.214-227.
16. Ocker, Rosalie J. and Gayle J. Yaverbaum (1999), Asynchronous
Computer-Mediated Communication versus Face-toFace Collaboration: Results on
Student Learning, Quality and Satisfaction, Group Decision and Negotiation,
Vol.8, pp.427-440.
17. Sandercock, G.R.H. and G. Shaw (1999), Learners’ Performance and Evaluation of Attitudes Towards Web Course Tools in the Delivery of an Applied Sports Science Module, ALN Magazine Vol.3 Issue 2 Dec., 9p.
18. Sener, John and Mary Liana Stover (2009), Integrating ALN into an
Independent Study Distance Education Program: NVCC Case Studies, JALN, Vol. 4
Issue2.
19. Thoennessen, M., E. Kashy, Y. Tsai and N.E. Davis (1999), Impact
of Asynchronous Learning Networks in Large Lecture Classes, Group Decision and
Negotiation, Vol.8, pp.371-384.
20. Turoff, Murray and Starr Roxanne Hiltz (2009), Effectively Managing
Large Enrollment Courses: A Case Study.
In Bourne, J. and Moore, J.C., eds, Online Education: Volume 2. Learning Effectiveness, Faculty Satisfaction,
and Cost Effectiveness, pp. 55-80.
Needham, MA, Sloan Center for Online Education.
21. Wade, Vincent P. and Conor Power (1998),
Evaluating the Design and Delivery of WWW Based Educational Environments and
Courseware, ACM, ITICSE Proceedings, July, pp.243-248.
22. Worrell, Malcolm, Starr Roxanne Hiltz, Murray Turoff and Jerry
Fjermestad (1995), An Experiment in Collaborative Learning using a Game and a
Computer Mediated Conference in Accounting Courses, Proceedings, Hawaii Int’l
Conference on System Sciences, 28th Annual, pp.63-71.