2.     Research Models, Methods and Measures for Assessing the Effectiveness of ALNs

Asynchronous Learning Networks may be considered to be one type of information system: a computer-based system designed to support the work of teachers and learners. There are two dominant research models in Information Systems and in other fields using social science methodologies to study human subjects: the "positivists" and the "interpretivists." The positivists strive to follow the model of scientific inquiry developed in the natural sciences, with the objective being to specify quantitative measures of all variables, state hypotheses, collect data using random sampling and other procedures that will enable the testing of hypotheses using inferential as well as descriptive statistics, and then analyze the data, report the results and their limitations. The "interpretivists" strive for an "in depth" understanding of the processes that are occurring in human social systems; they generally start with research questions, use qualitative methods such as participant observation, unstructured or semi-structured interviews, and content analysis to obtain a rich description of the phenomenon and arrive at an interpretation of why and how things work-- or do not work (Ngwenyama and Lee, 1997). The most sophisticated research projects combine both quantitative measures (describing quantitatively "what" is happening") and qualitative measures (describing "why" the results are occurring, in terms of the details of behavior and interaction that transfer an "input" of course materials provided in various modes to an "output" of what the student does or does not learn.)

To qualify as an empirical study using generally accepted methods, a research project should start with specific hypotheses and/or research questions, which guide the selection of methods and measures. Unfortunately, the majority of publications related to ALN's do not have any explicit research questions, let alone specific hypotheses. They tend to be accounts by instructors of courses they designed and of their experiences and impressions, whose value in building a scientific body of knowledge is questionable.

The paradigm of positivist research has enshrined experimental design as the most valid method for determining "cause and effect," specifically, the "pre-test, post-test, control group design" using random assignment of individual subjects to conditions. It is basically impossible to randomly assign students to take a traditional section or an ALN section of a course; they may be unable to travel to campus if they live 2009 miles away, or unable to take an ALN section if they have no PC and Internet provider. One is therefore left with "quasi-experimental" designs at best, in most cases, in which student’s self-select mode of course delivery, but the study designer and instructors try to "hold constant" everything else, such as the syllabus, assignments and exams. The "pre-test, post-test" design means that ideally one measures the dependent variable ( such as knowledge about accounting or data bases or English Literature) before the course, then measures again after the course, to determine "amount learned" as the difference between scores. Most ALN studies do not do this either, since it is not usual to give students the equivalent of a final exam on the first day of the course, and in many project-based courses, it is not appropriate, since there will be no "final exam" either. (One exception is the Worrell et. al study of a graduate accounting course [#], for which standardized professional exams are readily available to test "knowledge" of accounting). Thus, we often do not know whether differences in grades at the end of the course are caused by differences in mode of course delivery, or differences among the students who self-selected the different modes. Positivists would thus tend to say that most ALN research to date is not very rigorous.

There are several different types of measures of effectiveness of ALN's for students that are commonly used. Objective measures of performance and subjective assessments by students have been used about equally (See Table 1). Objective measures include the following; the number of studies shown in Appendix 3 using each of these measures is noted in parentheses:

Grades, for specific projects or exams or for the entire course, compared to sections or students using other delivery modes (16).

Measures of the quality of work (e.g., group projects may be judged on creativity, completeness, length, etc.) (9). Such judgements may be biased if they are made by the instructor who designed the online course, and who knows who did a particular piece of work and the mode that student was in. Thus, procedures need to be designed to make the quality of work measure as reliable as possible by using multiple judges, who are "blind" to the identity and course delivery condition of the student (e.g., see Fall, 2009).

         Course completion rates (3 studies)

         Counts or measures of activity levels or patterns (5)

"Subjective" measures are frequently used in the current body of ALN research on effectiveness, though they are not usually considered as valid as objective measures.These include student self-assessments (through questionnaires or interviews) of course learning outcomes (absolute or compared to traditional courses; 8 of the studies included in this paper use such a measure); the effectiveness of the mode or system used for delivery (including convenience, motivation, usability, time required, access to professor; 18 studies); or of the quality of the instruction or materials (3 studies). Note that the total number of studies using each type of measure adds up to more than 100% of the studies, since many used more than one measure of effectiveness. Generally, multiple measures enhance the reliability of the conclusions about effectiveness.

Table 1: Summary of Measures and Results for All Nineteen Studies