Learning Networks Effectiveness Research Program

Overview

         The goal of this research program is to increase the quality, quantity, and dissemination of results of research on the effectiveness of Asynchronous Learning Networks (ALN).  It will  synthesize existing knowledge and create new knowledge about the methods and findings of research on the determinants of effectiveness  of (ALN, and to make the results available worldwide via a project web site. The major activities will be:

1.  Create a “WebCenter for Learning Networks Research.”   This will include a series of  online knowledge bases that are regularly updated and will be available through the project web site to researchers, faculty, the press, and the public.   There is a great need for a high quality website that provides a comprehensive listing of research that is being done in the area of ALN Learning Effectiveness.  NJIT has almost two decades of research accomplishments in ALN and is technically qualified to construct and operate a large web site, as the “most wired” public university in the U.S. for the last several years (according to YAHOO!).  Two of the Co-PI’s. (Professor Sharon Derry of the School of Education at the University of Wisconsin, and Ricki Goldman-Segall, who prior to joining NJIT in the Spring of 2005 was at the School of Education at the University of British Columbia) will help to assure that the research findings, methodologies, and theories of educational researchers studying the effects of technology on teaching and learning are adequately represented in the knowledge bases.

2.  Build/strengthen an ALN evaluation research community that will create and share improved research methods, theoretical frameworks, and instrumentation for assessing the outcomes of online learning.  Face to face workshops and asynchronous conferences will be used to achieve this.  The results of these activities will be used to enhance the materials on the web site.

1.Create a knowledge base of ALN research results

Almost daily, there are articles in the press about online courses, that approach the subject from either a negative or positive angle.   Most of these articles cite a few faculty members or students who have been interviewed, or cite a single study of a single course, with unknown quality of research methods.  Many of them do not distinguish Asynchronous Learning Networks, which emphasize extensive student-student and teacher-student interaction, from distance modes which simply post materials on the web and use individual email assignments, without any collaborative learning activities or formation of a “class” of interacting students. 

Claims and counter-claims are likely to persist unless a comprehensive and authoritative “knowledge base” of information about evaluation research studies of ALN is created and made available to the public, on the Web as well as in published journals.   There is currently no place where faculty, researchers, potential students, or the press can go to find out what information exists about the following:

       What empirical studies have been conducted about ALN?  What were the methods, variables studied, and findings?

       What is the current overall picture of these research findings, in terms of comparisons of the effectiveness of ALN with other modes of delivery of college-level courses?  What do we know about relatively effective and ineffective ways of “doing” ALN?

       Who are the leading researchers in ALN, are they willing to be contacted by the press, and if so, how can they be contacted? (Video clips, some of them from NetLearning and other ones recorded at meetings, can be used to introduce the public to some of these leading researchers).

       What methods and research instruments (questionnaires, interview guides, etc.) are available for use by ALN researchers? 

      What are the strengths and weaknesses of various research methods for studying ALN, and what methods are recommended by experts in the field as especially appropriate for understanding this form of educational medium?

      What theoretical foundations are there for the field of ALN? What theories are most appropriate for framing research studies in this area?   And which methods best match different theoretical frameworks?

Learning networks are defined as groups of people who use computer networks (the Internet and World Wide Web) to communicate and collaborate in order to build and share knowledge.  The emphasis will be on asynchronous (anytime, anyplace) use of networks, but the project will include studies of courses that integrate some use of synchronous (same time, different place) technology or which compare face to face, synchronous and asynchronous learning processes.   Secondly, the emphasis will be on post-secondary, for-credit courses, but information will also be collected about studies of the use of ALN in pre-college courses and in continuing professional education (not for academic credit) courses or learning communities.  Effectiveness will be defined in this project to focus on  both learning outcomes for students, and positive or negative impacts on faculty.  To the extent that other measures of effectiveness are reported in empirical studies (e.g., fiscal impacts on educational institutions, cost-benefit analysis, or societal level impacts in terms of educational access and equity), they will also be included.

The planned major components of the web site are:

1.    A knowledge base and accompanying analysis of all ALN effectiveness research studies and results.

2.    An ALN research methods section, with a hypertext tutorial, a research instruments collection, and a multi-media “gallery” of leading ALN researchers.

3.    A theoretical frameworks section, with a hypertext tutorial and bibliography.

Each of these is described further below.   Additional sections may be added based on feedback from research workshops and web site users. Each component will be posted first as a prototype that will be subjected to usability testing and improved before public posting.  Subsequent to first release, each part of the web site will be frequently updated.

        1.1.  A synthesis of all (English language) published research that evaluates ALN courses, in refereed journals, refereed conference proceedings, or book chapters. 

A data base will be created which codes for each study the theoretical framework used (if any), research methods, results,  and important conclusions.  Though the ALN Research Studies Knowledge Base will not include non-English sources, contributions of abstracts in other languages will be accepted for a more complete bibliography of publications, and partners will be sought to translate the data base into other widely used languages.  As part of this work, a list of online and printed journals that frequently publish articles in this area will be created, and subscriptions obtained to these sources, so that new studies can be added on a timely basis. This compilation of peer reviewed, published  ALN effectiveness research, available online, will give ALN evaluators an efficient way to be able to build on past research.  It will also make up to date and complete information on ALN effectiveness available to the press, faculty considering teaching online, and the public.

A start has been made on this knowledge base, as part of the dissertation work for a student, with some initial output attached as Appendix 1.   Since the data are in a data base, they can be statistically analyzed, similar to the analyses of all experiments and field studies of Group Support Systems by Fjermestad and Hiltz (1999, 2005).  In addition to availability online, the results of the analyses will also be presented at ALN meetings or research workshops, and submitted for publication.

    1.2. Organize and disseminate knowledge about appropriate research methods and procedures for evaluating the effectiveness of ALN. 

A hypertext tutorial will be created which reviews the application of various research methods in the study of ALN, with their advantages and disadvantages, and pointers to published work that uses these methods.  (It is proposed to submit a non-hypertext version of this as a paper to JALN).   An online segment will be a collection of research instruments used in past ALN research (questionnaires, interview guides, content coding schemes, etc.  As with the IS World collection of survey research instruments, active researchers will be solicited to place their instruments in the collection.) 

As part of this work, a section will be created for “methodological innovations” that represent novel and potentially useful adaptations of research methods used in the study of other forms of human communication and behavior, to the unique context of learning networks. 

2. Stimulate methodological innovations to meet the needs of research in this unique area. 

A series of synchronous and asynchronous computer conferences and workshops will be created to engage active researchers in developing and extending methodological innovations that will improve the efficiency, validity, and completeness of evaluations of ALN. The two Autumn face to face workshops will either continue the series of  the Sloan sponsored workshops on effectiveness research (U. of Illinois, Summer 1999; Lake George, September 2005) or will need to be coordinated with these events.  

When the initial tutorial material has been posted, it is planned to kick off the methodological innovation activities with an online asynchronous conference or conferences for  active researchers on methodological issues and innovations in ALN evaluation.  This will lead to a workshop to be held in the new Collaborative Hypermedia Laboratory at NJIT in the Autumn of 2006.    This facility , funded through grants from the National Science Foundation, the state of New Jersey, UPS, and NJIT,it includes a “collaborative meeting room”. Up to 21 participants, each at their own computer, can participate in synchronous discussion and knowledge building, along with the hardware (such as three projectors) and software to support, capture, and organize this activity.  This will be a one day workshop (with a social gathering the night before), including approximately 15 of the most active researchers in the area of ALN studies today.  By holding it in the computer-supported meeting room, all of the material generated will be preserved digitally, for inclusion in the online data bases, reports, copying portions into the asynchronous continuing discussion, etc. 

          The results of the one day synchronous workshop will be organized and used on the project web site and as discussion topics in the   asynchronous conference that is open to all, to continue discussion and knowledge building about innovative methodological techniques for studying ALN.   This online conference will continue through the 2001-2007 academic year.

3. Build Theoretical Frameworks for ALN Research and Practice 

Theory and methods are the twin foundations of good research on ALN.  The strategy for   synthesizing and advancing theoretical frameworks and models in ALN research will be similar to that described above for research methods.  A hypertext tutorial will be created on theories and frameworks that have been applied in ALN and related research, to predict and understand the processes and outcomes of ALN (or other kinds of online knowledge-building communities). 

This will then be followed by a final synchronous workshop on theoretical and methodological innovations and issues in ALN effectiveness research, held at NJIT in Summer or early Autumn 2007 for a group of about 10-15 researchers, and a  follow on public asynchronous conference of about three months duration.  A major activity of this final workshop will be to build a proposed plan for future activity by the ALN research community after the end of the project.

4. Examples of Project Activities

4.1. Some Research Issues

What are the advantages and disadvantages of different media mixes, CMC features, teaching techniques, examination modes, etc., within the broad scope of “ALN delivery?”  In other words, can ALN researchers gather data that convincingly answers questions about relatively better and poorer ways of “doing ALN?”  This is one of the most important objectives of the project, to assist researchers to be able to frame and carry out studies that will help faculty and universities to “do it better.”  We hope to first post suggested methods and instruments for studying such questions, and then be able to post a growing compendium of findings to guide practitioners.

For example, there are now a range of technologies available for supplementing asynchronous (any time, generally meaning different times) interaction online with synchronous (same time, different place) online interaction (text chats, audio, video conferencing, whiteboards, etc.)  There are also a set of “pervasive,” wireless technologies (web-ready computers in your car, your pocket, maybe in your eyeglasses or built as chips in your body) that are emerging that might support more truly “anytime anywhere” interaction.  What happens when one introduces such technologies into an otherwise ALN course for use in  required or optional synchronous sessions or assignments?  How can they improve ALN effectiveness, and what modes of use have more problems and disadvantages than advantages?  The proposed Web Center for Learning Networks Effectiveness Research can help researchers learn what is out there to be studied, and how other researchers are studying it.  A growing compendium of findings, posted in a timely manner,  can help faculty and faculty development programs to decide which of these emerging synchronous multi-media technologies they should consider incorporating into their courses. 

4.2. Some Methodological Issues

1.    How can we get much higher response rates on surveys of students?  Short online surveys, perhaps with some incentive to complete them?   “Interviews” online with new voice interface technologies?  

2.    Can we build and test better “scales” to measure different types of impacts on various stakeholders?  Can we post a “standard” set of measures suggested for research use, so that studies are more comparable and can better build a cohesive body of research findings?

3.      How can be better measure differences in “learning” and in “retention” of materials of different types delivered by different modes and pedagogical strategies (e.g. individual work online vs. small collaborative groups.)  For example, could we develop some standard “skills mastery” questions for certain subject matter, and post automated tests that could be used “before and after” a course to measure learning?

4.     What longer term, more “global” measures can be developed and used to measure the impact of whole programs delivered via ALN rather than individual courses, on a the whole set of stakeholders, for those who participate in ALN vs. those who don’t?  ‘For example, what are the important potential impacts to look for, and how can one measure these impacts on the families of students, employers, the university as a whole, the communities in which the universities are located, and the socio-economic structure of the society as a whole?     E.g., are ALN alumnae as loyal as traditional alumnae, and do they give as much money after they graduate?  Do part time adult working students in ALN programs graduate at a much faster rate than those in traditional on campus programs, and if so, how much is this faster completion “worth” to them and their employers?   There are whole sets of questions here that nobody seems to have tackled; a web site that suggests such issues and how to measure these variables would stimulate the study of   such broader research issues.

5.     How to set up quasi-experimental study designs to more rigorously and definitively answer questions about relative effectiveness of two or more different educational “treatments.”

6.     What techniques and software are available to do “content analysis” of the transcripts of ALN course discussions and online work?   What do we most want to look for in such studies of what actually goes on “inside” ALN learning spaces, to be able to draw conclusions abut how differences in the nature and dynamics of interaction are related to differences in outcomes.

5. Advisory Board

          The Project Advisory Board will have 6-8 members, plus the project officer (ex officio).  There are at least three important functions for this advisory board.  The first is to have broader expert input into the kinds of information and facilities that are most needed by the ALN research community , so that the Learning Networks Effectiveness Research web site is as useful as possible, as soon as possible..  The second is to better assure coordination with other activities that are occurring in this community, particularly the main ALN web site, conference, other workshops and events and projects, etc.  The third is to broaden the base of sponsorship and association with the project, to improve the breadth of data included.

Timetable: Completion targets for project activities

Go Back