CTER OnLine is a master of education program developed in 1998 at the College of Education, University of Illinois, Urbana-Champaign, aimed at precollege teachers interested in issues concerning curriculum, technology, and education reform. The first set of 26 students graduated in May 2000. This paper presents an evaluation of CTER OnLine over its first two years, describing the initial stakeholder analysis that motivated the design of the evaluation, presenting data from surveys and from in-depth case studies, and outlining the overall theoretical framework for the design, implementation, and revision of CTER OnLine. It concludes with recommendations for online education.

In the summer of 1998, the College of Education at the University of Illinois in Champaign-Urbana began offering courses in an online format that would culminate in a Master of Education (Ed.M.) degree (see Ed-Online at http://www.ed.uiuc.edu/ed-online/ ). With financial assistance from the UI OnLine initiative http://www.online.uillinois.edu/ , courses were redesigned and offered to education professionals through different departments in the College. Initially, the intention was to fill the needs of two groups of professionals. CTER OnLine focuses on topics related to Curriculum, Technology, and Education Reform http://cter.ed.uiuc.edu , and is geared toward practicing K-16 teachers and administrators while the other initiative, HRE OnLine, targets human resource educators.

For the past two years, we have conducted both formative and summative evaluations on the Ed-Online initiative. This paper focuses on the evaluation of CTER OnLine from both a program development and student learning prospective.

The evaluators identified five groups of stakeholders and specified their interest in the development and conduct of online degree programs. These stakeholders include administrators at the university, college and department levels; college faculty, and students.

University administration was interested in funding an online initiative. At the college level, there was an interest in increasing departmental and college capacity, maintaining high academic standards, providing programs that are self-supporting, and utilizing college resources wisely.

Department administrators share many of these concerns and must also deal with issues within their departments. For example, departments must be concerned about the amount of money spent on personnel (e.g. faculty and staff), and on computer technologies (e.g. hardware and software). They must also be concerned about providing quality programs and faculty satisfaction with their online teaching experiences.

Faculty members who teach online courses have a stake in the design, development, and conduct of on-line graduate programs. They must consider the amount of time it will take to learn new technologies, develop course content, and interact with students via the use of information technologies. College faculty who are not participating in online courses are also stakeholders because the decisions made will set precedents in the college that influence the allocation of funds for teaching online classes and the use and purchase of computer technologies within the college.

Lastly, students have a vital interest in the development and conduct of these online Master of Education degrees. Students expend a great deal of time, money, and energy completing a degree program. They are concerned with the quality and usefulness of their learning, the quality of interactions between themselves and the instructor, the quality of interaction between and among their classmates, and the ways in which this learning is useful for their current jobs as well as how the program is viewed by their administrators. Potential students are also watching the outcomes as one online program after another report on their successes and failures.

The stakeholders’ concerns framed our selection of evaluation questions and the selection of data collection procedures. Evaluation questions were classified into eight areas: Interface design, Instructional design, student satisfaction, faculty satisfaction, economic viability, departmental capacity, interdepartmental collaboration, and college level infrastructure. A combination of survey data, interviews, and observations were used to determine the effectiveness of the program and the extent to which it is developing capacity that promotes its long-term viability.

Survey Data

Formative and summative data was collected from surveys administered online. Twenty-six students were asked to complete three program surveys throughout their two-year program. A pre-survey was administered during an orientation meeting that assisted faculty in determining the students’ level of computer expertise and their attitudes toward learning in an online environment. Mid- and post-surveys were administered after the first year and at the end of the two-year program. These surveys tracked students’ progress and changes in attitude over the two-year period.

Course surveys were also administered at the end of each semester and provided faculty members with information about their instruction, course content, and use of technologies. In some cases, faculty implementing innovative concepts (e.g. online simulations) into their teaching administered mid-course surveys to provide them with student feedback and the ability to modify their courses in progress.

In addition, during the same two-year period, two college-wide surveys were administered that asked faculty about their use of technology and their beliefs/reactions/opinions concerning online programs. The College also conducted an evaluation of the Office of Educational Technology, the office that coordinates Ed-Online and manages technology resources in the college. These detailed formative evaluations were designed to improve the support for online education in the College of Education.

Interviews and Observations

Faculty teaching CTER OnLine courses were interviewed after their course was taught. CTER support staff were also interviewed to determine the type and amount of technical support needed both for students to learn and for faculty to develop and teach online courses.

Four CTER students were interviewed and observed as a part of four in-depth case studies (Waddoups, 2000; Waddoups, et al., 2000). Eight interviews were conducted with each of the case study students over the two-year period. Exit interviews are still in the process of being conducted. Periodic observations were also made of the case study students in the context of their online classroom participation, in the context of working online and face-to-face in small groups with other CTER students, and in the context of their own classroom teaching.

A combination of synchronous and asynchronous technologies has been used in these courses. Course descriptions, syllabi, grading procedures, student participation and expectations have been available on the web. A variety of communication tools have been used depending on the type of interaction needed. Instructors have used the WebBoard® conferencing system for asynchronous discussions and synchronous text chat sessions. Individual email and group reflectors have also been used for one-to-one or one-to-group correspondence. RealPlayer® has been used to stream and archive audio and video files. Text translations of audio files have been provided for hearing impaired students. Audio-narrated PowerPoint® presentations have been created for some topics. Tapped In, (Schlager, Fusco, & Schank, 2000) a multi-user object-oriented environment (MOO) coordinated by SRI, International provided interactions between the CTER students and educators across the country. A specialized tool, known as CTERbase/TEbase, was developed to help faculty provide individual feedback to students, grade assignments, and publish exemplary work on the web (Levin, Buell & Levin, 1999). In some courses, faculty members have used CourseInfo, a course management system. In each course, the instructor first thought about the type of interaction he/she wished to have with the students then picked a technology that best handled the interaction.

CTER OnLine faculty members have found that utilizing multiple technologies and multiple learning frameworks allowed them to create a learning environment in which web pages, communication tools, and electronic portfolios were woven together relatively seamlessly. The instructors have combined multiple methods of instruction to enhance student learning (Levin, Levin & Waddoups, 1999). The frameworks developed for online asynchronous discussions have allowed students flexibility to participate on their own time. Students currently teaching in K-12 and college settings have been given the opportunity to implement their coursework into their own classroom practice. Simulations have been used to provide experiences dealing with sensitive topics and issues. Various grouping arrangements have allowed students to learn from one another. These multiple methods have been used to also demonstrate how current educational theories on learning can be incorporated into practice.

Course evaluations.

Student satisfaction was assessed using the University of Illinois’ Instructor and Course Evaluation System (ICES) that is used to evaluate all campus courses. Students were asked to rate the overall quality of the course on a five point Likert scale (5 = exceptionally high rating).

In general, CTER students were satisfied with the CTER OnLine courses. Of the eight courses taken by the first set of CTER students, the ratings for the courses ranged from 4.80 to 3.05, with a mean rating of 3.83. This pattern of course evaluation rating corresponds closely to the general pattern of course evaluations on on-campus courses at UIUC.

Program evaluations.

In addition to the evaluations of each course, the CTER students filled out three surveys designed to help evaluate the overall program. They filled out a "pre-survey" in June 1998, a "mid-survey" in June 1999, and a "post-survey" in May 2000. Part of each survey was a self-evaluation of the students' technical skills. As you can see from Table 1 below, their self-ratings of skills rose in most all cases as the program progressed.

CTER students were also asked to rate their attitudes toward technologies. As shown in Table 2, they started out largely positive toward technology (as shown by strong agreement with questions 1, 4 and 5 and strong disagreement with questions 2 and 3) with only minor changes over the course of the program.

In addition, students were asked to rate their attitudes on relevant issues related to different approaches to learning. For all questions except 3, 7, and 11, there are mostly minor changes in the mean values. The biggest changes are a increased disagreement with the superiority of the lecture format (#3), an increased appreciation for their abilities to budget their time and prioritize (#7), and an increased disagreement with the ineffectiveness of the web (i.e. an increased appreciation for training via the web) (#11).

Finally, the CTER students were asked to directly rate their satisfaction with CTER OnLine. More students said the program exceeded their expectation than who were disappointed. More said that they spent more time working on their Master degree than they expected. Finally, the students gave the program a grade of A-/B+.

Faculty were interviewed over the course of the first two years of CTER OnLine and generally expressed satisfaction and excitement about being involved with CTER OnLine. For several of the faculty with considerable technical expertise, this was a chance to put into practice innovative ideas that they had previously developed on a much smaller scale. For faculty with less technical expertise, it was an opportunity to learn a range of educational technologies in a supportive context.

One important factor contributing to faculty satisfaction has been the monthly CTER faculty meetings, which allowed an open discussion of experiences, ideas, and plans. The faculty with less expertise used these meetings as a way to start thinking about which of the range of technologies to plan to use; those with more expertise used it as a way to reflect on experiences and to plan for future uses. Out of these meetings came a sense of an overall "CTER philosophy", and shared lessons about how online courses, especially with adult learners, is different from face-to-face courses. For example, faculty discovered that CTER students were generally more willing to challenge faculty members than on-campus graduate or undergraduate students. Each faculty member initially thought that the challenge was unique to his/her own course, but was surprised to learn that student challenges of various sorts reoccurred across most courses. The "consumer mentality" of many of the students was a source of considerable discussion, especially when thinking about the future directions of education and of universities more specifically. This refers to the ability of students to "shop around" for courses and program and the tendency for this to increase the likelihood that students will superficially judge a course or program on its ability to provide immediate benefit while ignoring possible long-term benefits.

Another support mechanism that contributed to faculty satisfaction has been the monthly Ed-Online meetings coordinated by the Office of Educational Technology in the College of Education. In these meetings, the two different online master of education programs in the College of Education have discussed their experiences, lessons learned, and have demonstrated new technologies and techniques. This has provided a broader context for faculty in each program to understand their teaching and the nature of online media.

CTER teaching assistants have provided technical support to CTER faculty and students. This support has been crucial for faculty satisfaction with course development and course implementation. Technical problems have occurred, but the technical support staff has been able to solve these problems quickly and with minimal disruption for the faculty and their courses.

Another reason many of the initial faculty members participated and were satisfied with their participation is that teaching in CTER fit into their program of research. This reason may be less salient as more faculty members become involved whose research interests lay in other domains.

The drop-out rate of CTER students has so far been zero. All 26 of the students who started in the summer of 1998 graduated in May 2000, and all 29 current students who started in the fall of 1999 have remained in the program as of May 2000. While we do not expect this rate to remain at zero forever, it is considerably lower than most other online programs (and considerably lower than most face-to-face programs as well). Why has this drop-out rate been so low? Based on our evaluation so far, there are four contributing factors. 1) Students formed informal support groups, so that they could work together on course work. Many students said that they would not have made it without these informal support groups. 2) The program director and program coordinator have provided a "back channel" for communication for the students. This gave the students a feeling of continuity over time and a feeling that if they had a problem they had a potential outlet. 3) The ongoing evaluation provided feedback to improve the communication and support between students and faculty. 4) Flexibility in the program has allowed students to shape the program somewhat to their own specific needs. This flexibility is being institutionalized in the program, which is now defining a set of core required courses and elective courses.

The CTER OnLine courses have been largely project-based. Across the eight courses taken by the first group of CTER students, there have been a substantial number of exemplary projects created by students, which are important indicators of the success of the online program.

In the very first course the CTER students took during Summer 1998, they were introduced to the concept of an "ePortfolio", a web page that they created from a template and updated throughout the program that linked to their course work. In this first class, students conducted a major project, integrating technology into a lesson that they could use in their own classroom in the fall. Three students in that class choose as their projects the development of frameworks for having their K-12 students develop ePortfolios. It is a good sign when students who are practicing teachers actually take something that they themselves do as students and integrate it into their own teaching practice.

One of our case study students extended this ePortfolio concept to apply to her first grade students. They each developed an ePortfolio in PowerPoint, and then displayed their ePortfolio at the first parent-teacher conference of the semester. (See http://lrs.ed.uiuc.edu/students/jacobs2/ci335/stickies.html .) She, her students, and the parents of her students all judged this use of technology to be appropriate and powerful for supporting student learning.

Another good example of exemplary class projects is the set of "white papers" developed by groups of CTER students in their course on "Ethical and Policy Issues in Information Technologies". Groups wrote "white papers" on all of the major topics covered in the course: Access Issues; Credibility and Web Evaluation; Free Speech vs. Censorship; Privacy; Commercialism; Intellectual Property, Copyright, and Plagiarism; Computer Crime and Technology Misuse. These white papers were subsequently published electronically (see http://lrs.ed.uiuc.edu/wp/ ) and have been accessed widely by educators around the world -- during March 2000, there were 7357 accesses to these white papers from people in 36 countries around the world.

The economic viability of online programs is directly related to their sustainability. Therefore, the cost of providing an online program has been a major concern for all levels of administration. A number of issues related to viability were raised:

The University of Illinois administration provided "seed" funding to colleges and departments to begin developing online programs with the agreement that the programs be self-sustaining after the initial funding expired. Given this stipulation, the Dean, department chair and program director wrote budgets and estimated tuition revenues. It quickly became apparent that the cost of developing and sustaining an online program was not possible under the current off-campus tuition rate and it was suggested that a professional tuition rate needed to be established for online programs. The University trustees approved a new tuition rate that eased the budget burden.

While it was difficult to estimate all costs and the percentage used by online courses, college and department administration came to an agreement about online program budgets. Program budgets included the cost of program administration and secretarial staff, technical and course development support, equipment and supplies. Some of these costs are expected to change over time and budgets must be reviewed annually. Faculty members have been expected to teach courses on-load or as part of the department’s summer budget.

In the past two years, online education has been highly publicized. Virtual universities were sprouting up, higher education institutions were jumping on the bandwagon, and private companies were advertising online courses and degreed program. While faculty members have received increasingly frequent requests from prospective students interested in a masters degree program online, the question of how much of a demand is there for an online program was unknown.

The College announced the development of CTER OnLine to several school districts throughout Illinois that had previous ties to the university. The announcement was sent in March 1998 that we would be accepting students beginning in the summer of 1998. Forty-five students applied to the program by May, 1998 and 26 students were accepted. Due to the initial online course offering and the evaluation research component, faculty members insisted that class size be kept to a reasonable amount. Since the initial announcement in 1998, we have received over 700 requests for information about the program from all over the world. During the second year of the program, 29 new students were admitted. All 26 students in the first cohort stayed in the program and graduated in May 2000. We have not had any students drop out of the second cohort and we are expecting to admit a third cohort in Fall 2000.

Department capacity refers to issues of developing and conducting an online program. We attempted to answer the following questions.

• What does it take to build capacity?
• What are the barriers and possibilities?

• Is there a meshing of these online course offerings with other department programs, on campus courses and the like?

Department faculty members play a vital role in how online courses and programs are developed and offered. CTER OnLine was fortunate to have a group of faculty members who were willing and able to get this started. At the department level, faculty members were given permission to teach an online course on-load. Issues of equity arose when some departments offered faculty members release time for course development when other departments insisted that faculty members continue to teach their normal course load. The same issues of equity arose across the university as different colleges offered a variety of incentives for faculty. A university-wide committee was formed to discuss the issue of faculty salaries and incentives and make recommendations to the administration.

After the first year, CTER OnLine began to offer additional courses and new faculty members participated. These additional courses not only included more faculty members in the online initiative, but also allowed students more flexibility in the courses they were able to take. As the initiative moves into its third year, we continue to develop and teach new or revised courses in an online format. From our initial eight course offerings, we have now expanded to offering five required and nine elective courses.

Faculty members from four departments volunteered to develop and teach courses in the online format. This directly related to the sustainability of online efforts in the College that led to another set of questions:

• What department, college, and university resources are being used and created to insure ongoing capability to conduct online programs?

CTER OnLine drew support from resources at the department, college, campus, and university level. The Office of Academic Outreach provides assistance with course registration, marketing, off-campus library services, server space and student accounts. The UIUC campus provides these services for all off-campus courses. UIUC also supports the Center for Educational Technology that provides course management systems and maintains course information for both on-campus and online courses.

The Office of Educational Technology in the College holds monthly Ed-Online meetings for all faculty and staff involved in online courses throughout the College. These meetings are an opportunity for different online initiatives to share their experiences, both good and bad, with one another. This office also maintains a Faculty Development Center, providing all College faculty members with training on a variety of educational technologies.

Most of the initial faculty members in CTER OnLine were proficient using technologies and assisted other faculty members who were less experienced or uncomfortable with teaching in an online format. Monthly scheduled meetings of CTER OnLine faculty and staff were also held to discuss the technologies being used and problems that have risen, jointly finding solutions to the problems. As an incentive, CTER OnLine faculty members were given additional teaching assistant support for two semesters in order to develop and teach the online course. In addition, faculty members who wanted to work from home were given higher speed modems and an outside Internet Service Provider to use when university access was slow.

Instead of offering four half-unit foundation courses for CTER OnLine, two one-unit courses were developed and offered, reducing the number of courses for a Master of Education degree from ten to eight. Two departments are affected by providing these online courses. In one department, the course is taught on-load, in the other department, the course is taught in the summer. As the program continues, we are seeing a greater demand for foundation courses taught in an online format. Staffing issues are being addressed. The home department of CTER OnLine also provides office space, computers and servers for the program and technical support graduate assistants and secretarial support to process new student applications.

Over the past two years, the College infrastructure has seen extensive growth in the area of technology. New computers were purchased for a student computer laboratory and for the Office of Educational Technology, the building network is in the process of being upgraded, and the Faculty Development Center was started. Resources provided CTER OnLine at the College level through the Office of Educational Technology include server space, backup maintenance, classroom and computer lab space for face-to-face meetings, equipment checkout facilities, and overall coordination of technology used for both on-campus and online courses.

Based on data gathered from most of the stakeholders, CTER OnLine has been successful during its first two years. What lessons can we learn from the nature of the success, both to improve this online program and for online programs more generally? There are several major lessons that we can generalize from our experiences, that relate to broader theoretical frameworks developed in cognitive sciences.

One general aspect that stands out from our research on CTER OnLine is the importance given to the active learning community that was created and maintained over the first two years of CTER OnLine.

Community Building.

Sometimes a learning community will "spontaneously" emerge from a given course or program of courses. But many times such a community fails to grow. In CTER OnLine, explicit efforts at community building were taken from the very beginning and community maintenance efforts continued throughout the first two years. All of the students and many of the initial faculty participated in a two-and-a-half day face-to-face orientation in June 1998. A wide variety of techniques were used to help establish the community: nametags, icebreaker activities, group activities, social activities, welcome greetings from the Dean of the College and others, taking a digital individual picture and establishing an personal website with a template eportfolio for each student, group pictures posted on a CTER website, etc. These techniques contributed to the idea that the students were part of something larger, CTER OnLine.

Community Maintenance.

Once a community is created, sometimes it will "spontaneously" grow and reach maturity. But often a new community will wither and die. In CTER OnLine, explicit efforts at community maintenance were taken over the first two years. A wide variety of techniques were used to maintain the community: high interactivity in courses (both with faculty members and TAs, and with other students), substantial and timely support (technical, administrative, and social), active involvement by the CTER director and the CTER coordinator to provide alternative channels of communication rather than just through the course instructors, etc. These techniques reinforced the idea that the students were members of CTER, not just fellow students in a course or set of courses.

The importance of communities of practice for the professional development of teachers is becoming increasingly clear in the literature, and the evaluation of CTER OnLine presented here certainly supports that concept.

Recent cognitive science research has clarified the nature of expertise in a given domain. Expertise consists of having multiple ways of thinking about a given domain, multiple conceptual representations that are coordinated so that the expert can move from one representation to another. In addition, the expert has the "meta-expertise" to know which conceptual representation to select at each point in the process of dealing with the domain and when to switch from one to another (Levin, Stuve, & Jacobson, 1999).

This conception of expertise was used in CTER OnLine in two major ways, multiplicity and connections to contexts of practice.

Multiplicity.

In CTER OnLine, we explicitly built "multiplicity" into the program and the courses (Levin, Levin, & Waddoups, 1999). We used multiple media of teaching and learning, multiple instructional formats, multiple student activities, multiple student assessment and program evaluation techniques, and multiple context for learning. This use of multiplicity was motivated by the goal of helping students acquire multiple coordinated conceptual representations of the knowledge domains.

Connection to the students' contexts of practice.

The courses in CTER OnLine were largely project-based in design, with student projects in most cases selected by the students to have strong connections to their own teaching context. In this way, students could learn and become expert with abstract conceptual representations in a number of different domains, but with a strong coordination with the concrete conceptual representations they have developed in their own teaching practice. Through this explicit "connection" process, the knowledge and skills that the CTER students acquired is much more likely to "transfer" to their ongoing practice. Their learning was also much more motivated, as they could see how and why they needed to learn the knowledge and skills in the courses of the program.

Current theories of learning have pointed to the importance of the multiple contexts within which the learning occurs. Not only are the CTER students' contexts of work important, but also the other contexts as well (family, community, etc.). These students are busy adults, and their identity as "student" is only one of many. We have found that identity formation, extension, and maintenance mechanisms play an important role in student learning (Waddoups, 2000; Waddoups, Levin, & Levin, 2000). Recognizing these multiple identities and the constraints that they put on student involvement is crucial for a successful online program. This is why most of these people chose an online program, because of the flexibility it provides in time and place of learning.

Given the extensively evaluation of the first two years of CTER OnLine, we are now in a position to make a set of recommendations for future action. Some of these are somewhat limited to online programs with a similar set of goals and students as CTER OnLine, but many are relevant more generally.

The first set of recommendations relate to the importance of a functional learning community for creating a powerful learning environment.

• employ systematic group formation techniques
• employ systematic group maintenance techniques
• provide for flexibility so that community members can shape the program to fit their own needs
• provide for adequate support for participants in the learning community, from multiple levels of the university
• arrange for the most important mediator roles to be effectively performed

The potential conflict between providing a cohort model and providing flexibility has been resolved for CTER OnLine by moving to a "semi-cohort" model. In this model, all students in a "cohort" go through a common orientation and two core courses at the beginning, then have elective courses to choose among, but then come back together as a cohort for the remaining core courses at the end of the program. Other online efforts may well have other solutions to this issue.

The second set of recommendations relate to the nature of expertise and current learning theories based on this conception of expertise.

It is important to consider the value gained vs. the costs incurred for any given choice of:

• instructional media

• asynchronous vs. synchronous interaction

• place (distant vs. face-to-face)
• use of groups in the instruction
• use of multimedia
• use of simulations vs. "real" activities vs. exercises

For any of these considerations, it is important to consider the value of simplicity and consistency in instructional design vs. the value of multiplicity in encouraging the development of multiple coordinated conceptual representations in the domain of expertise.

The final set of recommendations relate to the importance of systematic program evaluation in guiding online education efforts. Especially when developing and implementing educational programs utilizing new media, it is critical to build in mechanisms for providing guidance and corrective feedback.

• implement systematic program evaluation
• use the evaluation formatively along the way to improve the program
• learn by contrasting things that work well with things that do not work so well

We hope that this paper provides support for this final point, in that the systematic evaluation of CTER OnLine has provided a valuable set of formative and summative outcomes that have improved the program while providing guidance for future online efforts.

We would like to thank UI Online, Ed-Online, the College of Education, and the Department of Educational Psychology at the University of Illinois for their support of CTER OnLine and the evaluation reported in this paper. We also would like to thank the CTER OnLine students, faculty and support staff, without whom CTER OnLine would not be such a powerful context for learning and research.