The College of Education received funding from the Vice President’s for U of I Online initiate two masters degree specializations, one in Curriculum, Technology and Education Reform (CTER), geared to practicing K-12 teachers and administrators, and the other in Human Resource Education (HRE), aimed at human resource professionals. One purpose of this evaluation is to provide formative feedback to faculty members, support staff, and department administrators to improve the development and delivery of online degree programs in the College of Education. This evaluation will also account for the use of departmental, college, and university resources in developing and delivering online degree programs. In general, the purpose for this evaluation is to determine the effectiveness of these online programs and the extent to which these programs are developing capacity which will lead to their long term viability. In the following proposal, we summarize the background of Ed-Online, identify the stakeholders, identify a specific evaluation design, and specify a timeline for completing the evaluation procedures.

The Curriculum, Technology, and Education Reform (CTER) masters degree specialization is designed to provide practicing teachers with the opportunity to earn a coherent high quality masters degree at home or at their local school through using personal computers and Internet connections. Over the past several years, the Illinois State Board of Education has revised the state’s pre-college (K-12) curriculum standards. One focus of these standards is the use of information technology to facilitate a project- oriented approach to learning and instruction. The College of Education has received increasingly frequent requests from practicing teachers interested in a masters degree program to upgrade their knowledge and skills in these areas, but who are unable to do so because their schedules prevent them from attending on-campus classes at the University.

Stakeholders for the CTER and University level administrators, College of Education (COE) administrators, C&I Department heads, COE faculty members, and students in each masters degree specialization. Although all stakeholders interests are important, those involved in the daily decision making (department heads and faculty) will be considered the primary stakeholders for this evaluation. In this section, we identify the stake holders and specify their interest in the development and delivery of online degree programs in the College of Education. The stakeholder’s interest in these programs will guide the selection of evaluation questions and data collection procedures. In addition, the stakeholders are the audience for whom formative and summative evaluation reports will be written.

(A) University Administration: Associate Vice President for Academic Affairs

The Associate Vice President for Academic Affairs, Burks Oakley, released a RFP in December 1997 for the development of online post-baccalaureate programs at the University of Illinois. Due to ongoing research and outreach efforts in the area of technology, training, and education, faculty in Human Resource Education and Curriculum and Instruction were well positioned to respond to this RFP and to receive funding for the development of an online masters degree specialization in their fields. The Ed-Online proposal was funded because it was determined that the College of Education had adequate personnel, technical support, appropriate technologies, and facilities needed for the delivery of online courses. In addition, there were plans at the department and college level for the programs to become self supporting and a regular part of the departmental teaching efforts (i.e., outlasting the project period, and not dependent upon long term involvement of key faculty).

Given these criteria for funding Ed-Online, Burks Oakley and those affiliated with UI Online will be interested in the ways in which CTER develops, support, and deliver their online masters degree specialization. In particular, they will be interested in the ways in which departments develop capacity to integrate these programs into their departmental teaching and service missions.

(B) College of Education

A second set of stakeholders for this evaluation are administrators in the College of Education. Dean Mildred Griggs and Associate Dean Susan Fowler are supportive of the development of Online programs in the College of Education. They are interested in increasing departmental and college capacity, maintaining high academic standards, and utilizing college resources wisely. In terms of college capacity, COE administrators have a stake in how satisfied faculty members are with the time and effort necessary to put a course online. COE administrators are also interested in how departments who offer foundation courses respond to requests to develop and deliver online courses. In addition, COE administrators have an ongoing concern in maintaining high academic standards in both online and more traditional degree programs. Finally, administrators in the College of Education are interested in the cost per instructional unit to deliver an online degree program. More specifically, they are interested in the amount of time needed to develop and delivery online courses, the kind of hardware and software needed for online courses, and the support staff that is required for providing quality online programs.

College of Education administrators will be interested in the formative and summative evaluation of these on-line masters degree programs to facilitate decision making concerning the ongoing investment of resources in these and other online programs. In addition, they will be interested in identifying potential barriers for developing capacity at the college and departmental level.

(C) Department Administrators

Department administrators share many of the concerns of college administrators: however, their concerns are focused on issues within their specific departments. For example, department heads are concerned about the amount of money spent on personnel ( e.g. faculty and staff), graduate student assitantships, and on computer technologies (e.g. hardware and software). In addition, they are concerned about maintaining stature for the department by providing quality programs. Finally, they are concerned that faculty members are satisfied with their online teaching experiences.

Department heads will be interested in the aspect of the evaluation that documents faculty and staff time spent in developing and delivering the program, the amount of additional hardware and software necessary in the development of these programs. In addition, as administrators in the department, faculty satisfaction in teaching online and student satisfaction in learning online will be of interest to the department chair.

(D) College of Education Faculty

Faculty members in the College of Education have a stake in the design, development, and delivery of on-line graduate programs. Faculty members who teach in the CTER specialties will be particularly interested in this evaluation because they will spend a considerable amount of time learning new technologies, developing course content, and interacting with students via the use of information technologies. In particular, the faculty member who is teaching will be interested in the mid-semester formative evaluations his/her online course so he/she can make mid-semester corrections in the content and delivery of their classes. In addition, the ongoing evaluation will provide formative feedback for instructional designers as they make decisions about the development of future classes.

Those COE faculty members who are not participating in either degree specialization are also stakeholders because the decisions made will set precedents in the College of Education that influence the allocation of funds for teaching online classes and the use and purchase of computer technologies within the college. Understanding their perceptions of these online programs will be important for the political viability of these and future online programs in the College of Education.

(E) Students

The students are another set of stakeholders that have vital interest in the development and delivery of these online masters degree specializations. The students are the clientele for whom these programs are aimed. Indeed, students in each cohort group are expending a great deal of time, money, and energy to complete their masters degree program. The students are concerned about the quality of interaction between them and their instructor, the quality of interaction between and among their classmates, and the ways in which this information is useful for their current jobs.

The stakeholders’ concerns have framed our selection of evaluation questions, the selection of data collection procedures, and will be the audience for whom our evaluation report will be written.

(A) Evaluation questions

The evaluation questions have been assembled through transforming the stakeholder concerns into specific evaluation questions. (See Appendix #1)

• Does the online learning environment facilitate easy exchange of information between and among students and the instructor?
• Does the online learning environment provide students with access to all information necessary for completing the class? (library, student services, content material).

Questions about instructional design are of interest to department administrators, faculty members, and students.

• Does the instructional design encourage active learning , peer learning, and group collaboration?
• Does instructional design encourage learning in realistic contexts and practice opportunities that promotes transfer of knowledge?
• Does instructional design provide for presenting the material to be learned in a variety of forms (e.g. text, graphics, simulations, figures, pictures, sound).
• What proportion of the class is synchronous versus asynchronous?
• To what extent does curriculum content match the medium?
• How does curriculum meet the expectations of students? If it does not match expectations do students’ have a positive or negative reaction to the curriculum
• How relevant is content to students’ careers?

• How does students’ technical background influence their participation in the online classroom and their "feelings" about the program.
• What is the impact of this new kind of course delivery on students? (impact can be measured in terms of transfer of knowledge into work context)
• Is there evidence that students are transferring knowledge into their work?
• Are students satisfied with online instruction? What do they like about online instruction? What do they think is missing?
• What kinds of technical support do students need? Are their technical support needs being met? Are they able to contact someone when they have a technical problem?
• Are students able to access the library and other university resources as needed?
• Are students able to construct relationships with a faculty advisor? Is this something students feel should be an integral part of their masters degree program?
• Do students feel connected to other classmates and the teacher? What is it about the content and delivery of the course that does or does not promote this connectedness?
• Does the suite of technology used complement the technical capabilities of prospective students (equipment, software, knowledge) and/or are aids available to guide learners?

The question concerning the process of teaching online and the extent of satisfaction with using online technologies will be of interest to faculty both in and out of the programs, department heads who must make decisions about how to allocate time and resources to cover departmental teaching loads, and college level administrators.

• What resources do faculty utilize to design their online classes?
• How much time do faculty members spend in the development and delivery of online courses? How does this compare to face to face teaching?
• Are faculty members satisfied with their online teaching experiences?
• Are faculty satisfied with financial arrangements? (e.g., buy out time, TA assistance)
• What knowledge, skills, and attitudes are necessary to plan, develop, and deliver online programs.
• Do faculty members receive sufficient training to develop sufficient knowledge, skills and attitudes toward online instruction?
• What difficulties are there for teaching in an online classroom versus a traditional classroom?

(5) Economic viability

The economic viability of these programs relates directly to the sustainability of these programs and is important for most stakeholders including university administrators, college administrators, department heads, and department faculty.

• How much does it cost to deliver an IU using distance learning technologies, versus face to face?
• How much demand is there for the online courses on campus versus off campus?
• To what extent do costs change over time?
• How much additional revenues are brought in by online programs?
• Do these programs provide opportunities for gaining additional revenue to the College of Education?
• Is the cost of using resources outside the College more or less expensive than building capacity within the college?

These questions focus on the concept of building departmental capacity to develop and deliver online degree programs. What does it take to build capacity? What are the barriers and possibilities? These questions are important for university, college, and departmental administrators. In addition, faculty members have an interest in the development of departmental capacity.

• Is their a meshing of this online specialization with other department programs, on campus courses and the like?
• Grad student support needed.

• What department, college, and university resources are being used and created to insure ongoing capability to deliver online programs?
• How comfortable do departments who deliver foundation classes feel about putting their classes online?
• Are there collaborative relationships being developed between and among faculty and staff to facilitate the development and delivery of online classes?

• To what extent has Office of Educational Technology functioned to build capacity or infrastructure for developing and delivering online classes in the College?

The data collection procedures are designed to answer the evaluation questions we articulated in the previous section. We will use a combination of survey data, interviews, and observations to determine the effectiveness of these online programs. In addition, the data collection procedures will provide information concerning the extent to which these programs are developing capacity that promote their long term viability. (See Appendix #2)

(1) Survey Data

We will design several surveys to measure faculty and student satisfaction with the teaching and learning taking place in the online classroom. We will develop surveys concerning the student’s reaction to the interface design, instructional design, the use of specific software applications, and the overall quality of the online classes. In addition to student questionnaires, we will gather survey data from all faculty members in the College of Education concerning their beliefs/reactions/opinions concerning online programs. Understanding the faculty’s viewpoint is important for the political viability and sustainability of an On-line program within the College of Education.

(2) Case Studies

The qualitative methodologies we use will include interviews of students, faculty members, and program support staff. In addition, we will observe synchronous and asynchronous activities in the online classroom. The interviews and observations will provide data for constructing rich case studies of teaching and learning within the online classroom. Through analyzing the interviews and observations we will develop case studies of student participation in the online classroom and will document their reaction to the interface design, their utilization of software, their reaction to the instructional design, and their overall satisfaction with the course and their masters degree program. In summary, student case studies will provide information concerning the teaching effectiveness and overall quality of the online learning experience.

We will also develop case studies documenting the ways in which faculty members use information technologies in the context of the online classroom. The following questions will inform the development of faculty case studies: How did faculty members transform their teaching practice to facilitate teaching in the online classroom? How satisfied are they with the teaching primarily using distance learning technologies? How much time did they spend on the program? How does the time spent compared with time in the traditional classroom? Did they collaborate with other faculty members? What resources if any did they use within the college and with other units on campus to complete their program? What software do they think works bests for their instructional purposes? These case studies will be important for faculty members who are planning to teach online courses because they provide models for teaching in the online classroom. In addition, department heads will be interested in understanding the kinds of support resources and time issues that must be dealt with for the development and delivery of these programs.

Another set of case studies will center on understanding the role and function of the development team and program support staff. The case studies of program development staff answer questions relevant to the economic viability of these programs and questions of infrastructure and interdepartmental collaboration. For example, questions leading this inquiry will be: How is the development team organized? Who makes up the development team? Do they make use of interface designs from previous classes? How much time is spent by the development team? Does the development team rely on colleagues within the college or from extramural affairs?

The case studies of the students and faculty will provide rich descriptions of the effectiveness of CTER and their ability to provide quality teaching and learning opportunities. The case studies of the development team will provide formative feedback concerning the ways in which they can improve their process of development. More generally, the development team case studies provide a detailed description of the process of developing capacity to support online programs at the departmental level.

(3) Documentation of the development of College capacity

Another set of data central to this evaluation is the documentation of college wide capacity to support departmental efforts. To this end, we will document the ways in which students, faculty, and the development team access resources from the College of Education and the campus. Understanding the development and use of capacity in the COE goes directly to the sustainability of these programs and the development of new programs.

(C) Evaluation Timeline

(1) Survey Data

The student survey data will be collected at mid-semester and in the last week of the semester. These data will be reported to the faculty member teaching the class and the support staff within one week of collection. The faculty survey will be distributed September/1998, at the end of year one, and again near the end of the program to document the changes in opinions among faculty in the College of Education.

(2) Case Studies

In addition to surveys, we will conduct case studies of four students from CTER. The student case studies will consist of three interviews (beginning, middle, and end of semester) and periodic observations of their participation in the online classroom. Near the end of each semester a case study report will be written for the development team to facilitate improving the effectiveness of the program.

The faculty case studies will consist of interviews in the beginning, middle and end of the semester. These interviews will be coupled with observation of their teaching in the online classroom. After each semester a case study report will be written for the development team and the faculty member to facilitate improving the effectiveness of the development and delivery of instruction within the CTER program.

(3) Documentation of the development of College capacity

We will also document the development of college capacity in terms of technical support and the availability of computer technology. Each semester a report will be written for the Office of Educational Technology, the C & I Department heads, the CTER development team, and key faculty embers. The purpose of this report is to document the support being utilized by faculty, staff, and students, and to make recommendations to further develop capacity in the College of Education.

The preceding proposal outlined the stakeholders, evaluation questions, data collection procedures, and timeline for evaluating online graduate programs in the College of Education at the University of Illinois. Recently, the College of Education received funding from the Vice President for Academic Affairs to develop the CTER masters degree specializations. The proposed evaluation will account for the use of departmental, college, and university resources in developing and delivering CTER. In addition, we will determine the effectiveness of these online programs and the extent to which these programs are developing capacity which will lead to their long term viability.