The following evaluation was conducted and prepared by five students enrolled in the Curriculum, Technology and Education Reform master’s degree program at the University of Illinois, Champaign-Urbana. This endeavor was a requirement of the course Educational Psychology 398-OnLine, Program Evaluation. Brian Pianfetti was the instructor of record and served as liaison between the school district involved and the participants in this evaluation.

This summative evaluation was conducted from an objectives- oriented perspective. The authors’ goal was to determine the "extent to which outlined objectives were achieved" (Worthen, et. al., 1997). It was an external evaluation as none of the evaluation team members were from within the school district being evaluated. The summative nature of this project is reflected in its purpose. The role of this evaluation is to render judgment about the mature program’s value, demonstrate accountability and document the progress and process of the InTech program. (Worthen, et al, 1997). The administrative staff and focus group will use findings to support the revision, continuation, and expansion of InTech.

Our target audiences are the InTech administrative staff, District 87 administrators, participating teachers and the program focus group members. Secondary audiences include other faculty members, parents, funding institutions and the community at large. Our audience loosely articulated the process of the evaluation we adopted by defining general outcome expectations. We understood our job to be limited in scope, focusing on three specific objectives. Our method of presentation was left open-ended but we were encouraged to use data to support analysis rather than to focus on the presentation of statistical information. Thus, our point of view is non-technical.

A major limitation of our report involves difficulties obtaining responses from program participants. Every attempt was made to communicate directly with respondents, accommodating their schedule and time constraints. However, some questionnaire responses were incomplete and one participating teacher declined to respond at all. Follow-up email correspondence was generally ignored as well. Another constraint was distance, as evaluators would have preferred to observe the participants directly. Samples of project work were solicited but none were provided. Web based project samples were difficult to identify, although some will be included later in this report. The timing of the evaluation was also an issue as the InTech program culmination has yet to occur. Some projects are still in the working stage and have not yet yielded adequate product for final evaluation. As a result, most data is rather one-sided from the participating teachers’ point of view.

The goal of this evaluation was to determine the extent to which each InTech team:

• achieved their goals as stated in their InTech proposal
• acquired and applied new skills related to technology
• achieved new objectives related to technology not stated in proposal

The questionnaire method was employed to accommodate the schedules of the participating InTech staff. Email correspondence was a prerequisite of the evaluation protocol. Although telephone communication was allowed as well, this evaluation team chose not to use this approach. Logistical issues precluded any face-to-face meetings. The research team limited interaction to email correspondence primarily in an effort to avoid compromising the cooperation of the evaluation subjects. We did not want to become obtrusive or burdensome in our effort to collect data.

The evaluation was conducted between February 26, 2000 and March 30, 2000. Electronic mail questionnaires were sent to InTech program participants and administrators. The  complete evaluation proposal is located in Appendix A . Response dates were indicated, as was the possible need for follow-up communication. Preference times and dates for such interaction were requested as well. Following the receipt of responses, the evaluation team met to discuss, compare, and analyze those responses. Specific rebuttal questions were formulated and then sent to respondents as deemed necessary. Additional evaluation team meetings were conducted at which further statistical analyses were completed. Consensus recommendations were developed and the reporting of findings was a collaborative effort among all five evaluation team members.

Since the Bloomington District 87 InTech program is the focus of this evaluation, we felt it important to include some background information to put things in perspective. The program was developed to jump start technology integration in the classroom. Funding came from the District’s 5- year technology plan and the District Technology Focus Group will determine the continuation of this financial support. This report will be an integral part of that decision making process. This is the first year of the InTech program’s three-year plan.

The program is the brainchild of Jim Peterson, the district technology director. Loosely based on a similar program undertaken in Urbana, Jim was looking for a way to develop engaged learning programs to use in classrooms as part of the technology plan. A comprehensive approach was adapted that would provide equipment, training and teacher compensation to support projects that would promote both engaged learning and technology integration. Funding was limited. Therefore, to ensure equitable distribution of resources, a grant/proposal format was adopted.

The Technology Focus Group created the InTech goals. These were a compromise of previously stated goals from the district’s established technology plan and new goals identified necessary to direct future planning.

Formative assessment so far has been "sloppy" according to Mr. Peterson. Monthly meetings provide an informal arena for problem solving but often result in more questions than answers. Short-term needs are identified and accommodated; training opportunities have been made available and additional tech support staff has been provided.

1. To what extent did you achieve the goals stated in your proposal?

While one of the respondents felt he/she "not only achieved the stated goals but went way beyond the original goals", three stated they had either already met their goals or would fully meet them by the end of the school year. Two subjects believed they met some goals, exceeded some goals, and did not quite fully meet the remaining goals. Due to inability to contact people needed for a project, preparing for ISAT (Illinois Standard Achievement Test), and having students with lower ability than expected were the causes two respondents gave for not meeting their stated goals.

2. How did your individual proposal goals facilitate the InTech program objectives?

Of the responses, 50% mentioned engaged learning as being an important factor in their proposal goals as well as InTech's objectives. One respondent communicated having modern computers in the classroom and access to the internet were project goals which are aligned with InTech goals. Various other responses included showing other building staff how to use the technology, bringing a variety of academic subjects around a central theme using technology as a communication tool, keeping parents better informed, and being innovative and fun. Of the respondents, 25 % did not think their proposal goals facilitated the Intech program objectives. Not receiving equipment and not having the room wired until late in the school year inhibited one respondent's project and goals since the particular curricular area had already been taught by the time the needed technology was in place. Another subject stated, "the InTech program objectives seemed to be in a different place than we were most of the time. A lot of time was spent on video elements"…"our project didn't involve as much video".

3. How has InTech encouraged your use of engaged learning?

Responses to this question varied greatly by grade level and type of project. The various indicators of engaged learning are as follows:

Students

• Creating articles
• Budgeting time
• Being motivated to complete assignments for audiences to view
• Working in groups
• Becoming comfortable with technology presentations and tools
• Brainstorming
• Teaching peers
• Making choices

• Becoming a facilitator along with the students
• Assessing through discussion with students
• Taking a more passive role

4. Do you believe engaged learning has improved the quality/quantity of student work? Why or why not? Cite examples.

Of the seven subjects who answered this question, all seven believed engaged learning has improved the quality/quantity of student work. The table below shows the five main areas where quality or quantity of work were improved.
 

Respondent	Audience of peers/parents is important	Motivated/Put more time into projects	Work Collaboratively	Take pride in work	Revision of work
1	x	x			x
2	x	x		x
3	x		x	x
4	x	x	x		x
5		x		x
6		x		x
7	x	x		x

Having an audience of peers and/or parents, motivation to work on projects, and taking pride in their work were the top three ways engaged learning improved student work with at least five of seven respondents cited them as examples.

5. How are you able to make your students responsible for their own work?

Only two of the eight InTech teachers grasped the concept of making their students responsible for their own work. The two teachers had detailed plans including presenting goals, indicating paths of pursuit, requiring presentations and defending them, and setting the minimum expectations or requirements. The remainder of the InTech group ran the gambit of not grasping the need to apply responsibility to resisting that it was even possible. One even responded, "Make? Make is a strong word. I would just say, ‘Let’s try to do this.’" One of the teachers did indicate they used a designed rubric for presentations. In addition, there was great commitment to the work, but no direct indications of assignment of responsibility were available.

6. Has the InTech training been beneficial to you? How?

Six of the nine respondents indicated that InTech training was beneficial to them in some fashion. The benefits derived varied from an insight as to whom to contact for assistance and what resources are available. The other end of the spectrum was the teacher who thought that they knew a lot about computers until taking the training. One quote for illuminating the value of the training is, "InTech has made me an engaged teacher again. I’m interested, excited and devote more time and energy to school." Another teacher said that InTech has provided the introduction and was an on-going resource.

One negative reply was that the training given left a lot of trial and error, and frustration. For busy teachers, the training is not viable. Another stated that they already knew most of the software and hardware that was taught, and the video editing was limited due to the fact that one camera, one computer and three classes of twenty-five students is not feasible.

One response was indifferent. "It’s given me time to think about technology."

7. Has this training been sufficient to implement your projects and goals?

Four respondents answered that the InTech training was sufficient. Four responded yes, and one responded no.

Most that answered the training was sufficient went on to say that more time was needed to perfect the skills taught. Another stated that they could always use more time. One was "blind, and lost, dazed and confused" at first, but now knows what is necessary to learn. Another respondent stated that the training regarding engaged learning needed enhancement, since this was the basis for the completion of his project.

Although the InTech Training was inadequate for one of the interviewee’s, the individual help from computer services was beneficial.

As in many cases of technology training, the participants all start at different levels. It is very difficult for the trainers to present new materials in this situation. It is important to analyze the participant’s prior knowledge, and give them individual training as appropriate.

8. What technological skills did you acquire during the InTech training?

The overwhelming response to this question (8 out of 8) was video editing and digital video. Four responded that the web page design was beneficial. Utilizing the scanner had three votes, and two mentioned CD burning. Other responses were: everything from e-mail, Internet, PowerPoint, pagemaking, digital pictures, scanning, digital video, CD burning…everything!

Four of the answers to this part of the question showed positive applications of the teacher’s skill. The answers varied from "I would demonstrate and convey the skills to the students for use in their own endeavors." "Students and parents could check assignments, projects, and cool sites." "My students will be teaching me soon." "I’m not afraid to let the children use the computer, and the camera."

One replied, "I did not apply these skills in my classroom." Three of the teachers did not respond.

9. What type, if any, InTech technical support have you used?

75% of the respondents were very pleased with the technical support they received from the support staff. They were able to get help via email, phone calls, and direct meetings with support staff members. One respondent attended an extra Saturday workshop to learn more about movie capture/editing. That person also received help from the Learning Center teacher and student teacher. One person ended up solving most of her own problems "because the technology department was not available to solve the problem."

10. Did you receive support in a timely manner? What was the average response time?

All the respondents except the one who solved her own problems were able to receive help in 24 hours or less if the help came via email or phone call. A physical presence could take 2-3 days, which left 1 respondent "feeling somewhat vulnerable to a technological meltdown that would be disruptive to my teaching." 7 out of 8 seemed satisfied with the help they received. Most were complimentary of the technical support staff, particularly Chris and Jim. The person who solved her own problems seemed dissatisfied with the technical support.

11. Why did you choose the particular technology package?

50% of the respondents chose to use Macs simply because that is what was already being used in their buildings. They felt it was easier for students to stick with one kind of computer. 25% of the respondents were partial toward PCs, therefore they chose to have that type of computer. The other 25% chose whatever their teams were most comfortable with.

12. Describe the kind and amount of technology you had access to prior to Intech?

Only 6 out of the 8 people responded to this question. Of the 6, 2 had no technology in their classrooms prior to Intech, although one of them said she had access to a computer lab with good connection to the internet. All of the remaining 4 had 1 or more older computers, many of which were refurbished and/or donated by State Farm. In addition to the older computers, 2 of the 4 people also had 2 or 3 PowerMacs in their classrooms. One person also had a scanner, a 56 baud modem hooked up via telephone line, and access to a digital camera. Another person had new ramjet computers in the computer lab, a digital camera, and an 8mm video camera that she purchased herself. The break down of technology in those 6 classrooms prior to InTech is as follows:
 

Teacher	Older Computers	PowerMacs	Access to Computer Lab	Additional Technology
1
2	several	2		Scanner modem hook-up digital camera
3	1		yes
4	2 (1 PC, 1 Mac)		yes	Digital camera 8 mm video camera (personal)
5			yes
6	3	3

In addition to the package you received, what other technology resources have you acquired to support your InTech project?

The 7 teachers who responded to this question received a wide variety of technology through intech money, additional grants, school funds, and science department money. The additional resources they now have are as follows:
 

Teacher	Computers	Printers	Software	Additional Technology
1				2 Projection devices
2			QuickTime VR	CD Burner
3	1	Color laser jet		Video camera Scanner
4			PaintShop Pro 5	Digital camera
5			Web page software	CD Burner tapes for camera and CD
6				Sony digital camera (personal) boosted the ram of another computer so it is on line also shared scanner
7	PowerMac	yes		external floppy disc drives for G3s

13. What kind of access did you have to the District 87 network and the internet prior to InTech? What kind of access do you have now?

Prior to the InTech training, only one of the participants had internet access in their classroom. All others were able to access the internet only by going to a computer lab. One participant mentioned that they were only allowed 45 minutes each week at a scheduled time in the lab. One hundred percent of the participants now have full access all day every day to the District 87 network and internet in their classrooms. They all have access to the network, which allows them to print to a remote location using a high quality laser printer. One participant was quick to point out that all teachers now have access to the internet in their classrooms, not just the InTech participants.

14. During the course of this project, what new technology related objectives did you achieve that were not stated in your original proposal?

Aside from the one participant who is still working on their project, all but one participant cited many new technology objectives that were achieved. They learned about video conferencing, video capture, video editing, web links, and posting to the web. Technology became a part of one participants' daily teaching. He is now providing information to parents and absent students regarding activities at school through a web page that grew out of the InTech experience. Another participant has correlated technology to the School Improvement plan, the Language Arts essential outcomes, and the indicators of engaged learning. A third participant indicated they basically had no idea what was possible. They learned how to use a CD burner, which has allowed the students to take home the products they make in the class.

15. What were the unexpected benefits from participating in the InTech program?

The participants cited many unexpected benefits from participating in the InTech program:

• The chance to work with others on a group effort that coordinated academic and technology goals.
• The immersion in aspects of technology that came from exposure to the program and the presence of the InTech package in the classroom.
• Seeing student excitement, motivation, lack of discipline problems because of task commitment.
• Being on the cutting edge of things in the area of technology.
• Seeing the students really interested and having a good time learning.
• Having all this knowledge available and the resources assist us when necessary.
• Learning so much and being able to use what was learned.
• Success of peer assessment.
• Becoming bolder in attempting new technology.

The InTech program-

• The training aspect needs to be more organized and reflective of the needs of the participants. It would be beneficial if the training topics were generated by the participants, and if training was voluntary.
• Not spending as much time on video early in the year. Other kinds of help are necessary.
• Once the grant and hardware are explained to the applicants, don't change the hardware after grants are granted.
• Become a two year process with the first year as an apprenticeship with someone who has done similar things and the second year for full implementation.
• Emphasize Think Quest projects with support from the technology support team.
• One participant offered no suggestions because "the training was perfect for me. It was exactly what I wanted and more."

• Have more people available for support.
• A more proactive approach to support would be beneficial because participants may need direction and advice that only someone with experience would be aware of and able to provide support.
• Support for installing and servicing hardware and software needs to be more timely.
• Continue support into the second year.
• Having more one on one help because the large group training was difficult at times.

• Center around a more defined theme in order to make it easier to provide training and support that is of benefit to all involved rather than having specialized projects that there simply isn't enough time to fully support.
• The majority of the respondents felt that the proposal process was okay. They felt it was detailed enough. One participant said the process was "gruesome, but necessary."

• The process currently appears to be one way with participants sending in information. It would be helpful to know what other participants are doing.
• Other respondents found the evaluation to be necessary and okay.

Mr. Peterson had a vision but didn’t know exactly what to expect the first year of the InTech program. He hoped that participants would develop projects that would utilize the new technology in innovative ways. Satisfied with the results so far, Jim is encouraged by the enthusiasm and growth of the teachers involved. His initial concerns regarded lack of teacher experience with the tools and network technologies. Although proposals may have reflected lofty expectations, projects have realized great success.

Web sites associated with each InTech project yield the most visual measure of success. However, Jim prefers to make judgments based on the feedback he gets from those in the field. Principals are thrilled by the excitement they see in the students and teachers involved and participating teachers continue to ask for more from the InTech support staff. Jim expects the Tech Open House on April 26 will provide the greatest arena to measure the success of the program, as students will have the opportunity to showcase the products resulting from their involvement with InTech.

Jim feels that final evaluation of the success or failure will depend on several measures. Anecdotal evidence, the external evaluation by the CTER students, and summary evaluations by the participating teachers are all important indicators of the ultimate success of the program.

Jim has already considered changes to the program. He’d like to adopt a more cohesive model to guide project planning and doesn’t feel the need to do everything in-house. He hopes to continue the grant process, provide participants with high-end equipment and paid project development time. However, he’d prefer to provide a framework for project development along the lines of the ThinkQuest model. This would provide structure for developing classroom projects that could alleviate some of the burden from the teachers.

Based on the data we collected, our team was able to draw the following conclusions to these evaluation questions.

1. To what extent did each InTech teaching team achieve their goals as stated in their InTech proposal?

Of the 8 InTech teaching teams interviewed:

• 50% met or exceeded their goals
• 25% met some, exceeded some, and did not meet some goals
• 25% did not meet their goals

100% of the teachers interviewed acquired new skills relating to technology, but 3 of the 8 teachers claimed they did not apply those skills in the classroom. The new skills that were most valued were digital video and video editing. The degree of value varied among the teachers because of varied amounts of prior knowledge. See Graph 1.

3. To what extent did each InTech teaching team achieve new objectives related to technology not stated in their proposal?

All but one of the teachers interviewed felt they had learned things that they had not expected. Some of the more enthusiastic responses were:

• "Intech has made me an engaged teacher again. I'm interested, excited, and devote more time and energy to school."
• "We basically had no idea what was possible. We now have a burner that I'm really excited about that wasn't part of the original deal. The students are able to take home the products they make! The videos are more interesting than I realized they would be. The students are able to do more than I realized they would. Students who used to skip school now stay after every night. Students love teaching others what they know. I'm impressed with web pages they make. We've done a lot of things in class with the technology that have been engaging for everyone."

Objective 1: Determine the extent to which the InTech teaching team achieved their goals as stated in their InTech Proposal.

• Some participants of InTech presented a set of goals that were so broad that they were unattainable. The training team should be more sensitive to this and provide training and sufficient information to insure that the InTech teams can set realistic goals.
• InTech goals must align with curriculum sequence.
• When writing proposals, participants should consider the impact a particular student population will have on their ability/inability to achieve set goals and  include guidlines within the proposal that allow for modications of expectations.
• The InTech personnel need to assess the technical expertise of participants to insure efficient and comprehensive training tailored to individual needs.
• Equipment must to be in place, fully functional, and wiring completed before the start of the school year so project implementation can align with curricular sequence.

• Six of the nine respondents acquired new skills (related to technology). The degree of the skill growth was most beneficial to these participants. The other three stalled in acquiring sufficient skills because they understood the program to be of a different nature. In an effort to promote greater skill growth and application, the InTech administrators must more accurately inform prospective participants as to what they can expect the program to provide in terms of training, resources and technical support.
• Allow sufficient time after training, before project implementaion, to perfect the skills taught.
• Review the training regarding learning enhancement to determine whether it needs enhancement.
• The InTech support staff must be aware of other resources available within the district so that they refer individuals appropriately when they cannot answer questions or solve problems.
• Instill InTech support with a higher sense of urgency when responding in person to the classroom.
• Emphasize the importance of the PC in business and industry and condition teachers to know the importance of training people on various platforms. People who are very competent on the MAC platform are afraid of PC’s, and visa-versa. The choice of platform should correspond to what is best for the student, not what is convenient for the teacher or school district.

• Extend the InTech training to each classroom in the District.
• Review each suggestion for improvement very judiciously and incorporate them as appropriate, just as soon as possible.
• Recognize the importance of secondary gains like improved student attitude and motivated teachers.