FINAL EVALUATION PROJECT
Patty Chiles
Douglas School
444 W. Reynolds
Springfield, IL
In the summer of 1998 I began a masters program with the University of Illinois in Champaign, Illinois. In the first two summer courses I developed a unit incorporating the teaching of basic computer skills with the teaching of aggression replacement training. The unit consisted of seven lesson plans.
In the fall of 1998 I began the course Evaluation of Instructional Technologies. One of the course assignments was to find a program to evaluate. I chose to evaluate the program I had developed over the summer.
The main text for the Evaluation of Instructional Technologies course was Program Evaluation - Alternative Approaches and Practical Guidelines by Blaine R. Worthen, James R. Sanders, and Jody L. Fitzpatrick. This text provided me with the knowledge I would need to begin an evaluation of my unit.
I had absolutely no knowledge or experience with conducting an evaluation. Before beginning this project I had a biased view of the outcome most formal evaluations produce. I had, in the past, always thought of evaluations as being value oriented, more judgmental than helpful.
Before presenting the evaluation of my own project I would like to clarify some of the issues of evaluation. Page five of the Worthen text states that evaluation is the identification, clarification, and application of defensible criteria to determine an evaluation objects value (worth or merit), quality, utility, effectiveness, or significance in relation to those criteria. This quote sounds so simple, yet I re-read it several times.
I believe that everything has value of some sort, but how we decide its significance or effectiveness is where the role of evaluation comes in to play. The Worthen text continues with a discussion of formative and summative evaluation. The authors state that formative evaluation is conducted to provide program staff evaluative information useful in improving the program (pg.14).
After reading the definition of formative evaluation I had to stop and assess my own thoughts about evaluation. I thought I had no experience with evaluation, but I had always thought of evaluation in a summative format. As an educator I use evaluation every day. I use evaluation to improve my own skills or the skills of my students.
It was as if a light bulb clicked on at this point. Previously I had a negative view of evaluation. I had always thought of evaluation as being a collection of data manipulated around to suit the purpose of the evaluator. The thought of using evaluation to change and improve appealed to me.
I dont feel that I am in the minority with my view. The authors comment that unfortunately, far too many agencies conduct only summative evaluation of their programs. The summative evaluations are what we hear and read about. We are bombarded with statistics that can be manipulated to present either pro or con. These are the evaluations that can make or break a program, not improve it.
I knew at this point that I wanted to conduct a formative evaluation of my unit. I believed that I had created a unit that had value and worth, but I was open to finding ways to improve it.
As the course progressed I knew that I wanted my evaluation to be qualitative. On page 68 of Worthen, the authors cite the description of qualitative inquiry by Schofield and Anderson to be the most useful. Schofield and Andersons view of qualitative inquiry generally
In the unit I had created I had also developed an assessment tool for each lesson. Due to the context of what would be taught the assessments were created to be either pass or fail. The qualitative approach would fit exactly with my needs.
Worthen lists six areas of classification for evaluation approaches. The closest to my needs for this evaluation I felt would be the expertise-oriented approach. This approach depends primarily on the direct application of professional expertise to judge the quality of whatever endeavor is evaluated. I dont consider myself to be an "expert" in any area, but the evaluation of this particular unit would be very subjective. The decision of a pass or fail mark would depend primarily on my decision as an educator to whether or not the concepts of the aggression replacement material were comprehended.
At this point in the course I felt ready to begin the evaluation of my unit. I knew I would make many mistakes, but I was beginning the evaluation on a positive note. My personal goal was to be able to decide ways to improve the unit.
I teach at Douglas School. Douglas offers an alternative educational program. Part of the school consists of self-contained behavior disorder classrooms. The other part of Douglas offers alternative education. I teach science, literature, and aggression replacement training to approximately thirty junior high students in the alternative program.
Each student coming to Douglas goes through an intake procedure. Through the intake procedure, cumulative folder, and home school records I often know the goals to be obtained for the students aggression replacement training. I have no initial knowledge of the students ability with computers.
After reading the Worthen text I decided that I needed to administer a pre-survey to assess my students attitudes and abilities in the area of technology. I also decided that it would be beneficial to me to conduct individual and group interviews in regards to technology.
I started my evaluation with a pre-survey. I tried to keep my questions simple and open-ended. I asked the students questions such as; what are the positive things about computers and what are the negative things about computers in the educational setting. I asked the students to list everything they could think of that they had ever done on a computer.
The pre-survey was probably one of the most valuable tools in my evaluation process. In regards to evaluation it would provide me with a more tangible way to assess learned skills. As an educator it prepared me to approach the unit at a pace that would be most appropriate for my students.
After the pre-survey I conducted individual and group interivews of technology skills.
During the group discussions students tended to over state their computer abilities. In the individual interviews students were more open to sharing their lack of computer experience.Less than twenty percent of my students have access to a computer in the home. In response to listing computer skills I learned that the majority of my students had experience mainly with skill and drill applications. All of my students were able to list positive aspects to computer use. Most of the students saw the computer as being "fun". The negative aspects of computer use ranged from explicit chat rooms to restricted access to the internet in the school setting.
Lessons one and two of my unit centered around the skill of introducing yourself and the computer skill of word processing. I had created assessment tools to determine if these skills were satisfactorily acquired. Through observation I realized the importance of adding a role playing component to the introduction skill.
Lessons three and four of the unit centered on communication skills. Students needed to have the ability to open Netscape and enter the e-mail accounts we had created. Lesson three focused more on the skill of using e-mail as a communication tool. Lesson four had students making positive communication with friends and fellow students. The assessment I created for these lessons allowed me to determine that students had mastered the skills to be learned. The observation of the lessons went beyond the actual assignments.
Through observation it was apparent that students enjoyed writing when it was student directed. The majority of my students usually object to any writing assignment. Students would ask for extra time to be able to write to someone. Instead of writing being viewed as an assignment, it was as if it was a reward. Some students would even request additional help to learn how to use spellcheck with the e-mail program. The lessons were so popular that we have kept the e-mail accounts open.
Lesson five of the unit dealt with empathy. Students worked in small groups to offer positive, empathetic advise. The groups were given three fictional letters to respond to. The groups then worked together to compose their advice column. The assessment I created for this lesson allowed me to assess that the assignment was completed in an appropriate manner. It did not allow me enough information about individual input. I was only able to determine individual participation through observation of the groups.
Lessons six and seven both concentrated on teaching the skill of anger management for the aggression replacement component. The computer skills component of the lesson was to use a slide show program and to learn to use and copy graphics. We did not get the slide show program installed on the building computers, so changes in the lesson had to be made.
I did not see this as a failure of the lesson, but as an adjustment in plans. Most educators realize that these occurances happen much more frequently then we would like, but we must remain flexible. Instead of the slide show students created banners to teach the skill of P.E.A.C.E.
The unit ended with the seven lessons, but I continued with additional lessons. I wanted to teach the skill of using the quick cam and created a lesson where students would create simple resumes. This was a failure. The lesson failed due to poor planning on my part. The students were uncooperative and had the "I don't have a clue what you are talking about look" written all over their faces. I changed the lesson to using the quick cam and a writing assingment that focused on the skills each individual would need to acquire to return to his or her home school. Every student participated in the revised lesson. Students who were absent on the day we did the lesson even sought me out to ask for the opportunity to complete the lesson.
The unit has had a positive impact in other subject areas. Each year in literature we do a section on poetry and I include an assignment of writing poems. Students requested computer time to use ClarisWorks to type their poems. Instead of being "told" what to do students would ask how to change font styles and sizes, and how to use spellcheck, or add graphics.
In another aggression replacement training skill that focuses on making thank yous some students requested to create thank yous on the computer. Some of the students decided to create without the computer, but it was rewarding to see that some of the students asked to use technology. I even observed students asking other teachers in my team to help them with using computers to complete assignments.
I have no fancy charts or graphs to offer on my evaluation. As I was working through the unit I often doubted myself. I had kept the computer skills basic enough that they could be mastered by all of my students. I felt confident that each student comprehended the aggression replacement components for the lessons. If the students would "use" the aggression replacement skills taught would be another evaluation in itself. But, I did feel satisfied through the assessment sheets, through interviews, and through observation that my students had learned from the unit presented.
I referred back to the Worthen text for evaluation models. On page 101 Worthen discusses Alkin's evaluation model. Alkin (1991) makes four assumptions about evaluation in his model:
As I looked over the four assumptions offered by Alkin I felt that I was able to relate my own evalution to these four assumptions.
The text also encourages the evaluator to assess the strengths and weaknesses of the evaluation. As I evaluated my unit I could see both. It was a strength to be able to use a variety of methods to evaluate. Evaluation through interview and observation were invaluable, but the weakness would be that they are also very subjective in nature. Another educator could teach the same unit to the same class and not feel that the responses to the aggression replacement components were satisfactory. It is also a weakness of my evaluation with so much being subjective and done with observation to make comparison studies from year to year, or class to class.
Overall I feel the evaluation method I chose for this particular unit was the most appropriate. The unit was teaching aggression replacement training skills in conjunction with basic computer skills. My students do not receive a letter grade for aggression replacement training. These skills are taught on a daily basis. This evaluation approach allowed me to assess each student in a manner that took the student's ability level and behavior needs into consideration.
The assessment sheets I created allowed me to determine if certain skills had been obtained, but not much more. The interviews and observations allowed me to reflect on a daily basis if the students were comprehending the lesson. Observations allowed me the opportunity to reteach concepts that needed clarification.
I realized that I evaluate on a regular basis and that I have been doing it for years. I believe that most educators also evaluate on a regular and consistant basis. The assignment for this course just made me more aware and critical of the steps I use.
I was pleased with the results from teaching my unit. I will use the unit again next year, but next year will be different. I will have different students with different needs and abilities. I will use the same type of evaluation methods because it will allow me to change and improve as I teach.
It was important to me that my students felt successful as they worked through this unit. It was important to me that their attitudes towards technology remained or became positive. What a nice reward for me that upon completion of the unit that I felt I had achieved these goals.