Case Study Report #5

by

NEF Team Davis Evaluation Team

Sue Bogren, Ascension Parish Schools, Donaldsonville, Louisiana

Megan Forness, Rochester CUSD, Rochester, IL

Lynn Gilmore, Springfield Public School District, Springfield, IL

Walt Huntsman, University of Illinois, Champaign-Urbana, IL

Michele Jacobs, Tri-City School District, Buffalo, IL

David Stone, University Laboratory High School, Urbana, IL

Formative Evaluation of Wynne County, IL/Wildwood Junior High School's Progress in Achieving NEF-Funded Technology Goals

OVERVIEW OF REPORT COMMENTS

•Purpose of the Evaluation

•Focus of the Evaluation

•Overview of Evaluation Plan and Procedures

•Presentation of Evaluation Results

•Conclusions and Recommendations

•Appendices

PURPOSE OF THE EVALUATION

Why was the evaluation conducted?

The Davis Team is an ad hoc review panel hired by the National Education Foundation (NEF) to perform an early formative summary of progress in achieving the goals and activities set forth in Wynne County's, April 1998 "Awards to Facilitate Research and Development of Technology-Based Education" proposal. This proposal was written in response to NEF's request for proposals. As an early formative evaluation team, our role includes that of short term consultants to assist in planning and anticipatory troubleshooting, if that appears to be a need of the funded organization. Wynne County's preliminary budget is available online. This plan allocates funds for technology acquisition, leadership/staffing, student training, teacher training, and curriculum development.

Audience

In order to fully understand the "Learning with Computer Technologies" (LCT) program and the impact that it will have on this school district, we felt it necessary to reach out to the WJHS stakeholders. We have gathered information by contacting several teachers, administrators, and support staff via email. We have also participated in a WebBoard chat with Lee Daniels (principal of WJHS) and Roger Wildabette (technology coordinator of WJHS). The table below summarizes contacts made thus far that were initiated by our evaluation team.

What do we intend the evaluation to accomplish?

The goals of the evaluation team are multi-dimensional. First, we intend to examine how the implementation of technology has progressed both before and during the grant period. We intend to obtain a sense of how the grant has impacted both how teachers teach and how students learn with technology. A decision has been made to focus our evaluation specifically on the aspect of teacher training. This issue is paramount based on the presumption that teachers can better implement the technology when they are equipped with more extensive training.

What questions is the evaluation intended to answer?

It is the intent of our evaluation to answer many questions as to exactly how the grant has helped the teachers accomplish goals as set forth in the LCT plan. Our questions focus primarily on how teachers are trained and the extent of this training. What teachers do with this training in the classroom is an issue that has been researched by our team. The vision of WJHS is an important issue that our evaluation team has discussed. Does WJHS have a focused plan for the future of technology in their school, or is it in need of a more definite plan?

Why was the evaluation conducted in the manner used this far?

For reasons of efficiency, most of our data has been obtained by reviewing the WJHS web site, corresponding with stakeholders via email, and chatting synchronously with Principal Lee Daniels and Technology Coordinator Roger Wildabette. Since the teachers, administrators, and support staff at WJHS are busy people with not much time to spare, we found that asynchronous communication was a true advantage. In one email correspondence, the Davis Team was actually invited to the virtual school to look through files at our leisure. We chose to decline and arrange an online meeting in lieu of the "virtual" voyage.

Limitations of the evaluation

Due to the brief time period in which we are conducting this evaluation, our range of investigations is somewhat limited. Originally, our team had hoped to focus our evaluation on the three areas of technology planning, training, and inventory/expenditures. Again, time constraints forced us to narrow our focus to just one of these topics. As a team, we agreed that teacher training seems to be one of the most important aspects of the LCT program. Thus, teacher training became the primary focus of our evaluation. Another limitation of our evaluation involves the medium through which we are collecting data. All of our correspondences thus far have taken place electronically with a computer. However appropriate this may seem to our task at hand, it is much more difficult, if not impossible, to develop a true sense of the stakeholders' feelings as they respond to our questions. Although we gleaned some cooperation, excitement, confusion, and even some bits of humor in our respondents, nothing can replace the telling eloquence of body language, which has been absent throughout our evaluation thus far.

Disclaimer

The information contained in this report has been researched and collected solely by Team Davis and reflects the opinions and recommendations of its six members. Any information presented in the contents of this report is, to the best of our knowledge, true and complete.

FOCUS OF THE EVALUATION

Description of Wynne County District and Wildwood Junior High School

Wildwood Junior High School is located in Wynne County, Illinois. It is nested between two large urban schools and has a widely diverse student population. WJHS is currently integrating their self-created "Learning with Computer Technologies" (LCT) program into their school in attempts to create a "model technology district in the state". It was mentioned that Wildwood is the "leading school" in the area of technology and was selected to apply for this grant in part due to concerns about violence. Although no specifics were reported as to the nature of the violence, it was felt that an innovative curriculum would be one way to address this concern.

The school has one computer lab with 24 computers. They do have an Internet connection, but it is not very fast. Most of their machines are Apple, and they mention that the grant has allowed them to purchase more equipment and plan new training.

One can find a plethora of information about the school online by browsing through the WJHS web site. The site itself contains a drawing of the school by a ninth grade art student. It currently hosts animated seasonal graphics that are eye-catching but in need of updating. School spirit seems to weave through the site with the presence of "flaming" torches and an emphatic, "Go Flames!!!" A link to the Wynne County School District Homepage is introduced at the top of this page to provide users with a convenient source for more information about the district. The WJHS page itself has sixteen links, twelve of which are active. One of the non-active links is the library, which seems a bit peculiar due to the fact that in many schools, the library plays a major role in the implementation of technology. The three other non-active links pertain to a "Blue Flame" award, clubs, and athletics. Seventh and eighth grade students are represented on this site by their Student Body Officers. These officers include two students who have designed their own web pages, one of which has a Spanish version available. Other links included on this site connect users to administrators, departments within the school, teachers, P.E. rules, PTA, June 1997 activities, subject area objectives and behavioral outcomes, information about the NEF grant, the 1997 School Report Card, and a schedule for teacher workshops.

Some of these links are more developed than others, and some seem to contain more "useful" information. For example, the link to the departments contains only two sublinks, one of which can be accessed through the teacher's link. An entire link is devoted to P.E. rules, and this exists within the realm of the much more broad 1997 School Report Card. Our team agrees that there may be some organizational issues that need to be addressed within this web design. At the time of the completion of our evaluation activities, this site had not been updated in over one month.

Rationale for our selection of area(s) of focus

Because the focus of our evaluation deals directly with the implementation of technology, we decided to focus our research on teacher training. Since the teachers are the main cogs in the education wheel, our team feels that it is of utmost importance that they possess the skills necessary to steer the students on their road to success.

Questions used to focus the study

• What training have teachers already received in using both the hardware and software the school has purchased?
• What future training will they receive?
• What form(s) has this training taken, and what form(s) will it take in the future?
• What kind of technical training have instructors received in web design, and what kind (if any) will they receive in the future?
• How does the web tie into the overall technological vision?
• How is technology used with your curriculum? (to specific subject area teachers)
• What kind of training or instruction have teachers received in terms of envisioning and using the web as a teaching tool? Has there been any pedagogical instruction in this area?
• What will the teachers be expected to do with the web or with the technology in terms of their teaching or the district's educational mission? What training will they receive to help them fulfill such expectations?

Information obtained thus far

Specific to the financial aspect of the proposal, WJHS has proposed a preliminary budget for 1998-1999 that is summarized in the chart below.

The Davis team has received a variety of responses from the stakeholders connected with WJHS. This link contains several verbatim responses from various stakeholders mentioned in the earlier portion of this report. It is apparent that some teachers at WJHS have substantially more technology experience than others. It was reported that many of the teachers do not need to be prodded to use the technology, although a few are still a bit apprehensive. The grant money has seemed to supply teachers with more resources, such as hardware and software which, in turn, have been causing them to "re-examine" their goals and methods. It was mentioned that this period of reexamination will continue throughout the grant period.

Use of Technology

The math department reports successful integration using Geometer's Sketchpad in the classroom. For the NEF grant, they are focusing on having students create a high school math web site, which aims to include exemplary resources in this area.

The physical education department reports that the NEF funding has allowed them to purchase necessary heart rate monitors and software needed to track this student data. They also use the Internet to research topics related to general physical education.

The English department praises the inclusion of technology with respect to its word-processing applications, which thankfully allow students' reports to be more legible than the usual "illegible hurried scrawls" of the pre-computer days. Coupled with the former positive remark was a sincere concern relating to the use of the Internet for research. Although the use of the Internet allows students to find perhaps more related material, is was reported that it, "...does not seem to drastically alter their overall outcomes when self expression and creativity are looked at... I guess I am trying to express the frustration involved with creating focused analysis when they are awash with seemingly relevant information at their finger tips. It is almost as if the technology interferes rather than contributes".

It is clear there exists a wide range of perspectives regarding technology use at WJHS.

Training with Technology

It was mentioned that staff development in the area of technology has been offered by the Wynne County School District for three years, but with somewhat inconsistent results. In conjunction with this response was a remark of the varying degrees of teacher expertise in this area. More specific questions concerning training were presented to Judy Frank and Tom Barrett. In a response from Judy Frank, it was learned that, to her knowledge, there has been no formal evaluation of technology training in the district and/or WJHS. Only informal assessments of technology training in the district have been administered. She does mention, however, that there is an intent to begin more formal evaluations of this and other technology-related areas. Our team feels that it might be more beneficial if the district were to develop a more formal procedure for evaluating this type of training.

Closer examination of teacher training reflects the need for WJHS to develop a technology plan. They have bits and pieces of planning in different places, yet a single focused plan has not been formulated. Has the cart has been put before the horse in this instance? Although there is no specific plan, money is being spent on programs. Three lead teachers are being funded at one period per day to serve as the initial group of lead teachers. They are developing curricula right now and will eventually present mini-workshops. We wonder what the focus will be for the mini-workshops that they are developing. It is mentioned that an incentive to learning new technologies is that one can become a lead teacher. The specifics as to what the "incentive" is for becoming a lead teacher are never discussed. They do not expect 100% involvement, but if they get some key people involved they feel they can make a big impact on the district.

It seems that the majority of training received by teachers at WJHS has been independent and somewhat informal. Many respondents have mentioned being self-taught. Some have mentioned learning informally from colleagues and from their family members. There has been mention of a few district workshops which have been offered, but the general perception gleaned in our research is that the district training is perhaps not as effective as it could be. For instance, one teacher notes the lack of discipline in specific training and says that the trainers "have no relevance to me."

Two workshop opportunities were recently offered to teachers. They were hosted by Dr. C. Wright Webb from Northeastern Missouri State University. The first workshop, entitles Getting Started with the Web, was designed to help teachers feel more comfortable searching the web, to explore useful educator sites, and to create bookmarks and examine ways to conduct more efficient web searches. The second workshop, entitled Introduction to Designing Web Pages, was geared to address issues of producing more effective web pages. At the time of this evaluation report, no feedback on these workshops has been collected.

OVERVIEW OF EVALUATION PLAN AND PROCEDURES

Team Davis collectively developed a plan for evaluating WJHS that included a variety of methods and activities. We attended an open forum meeting lead by Mark Jackson, who is a representative of the NEF. We gained some insight as to what our mission was as an evaluation team. We also learned that our mission was not as cut and dry as we may have originally expected. Initially, we used the WJHS web site to familiarize ourselves with the school itself. In the evaluation's infancy, this site was not nearly as informative as it is presently, but nonetheless it provided a starting point.

Our team met biweekly to discuss how to go about the evaluation process and decide how to go about contacting the various stakeholders and what initial questions were relevant to our evaluation. We agreed that email contact would prove to be the most efficient method for all parties involved. In addition to the asynchronous medium of email, using a synchronous method of communication was a possibility that we did not want to discard. Some of our witty team members tossed around the idea of voyaging to this virtual place for a virtual tour, and, although we were invited, we chose to decline.

As the WJHS web site developed, we began to communicate with some key stakeholders via email. Administrators and individuals with technology-related positions were contacted first. At this point, obtaining general information was perhaps our initial goal. As the respondents addressed our questions, we continued to discuss the focus of our evaluation. Perhaps being too tunnel-visioned about receiving answers to our questions, we received a few email responses that were returning questions for us to answer. During this process, our team realized the imminent need to be more sensitive of our stakeholders' perspective. After a few confused responses, we were mindful to be a bit more personable and to share with these people some of our own experiences. WebBoard proved very useful as a means of sharing our responses. WebBoard was also used to communicate synchronously with two stakeholders.

On October 13th at 8:30 PM, WebBoard hosted our team for a chat session with Princi-PAL Lee Daniels and Technology Coordinator Roger Wildabette. A list of questions was sent to each man prior to the meeting to prepare them. Although we thought this was a good idea at the time, it seems that our sending questions was more beneficial in helping us organize ourselves than it was in preparing our guests to respond. Due to their busy schedules, this was quite understandable. The chat lasted almost an hour, and, although we did not ask all the questions we had generated beforehand, we were able to adjust our inquiries to the flow of the conversation. The meeting was very relaxed in nature, and at one point we were asked to tell our superiors to give WJHS more money...lots of money. We were also offered the task of writing a technology plan for the school. One of the problems with text-only communication is that the absence of tone of voice and personal presence creates room for misinterpretation.

PRESENTATION OF EVALUATION RESULTS

Summary of findings

WJHS seems to be comprised of several teachers and administrators who view technology as a positive part of the educational environment. There exists a broad range of teacher abilities with respect to computer literacy. Responses indicate a desire to integrate technology into the curriculum, but little formal training on how to achieve this task is in place currently. Instances of informal and self-training are quite common. Currently, there exists no formal evaluation plan for the LCT program described in the NEF grant proposal. Teachers are not required to document how they are using technology, nor are they expected to report how its use compliments their curriculum. The paper trail is very lacking in this program. There is no clear and easily accessible documentation to support that the goals of the proposed project are being met.

Even though WJHS has a web site currently up and running, it seems to lack some basic web design protocol. The overall design of the site is not consistent, and some pages are very hard to read and lend themselves to some scrutiny in terms of accessibility issues. As for content, it is unclear if this site has an editor or a group of editors who monitor what information is being added to the site. Some links on the initial page seem to be out of place, and some content within the links may be of questionable importance. For example, there exists a link to the P.E. rules on the first page, along with the link to the teachers. Perhaps this link could have existed on the P.E. teacher's page or even the page that lists the specific departments. Additionally, the link to the departments only contains two departments. Perhaps a better plan might be to host one link to the departments and include teachers' links as subheadings under the appropriate department.

Interpretation of findings

The various levels of teacher ability is a concern in the implementation of the LCT program. When teachers are self-taught and/or taught by each other on an informal basis, this creates a greater possibility for misinformation and/or improper use. However, this is not to say that all of these training options are not beneficial. It seems that many teachers have benefited from these types of informal training. Nonetheless, a more formal system of teacher training and accountability may help to unify the efforts of the teachers. Training should be presented in a way that is encouraging and that reaches all teachers.

The lack of a technology plan is a major oversight in both the LCT program and in the district. A technology plan would not only provide a sound structure for the present but a vision for the future. From the lack of responses we have received about where this district will be in five or ten years in terms of technology, we anticipate a great need for a specific plan to be developed.

CONCLUSIONS AND RECOMMENDATIONS

Judgements About WJHS Technology Proposal Progress (Strengths and Weaknesses)

Efforts have been made to integrate the LCT program at WJHS. The acquisition of hardware and software provided in part by the NEF grant has allowed both teachers and students to heighten their awareness and skills of these technologies. It was discovered that most of the teachers need little"prodding" to participate in the LCT program. In fact, some are extremely excited and motivated about this program and have willingly donated their time and money to sharpen their skills. WJHS has made a valiant effort to place its students and teachers on the information superhighway. Networked computers allow students to use the Internet as an educational resource. Students seem to have the opportunity to use computers quite a bit, and efforts from the PTA hope to allow this use to continue in the home by making sure all families have computers and Internet access. Although this process may seem presently unrealistic, the desire to provide access to technology for all is clear.

Although WJHS has made valiant efforts to successfully implement its LCT program, it is still in its infancy, and several issues need to be addressed before it can mature. First and foremost, the absence of a material technology plan is a major oversight. A school technology plan should be created to outline specific goals for technology integration and provide the district with a futuristic "vision" of how it plans to move forward as technology changes. There is no significant documentation of efforts made to address this issue thus far. Given funds for significant improvement of their technology infrastructure, we have little evidence of an effort to expend funds for the hiring of a full-time LCT coordinator and training activities which will have the potential for the longest term impacts. We have found no significant evidence of a coordinated teacher training program and no observable efforts to evaluate those sporadic training activities we have encountered thus far. Although the WJHS web site is available and contains some usable data, the structure is not consistent, and our team questions the editorial process, if any, involved in the production of this site.

Recommendations, Criteria and Standards WJHS Should Anticipate

1) Advertise and fill the full-time LCT Coordinator position proposed in the grant. This hire should, and will, be key to Wildwood achieving the goals proposed in the NEF grant. It appears to us that those responsibilities have been added to Roger’s already full plate. Continually adding responsibilities to the workload of a teaching technology coordinator often ends up being in the best interests of no one. We suggest you consider stating the following as desirable attributes for the full-time position: experience in 1) computer network administration, 2) technology-related training, 3) chairing or extensive experience as a member of technology-based planning committees, and 4) experience in the development of school technology plans and acceptable use policies.

2) Develop a working relationship with Judy Frank and develop goals and methods of assessment prior to the end of the funded period. This will allow the entire Wildwood staff to better focus their efforts in the short term, and will make grant writing and documentation required for future funding much easier.

3) Continue and follow-up the recent staff technology training. Informal surveys regarding teacher technology training needs and desires can be used as a base for technology-related record keeping. You may want to involve Judy Frank in this effort, though time is of the essence in maintaining teacher enthusiasm and momentum. Find out who among the teaching staff might be interested in serving as technology mentors and mentees. Let the teachers make their selections of pairings and reward those who are interested in developing mentor-mentee relationships with something informal (e.g. provide an open computer lab with assistants for the kids of the teachers interested in mentor-mentee relationships while the teachers get together for a one hour period every other week, then let the kids and the teacher-parents work for a half hour in the computer lab on whatever activity or activities involved the kids).

4) Make copies of technology plans from other middle schools available to teachers so that those who may be interested in participating in the development of a technology plan can gain exposure. A simple query using the phrase "middle school technology plan" to any of the major search engines will result in a myriad of technology plans for review and possible selection as a model upon which to base the skeleton of the WJHS technology plan.

5) Begin efforts toward the development of a technology committee. This committee will eventually deal with issues including the development of a technology plan, Acceptable Use Policy, etc. We recommend that the initial efforts of this committee be oriented toward teacher training, the development of a teacher training needs assessment and coordination of training activities. Information regarding the development of various needs assessments can be found in chapter 2 of the online book, Technology @ Your Fingertips.

6) Consider offering training and meeting time for users with two different levels of training - a novice group and a more advanced users group. Base initial training on the needs of those with the differing levels of expertise.

APPENDICES

NEF's Request for Proposals

Wynne County's Proposal to NEF

1998-1999 Budget Proposal by WJHS

Wynne County Home Page

WJHS Homepage

Questions/responses proposed to WJHS Stakeholders

Team Davis Case Study Report #1

Team Davis Case Study Report #2

Team Davis Case Study Report #3

Team Davis Case Study Report #4

NEF Evaluation, Team Davis

November 8, 1998