1.0 Introduction

Fitzgerald Associates (hereafter F.A.) were contracted by the National Education Foundation (NEF) in August 1998 to conduct a formative evaluation of the initial stage of the implementation of the Learning with Computer Technologies (LCT) program at Wildwood Junior High School in Fordham, Illinois, the pilot school for the Wynne County Unified School District (WCUSD) program. The LCT program was initiated in January 1997 prior to the current NEF grant and was originally funded by a one-year ISBE seed grant of $55,000. The current NEF funding period began with the beginning of the 1998-99 academic year, August 1998, and continues for two years through June, 2000, in essence positioning the present evaluation at the barest infancy of the grant's lifecycle. Total funding requested in the Wynne County proposal to NEF (April 1998) was $270,000, which, as far as we can determine from Wildwood correspondence, was granted in full (NEF grant 98-105).

1.1 NEF Request for Proposals

The grant for which WCUSD applied and qualified was NEF 98-105 "Awards to Facilitate Research and Development of Technology-Based Education". The grant is meant to be "catalytic", not long-term, preparing projects to qualify for other funding in future.

In its request for proposals, the National Education Foundation sees as a "high priority" the implementation and study of new computer technologies in public schools since it firmly believes that technology-based education (TBE) has far greater educational benefits than traditional classroom education, especially in career preparation. The aim is to fund "innovative educational activities" (and their evaluation) in schools without recourse to other funding.

Proposed projects it favors involve student-centered technology, partnerships that seek to implement NTES (National Technology Education Standards), linkages of small schools and districts, Web-based instruction, collaborative internet partnerships between teachers and experts, IT training workshops for teachers and students, interdisciplinary research on TBE, and state-based alliances of researchers, educators and technology experts. Implementation of NEF-funded technology projects is to rely on such technology-based education alliances with outside experts and undergo both internal and external evaluation upon project completion.

Proposals are judged according to two NEF Merit-Review Criteria: the intellectual merit of the project and the potential impact on learning and educational reform. To elaborate, the intellectual criteria deal with such issues as the importance of the project to furthering knowledge, the qualifications of the proposers, originality, clarity of project conception, access to resources, while the educational criteria deal with issues of promotion of student-centered learning, social inclusivity, enhancement of research and educational collaboration, social and scientific benefits, supportive political infrastructure, and quality of evaluation to promote student achievement and educational reform.

1.2 Wynne County Unified School District Proposal

After an initial one-year ISBE (Illinois State Board of Education) technology seed grant January 1997 to January 1998 to launch its Learning with Computer Technologies Program, WCUSD submitted a proposal to the NEF for further funding to expand the program in April 1998. The one-and-a-half-page proposal written and submitted by WCUSD superintendent Jan. D. Swenson states the original goals of the LCT Program as (a) enhancing technology in district schools to "maintain excellence in mastery learning and citizenship" and (b) connecting students to outside "worlds of information" on the Internet and other cutting-edge technologies to prepare them for the careers of the future.

With WCUSD's new proposal to NEF came new stated goals of its becoming a model school technology hub drawing and inspiring other Illinois school districts with a working vision of the educational future. The five major components of this expanded LCT are (1) technology acquisition, (2) leadership/staffing (including two new positions: a full-time LCT coordinator and a part-time instructional leader), (3) training for a diverse student body in cross-curricular integration, (4) teacher training using in-house and outside resources, and (5) a major rewrite of the curriculum to integrate technology to be done by an in-house-developed team of LCT Master Teachers at Wildwood JHS, a model which would then be available to other district schools. Internal evaluation plans include grant-long evaluation efforts, including surveys of students and teachers as well as other unspecified forms of data collection. The LCT evaluation is meant to align with other ongoing educational evaluation in the district.

1.3 General Introduction to Evaluation Format Used

In the following sections of the evaluation, F.A. discuss our evaluation orientation (Section 2), our methodology and the limitations we operated under (Section 3), our findings organized according to a modified ISBE Guidelines list of topics (Section 4), a general discussion and meta-evaluation and interpretation of our own evaluation (Section 5), and practical recommendations for Wynne County USD and Wildwood JHS administrators and staff (Section 6) to help in the ongoing development and improvement of the NEF-funded LCT Program.

2.0 EVALUATION ORIENTATION

In considering which of the many evaluation orientations available would be most appropriate for assessing -- sight-unseen -- the initial stages of the implementation of a fledgling technology program such as the LCT in a non-urban school district such as Wynne County USD, Fitzgerald Associates determined that management-oriented, objectives-oriented, and participant-oriented approaches provided the most reliable way forward.

Because the LCT project is currently receiving outside funding and in future will no doubt continue to require it, F.A. assumed the most useful approach to the evaluation for both the district and NEF would be management-oriented, as NEF has a clear financial stake in the outcome of the present grant and WCUSD has a financial stake in maintaining eligibility for continued funding. Furthermore, because both NEF in its request for proposals and WCUSD in its NEF proposal both list written objectives and priorities, we felt it was incumbent upon us to evaluate the extent to which the LCT is poised to live up to these respective objectives, thus entailing an objectives orientation for this evaluation project.

In addition, because our evaluation is necessarily formative and heuristic at this very early stage of the project's expanded implementation, we thought a participant-oriented approach to evaluation would serve as a balance to the previous two approaches and provide the best methodological approach, i.e. approaching the situation objectively with few preconceptions and talking to all the stake-holders to hear as many sides as possible, to understand the entire situation as much as possible before passing any interim judgment as to how well objectives stated on paper are being lived up to in reality. Again, due to the very early stage the project is in, we felt a strong compulsion to be fair, positive wherever possible, and restrained in making any dogmatic statements, especially in view of the admitted methodological constraints under which we operated. Our aim has been first to understand, and only then to advise and encourage what appears to be a worthy venture.

2.1 Evaluation plan

As an external evaluation team, Fitzgerald Associates, given the broad freedom of NEF's evaluation parameters, took the liberty of drawing up the following overall objectives for the present evaluation:

Because WCUSD is an Illinois public school district, to receive state funding in future once NEF's one-time grant ends, the district will be mandated to present an (updated) formal technology plan following the Illinois State Board of Education (ISBE) Technology Plan Progress Guidelines -- which themselves integrate the criteria used by the Technology Literacy Challenge Fund (TLCF) and the Federal Communication Commission (FCC). According to ISBE regulations, to qualify for state funding, an Illinois school's technology plan must meet at a minimum the description for the "emerging" category (the second of four categories: beginning, emerging, advancing, and exceeding). To ease the necessary transition to state or federal funding (and accreditation of its programs) at the end of the NEF grant, we felt it in the school's best interest to organize our evaluation of their technology program in categories that will aid them in aligning their technology plan more closely with the ISBE guidelines as early in the process as possible. (We are assuming the existence of an original WCUSD technology plan meeting ISBE's "emerging" level description for its ISBE grant proposal to have qualified for the initial ISBE "seed funding" given in 1997, although neither the District office nor Wildwood Junior High School have so far provided us a copy of this document.)

3.0 METHODOLOGY

- contributed by K. Lunsford and P. Sundberg

While our overall goal (in line with a participant-oriented evaluation approach) was to sound out as many participants as possible, our initial plan was to follow the correct political protocol, informing first the administrators of our desire to conduct the evaluation and asking for their permission. We gave initial priority to contacting the following participants: Jan Swenson, district superintendent and writer of the proposal; Lee Daniels, Wildwood JHS principal (and only Wildwood administrator online initially); Jack Warner, the business/tech teacher; Roger Wildabette, the technology coordinator; LCT administrator (we were not initially sure who this might be); any teachers who are implementing technology as part of the grant; parents; and students. We realized early in the process that a major barrier to accomplishing our goal of understanding and talking to all sides was one of (lack of) access. The Website offered us initially a tiny and incomplete pool of faculty that had e-mail links or even telephone numbers.

Fitzgerald Associates employed three technological media for gathering information about WJHS's technology program and general context: (1) e-mail communication, (2) WebBoard chat, and (3) a web site overview. In addition, we relied on any documentation various participants could provide to gain more objective data for our evaluation. Specifically, we requested WJHS's technology plan, a list of hardware and software purchased with the grant, a program indicator with measures of the technology program, the original needs assessment from which the LCT Program developed, any PTA surveys, surveys of teachers and students, and the self evaluation mentioned in the NEF proposal. Many of these have been promised, although not yet delivered, and we look forward to the extra insight they will give us.

Questions posed to LCT Project participants have dealt with issues such as curriculum development, staff development, hardware and software acquisitions, learning goals for the overall program, development of the district/building technology plan.

Team members of F.A. participated in different activities; ordinarily, the five team members reported findings back to a team coordinator each week via e-mail or WebBoard. Although there were some limitations on our access to WJHS's staff and school environment, we believe that we presently have sufficient data to make a tentative preliminary evaluation.

1) E-mail Communication

The five members of our team were responsible for communicating with two to three
specific district and Junior High School administrators or faculty on a regular basis throughout our evaluation. We appreciate the cooperation we have received from both Wynne County USD officials and WJHS faculty. In the following chart (table 1), we tally both sent and received e-mail messages over the past two months:

2) WebBoard Chat

Five team members participated in an hour-long WebBoard Chat on September 24, 1998 (cf. chat log) with three Wynne County/WJHS administrators: Superintendent Jan Swenson, Judy Frank, and Dr. Ann Felix. Although we also invited Gloria Panowski and Roger Wildabette -- the two faculty members with release time for the technology program -- we did not receive a response from them at that time. (They did correspond with us later on e-mail.) During the Chat session, we discussed topics such as the district's evaluation and technology plans, the integration of software (especially) in the WJHS curriculum, and the stakeholders in the LCT Program.

3) Wildwood JHS Web Site Overview:

Being unable physically to visit the Junior High School in person, we were forced to rely on its newly-developed Website (http://talon.extramural.uiuc.edu/cter/wjhs) for much of our background investigation of school life, personnel, and contact information (e.g. e-mail addresses). We found ourselves skimming and reskimming the pages as new ones appeared to gain a "feel" for the reality behind the site. In addition, since all six F.A. team members are experienced Web authors, teachers and users, we applied our knowledge of Web site design to the Wynne County and WJHS web pages. Each team member evaluated 3-4 pages within the overall site. We were particularly concerned with how well (or poorly) the Web site might portray WJHS as a "technology hub" to the various audiences that will use site. This is not to detract from the real progress and positive aspects of a Junior High School actually possessing a site and continually updating and adding to it.

Possible Limitations of Current NEF Evaluation Procedures:

F.A. were mandated by NEF to conduct this evaluation using the new distance communications technologies available both to us and to WJHS (e-mail, WebBoard Chat, the World Wide Web). As we have experienced whenever we have implemented or evaluated similar new technologies, we encountered a few limitations. Although we were able to communicate with WJHS teachers immediately involved with the NEF grant and others connected with the grant, we were unable to communicate directly with those who were not involved or who chose not to respond. Several administrators estimated that approximately only half of the teaching staff at WJHS were accessible to us via e-mail. In addition, we did not physically visit WJHS to observe the computer facilities directly or to speak with students and teachers, so our report may include the views of only some of the program's stakeholders. We are also well aware that a fledgling Website such as Wildwood's can offer only a partial and perhaps skewed glimpse of the institution behind it; we therefore considered it only a stepping-off point. Finally, we began this evaluation just as WJHS was beginning to implement the LCT program. We recognize that this was not the optimal time for an in-depth evaluation, and we therefore appreciate all the more WJHS and the district's efforts to accommodate us with the requested information wherever possible.

4.0 EVALUATION FINDINGS (based on Holinga's 4 steps)

For the reasons stated in section 2 above, F.A. have chosen to structure the Findings section of this evaluation report according the ISBE Guidelines synthesized with the more simplified four-step Springfield School District rubric on "Developing a Technology Plan" created by Mike Holinga, Springfield District 186 Technology Support Liaison. He divides the process of developing a technology plan into 5 steps (whereas the ISBE Guidelines are made up of 14 separate components):

Step 1: Assessing Current Position (hardware, software, etc.)

Step 2: The Vision

Step 3: The Plan

Step 4: Staff Development

Step 5: Evaluation & Adjustment

A. PREPLANNING (District/Community Profile)

- contributed by T. Greer and M. Tjelmeland

In Holinga's recipe for developing a technology plan, the standard evaluation of the context and participants is perhaps assumed, since his steps are intended for educators planning within their own familiar context. The ISBE Guidelines and most standard evaluation formats, on the other hand, state this component explicitly, requiring schools and districts to furnish a formal "District/Community Profile" to qualify for minimal acceptance, and for any external  evaluation, such as the present one, an understanding of this area is essential.

The Wynne County Unified School District pilot school for the LCT Project -- Wildwood Junior High School -- is located in Fordham, Illinois, between two large urban centers. The District serves a widely diverse student population, families poised on the extremes of income (wealthy or poor) with a smaller middle class.

The junior high school appears to receive strong support from the community. The PTA has been an extremely active participant in advocating the integration of technology into the school system, taking a role even in evaluation by passing out their own surveys on technology. Parents of college-track children especially seem interested in the LCT project. The PTA leader mentioned that, in addition to positive preparation for college, computers are seen as a means to solve a local social problem: violence in the schools. Technology is seen by some in the community as a way to divert students from violence by engaging their interest in more positive pursuits.

Participants/Stakeholders

The immediate participants in the LCT Project are administrators, faculty and students. In administration, there is Jan Swenson, superintendent, a firm believer in technological innovation, and Lee Daniels, Wildwood JHS principal, who, while open to technology, apparently was not involved in the initial district-level push for the NEF grant.

The faculty comes with very mixed attitudes towards the LCT Project and technology in general. It is in part a generational difference, with more long-term established teachers such as Sally Laughon initially uncomfortable with technology (but now with her own Website) and younger faculty with greater technology backgrounds. Many teachers appear to be less than enthusiastic at present about implementing technology in their classrooms due to the perceived effort involved; despite these concerns, however, many are in fact making an effort to work with the technology. Another perception is that computer work is not "real" school work and takes time away from serious work. Moreover, some teachers are required to travel between schools and have no main base. For them, teacher training in technology, hardware availability, and the opportunity to integrate technology in their classes is problematic. In general, there seems to be a feeling among the teaching staff of being unsure of themselves, "feeling their way" through the initial implementation, which is a completely natural response to the situation.

Outside experts have been a smaller, and so far less influential, group. NEF strongly encourages the input and collaboration of such outside experts. Burks Oakley of the University of Illinois (UIUC) has been to Wildwood to conduct a staff training day. The input of more such persons, especially those who have had direct contact with the LCT Project would be extremely valuable resources in future project evaluations.

The Wildwood students, while not highly represented on the school Website (only two children with sites linkable by the general public, and both siblings) are reported to have very positive attitudes towards computers and even do some of the Web development for their less technologically-inclined teachers. Some have found teachers' limitations on their use of school computers during class frustrating.

Last but not least, the five members of Fitzgerald Associates were of necessity participants with our own role in the project context. Just by appearing when we did and posing the questions we did we cannot but have an influence of the direction of the project and the thinking of the other participants. There are two dangers faced by outside evaluators contracted by the "deep pockets" that sponsored the Project: either we will be seen as too closely tied to the contracting agency and thus not be trusted, not be spoken to candidly -- or on the other extreme flattered and given views and information that the other participants believe NEF wants to hear. Two of the participants (Barb and the PTA president) saw us as a "consultant group," which is undeniably what we are!

B. ASSESSING CURRENT POSITION

- contributed by T. Greer and M. Tjelmeland

Holinga's first step "assessing current position" is an opportunity for schools developing their technology plan to take stock of their present technological status -- hardware, computer literacy of staff, equitable distribution - before seeing where they would like to go with technology development. The following section addresses three of the five components stated in the original LCT proposal: technology acquisition, and teacher/student training.

Hardware acquisitions

• 24 computers including a building server and network have been purchased.
• Networking arrangements are one building server at WJHS connected by a T-1 line to the district hub in a Wide-Area Network (WAN) and locally networked with a LAN linking the main office and two other locations.
• Several teachers received additional computers for their own use.
• The swim and dive coach received two computers.
• There is a computer lab, although we never discovered who was in charge of it.
• Both the drama and the computer clubs use the lab after school
• There appears to be no plan for the equitable distribution of computers which has led to reports of animosity among some WJHS teachers about the present distribution arrangement
• There is no evidence of any floating technology-computer hardware or LCD panels on carts that can travel among departments. It seems that teachers believe that hardware and software is dedicated by departments-little or no sharing of equipment.
• Some classes are already participating in outside technology projects such as Prairie Flowers (plant biology).

• Other software in use is J and S GradeBook (grade book system), Pacer Forum (for student communication), Web browsers (a big focus), Geometer's Sketchpad (Math), and e-mail between students (program not specified).
• Some software was purchased under the original ISBE grant: PowerPoint, Adobe PhotoShop, FrontPage, First Class Conferencing.
• The art and band departments did not receive the technology they wanted -- mainly software programs, such as music composing and graphics programs. It could not be determined what departments have software available to them.

Acquiring the technology is only a first step, however. The real challenge to any school or school district is in how well the technology is integrated into learning activities. At Wildwood, the major focus at present is on use of the World Wide Web, according to some teachers. There has been mixed student reaction to doing searches on the Web: they are "cooler" than looking at encyclopedias but much more confusing because it is hard to narrow the searches to get precisely the information needed. Not much was mentioned about concerns for teaching students to judge the reliability of the information on the Web; rather, the typical concern expressed was about children finding obscene material. Another concern voiced by respondents was over students' receiving e-mail messages from strangers and the potential in that for danger. There has also been major concern about unequal student access to computers -- especially for homework assignments. Many students' families cannot afford to buy computers for home use, while other families can.

There seems to be a growing potential for discord among both Wildwood teaching staff and among parents in the community based on real or perceived inequalities in the distribution of technology in the school and community. There appears to be no formal plan for equal distribution of the existing technology at Wildwood (according to Judy Frank) nor any community-wide plan to provide the children of lower-income families with additional access to school technology to put them on equal academic footing with children from homes with computers. If allowed to continue and grow, this issue (whether perceived or real) has real potential to irreparably harm the LCT project long term and become a source of contention in the school and larger community.

One other point mentioned was that many students complained that the computers were old and dirty. However, Lee Daniels reassured us that WJHS had not bought any used equipment. Mr. Daniels referred us to the technology coordinator, Roger Wildabette (who has not yet responded to this question); Mr. Daniels did not address this question personally.

C. VISION - integration of technology into curriculum, teaching and learning,

- contributed by D. Lerch and P. Sundberg

Holinga's second step, "vision", involves the school's goals for integrating technology into the curriculum, curricular development, and the desired impact on teaching and learning ("where you are going," in H's phraseology). In the ISBE's terminology, the "vision" (section 5) involves incorporating both the district and general community in discussions of participants' "preferred future of technology's role in promoting educational excellence and opportunity all learners." The NEF proposal does not specifically mention any participating role for any stakeholders beyond the educational institution (except for visiting educators) -- what would be considered the minimum "Emerging" version of an ISBE vision.

However, there are references to wider participation in other data from Wildwood: the 1997-98 School Improvement plan (on the Wildwood Website) which indicate that the conception of the vision of LCT at present falls somewhere in between the "Advancing" and "Emerging" phases of the ISBE "vision". The School Improvement Plan mentions goals of incorporating business, community, and parent volunteers into their instructional and mentoring efforts to increase opportunities for success for all students and making the PTA an active instructional partner. This approximates the ISBE description of the "Advancing" phase ("Some stakeholder groups were involved in the development.") more closely than it does the "Emerging" phase ("Limited stakeholder groups were involved in the developments.")

It would appear, therefore, that Wynne County USD's conception of its vision has been deepening since the original NEF proposal was drawn up in April 1998, one that more closely approximates ISBE's conception. This is a trend we strongly encourage. On the other hand, explicit planning to bring about the participation of outside school districts in Wynne County's developing "hub" (a goal mentioned in the NEF proposal) and participation, at least online, of outside experts, as envisioned by NEF is harder to document, although the general vision is shared.

D. THE (TEMPORAL AND FINANCIAL) PLAN

- contributed by P. Sundberg

Timeline

Holinga's third step "the plan" involves the make-up of the technology plan itself: a timeline, established priorities, and budget which specifies allocations to hardware, software and training. This is also a major focus of the ISBE Guidelines as well.

The overall timeframe for the LCT grant is quite clear: two school years. ISBE Guidelines, however, require that a school's plan's timeline be defined for a minimum of three years (12) -- requiring a vision of technology development beyond specific grant periods. There is a curious silence in the LCT proposal about the timeframe predicted to reach the various objectives WCUSD has set for itself. Are the goals expected to have been met by the end of the grant period, or are some of them longer term? Can Wildwood reasonably expect, for example, to be a fully functioning technology hub with technology everywhere integrated into its curricula at the end of two years? Likewise at a micro-level, part of ISBE's "Community Involvement" guidelines (8a., 8b., 8c.) requires participating schools to include a timeframe for each of their goals and strategies for community involvement, engaged learning, and professional development to minimally qualify as "emerging", yet neither in WCUSD's NEF proposal nor in the budget are timeframes mentioned.

Budget/Financial Planning

The WJHS site thoughtfully provides a link to the budget for the LCT grant, allowing F.A. a glimpse into the implementation of the project during its important initial expenditure period. The five items in the budget reflect the five components of the original proposal: (1) technology acquisition, (2) leadership/staffing, (3) student training, (4) teacher training, and (5) curriculum development. The 1998-99 budget appears reasonably close to an expected half of the total grant monies ($135,000) allocated for the first year. However, in view of the ISBE Guidelines, one would expect to find not only the 1988-99 budget, but a budget for all phases of the life of the grant in the document. To meet ISBE's requirements, the financial committee (or person responsible) would do well to at least consider the following points, which stood out in a comparison of the first-year's budget with the proposed budget in the NEF proposal:

1. While technology acquisition expenditure is exactly on course (half of the total $125,000 has been spent), the kinds of hardware (24 "computers") and names of software acquired are not mentioned on individual budget lines. In the latter case, the lack of clarification is of more concern, since the variety of software types (and subjects for which it is meant) is far greater than the kinds of basic personal computers for school use. Are all departments benefiting equally from the software purchases? Of prime concern to us was equitable distribution of the technology purchased. For instance, the swim and dive coach has received two computers, but the arts department never received the software they requested.
2. Funding for staffing seems to be the most divergent from its description in the proposal: the "full-time LCT coordinator" mentioned in the proposal is now listed as a "temporary LCT building coordinator" with a 40% salary buy-out (is s/he in charge of coordinating "building" or coordinating the tech program?), while the proposed part-time LCT "instructional leader" is not mentioned, only a vaguely worded "teacher preparation buy-outs for technology integration." Does the latter refer to one employee or funded slots of time for release time for various teachers? Is any job hunt underway for a full-time coordinator, thus explaining the temporary nature of Mr. Wildabette's position?
3. As for student training, the original proposal details one aspect as "new course development in technologies and learning", which appears to fit under the later "curriculum development". The original goal was to develop courses across subject areas, learning styles, and populations. The reality suggested in the budget is understandably more limited at the early stage: support materials for a Language Arts-Tech course and a Business and Technology course, two attempts to develop cross-curricular courses. It is still early in the game, but at a strategic time in the 2-year grant when long-term curricular development plans need to be underway if they are ever to be realized, it is not encouraging that only a third of the year's student training budget of $15,000 has been allocated. Furthermore, are any funds to be set aside for accommodating disabled or economically-disadvantaged children, two vital ethical aspects of the proposed student training goals?
4. The teacher training budget for 1998-99 seems to be fully allocated, mainly to workshops, with 12% going for conference mini-grants. It would be helpful, however, for a break-down of exactly what the $11,000 allocated for workshops is going towards. Outside experts were mentioned in the NEF proposal. What percentage of the $11,000 is therefore allocated for honorariums? How much for other expenses? As part of WCUSD's vision of becoming a technology mecca for other schools, how much of the conference budget is being allocated to bringing outside teachers and administrators in to profit from Wildwood's technological experimentation, or is this money only for in-house use?
5. Slightly more appears to have been spent on curricular development than budgeted ($9,000 spent versus an expected $7,500), although this may in fact be called for, given that curriculum development is something which requires immediate attention if it is to bear fruit within the life of the grant. One might wonder, however, what kind of "materials" at $500 per department there might be for a technology project that aren't already included under "technology acquisition". More specificity here would be appreciated. It is also good to find that paid release time is explicitly written into the curriculum development section, since if such work isn't paid for, it rarely gets done. This aspect is not merely in the planning stage, it would appear, for Jay Andrew, match teacher, has been given release time through the NEF grant to develop his math site.

E. STAFF DEVELOPMENT

- contributed by K. Erlinger

Holinga's fourth step, "staff development," recognizes the need for the technology first of all to be accepted and understood by the teaching staff before there is any hope for its successful implementation and integration into the existing curriculum.

Along these lines, Wildwood Junior High School and the Wynne County School District has provided and is planning to provide technology training for its teachers. According to Judy Frank (Wynne county District Internal Evaluator), Burks Oakley was hired to conduct an initial in service at the beginning of the 1998-1999 school year.  Burks Oakley is an instructional technology expert from the University of Illinois (Urbana-Champaign) and his in-service focused on instructional technologies showcasing instructional web sites. Subsequent training has been planned for the school district. This training includes two workshops to be given by Dr. C. Wright Webb of Northeastern Missouri State University. The first workshop, "Getting Started With The Web" focuses on learning to use search engines and creating "book marks". The scheduled date for this workshop is Monday, November 2. The second workshop, "Introduction To Web Design" focuses on writing HTML and designing Web pages. This in-service is scheduled for Wednesday, November 25.

There is talk, but no formal plans at this time, that the Wildwood LCT staff will provide additional training in applications such as spreadsheets, databases, word processing , as well as specific software training such as Geometer's Sketchpad. F.A. have so far been unable to ascertain, however, whether participation in the in-services is mandatory or voluntary, and if it is voluntary, what incentives and compensation have been provided to encourage full staff participation. Additionally, some staff members have voiced complaints that the in-services are not providing the information they need on how to integrate the technology taught into their lesson plans and overall curriculum.

F. INTERNAL EVALUATION AND ADJUSTMENT

- contributed by K. Lunsford

Holinga's final step, "evaluation and adjustment" helps remind schools that merely planning for and implementation of technology is not the end of the story. In order to assure quality during the technology plan's implementation and successful accomplishment of the original goals stated in the plan, internal evaluation is a vital component.

According to the ISBE guidelines, the district will need to supply future grant agencies with an evaluation plan that, at minimum, contains vaguely worded expected results/success indicators for several goals, identifies at least one quality of effective evaluation and assessment, and outlines a process for evaluating how acquired technology are integrated into the classroom and impact student learning. Stronger applicant institutions will specify, in addition to the on-going evaluation plan, more exact expected results/success indicators and several qualities of effective evaluation and assessment. The strongest applicants will employ both quantitative and qualitative measures to conduct their evaluations.

Our overall impression is that WJHS and Wynne County representatives have been well intentioned and responsive to the needs of students, teachers and parents. For example, four administrators mentioned their concern about guaranteeing computer access to all students in the school and in the district -- one of the goals specified by both NEF and the ISBE guidelines. Judy Frank also detailed how WJHS students work online with graduate students at a nearby university, thus providing a qualitative account of how the technology program is beginning to encourage the collaboration that NEF and ISBE wish to foster.

However, we are concerned that without more systematic record-keeping, these efforts will not be rewarded by future grant agencies. Since WJHS has already had an ISBE grant in addition to the NEF grant, it is likely that state agencies in particular will be expecting fairly extensive accounts of WJHS's activities in any future proposals for funding. Documenting the program is crucial for its survival and future funding; therefore, F.A. would like to make specific recommendations towards that end at this point (in addition to the recommendations we provide in the final section).

First, we suggest that it may be prudent at this point to include funding for internal evaluation services in next year's budget for the technology program. The district evaluator may have been unaware of the two-year cap on the NEF grant; she mentioned that she was following a three-year evaluation schedule -- ISBE's minimum timeline for technology plans. Since the documents will be required sooner, we recommend that WJHS budget monies to be used to hire a full-time or part-time evaluation assistant whose duty will be to help her compile the document portfolio.

Second, a document is needed that outlines the on-going process for internal evaluation of how well technology is being integrated into the curriculum and how it affects students' learning. Judy Frank mentioned that she already assists individual instructors with assessing how effective certain software/hardware is for their classes. Accounts of these sessions might be kept in a work-log to provide the qualitative measures that are needed. In addition, the evaluation plan might take into account any in-service training that teachers receive -- their attendance and their reactions to the sessions. We heard from several teachers that they believed that the in-services focused more on product how-to's (such as how to design a Web page) instead of how to integrate the technology in their classes. If so, then an internal evaluation may determine how to change the emphasis of these in-services to align with the ISBE guidelines. Most important, the plan needs to integrate the several evaluation procedures that are already in place to eliminate redundant efforts. Several faculty members mentioned that they felt somewhat overwhelmed by additional paperwork; perhaps these procedures could be streamlined to reduce this perceived burden.

Third, there are several simple quantitative measures that could be documented. Several teachers mentioned that students are "always" using the computers, which is a credit to WJHS. However, we received no numbers on how many students were using them and when; if the computer lab coordinator would keep a sign-in sheet, those data would be available. Likewise, the computer lab coordinator should be able to summarize how many teachers reserve the lab each week and what software/hardware they employ the most.

Finally, WJHS may wish to review questionnaires from the Flashlight Project to help design a survey that identifies students' and teachers' attitudes toward technology. We were surprised to hear that students apparently felt that the computers were old and dirty, although we were assured that the technology was purchased new. Likewise, we were concerned to hear that some teachers were still resistant to the new technology. The Flashlight Project provides surveys that track these attitudes to alert administrators/teachers to potential problems, and since the surveys have been field-tested, they are likely to be accepted as further quantitative measures. If the cost of the Flashlight Project is prohibitive, then it would be useful for WJHS and Wynne County to design their own survey. The survey could be administered as part of a general course evaluation, and it would enable educators to identify and to adjust to developing needs.

5.0 DISCUSSION / META-EVALUATION / INTERPRETATION

Despite the impression that preceding pages might give of extensive coverage of the evaluation context due to the sheer quantity of information, Fitzgerald Associates would like respectfully to remind all the stakeholders in the Learning with Computer Technologies Project -- both Wynne County USD and NEF -- of the caveats mentioned earlier in the document: our limited access to firsthand sources of data and the possibly premature timing of this evaluation effort. These two, especially, have an impact on the validity and reliability of the information for decision making. In other words, the information in this report should be taken cautiously -- and with a grain of salt by decision makers and other stakeholders in the project.

F.A.'s evaluation efforts were hampered by the artificially-limited access we had to participants. Our contacts at Wildwood were limited by unavoidable methodological constraints to those staff members with e-mail listed on the school Website, and of those with e-mail, to those who responded. Thus, our sampling of the potential pool of participants in LCT is necessarily skewed, although we stress that the qualitative data comprising their verbal communication is authentic and presented, we hope, in the sense intended in the original context.

In addition, F.A. were contracted to begin our evaluation of the Program in the first month of its implementation, and consequently the findings above are extremely tentative and the evaluation necessarily formative. Due to the limited time the current proposal has had to gain its footing, F.A. experienced some internal division regarding our proper role as evaluators in this context: should we be more directive, giving more advice to the district and to Wildwood, less directive, or totally non-directive until we understood the situation more clearly? During data collection, team members adopted different approaches, some members limiting their role to gathering data in order to describe the current state of the project, other members advising LCT participants on location how better to align their efforts with outside guidelines. We would appreciate knowing from the participants with whom we were in communication what their experience of these approaches were on the other end in order for Fitzgerald Associates to hone our own evaluation strategies and skills for future evaluations.

A slightly graver limitation we encountered, one which should not have been so extensive or necessary, was the lack of documentation available when requested. During the two months of our research, F.A. frequently asked for a number of essential documents, including:

In formative evaluation, it should be remembered that any negative findings are to be viewed merely as fuel for future improvements, not final and binding judgments. In spite of the limitations and problems listed above, F.A. is extremely encouraged by the enthusiasm shown by the participants in the LCT Project and their obvious willingness to learn and adapt. The very visible growth in the quantity of the pages and information available on the Wildwood Website over the course of the evaluation is, to us, a clear indication of the general health of the plan during these very early days in the current Project's lifeline. It is to be expected that the inception of any major effort at technology implementation in a public school is a shaky and unsure time. With the proper outside input and continued collaboration with technology experts such as Burks Oakley from the very beginning, we feel that the current Project has good prospects for ultimate success.

6.0 RECOMMENDATIONS

Perhaps the most useful role Fitzgerald Associates can play at this stage in the current project, we feel, is that of recommenders. While Wynne County USD deserves credit for its forward-looking vision of the promise of technology in educational settings, there are still many areas for improvement in both the conceptualization and implementation of their LCT Program, as they themselves freely admit. To this end we respectfully submit the following well-intentioned list of recommendations for consideration by WCUSD and all interested parties in their continued decision-making on the LCD Project. The recommendations are useful only to the extent that they are based on an accurate understanding of the actual context (something we have little right to claim), being informed by the collective years of experience and expertise of the evaluation team in technology implementation in public schools. They are not to be seen, we would hope, as divine revelation, something inflexibly to be applied in their totality, but rather as well-intentioned pieces of advice that may -- or may not -- fit the Wildwood context. In that knowledge, the Wynne County participants in the Project have the clear advantage over any external evaluators. Finally, some of the F.A. recommendations below are provided for the benefit of the National Education Foundation as well in its decision-making for future evaluation efforts of its grant recipients.

Next external evaluation

• The present evaluation, in hindsight, appears to have been initiated somewhat early. Future evaluations of NEF grant recipients would be more profitable for all parties concerned if the recipients are given at least one semester to begin implementation of their goals.
• In future NEF evaluation efforts, evaluators should be able to visit evaluation site in person, to talk to all the stakeholders concerned, not only the subset (50% at Wildwood) online at present. Evaluation from a distance will always hold problems of validity, which can be avoided by at least part-time evaluation on site.
• Regarding a follow-up formative evaluation of Wynne County's LCT Project, we strongly counsel waiting until at least Fall Semester 1999, after LCT has been had more time to integrate the recommendations contained in this first evaluation.

• As all serious future funding agencies will require a well-developed technology plan, we cannot stress too much WCUSD's reviewing the current plan and carefully adapting it to align more closely with the ISBE Guidelines' minimal standards ("Emerging"), and preferably with the "Advanced" category.
• Such a plan should contain very specific -- and realistic -- goals with specific timeframes for each goal, regardless of the funding period of any grants applied for (ISBE suggests a minimum of three years in a technology plan's timeline).
• Plans for collecting the data necessary for the various ISBE rubrics should be made as soon as possible, both to complete an eventual technology plan and for current internal evaluation use.

• A major administrative weakness in the district discovered during the current evaluation was the absence of important documentation either because of loss, misfiling, or a document's never having been created. We do not need to emphasize how important this is not only for the district's reputation outside Wynne County, but for future grant-writing efforts. The district and Wildwood JHS need to keep better records and immediately review its current filing procedures to prevent future loss or misfiling of vital information.
• Be proactive. The respective administrations should determine in advance which documents the state, the grant provider, or other schools might need and make a list of these ahead, and begin to locate/recreate the necessary documents. It goes without saying that copies should be kept in several locations for security.

• Another major weakness we encountered was a lack of clear communication between the various stakeholders, which gave rise to animosity and gossip. The administrations need to foster better communication between Wildwood teachers by requiring faculty meetings dedicated or partially dedicated to discussion of the technology issues raised above to provide a forum where all views can be voiced.
• The administrations should also foster better communication between the school district and the community (the PTA shouldn't have to initiate its own surveys) and provide several forums throughout implementation of the NEF grant, where interested parents and community members can voice their views.
• The administrations should also foster better communication within its own ranks, the district office staff concerned with LCT meeting frequently with the principal, vice-principal, and NEF-funded technology staff to guide this large undertaking with a unified voice.

• The NEF proposal mentioned WCUSD's desire to become a technology hub, a magnet district for other Illinois schools. This will require specific planning to bring about the participation of outside school districts in Wynne County's developing "hub". It won't happen spontaneously of its own accord.
• A major NEF concern mentioned in the original request was for participation, at least online, of outside experts. In addition to bringing in outside speakers and trainers such as Burks Oakley, the Wynne County administration needs to specifically seek out links with such distant experts -- at the University of Illinois, for instance.
• It would be worthwhile for WCUSD to begin to see their usefulness to other districts not only as high tech experts but as instructional technology integration pioneers whose experiences positive and negative can help other districts that are just a bit further behind technologically.

• Keep detailed lists of technology acquisitions, separated according to which grant they were purchased under (e.g. science classrooms have tech from the previous grant from ISBE). This will ease the job of grant auditors and school accountants and allow the district to evaluate its progress in acquiring hardware and software.
• WJHS should have a formal (but not too bureaucratic) system which allowed teachers to request software and other equipment or to be reimbursed for necessary purchase (instead of them having to purchase software at their own expense). Software acquisition, especially, should proceed from the "bottom up", with the main impetus coming from the teachers themselves and not mid- or upper-level administration.
• For a more equitable distribution of existing technology, WJHS should develop "traveling technology" (e.g. computers on carts) as much as possible as long as equal equipment cannot be stationed in each participating teacher's own class.
• Likewise, lab access should be fairly distributed among all the teachers who request it. There should no longer be any bias towards math and science in lab use, as unfortunately was the norm in the 1970s and 1980s.
• Another equally important access issue is how to assure equal access to technology for children from families without computers and disabled children. Such concerns should be explicitly addressed in any technology plan.

• Unless the local job market prevents it, WCUSD should try to find and fund the two LCT positions it specified in its grant proposal as soon as possible rather than expecting two part-time staff members to fill the role specified for a full-time and part-time employee. It is unrealistic to expect the same level of productivity or expertise in such situations. NEF agreed to fund these positions, and the budget money allocated to this should be used as agreed on.

• In addition to seeing students as technology learners, the WCUSD should include students in other appropriate aspects of the Project. For example, the maintenance of computers is a task some students might aspire to (students said some machines looked "dirty"), and perhaps more technically savvy students could even function as lab monitors.
• Students expressed frustration at not having enough access to computers at school. Therefore, time should be blocked out for open student use of computers during free time or before or after classes, especially for those with no computers at home who have been given technology assignments.
• In addition to the mechanics of WWW use, students need to be taught to judge the reliability of the information on the Web and how to protect their privacy by keeping photos or contact information off their pages and not sharing URLs outside the school context. (It concerned some of the F.A. team that two students had Webpages with private information connected to the public school site.)

• There is a marked spread in the level of technical competence among Wildwood teachers (some have never used a computer; others are writing away for technology grants). The administration needs to make more of an effort to bring all faculty up to minimum level and stave off potential feelings among teachers of animosity towards the technological "haves" and fear of being left out. Perhaps separate training sessions could be held, allowing beginners to learn with other beginners and that way avoiding embarrassment.
• In-service training sessions, in addition to teaching how to use specific hardware or software applications, should also teach teachers how to integrate the technology into their lesson plans and overall curriculum.
• Given the example of one enterprising teacher who is applying for her own grants for funding (e.g. Sunny Beach in physical education), it would be a valuable skill to train other Wildwood teaching staff how to write their own funding proposals to supplement existing NEF and other all-district grants. Outside experts might be brought in for a one-time workshop on this topic.

• Now that a substantial amount of technology has already been acquired, a more important priority should be to integrate the existing technology into Wildwood's curriculum. If local teachers and administrators cannot think of such ways in the initial stages of the Project, outside experts can be brought in (or contacted via internet) to "inspire the troops".
• Although the innovative concept of a "team of LCT Master Teachers" is mentioned in the NEF grant proposal, F.A. could discover no evidence of any specific plan to bring this about. Some teachers were apparently doing technology curriculum development on their own with or without release time, but there was little sign of organized cooperation among them. The administration needs to begin by forming an initial team of its best and most enthusiastic teachers, and then granting them the release time and compensation needed for the Master Teacher team to ever come to fruition.

• The financially-trained staff at WCUSD need to develop a much more detailed budget that covers the entire length of the grant -- "all phases," as specified by ISBE. For example, a budget line titled "software 8,300.00" is not specific or useful enough in itself to satisfy either the district's own needs or, more importantly, outside agencies -- state auditors, grant agencies, etc.

• F.A. were initially frustrated at the district's and junior high school's seemingly vague plans for internal evaluation of the project. ISBE and NEF both require well-developed evaluation plans, and the required evaluations mentioned by both should be minimum starting points for WCUSD's own evaluation efforts. For a comparison and evaluation of the extent that goals have been achieved, it is vital that an initial evaluation of equipment, personnel, etc. be carried out as early as possible so that any change will be detected. Developing a clear evaluation plan and implementing it as early as possible should be one of the district's immediate priorities.
• In addition, F.A. advocate that any LCT project evaluation should make better use of the existing district evaluation specialist (Judy Frank), who has expressed willingness to fulfill this role. A role for the district specialist should be explicitly written into any project evaluation plan.