The TEbase Initiative:
Research, development and evaluation for educational reform
Sandra R. Levin, James G. Buell and James A. Levin
University of Illinois, College of Education
Preparing pre-service teachers for classrooms of the 21st century poses new challenges in the wake of rapidly changing technologies throughout our society. As electronic resources for learning become increasingly important to schools across the country and around the world, technology instruction needs to become a major integrated component of preservice teacher education programs. Too often, this is still an unmet challenge, owing to lack of time in the curriculum and unclear instructional focus. Curriculum restructuring is an important step on the road to reform.
In 1996, the International Society for Technology in Education's (ISTE) Standards Committee addressed this challenge by recommending a set of Technology Standards for All Teachers that has been adopted by the National Council for Accreditation of Teacher Education (NCATE). Since then, several state departments of education, including Illinois, have been developing their own Technology Standards for candidates seeking teacher certification. This article describes a framework used by the University of Illinois to integrate technology instruction based on these technology standards into a redesigned teacher education certification program.
For the past three years, the teacher education program at the University of Illinois has engaged in curricular reforms to promote constructivist learning and provide strong foundational skills in instructional technology. What began as a pilot program for a select group of senior-year students was expanded in Fall, 1998 to cover all juniors and seniors pursuing certification as elementary or secondary school teachers.
In support of these program reforms, there has been ongoing development and formative evaluation of a set of instructional resources known as Teacher Education Databases, or TEbases. This paper describes the historical, current and planned uses of TEbases as aids to educational reform.
TEbases first took shape as a Technology Competencies Database (TCD), built directly upon the ISTE/NCATE Technology Standards. The TCD was developed at the University of Illinois during the 1996-97 academic year and first piloted with a small group of students there. Our inspiration was an article by B. Levin (1996), suggesting ways in which a portfolio approach could be used to document students' mastery of the ISTE competencies. The following year, colleagues of the developers became interested and piloted the TCD at their institutions (Southern Illinois University - Edwardsville and National-Louis University) in varying degrees to address the same preservice needs. Eastern Illinois University and Illinois State University began working with the tool for the first time during the 1998-99 academic year. All five institutions continue to reshape the database and to develop related materials for faculty and students.
The Technology Competencies Database is a World Wide Web-based software tool which allows students to enter the ways in which they can demonstrate the ISTE competencies. The TCD is intended to provide students with an avenue to communicate their experiences with instructional technologies to the faculty, and for the faculty to assess the students' accomplishments in these specific aspects of instructional technology. Each student describes the activity which fulfills the specific competency and provides evidence supporting the description. The faculty member then has the option of accepting the student's submission or requesting further information and evidence from the student. With the students' permission, the most outstanding examples become available for other TCD users to see and learn from, so the tool continues to grow in utility as students add to it.
Working with the TCD throughout their preservice program, students develop the competencies through different course experiences. Their own matrix becomes a fluid document, easy to revise as their skills further develop. Students might first satisfy a competency during a course experience by giving a simple example, but as they have opportunities to observe a body of best practices they might later meet the competency in another way and revise or supplement their first efforts.
The TCD is also an avenue through which elementary and secondary education faculty increase their own technological competencies because they need to assess the appropriateness of student submissions. Faculty at each institution work together to create the exemplars and criteria needed to assess student submissions.
National standards have become part of the landscape across the entire field of education. Standards-based reform efforts have both adherents and detractors, but from either side of the fence it is clear that any checklist or set of guidelines created by an external organization must have its terms operationally defined in use, in actual educational settings. If educational programs are to do more than pay lip service to a set of standards, they must incorporate them as part of a continuing dialog about their educational goals, methods and practices. The Technology Competencies Database was built around this realization.
When students interact with faculty members via the TCD, they engage in reflective processes, display their knowledge, and negotiate meanings in individualized ways. That is, students select which competencies to attempt, in which order, and what to propose as evidence of satisfaction. Faculty members consider each student submission separately, and have opportunities to give individual feedback in addition to a grade. Student work which has met with faculty approval can automatically become available for other students to see and learn from. In this manner, the somewhat dry language of a set of competencies becomes the basis for a rich exchange of ideas among a group of learners. For students, the end result is an individualized "electronic portfolio" linking to their best work, which can be of use to them in reviewing what they have learned and in searching for work after graduation.
Evaluation of the Technology Competencies Database
During the Spring semester of 1997, 13 competency areas were included in the TCD. These were the original set of standards recommended by ISTE/NCATE. (Changes made by ISTE/NCATE in mid-1997 resulted in 18 rather than 13 standards.) In the Spring, 1997 version of the TCD, students were rated as having achieved at one of three levels of accomplishment for each of the 13 competencies; in general, Level 1 represented basic familiarity with a competency, Level 2 indicated that the student had used the competency with elementary-school classes, and Level 3 showed that the student teacher had made the competency an integral part of an extended set of lessons in the elementary school (B. Levin, 1996).
In Spring, 1997, 12 out of 40 senior-year students in a teaching methods course volunteered to try the TCD. Of these 12, seven successfully completed at least one competency. Eight students had submitted descriptions which received no response from the instructor, and five students received feedback from the faculty member but did not respond to the instructor's requests for additional information about one or more of their descriptions. Twenty-five of the 39 competency/level combinations were attempted by at least one of the students.
An evaluation of this first TCD implementation revealed strengths and weaknesses (Moore, 1997). On the positive side, students reported high interest in using the TCD, particularly because they thought it would prove useful in job searches. As negatives, students listed the slowness of the system (it has since been made considerably faster to use), and the long delays that many of them experienced between submitting information and receiving any response from their instructor.
These findings led to substantial revisions in the TCD in Summer, 1997. The TCD was updated to reflect the revised 18-competency version of the NCATE standards, and the three-level performance distinction was eliminated. In the revised version of the TCD, students are asked to describe a single level of completion for each of the 18 competencies; the instructor is given the choice of withholding credit pending more information, grading the submission as acceptable, or awarding an exemplary rating.
In Fall, 1997, formative evaluation techniques were employed to seek an understanding of the diverse viewpoints of the various stakeholders involved in the implementation of the Technology Competencies Database project. Through these efforts, we gathered information from students and faculty members via surveys, personal interviews and direct observations. We then used this information to modify the design of the TCD throughout the semester. Among the changes which resulted from this approach during the semester were improved instruction hand-out sheets and programming changes to maximize the speed of interacting with the database.
The evaluation conducted during the Fall consisted of four main phases: 1) participant observations at the time of students' initial introduction to the TCD; 2) distribution, analysis and collection of surveys given to the students at the time of introduction; 3) interviews and discussions with a faculty member participating in the TCD project; 4) interviews with two students who used the TCD extensively.
The students were introduced to the TCD in late September. At that time, forty students submitted responses for at least one competency area. Thirty of the 57 students returned surveys which were distributed to them at the time of their introduction to the system. In these surveys, they were asked to write briefly about their likes and dislikes regarding the TCD, about what changes they would like to see, about anything they found confusing, and about how they thought the TCD might be useful to them.
Fourteen students said they liked the overall operation of the TCD, with five reporting that it was user friendly, seven commenting on its ease of navigation, and two commenting that the directions were self explanatory. The negative comments from students largely fell into two major categories. There were seventeen complaints about the speed of the system, and twelve about the wording of the competencies themselves. In writing about what they found confusing, sixteen students commented on the language of the descriptions (they did not understand the wording of the competency statements); four others were confused and irritated by a programming error which brought up an incorrect response page (the error was fixed immediately after the session). Finally, in responding to how the TCD might prove useful, 17 students commented that they hoped to show their records in job interviews or as a part of portfolios for potential employers to see.
As a result of the students' responses, two independent efforts were made to clarify the intent of the standards. The instructor participating in the study prepared one set of descriptions to help the students understand the competencies. He distributed his comments to the students on paper. In addition, the authors of this paper also paraphrased and revised the statements and added these to the TCD via hyperlinked text.
Following the introduction of the TCD, the programmer (JGB) met several times with the students' instructor to show how the system could be used efficiently, and to explore areas where the TCD might be improved. In general, the instructor spoke quite favorably about the TCD's ease of use, and about what it has enabled him to learn about his students' abilities. He expressed concern about finding ways to decrease the time required to evaluate responses. In the first month after introduction of the TCD, the instructor graded submissions in one of the eighteen competency areas for the thirty students who had attempted it. However, those thirty submissions were just one-fourth of the 120 which students had submitted up to that point. Owing to other demands on his time, the instructor was able to advance little in the task over the following month. By late November, nearly all students were still waiting for responses to descriptions they had entered as much as two months previously.
As solutions were found to problems associated with access speed, information collection and usability, a key problem regarding feasibility of use was discovered. It became obvious that faculty would have a difficult time providing the optimal level of individualized feedback to students that the TCD is capable of enabling. The success of the TCD initiative depended upon our discovering several acceptable models for assisting instructors in managing the effort involved in evaluating student submissions. Having one professor evaluate 42 students in eighteen different competency areas proved to be an unrealistic task.
This has raised interesting questions about the role of evaluation in a student-centered resource like the TCD. At least one faculty member has considered the idea that students themselves be required to evaluate, or pre-evaluate, one another's completions of the competencies. However, students who have been presented with this idea have responded that they do not believe they would be qualified to judge one another's work. An alternative strategy might be to devote the time of a teaching assistant to this task; this, in fact, was suggested by two students interviewed in Fall, 1997.
Priorities in Spring, 1998 were much different for the 42 participating students because they were engaged in student teaching full-time during much of the semester. This left little opportunity for continuing with the TCD, and so they and their teachers did not use it much. As of December. 1997, the participating students had submitted a total of 156 competencies descriptions; at the end of the school year in May. 1997, the total had risen by just 17, to 173 submissions.
During the Spring, 1998 semester, a graduate teaching assistant was asked to help with the evaluation of the remaining TCD submissions as well as any new entries the students made. She was asked to respond to those that were obviously exemplary, sufficient, or insufficient, and to pass on to the faculty member those that were more problematic. She reported some of the same problems as the undergraduate students and the faculty member with understanding how the very generally worded competencies matched up with specific activities described by the students. She also found it easier to go through all those submissions for a given competency rather than evaluating all the submissions by a particular student or by evaluating submissions in the order they came in. However, the evaluation still took a substantial commitment of time, and all of the student submissions were still not responded to in a timely manner. In fact, fewer than half of the 173 submissions were ever awarded credit; the vast majority of this number remained unevaluated by the end of the school year.
From TCD to Activities TEbases
Through the evaluation research described above, we came to realize that the major barrier to faculty use of the Technology Competencies Database in the 1997-1998 school year was not technological but stemmed from the social organization of the task. While the TCD framework was constructed to span the multiple courses that our undergraduate students take over two years, we had not yet figured out how to integrate it into the specific courses that students take during that time. Efforts in this direction began in Summer, 1998 and continued in the Fall, 1998 semester, when new curricula for the elementary and secondary certification programs went into effect.
A primary goal in designing the Technology Competencies Database was to facilitate open-ended learning experiences in which students compare what they need to know to what they have learned and are learning, and then provide evidence of their learning to their teachers. The TCD was at least somewhat successful in this — it engaged the interest of 42 of the 57 students to whom it was made available, and these students, working voluntarily and independently, submitted responses for an average of over four competencies apiece. However, it had become painfully evident to us that staging an open-ended learning task in ways that make sense to students was only part of the challenge of successful technology integration. In addition, the faculty had to be brought on board in ways that engaged their teaching styles and fit with their ideas of education. Our failure to achieve this is evident from the fact that the teacher and a graduate student together were able to evaluate fewer than half of the students’ submissions.
Our current effort is to work with specific courses to identify class assignments and corresponding competencies, so by successfully completing an assignment, the students are aware of which technology competencies have been accomplished. As the students proceed through the two-year certification program, they can view their progress toward proficiency in all the technology competencies. We are discovering that in this way, the evaluation of the submissions can be more naturally distributed across multiple faculty, and the work of submitting and evaluating become part of the normal course load for students and faculty. As described in the remainder of this paper, the TCD has proven to be adaptable enough to mold itself to this new set of roles, but in an adaptation so major that we have taken to calling it by a different name, the TEbase.
In the 1998-1999 school year, the College of Education at UIUC began implementation of a redesigned teacher certification program. Included in this redesign was a new course, Content Area Applications of Educational Technology. In this new program, pre-service teachers are not required to take a separate introductory course in the use of technology in education. Instead, the technology instruction is stranded through all the pre-service courses, early field and student teaching experiences. The activities for the technology strand are based on the ISTE Technology Standards for All Teachers. In this framework, students are actively engaged in learning how to use technology in their content area or grade level and how to develop and implement lesson plans that will include the use of technology in their teaching. From our previous research on the level of technology experience of students coming into the teacher education program, we assumed that students would have basic familiarity with word processing and web browsers, although there are some students who come into the program with much more technology expertise.
This stranded course is coordinated by a faculty member experienced in the use of technologies across content areas and grade levels. A course syllabus was developed to be implemented over a two-year period. A number of meetings between the content specialists and the technology instructor were held to discuss what kinds of technology would be most useful for faculty and students to use in their teaching and learning. Hands-on instruction was provided in a computer lab setting by various content specialists and teaching assistants with support from the coordinating technology instructor. Five major topics are included in the technology strand: Computers in the K-12 classroom, Computer networks, Instructional & classroom management strategies, Social issues, and Keeping current with technology.
The Content Area Applications of Education Technology "distributed course" began in the Fall, 1998 semester. Students in four content areas in the Secondary Certification Program were given hands-on instruction in a computer lab throughout the semester within the context of a required course in the redesigned program, Teaching in a Diverse Society. This gave the teacher education faculty an opportunity to refine the syllabus integrating technology activities while gaining familiarity with the TEbase tool described below.
During this first semester, students learned how to use email for one-to-one communication and web-based asynchronous conferencing systems for group discussions. Students were assigned readings related to the use of technology in K-12 classrooms, searched the World Wide Web for content related resources, and learned how to evaluate the content of web pages and CD-ROM software for use in their content areas. Students also learned how to create personal web pages and upload files to a student server. Based on the course content, some students also began using digital cameras, presentation software, graphing calculators and spreadsheet programs. As students completed these assignments, they submitted them to TEbase for grading.
The reconstituted TEbase has become an organization and management tool for students and faculty. Students use TEbase to submit individual and group assignments for each course taken over a two-year period. Using a web interface, these assignments can be typed directly into the database or students can enter a URL that points to the location of their assignment (usually linked to their personal homepage or web-based discussion archive) that can be accessed at a later date in the development of a professional portfolio which is a requirement for graduation. At the present time, all students are required to develop a hard copy portfolio although we are beginning to see many students developing both a hard copy and an electronic portfolio we call e-portfolio. This portfolio will be used by students looking for teaching positions after graduation and later in their professional careers.
While students use TEbase as an organizational tool, faculty use TEbase to provide students with feedback on their assignments, grade and record assignments, track students progress of their work, publish exemplary student work, and map technology standards to the different assignments. While teachers do have the option of requesting paper copies of the assignments, faculty have expressed appreciation for the ease of using the Web to access students’ assignments either from the university or their home.
Each secondary content area course is taught by different faculty members, each with their own teaching style and focus, so technology activities identified by the technology instructor have been adapted accordingly and integrated at different times during each semester. Since students use technology in each of their courses throughout a two-year period, TEbase was redesigned after the first semester of implementation to monitor technology assignments and competencies by student rather than by course. This redesign was needed to help the technology instructor monitor the integration of technology into the content area courses across the entire certification program, to provide a more flexible database to accommodate unlimited numbers of activities, and to provide a tool for faculty that is easy to use.
In Fall, 1998, there were six junior-year education classes using TEbases: two elementary education methods classes and separate methods classes for students preparing for secondary-level certification in math, science, English and social studies. In all, there were 74 elementary education students and 148 secondary education students using the TEbases. Each TEbase was built around a different set of activities, and all of these activities were related back to the 18 ISTE/NCATE technology competencies.
An extensive presurvey of these third-year students was conducted online via a web-based interface early in the semester. The students were asked about their experiences with educational technology, their access to technology resources, and their level of expertise with specific educational technologies. There were 195 valid responses, a return rate of 88 percent.
The presurvey revealed high levels of access to computing hardware:
Four respondents rated their own overall computing skills as expert, 20 percent (39) as above average, 64 percent (125) as average and 14 percent (27) as below average. Many respondents also reported high levels of experience with certain categories of software:
Web Use:
Email Use:
Office Applications:
Online discussion:
Multimedia Creation:
Use of CD-ROM Resources:
Given the students’ level of experience with general computing, it is perhaps not surprising that they adapted quite rapidly to using the TEbases. (Several of their teachers required more hand-holding.) By the end of November, it was apparent that the TEbases had been adopted as integral parts of their respective courses in all but one instance. It had been used to record five observation assignments in each of the two elementary methods courses, for seven activities in the secondary social science section; for five assignments in the secondary math section; and for 14 assignments in the secondary science section. In the secondary English section, the teacher did not use the TEbase with her students but the technology course coordinator used it to track two assignments with those students.
There were few difficulties with the technical operation of the TEbases. Nearly 2,000 logins had been recorded as of Nov. 30, with fewer than a half-dozen students reporting difficulties (in all but one case the problem was an incorrectly entered login name or password). Most encouragingly, there was no evidence of any shortfall in grading or provision of feedback for any of the courses using the TEbases, in marked contrast to the previous year’s experience with the Technology Competencies Database. In the TCD and TEbase grading scheme, work that is judged acceptable appears on the web page with a "star" icon for a grade, and exemplary work is given a "double star." Work awaiting a grade appears with an "hourglass" icon, while work which requires revision receives a "question mark" icon, and assignments which the students have not yet attempted appear with "blank" icons. As of Nov. 30, 1998, the breakdown of grades awarded by class was as follows:
In total, 604 submissions were graded acceptable (star) or exemplary (double star) during the period ending Nov. 30, 1998. Decisions were pending (hourglasses) on 32 submissions, for a grading completion rate of 95 percent. In many instances, the teachers supplemented these grades with written feedback, likely adding to the pedagogical value of the grades awarded. (It remains to be learned how this feedback would compare in substance and length to what instructors might have given had the assignments been completed on paper.)
Handler and Strudler (1997) recommend that the ISTE Foundation Standards should be integrated systematically into the curricula of teacher preparation programs, in ways which might vary from one context to another. Our experience provides strong support for this recommendation. Whereas the initial instructor and graduate assistant were unable to grade more than 50 percent of the 176 competencies descriptions submitted by their students even by the end of the school year, the teachers and coordinator of the Fall, 1998 classes were able to grade 95 percent of their students’ submissions soon after the submissions were made. Individual differences between this year’s and last year’s teachers and students may account for some part of this, as may the shift from voluntary to required participation. Yet it appears likely that in large measure the improvement is due to the difference in activity types. The activities in the Technology Competencies Database were the ISTE/NCATE technology standards themselves, and students were instructed to complete these in any order they chose. The activities in the TEbases were the instructors’ own assignments, and these were due in a specific sequence on specific dates. We believe the improvement we have found offers strong evidence that it is more efficient to give instructors the leeway to set their own assignments in the TEbase than to require them to grade students’ completions of the competencies.
The TEbase mechanism performs several important bridging functions within UIUC’s new teacher education curricula. First is the bridge between individual class activities and the ISTE Foundation Standards. Methods teachers and the technology course teacher work together to construct assignments which promote learning in each competency area. Second is the bridge across courses — the TEbases function as electronic gradebooks which operate much the same from one class to the next, permitting students and faculty members to track learning consistently across the two-year program. Furthermore, the TEbases offer bridges between individual learning experiences and a coherent body of knowledge. At the individual level, this coherence takes the form of an electronic portfolio which each student can point to as documentation of their learning about educational technology. At the social level, the TEbases’ ability to automate publication of exemplary work allows students to contributed to a high quality web resource that grows into something of value not only to the particular students and faculty, but to the educational world more generally.
There is a great deal more to be learned about the workings of the TEbases, and how they may be adapted further in order to ensure that the promise of instructional reform contained in the new teacher education curricula is in fact met. For this reason, we will be embarking in the coming year on more formal evaluation procedures, to investigate in an unbiased manner whether the benefits we think are occurring really are. Our course of action so far has been a continuing cycle of design-evaluation-adaptation. We believe this general course represents the best manner of attaining an optimal match of resources and curriculum. As evaluation efforts are formalized in the months ahead, we believe it will be important to maintain close ties to the evaluation itself, so as to benefit maximally from the process of investigation.
Bradley, A. (1997). NCATE told to emphasize technology. EducationWeek, 17(3) (Sept. 17).
Brehm, B., Moran, J., Bievenue, L., Andris, J., Handler, M., Levin, J., & Payne, A. (1998 in press). Integrating technology intoteacher and administrator preparation programs: Five models in Illinois. IASCD Journal.
Guernsey, L. (1997). Education schools must incorporate technology into courses, report says. Chronicle of Higher Education, Sept. 10, 1997.
Handler, M. G., & Strudler, N. (1997). The ISTE Foundation Standards: Issues of implementation. Journal of Computing in Teacher Education,13(2), 16-23.
Levin, B. B. (1996). Using portfolios to fulfill ISTE/NCATE technology requirements for preservice teacher candidates. Journal of Computing in Teacher Education, 12(3), 13-20.
Levin, J. A., Riel, M., Miyake, N., & Cohen, M. (1987). Education on the electronic frontier: Teleapprenticeships in globally distributed educational contexts. Contemporary Educational Psychology,12, 254-260.
Levin, J., Waugh, M., Brown, D., & Clift, R. (1994). Teaching Teleapprenticeships: A new organizational framework for improving teacher education using electronic networks. Journal of Machine-Mediated Learning, 4(2 & 3), 149-161.
Levin, J. A., & Waugh, M. L. (1998 in press). Teaching Teleapprenticeships: Frameworks for integrating technology into teacher education. Interactive Learning Environments.
Moore, J. (1997). Technology and the Year-Long Project. Unpublished report, Department of Educational Psychology, Champaign IL: University of Illinois.
National Council for Accreditation of Teacher Education, Task Force on Technology and Teacher Education. (1997). Technology and the new professional teacher: Preparing for the 21st century classroom. Washington DC.
Office of Technology Assessment (OTA). (April 1995). Teachers and technology: Making the connection. (OTA-EHR-616) Washington, DC: U.S. Government Printing Office. [Although the OTA report is out of print, it may be downloaded from the Web at http://www.wws.princeton.edu/~ota/disk1/1995/9541.html.]
President's Committee of Advisors on Science and Technology, Panel on Educational Technology. (March 1997). Report to the President on the use of technology to strengthen K-12 education in the United States. Washington DC.
Thurston, Catherine O., Secaras, Evangeline, & Levin, James A. (1996). Teaching Teleapprenticeships: An innovative model for technology integration in teacher education. Journal of Research on Computing in Education, 28(5), [ONLINE] Availableat <http://www.educ.ksu.edu/Projects/JRCE/v28-5/Thurston/article/main.htm>.
Zehr, M. (1997). Teaching the teachers. Education Week (Special Issue: Technology Counts) 17(11) (Nov. 10), 23-29.