Formative Evaluation of
Biology Workbench Education Enhancements
David M. Stone, University Laboratory High School, Urbana, IL, USA
Overview of Report Contents
I. Introduction
II. Project History
III. Focus of the Evaluation
IV. Summary of Evaluation Findings
- Project Structure and Approach
- Web Site Design and Organization
- Activity/Tutorial Evaluation
V. Conclusions and Recommendations
This formative evaluation is intended as both an historical document and current status assessment of Biology Workbench Education Enhancements, a computer-based molecular biology project which aims to incorporate molecular biology and computational science into middle school, high school and undergraduate biology teaching. As well as serving in an evaluation capacity, I am currently a Collaborating Educator in this initiative and believe it to be a project which:
Evaluation findings will be made available to the project's current collaborating constituents (the NCSA Biology Workbench Education Team and Collaborating Educators) as well as the students and staff of the Fall, 1998 C&I 490 EIT course.
Biology Workbench Education Enhancements is currently undergoing a major restructuring. The first meeting of all involved parties (NCSA Biology Workbench Education Team and Collaborating Educators) to be held in five months, occurred on October 28, 1998. It was clear from the onset, that the areas of restructuring include personnel, development approach and audience. A flurry of web site activity had occurred several days prior, with a number of minor changes in the interface and major restructuring of some of the tutorials and lessons, including more screen hots and explicit directions. It appears that a major change in the project leadership has occurred, with Umesh Thakkar's original role being reassigned to Raul Zaritsky and Scott Lathrop of NCSA's Education Division, though this reassignment was not made official at the meeting. A number of new modules including an evolutionary molecular biology and a developmental protein module were presented. Zaritsky suggested a new structure in which teachers and developers work together in small groups on development of teacher-initiated modules and perform several rapid trials with increasing numbers of students in each trial, rather than using the previous large group development model, which has yet to result in a final version of any of its modules. A new module dealing with glycolysis/fermentation/respiration is currently being discussed and will likely be written during the academic year. The NCSA Biology Workbench Education Team has been working with a BioQuest, a commercial software education company. Their discussions will likely shape the future direction of Biology Workbench Education Enhancements efforts. Based on the October 28, 1998 meeting, it appears that the development focus is moving from middle and high school to a much increased focus on undergraduate level activities, centered particularly within the area of biochemistry. A significant part of this evaluation is based on recent activities and structure made public by National Center for Supercomputing Applications staff during that meeting.
Information and views presented in this document are solely those of the evaluator, David M. Stone. Use of this document for reasons other than incorporation into Biology Workbench Education Enhancements development efforts is inappropriate and unintended.
Historical background regarding this development initiative is essential in understanding this project's current and future development efforts. Background regarding Biology Workbench is provided to give readers an understanding of the main software used for procuring data, data analysis and presentation of graphic- and non-graphic based protein and gene sequence information.
Biology Workbench (Professional Version) Background
Biology Workbench, a molecular biology software tool developed in 1996 by the Computational Biology Group at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign, allows scientists rapid access to protein and genetic sequence databases and sequence analysis tools which reside on numerous remote machines located throughout the world, via a web browser such as Netscape or Microsoft Explorer. Used by researchers in areas ranging from genomic analysis and taxonomy to the physical sciences, its potential applications extend well beyond its current use. Developed to facilitate the work of scientists well familiar with the myriad of established protein and databases, which often differ significantly in communication syntax, Biology Workbench possesses a fairly high learning curve and appears quite foreign to anyone unacquainted with the databases or analysis tools made more accessible through the Biology Workbench interface.
Biology Workbench Education Enhancements Background
In December, 1997, as part of its charge and in response to recent reports on the need for expansion of technological efforts to achieve the goals of a current education reform, the Education Division of NCSA began efforts toward development of an Education Version of Biology Workbench, presently known as Biology Workbench Education Enhancement. Targeted initially toward middle and high school audiences, current efforts are the result of the combined contributions of University of Illinois scientists, undergraduate and graduate students and a cadre of Collaborating Educators. The team is currently developing an underlying set of tutorials and activities which provide additional support and background for teachers and students using Biology Workbench Education Enhancements. Scaled-down from the full professional version, Biology Workbench Education Enhancements provides a more limited menu of databases and fewer options for database query, effectively leading the new user to more successful, early use experiences.
This evaluation incorporates a number of different approaches, collectively aimed toward better understanding the project in its entirety. The sequence of activities in this evaluation were: 1) initial assembly and reading of pertinent material, 2) selection of individuals to interview and development of questions for the project's Principal Investigators, 3) examination of the web site for organization, readability and interface considerations, 4) review of a newly written tutorial activity, 5) development of an evaluation instrument for students using the activity and finally, 6) student and teacher evaluation of the activity. Each of these components is discussed below, in the order presented above.
1. Assembly and Reading of Pertinent Literature (Completed October 6, 1998).
I began by performing a literature search using the search option on the National Center for Supercomputing Applications home page. The documents I was able to find, or which were recommended by staff associated with the development project, were limited to several NCSA Education Division in-house documents which contained brief paragraphs regarding the initiative and a five paragraph web document, entitled "Education Enhancements to the Biology Workbench," written by Dr. Eric Jakobsson, Lisa Bievenue and Dr. Umesh Thakkar. Much of this information is presented in the "Project History" section of this formative evaluation.
2. Selection of Individuals to Interview / Development of a Project Question Set
Dr. Eric Jakobsson has been the Biology Workbench Education Enhancements Project Director and Dr. Umesh Thakkar has been the NCSA Education Division Representative in this initiative since its December, 1997 beginning. I intended to ask both individuals the following questions, though I was able to schedule an interview only with Dr. Thakkar. I asked the following questions in the order presented, though I made an effort to allow for deviation from the questions as they were presented, steering back to my questions when appropriate:
Summary of Thakkar Interview
My interview with Dr. Umesh Thakkar took place at University Laboratory High School on October 6, 1998. Dr. Thakkar has been the NCSA Education Division Representative in this initiative since its beginning, and provided some of the historical information presented in the "Project History" section. Thakkar believes the major stakeholders to be 1) the Education Division of NCSA, 2) the local teachers involved in development, 3) a group called NCSA Alliance Partners, 3) University of Illinois scientists associated with Biology Workbench Education Enhancements development and 4) BioQuest, a commercial education product developer which has an interest in incorporating some (all?) aspects of Biology Workbench Education Enhancements Background into its development efforts. Timelines for project completion were established in NCSA in-house documents, and Thakkar stated that those timelines have been met. Project funding appears to have come almost exclusively from the Education Division of NCSA. A recently awarded Education, Outreach and Training grant pays the salaries of two undergraduate students who are working on the project. Thakkar believes the strengths of this project include the involvement of middle and high school teachers, equitable sharing of work between all of the parties involved in development and the provision of technology (retired NCSA Macintosh 8100s) and a $100 stipend for each of the teachers. Rather than dealing with weaknesses, Thakkar went on to discuss the next step in the project, which should include expansion of topics beyond the current four topics in development at that point, more presentations by teachers, funding for continued teacher involvement and expansion to include a number of additional teacher-developers. Thakkar also chose to address aspirations when queried regarding areas of concern. His main aspiration was that there be continued dialogue between the development team, teachers, undergraduates and University of Illinois faculty. He suggested that the major area of evaluation efforts should center around web design. Specifically, can teachers successfully navigate and use the existing interface? He suggested that I pick one of the two current lessons I had initially suggested (either sickle cell anemia or cystic fibrosis) and examine it in depth. At this point, neither version of Biology Workbench has undergone a formal evaluation. One of Thakkar's Summer, 1998 activities involved examination of how practicing molecular biologists use the current version of Biology Workbench, the professional version.Three practicing molecular biologists, considered experts in using Biology Workbench, were observed as they used this tool during May 31 - June 4, 1998. Two of the three scientists had difficulty using the query functions and understanding its required syntax. One scientist used Biology Workbench for initial searches and then used those results to continue her efforts using a different database she considered more robust. Participants liked the design of the site and the interface, though they found completion of tasks using Biology Workbench somewhat frustrating.
3. Examination of Web Site For Organization, Readability and Interface Considerations (Completed October 17, 1998)
I began this component of the evaluation by using six commonly used Internet search engines: AltaVista, Excite, Hotbot, Infoseek, LookSmart, and Lycos. Interestingly, two of the search engines (AltaVista and Looksmart) found several of the Biology Workbench Workbench Education Enhancements pages, two (Excite and Infoseek) found only the project home page and two (Hotbot and Lycos) were unable to find any pages associated with the project.
The project home page features an attractive logo and three sentence fragments which I assume to be placeholders for information yet to be stated. As currently presented - "Teacher resources. Tutorials using the Biology Workbench. to help students learn the science of biology, bioinformatics, evolution, and more." - this place holding section looks unprofessional, giving the reader a poor initial impression of the project and likely leaves them wondering about the degree of English literacy possessed by the project developers. The project home page at that time had links to five sections: 1) Introduction to Biology Workbench, 2) Tutorials, 3) Collaborating Educators, 4) NCSA Development Team and 5) Templates for Creating Tutorials. "Introduction to Biology Workbench," an introduction to the professional version, is well written and organized. Developers should consider adding a second introduction, entitled "Introduction to Biology Workbench Education Enhancements," that introduces the education version and includes a brief history and presentation of this version's goals and objectives. The "Tutorials" page is awkward in its appearance, and could be streamlined, with the topics divided into two major categories: Molecular Evolution Tutorials and Genetic Disorder Tutorials. The contributor pages ("Collaborating Educators" and "NCSA Development Team") differ dramatically in their appearance. The "NCSA Development Team" page is nicely formatted, though I suggest the major players, Jakobsson and Thakkar, be listed first and the rest of the development team be listed in alphabetical order. The "Collaborating Educators" page should use the presentation format of the "NCSA Development Team" page. Collaborating Educators should be listed alphabetically, with links to their personal and school home pages. Listing of the grades and disciplines taught is not necessary, though they can be included. The final section, Templates for Creating Tutorials, is clearly in a (pre-?) beta testing stage, so I won't deal with it at this point, though I'm very pleased with what I see of the template so far, and think once it is developed, it will significantly increase the potential of Biology Workbench Education Enhancements incorporation into specific classroom lessons, and may allow for successful use of this application for more activities at the middle school level. I suggest the development team consider expanding the number of sections and modifying their order to the following: 1) Introduction to Biology Workbench, 2) Introduction to Biology Workbench Education Enhancements, 3) NCSA Development Team, 4) Collaborating Educators, 5) Tutorials and 6) Templates for Creating Tutorials.
In terms of initial appearance, the site is well designed, with an attractive logo at the top of each page. Page designers might consider a smaller version of the logo to reduce, and in some cases eliminate, the need for scrolling. A counter indicating the number of visitors is included, dates of most recent update, and consistently positioned Home buttons allow for rapid navigation throughout the set of documents. The lack of Java-based items allows for rapid uploading of all documents.
4. Review of a Newly Written Biology Workbench Education Enhancements Tutorial
As a result of the October 28, 1998 meeting, I significantly modified my evaluation plans from those plan initially proposed in my original "Preliminary Evaluation Plan for Biology Workbench Education Enhancements". I am currently working with Ajit Chary, an undergraduate biology major and new member of the NCSA Biology Workbench Education Team, on his molecular evolution activity entitled "Myoglobin as a Probe for Understanding Molecular Evolution". He put a new version of his activity, on the WWW November 3, 1998. I reviewed the activity the following week, making suggested modifications to the activity,which can be found in section IV. B. below.
5. Development of an Evaluation Instrument for Students Using the Activity
In an effort to determine how well Chary's "Myoglobin as a Probe for Understanding Molecular Evolution" activity works in a classroom situation, my Introductory Biology classes, consisting of 80 freshmen, sophomores and juniors reviewed this activity during their scheduled class periods on November 9 and 10, 1998. They were introduced to this activity through a single page written introduction. Working either individually or in pairs, each student performed each of the activities while critiquing each page of the tutorial, using the tutorial evaluation form I developed for that purpose. Student feedback is summarized in section IV. B. below.
IV. Summary of Evaluation Findings
I began planning an evaluation of Biology Workbench Education Enhancements in June, 1998. Since that time its potential to enhance the teaching of many aspects of life sciences (e.g. genetics, evolution and visualization of biological compounds), increase equitable access to high quality biology instructional materials due to its presence as a web-based resource and its potential as collaborative model for the production of high quality teaching materials has become exceptionally clear. Evaluation findings are divided into three broad categories: 1) Project Structure and Approach, 2) Web Site Design and Organization and 3) Activity/Tutorial Evaluation.
1. Project Structure and Approach
From the Collaborating Educator perspective, I believe some aspects of the development have been in a holding pattern since Spring, 1998, and have had concerns regarding the future of the project. An article I encountered during Jim Levin's Ed Psych 387 class, Judi Harris' "Organizing and Facilitating Telecollaborative Projects," made clear to me how and why the teacher collaboration component of the project appeared to be stagnating. My analysis from July, 1998 can be accessed at http://www.uni.uiuc.edu/~dstone/harris.html . The project as a whole is undergoing a number of changes outlined in the first paragraph of the "Limitations of the Evaluation" section above which I believe will catalyze efforts required for this initiative to become broadly useful.
2. Web Site Design and Organization
This information is presented in III 3. above. A recently initiated reorganization of information and format is in progress, which is resulting in a more consistent style of presentation and more cohesive presentation within each of the current modules.
3. Activity/Tutorial Evaluation
Evaluator Critique
On November 9, 1998, I read through and critiqued Ajit Chary's tutorial "Myoglobin as a Probe for Understanding Molecular Evolution". I will be working with him on this tutorial as part of the new structure of a Biology Workbench Education Enhancements development team. I noted the following regarding the Biology Workbench Workbench Education Enhancements Home Page, the Introduction to Biology Workbench and the "Myoglobin as a Probe for Understanding Molecular Evolution" tutorial. The Biology Workbench Workbench Education Enhancements Home Page has an excellent introductory interface. The introductory set of sentences have been improved from those I encountered and discussed above in III 3., though it still is in need of rewording, so that individuals who have never used Biology Workbench can understand its use(s). The first document accessible from the Biology Workbench Workbench Education Enhancements Home Page, Introduction to the Biology Workbench, is very well written and easily navigable. The second link from the Home Page, Biology Workbench Education Tutorials, is nicely constructed, except that it lacks a logical order of presentation or grouping of topics. The first three pages of the tutorial following the title page are well written for the intended audience, though there is clearly a need for greater care in spelling and punctuation throughout those pages. The following six pages are quite good, though it should be made clear how those who are already registered for use of Biology Workbench can enter using their name and password, how to use the combination of pressing the "shift" key and the mouse simultaneously to select several different species whose names are not immediately adjacent to each other in the scroll boxes, and the words "select" and "click" need to be used consistently for the correct action to be performed, not interchangeably, as is the case in the current version of the tutorial. Page 11 of the tutorial is missing, which prevents students from being able to easily navigate the tutorial. The tutorial is becoming quite usable, though it is still in need of significant development before it can be used in any classroom situation.
Student Critique
Students found the Biology Workbench Education Enhancements Home Page, Introduction to the Biology Workbench, and "Myoglobin as a Probe for Understanding Molecular Evolution" tutorial to be generally quite good and the majority enjoyed working with it. Students commented very positively on the readability of the Introduction to the Biology Workbench web page, and the majority enjoyed the initial approach of the tutorial. Concerns consistently expressed include the need for spell, grammar and punctuation checking, as well as the need to explain why certain databases or software tools are selected for different operations. The missing page of instructions interfered significantly with many students ability to work with the software without significant teacher assistance. My concerns stated in previous paragraph regarding selecting nonadjacent species and the inappropriate use of the terms "click" and "select" were mentioned by a number of the students. Approximately 10% of the students mentioned that they ended up blindly following the directions regarding selection of various databases and software tools to the extent that they were less sure of why they had performed the activity after having spent two days working with it, than they were at the start of the undertaking.
V. Conclusions and Recommendations
Biology Workbench clearly has excellent potential to significantly impact student learning of biology concepts in middle school, high school and undergraduate teaching situations. The greatest strengths of this project include the expertise and broad experiences of the NCSA Development Team and the Collaborating Educators. The structural reorganization recently undertaken can be quite positive if the Principal Investigators can collaboratively develop a common vision of Biology Workbench Education Enhancement's versatility and potential teaching applications. A strong group of dedicated Collaborating Educators is paramount in the selection of topics which will complement topics taught at their grade levels and in their specific subjects. I like the new approach to collaboration in development and believe the revised approach will significantly increase the likelihood of production of a quality, easily disseminated product.The site designer possesses strong web page development skills and the graphics are visually appealing. Biology Workbench Education Enhancements activities of the Principal Investigators are funded through NCSA and their departments, so it appears that their activities with Biology Workbench Education Enhancements over the next several are likely to continue.
Concerns at this point center around several key elements. First, the Principal Investigators must be able to clearly establish project priorities and a common vision of direction of the initiative. Second, the Principal Investigators must be able to articulate that vision to widely varying audiences including NCSA, teachers, students and professors, as well representatives of funding agencies such as the National Science Foundation. Third, grade level of the intended audience needs to be determined AND instructors at that grade level MUST be able to influence direction of development initiatives so that the efforts will be appropriate for those grade levels. Fourth, Collaborating Educators need to be selected on a broader basis than the local audience used thus far. Selecting the Collaborating Educators from a national audience significantly increases the likelihood that we can establish a highly motivated group to play a more active role in the development of Biology Workbench Education Enhancements. Finally, the web site and its modules need to be more closely proofed and edited. Elementary, middle and high school students are expected to make use of various web-editing and proofing tools, and have their projects compared against a number of web-publishing guidelines. We should expect no less in terms of the quality of Biology Workbench Education Enhancements activities those students will use in their classroom.
The current Biology Workbench Education Enhancements cannot stand on its own as a teaching tool, or even as a single day activity. Teachers using Biology Workbench Education Enhancements need to have significant background knowledge regarding various protein and nucleic acid databases, as well as the tools used to access and analyze those databases. They must also be conversant in reading and understanding phylogenetic trees, particularly the type produced by Biology Workbench. I found the students who most enjoyed working with Biology Workbench Education Enhancements were those who finished the tutorial relatively quickly, and then developed their own protein queries and selected their own set of species for comparison. Toward this end, a list of proteins most commonly queried, and a brief description of their function, would be exceptionally useful so that all students could perform their own queries immediately after completing their first tutorial. A well written Teacher's Manual with numerous activities for several different grade levels is essential for the majority of teachers considering using the program, and initiatives toward its development should begin immediately.
I have several suggestions regarding 1) molecules to consider to better assist other teachers and students in understanding the potential of Biology Workbench, 2) introductory topics to consider in middle and high school module development, and 3) advanced topics better suited to students with previous Biology Workbench experience or currently taking upper level high school and/or undergraduate biology courses. I hope the following ideas will serve as a catalyst for further conversation and project emphasis consideration.
1. Comparison Molecules to Incorporate Into Tutorials
2. 1st and 2nd Year Biology Topics Topics Workbench Can Reinforce
3. 2nd Year High School/Undergraduate Biology Topics Workbench Can Reinforce
Lastly, a number of generally useful visualizations could easily
be combined into relatively easy tutorials, which would be broadly
useful for students regardless of grade level. I suggest the development
team begin with protein structure and mutagen impact visualizations.
The protein visualization should allow the user the ability to
select and/or make up a nucleotide or coding base sequence, seamlessly
examine its secondary and tertiary structure and see how the polypeptide
chain might interact with other chains (quarternary structure).
The mutagen impact visualization should allow the user to select
a base sequence, induce a mutation in a "hot spot" (repetitive
sequence) by exposing it to various different mutagens (e.g. uv
light, asbestos, etc). The visualization should allow each user
to have a split screen, showing the normal and mutated proteins,
as well as allow the user the ability to rotate each of those
molecules simultaneously. This would nicely reinforce the idea
that mutation impacts can range from negligible to severe.
Created October 3, 1998. Last modified November 15, 1998.