James A. Levin
University of Illinois at Urbana-Champaign
University of Illinois at Urbana-Champaign
Michael J. Jacobson
Vanderbilt University and University of Illinois at Urbana-Champaign
Paper presented at AERA '96, New York.
Given the phenomenal changes in the use of network computer technology in education, it has become important to better understand teachers' conceptions of this technology and its uses. This is particularly true in the case of conceptions, or "mental models", of the Internet and its uses. Two questions are addressed: How do novices and experts think about the Internet and its applications (e-mail and WWW)? Does a person's conceptual model have an impact on network use? To examine the role of these mental models in more detail, a survey and eight case studies were conducted among pre- and in-service teachers enrolled in University courses. The survey solicited mental models of the Internet, e-mail and the World Wide Web. The case studies consisted of a network problem-solving task using think-aloud protocols. All subjects identified themselves as to their technical expertise. The results show a surprising diverse set of plausible models of the Internet and the Web, not related to their expertise level. Epistemic relationships to mental models are also discussed. The case studies reveal some relationship between model and navigation strategies, where experts employ multiple, structurally and/or epistemologically different models at different stages in a network task. These findings are discussed with respect to educational applications of the Internet.
It has been four years since the Center for Technology and Education's (CTE) national survey of K-12 telecommunications was conducted (Honey & Henriquez, 1992). One important implication from this survey was that there is a need to better understand teachers' conceptions of technology and its uses. Honey and McMillan followed up this survey with in-depth interviews of 18 teachers, and found different "mental models" ("superhighway" vs. "ocean") among the teachers. Previous to and since these studies there has been little research in how mental models might impact network usage, navigation, and learning. Research of this type is vital if we are to understand how to use network learning environments productively in schools by teachers and students.
To examine the role of these mental models in more detail, we have conducted an exploratory study of mental models of the Internet, E-mail and the World Wide Web. These studies consisted of surveys of mental models and case studies of network problem solving using think-aloud protocols. This paper will address the following questions: How do novices and experts think about the Internet and its applications (e-mail and Web)? Does a person's conceptual model have an impact on network use? What are the implications for interface design, network training, server development, and other educational uses of the Internet?
We are interested in the "mental models" users have for using computer networks. Mental models are those internal, often informal and naive, representations that guide the user's actions on a task, creation, or problem (Gentner & Gentner, 1983; Johnson-Laird, 1983; McCloskey, 1983; Norman, 1983; Norman & Draper, 1986). Mental models often have strong visual images of the system to explain its function, or they can be metaphorical explanations as to its meaning in the mind of the individual. Together, these two features of mental models provide guidelines for acting upon the system, especially when no explicit strategies exist (Norman, 1988; Norman & Draper, 1986).
The goal of this study was not to find the most plausible models that explain the Internet or Web, but to empirically generate a categorization scheme for common mental models and to determine if those models are employed in any systematic manner in network tasks. It was also our goal to isolate the components and characteristics of mental models, their points of failure, and their adaptability to problem solving and network navigation. If a particular class of mental model predicts performance on a relevant task, then it might be used as an instructional model for training and designing network learning environments. The dimensions of this analysis include the classification of the models, their dynamic attributes over time and in relation to expertise, the extensibility of the models to other settings, and the epistemic qualities of the models.
This is a qualitative study using data from surveys, interviews, and observations. The subjects consisted of pre- and in-service teachers from the University of Illinois at Urbana-Champaign (UIUC) and the University of California, San Diego (UCSD). Other volunteers from UIUC were also studied. Three instruments were used to collect data. The first was a questionnaire to elicit self-descriptions of mental models from the subjects. Before eliciting the individual's mental model, we first provided an example of a mental model:
"For example, people often think about the atom as being like a miniature solar system. The solar system is then a mental model of the atom."
As shown in data from this study, this analogy was usually sufficient for eliciting mental models from the subjects. The questionnaire was administered at the beginning and end of the courses, and included interval scale assessments of the subject's computer and network experience and as well as their teaching experience. The remainder of the survey were open-ended questions about mental models of the Internet, Gopher, and World Wide Web.
The second instrument was a task analysis of the problem solving behavior and analysis of think-aloud protocols of subjects tackling two problems using the WWW on the Internet. The third instrument was a trace of network navigation during the think-aloud sessions. Those traces were used to augment the transcriptions from the think-alouds and to confirm navigational patterns. The variables in this study are: (a) experience of user, (b) the network task or situated problem, and (c) the robustness of individual models as they relate to actual use of the Internet in the context of the task. We are interested in the mental models of novices and experts, measured in terms of technical expertise, familiarity with the Internet and telecommunications, and professional expertise.
A pilot study was conducted in the Spring of 1995. Teacher education students enrolled in courses at UIUC responded to a survey of their mental models. Data from that pilot study were used to refine the survey and design a think-aloud protocol for observing subjects during a network problem-solving task. From the pilot study, we discovered a systematic problem with the phrasing of the questions. We were concerned that the subjects would not know what the term mental model meant, so we asked the subjects:
"What is your image of the Internet?"
This generated unpredictable responses, usually about the subject's opinion of the Internet. As a result, we re-worded using the term "mental model" the survey and provided an example of one (as mentioned above). The final survey is shown in the Appendix A. The network task and the think-aloud protocol is shown in Appendix B. This revised survey was used during the Summer and Fall of 1995 with three subject groups selected for their levels of network experience.
For the survey portion of this study, three groups of network users were studied. The first group was a math education class (C&I 336) at UIUC. The students in that class were mostly in-service math teachers working towards a Masters degree during the Summer 1995 session. As part of this course, the students were provided with laptop computers equipped with Internet tools. One of the goals of this course was to expose the teachers to math education resources on the Internet. The second group was a teacher education class (TEP 231) at UCSD. The students in that class were all practicing K-12 teachers enrolled in a master's program, also enrolled during the Summer 1995 session. The class focused on technology and education. The third group was an educational computing class (EDPSY 387) at UIUC enrolled during the Fall of 1995.
For the case studies, volunteers were solicited from each class, as well as from expert network users within the College of Education at UIUC. The subjects were informed that the interview would be recorded on audio tape, and that it would take less than one hour.
The survey was repeated at the end of each course, but without the questions regarding self-evaluation of skills. In all, 61 subjects from the three classes responded to at least one survey.
Responses to the self-evaluation were converted into scalar values from 0 (none) to 4 (lots). These indices were used to determine an overall measure of expertise. This is referred to as the Expertise Factor and is equal to the average of the six expertise questions (experience with computers in general, networks, e-mail, newsgroups, gopher, and Web). However, to better tune the Expertise Factor for the purposes of this study, the general computing value was weighted twice and the e-mail and Web values weighted 1.5 times. This measure was used to rate the subjects' overall computer and network expertise into three classes: novice, intermediate, and expert. Possible values for the Expertise Factor ranged from 0 to 5.67.
With respect to the questions about mental models, not all subjects responded to all the questions. Some subjects drew diagrams of the models, while others pointed to earlier responses. Responses from all three groups were collapsed into a single matrix. Table 1 shows frequencies of responses for each group for the pre- and post- experience questionnaires. Non-responses include both blank responses and those explicitly not responses (responses of "don't know", "?", etc.)
Table 1. Response frequencies
The most striking result was the diversity of models. Appendix B shows the responses from all three groups. The responses were then shortened into a few-word statement about the model. Table 2 is a summary of all the unique responses to the Internet and Web questions that could be interpreted as models.
Table 2 (a). Internet models.
|Count||Pre-Experience Model||Count||Post-Experience Model|
|6||highway, interstates||6||web, spider web|
|3||phone calls, lines||1||computer chip|
|2||solar system||1||connection of information|
|1||fungus with tentacles||1||Lattice - Interconnected|
|1||Maze||1||network of clients and servers|
|1||nervous system||1||neural networks|
|1||Star Trek Enterprise||1||teleconnections|
|1||toy jacks||1||telephone system|
|1||wave - interactive||1||wave - surfing|
|39||22 Total Unique Responses||32||22 Total Unique Responses|
Table 2 (b). Web models.
|Count||Pre-Experience Model||Count||Post-Experience Model|
|12||Web, Spider Web||5||Web, spider web|
|2||BBS||2||information flow, sharing|
|2||information, storerooms||1||ant trails|
|2||link - international||1||BBS|
|2||multimedia book, lines||1||city|
|2||streets, roads, highways||1||communication|
|1||Access to Internet||1||community|
|1||brain synapses||1||conference room w/ encyclopedias|
|1||fungus||1||funnels - visible|
|1||graphical Internet||1||global village|
|1||Homebase||1||grain of sand|
|1||jacks||1||library - floating pages|
|1||nervous system||1||nebula - amorphous|
|1||solar system||1||network operating system|
|1||visual model of culture||1||neural networks|
|1||Star Trek computer|
|1||supermarket of ideas|
|1||tangle of connections|
|1||window on the world|
|39||18 Total Unique Responses||33||27 Total Unique Responses|
As shown in Table 2 (a and b), there are similarities in how subjects perceive the Web and the Internet. For some, the Web and the Internet are the same thing. For others who may be more expert, the difference between the Internet and the Web are represented in different models of the two. For still others, the name "Web" evokes a model that is sufficient. Indeed, "Web" and its variants was the modal response for the Internet question pre and post. It was expected that Web would be the modal response for the Web question, and Table 2(b) bears this out. What is interesting in Table 2(b) is how the subjects discovered for themselves more salient models for the Web, than "Web". "Web" is still the modal response in the post column with 5 responses, but this is down from 12 in the pre column.
Expert Versus Novice Models
Table 3 shows the frequency of responses in which the subject replied that he or she did not have any model, or that the question was not answered. These are referred to as non-responses. The non-responses were sorted by expertise. Levels for expertise were determined by values from the Expertise Factor:
Novice 0 to 1.99
Intermediate 2.00 to 3.99
Expert 4.00 to 5.67
Table 3 shows a definite pattern that indicates that novices are less likely to report having a model than more experienced users. The totals are represented in the total responses in Table 1.
Table 3. Percentages of non-responses.
Impact of experience on model change
The diversity of models increases with experience, especially for the Web models. Table 4 shows all of the subjects who responded to the Web model question pre- and post-training, sorted by expertise. Thirteen (13) models show a substantive change in model, seven (7) show no substantive change, and four (4) show a similar model. It is the group of subjects (subject numbers: 2, 11, 35, 44) whose models were similar that is most interesting. Table 5 explores their actual responses in detail.
Table 4. Response differences between pre-and post Web models, sorted by increasing expertise.
|Subject||Group||Experience Factor||Pre-Web Model||Post-Web Model||Difference|
|2||TEP 231||0.33||encyclopedia||information flow||similar|
|10||EDPSY 387||1.08||library||USA Today||change|
|11||CI 336||1.17||library||encyclopedia - 24-hour||similar|
|14||CI 336||1.42||Web||Web with strands||no change|
|25||CI 336||2.25||BBS||international BBS||no change|
|27||TEP 231||2.50||Access to Internet||tangle of connections||change|
|31||TEP 231||2.75||fungus||fungus||no change|
|33||TEP 231||2.83||streets||freeway||no change|
|35||EDPSY 387||3.00||information storerooms||publishing||similar|
|36||EDPSY 387||3.08||multimedia book||TV channel||change|
|38||TEP 231||3.08||spider web||library - floating pages||change|
|41||EDPSY 387||3.17||spider web||web||no change|
|42||EDPSY 387||3.17||library||network operating system||change|
|43||EDPSY 387||3.33||link - international||information sharing||change|
|44||EDPSY 387||3.50||brain synapses||neural networks||similar|
|45||EDPSY 387||3.58||BBS||window on the world||change|
|46||TEP 231||3.75||spider web||spider web||no change|
|52||EDPSY 387||4.33||link - pictures||nebula - amorphous||change|
|53||TEP 231||4.67||library - linked||supper table||change|
|57||TEP 231||5.17||multimedia lines||connected computers||no change|
Table 5. Actual survey responses for subject whose Web models were rated as similar pre and post.
|Subject||Pre-Web Response||Post-Web Response|
|2||Huge encyclopedia with unlimited information available to everybody||Information going back and forth in all directions. No beginning/no end [a drawing of a network of nodes with interconnecting arrows]|
|35||Multi-faceted, Individually Based Storerooms Of Information||Opportunity for Individuals To Immediately "Publish" + Utilize Information|
|44||something like the brain, with tons of synaptic connections||neural networks|
Table 5 shows an emergence of more descriptive models, each in unique ways. For example, the models for subjects 2 and 35 take on a publishing component. This is an indication of their expertise gained in creating Web pages in their courses. Subject 11's model goes from a spatial library model to one of an encyclopedia, which represents the (graphical) information contained in the library. The models for subject 44 stay aligned toward the brain model, but change in structural specificity from the randomness of synapses to the structure of neural networks.
Ten volunteers were obtained to participate in an in-depth interview of using the Web on a task: two (novices) from CI 336, four (intermediate) subjects from TEP 231, one (expert) from EDPSY 387 and three (experts) from within the UIUC College of Education (two graduate students and one faculty member).
As in the survey study, we found diversity among the models of these subject. Initially we looked for a different in the kind of mental models used by experts from those used by novices.
In our initial studies of the case studies, there are two major phenomena that stand out. Both of these are seen most clearly by contrasting the case studies of the two novices with the case studies of the four experts.
NovicesFor the two novices, the transcripts of their network problem solving efforts indicate relatively vague and undifferentiated models for the Internet and the Web. Subject N1 described a "superhighway" model for the Internet, and described the Web as "sort of like the Internet." Subject N2 described both the Internet and the Web as "kind of like a spider web." This lack of specificity is not surprising, since both reporting only having seen a demonstration of the web during class the previous week.
ExpertsFor the four case studies of people with considerable expertise with the Internet, Email, and the Web, their models are much more articulated, especially in contrast to the novices. In addition, in their transcripts, three of the four experts described using at least two different models, and in fact in two cases, articulated reasons for using the different models in different situations.
Here is an example of what one expert had to say in the case study:
[describing his search strategy] "But I think although theoretically, I think that rhizomatic structures are interesting I still think....tap root hierarchical fashion. Let's go to the central sources first - let's get those names things and then.. "
So the two main phenomena are:
These findings are similar to the outcomes of previous studies contrasting experts and novices (Larkin, McDermott, Simon, & Simon, 1980; Chi, Feltovich, & Glaser, 1981). Novice physics problem solving have only one way to think about physics problems - they start writing equations, while expert physics problem solvers have multiple ways which they select among and switch between in the problems solving process.
In a sense, we were asking the wrong question earlier in our study. We wanted to know if expert models were different from novice models. They are different only in the degree of elaboration and detail, but not different in kind. Instead, the key difference is that experts have multiple mental models.
There are two main findings of these studies. The most striking is the diversity of mental models that people have, both for the Internet and for its uses. The second is that experts differ from novices not in the kinds of models they have, but in having multiple, detailed models that they switch between in the process of using the network to accomplish tasks.
These finding have obvious implications for teaching people how to use the Internet. The selection of any one mental model as a framework for training may be helpful to some students but not at all helpful to others. Teachers should be aware of the diversity of models, and build upon this diversity rather than trying to destroy it. The goal of instruction, instead, should be to help learners develop multiple, coordinated mental models that they can use at appropriate times to help them achieve their goals using networks.
The ability of the Internet and its most common applications to support a diversity of views may be to a considerable extent the basis of the "ease of use" that people report. As Internet applications become more heavily graphic, we may see the unexpected result that these applications become easier to use for some people (those who have and use mental models that contain the same visual elements), but actually harder for many others to use. This is one of the reasons that such studies of the mental models that people actually have of telecommunications may be very useful to the developers of the next generation of network tools.
Network Mental Models Questionnaire
We are conducting a study of how best to teach people about computer and information networks. We'd like you to fill out the questions below. We will protect your identity in any reports of our findings. If you agree to participate, please sign below.
Please contact me if you have any questions. Thanks!
Jim Levin 244-0537 Room 130 Education Building
Please circle the appropriate choice below:
My experience of teaching: none 1-5 years 6-10 years 11-20 21 or more
My previous experience with computers: none little some quite a bit lots
My previous experience with networks: none little some quite a bit lots
Previous experience with email: none little some quite a bit lots
Previous experience with newsgroups: none little some quite a bit lots
Previous experience with Gopher: none little some quite a bit lots
Previous experience with WWW: none little some quite a bit lots
Other network experiences:
We are interested in what "mental models" you have for using computer networks. For example, people often think about the atom as being like a miniature solar system. The solar system is then a mental model of the atom.
What "mental model(s)" do you use to think about the Internet?
What "mental model(s)" do you use to think about electronic mail?
What "mental model(s)" do you use to think about the World-Wide Web?
Would you be willing to talk with us in more detail about your mental models?
If yes, please write your phone number here:
What days and times would you be available this week to meet for about 30 minutes?
Imagine that you have a class of your choice to teach tomorrow and that you plan to use resources for the class from the "Web". You will have a total of 20 minutes to find relevant materials on the Web. Add the network documents you feel are relevant to the "Bookmark" list. Also, please tell us:
¥ the main topic or topics of the lesson plan,
¥ a general outline for the lesson,
¥ how the network materials are superior or inferior to non-network materials you would typically have available for a lesson such as this.
We want like you to "think aloud" as you do this task. Just say aloud the thoughts that come to you as you work on the Web and make your selections. Do you have any questions?
Feel free to write down any notes or thoughts you have on the front and back of this page.
In this study we are interested in what you think to yourself as you perform some tasks that we give you. In order to do this, we will ask you to think aloud while you are doing the network tasks on the computer. What I mean by think aloud is that I want you to say out loud everything you would ordinarily think to yourself silently. Just act as if you are alone in the room speaking to yourself. If you are silent for any length of time I will remind you to keep thinking aloud. At some points during the session, I might ask you about what you are doing.
Why don't we start with a practice problem for thinking aloud.
Think aloud while you tell me how many windows there are in your house.
After they are done, ask them how it was. Then ask if they would like to practice another think aloud. If so, then:
I want you to multiply two numbers in your head. Think aloud while you multiply 24 times 34.
At the beginning of our session, you indicated a mental model [or models--check their responses] you had about electronic networks. In what ways do you think your mental model [or models] of electronic networks influenced you while you were doing the task?Next, ask some probing or clarifying questions based on what she or he says. Any other comments you would like to make?Be sure to thank them for participating!
The overwhelming mental model I have for the Internet is what is called the rhizome...[the web is] also pretty rhizomatic. Rhizome is contrasted with the tap root model ...central tap root"
[describing his search strategy] "But I think although theoretically, I
think that rhizomatic structures are interesting I still think....tap root
hierarchical fashion. Let's go to the central sources first - let's get
those names things and then.. "
"My model for the Internet that I've come up with just recently has been
sort of a salvage ..... which comes from experience. And, I don't want to
say junk yard cause that -.... talk about .. that's now what I want ....
but rather it's a place where is a lot of stuff there that I don't want
and if I need a part for my car I'm going to go to the Ford section and
there might be a light working on one and a clutch cable on another and
just kind of strip from it what I want and that view of it has evolved and
I think away from the one I was given by the name web. And, thinking that
- a place I want to be and I jut need to find my way there and once I'm
there I'm - I don't think that way anymore because I'm going to have to
pull - I'm going to have to call it's much more that I'm going to have be
an active person....not trying to get somewhere..... ... salvage .... pull
the information .... I think that does have an effect on the way I use it."
"My model - well, let's see, how do - which - the evolved together. I
think it's really the web. "
"Well, I was reading yesterday that someone compared the WWW to a city. "
"the Internet .... as a spider Web than anything else cause I had that
image before the WWW came out so, I feel funny when I say when I think of
the Internet I think of the Web"
"oh another one that I use to explain is, I just use the phone company, how
we are all connected ...... and sort of works."
[on the relation between her mental models and her actions]: "I don't think
that there's any relation at this point - maybe when I was starting out.
But at this point I've got a pretty good concept of what's there that I
don't think about it anymore. Maybe subconsciously I do, but I don't think
about it really."
Internet as "computer network - all the computers are connected"
"All the books ... will be there and anyone can access"
Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121-152.
Gentner, D., & Gentner, D. R. (1983). Flowing waters or teeming crowds: Mental models of electricity. In D. Gentner, & A. L. Stevens (Eds.), Mental models. Hillsdale, NJ: Erlbaum.
Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge, MA: Harvard University Press.
Larkin, J. H., McDermott, J., Simon, D. P., & Simon, H. A. (1980). Expert and novice performance in solving physics problems. Science, 208, 1335-1342.
McCloskey, M. (1983). Naive theories of motion. In D. Gentner, & A. L. Stevens (Eds.), Mental models. Hillsdale, NJ: Erlbaum.
Norman, D. A. (Eds.). (1983). Some observations on mental models. Hillsdale, NJ: Erlbaum.
Norman, D. A. (1988). The design of everyday things. Reading, MA: Addison-Wesley.
Norman, D. A., & Draper, S. W. (Eds.). (1986). User centered system design: New perspectives on human-computer interaction. Hillsdale, NJ: Erlbaum.