A comparative evaluation of single and multipath tutors for solving
linear equations
Does more complex tutoring technology support more robust learning?
Maaike Sigrid Waalkens January 2011
Master Thesis
Human-‐Machine Communication Department of Artificial Intelligence University of Groningen, The Netherlands
Internal Supervisor: Dr. Niels Taatgen (Artificial Intelligence, University of Groningen)
External Supervisor: Dr. Vincent Aleven (Human Computer Interaction Institute, Carnegie Mellon University, USA)
Abstract
One feature that makes an Intelligent Tutoring System (ITS) hard to build is (strategy) freedom for the student. Strategy freedom is often seen as an important feature in ITSs, but does greater freedom mean that students learn more robustly? We developed three versions of the same ITS for solving linear equations that differed only in the amount of freedom. The strictest version supported a standard strategy and allowed no variations in the solution steps. An intermediate version supported the same standard strategy with minor variations. The free version supported multiple strategies. We conducted a study in two US middle schools with 57 students in grades 7 and 8 (12-‐14 years old). Students’ algebra skills improved with all versions; surprisingly, there was no significant difference in learning gain and motivation. Students tended to use only the standard strategy and minor variations of this strategy. Thus, the study suggests that in early algebra learning, only a small amount of freedom offered to students is useful, and large amounts of that freedom are not used.
Acknowledgements
In the first place I want to thank my supervisors Vincent Aleven and Niels Taatgen for their support, and feedback and for the opportunity to setup this study. Further, I’ll thank Vincent for sharing and transmitting his enthusiasm and passion for the research into intelligent tutoring systems.
Secondly, I would like to thank the people I worked with for their input and feedback while developing the tutoring systems and help/work so that the experiments went well.
Thirdly, I would like to thank Gert, for the conversations about the research, reading and support.
Last but certainly not least, I would like to thank my father, mother and brother, for always supporting me and giving me the opportunity to go the USA.
Table of Contents
1 Introduction ... 6
2 Theory ... 7
2.1 Robust Learning ... 7
2.1.1 Flexibility ... 9
2.2 Algebra ... 10
2.3 Motivation... 12
2.4 Intelligent Tutoring systems... 13
2.4.1 Cognitive Tutors ... 13
2.4.2 Example tracing tutors ... 14
3 Research Question ... 15
3.1 Hypotheses... 16
4 System design ... 17
4.1 Conditions ... 18
4.1.2 Strict standard strategy... 18
4.1.2 Flexible standard strategy ... 19
4.1.3 Multi strategy... 19
4.2 Feedback, hints, instructions ... 19
4.3 Problem set ... 22
4.4 Self explanation prompts ... 22
4.5 Freedom: Technical challenge... 23
5 Experiment... 25
5.1 Participants ... 25
5.2 Procedure... 25
5.3 Tests ... 26
5.3.1 Procedural knowledge ... 26
5.3.2 Conceptual knowledge... 27
5.3.3 Strategy flexibility knowledge ... 27
5.3.4 Motivation questionnaire ... 27
6 Results... 27
6.1 Overall Learning Gain... 28
6.2 Differences between the 7
thand 8
thgrade ... 29
6.3 Differences between the conditions in learning gains and motivation ... 32
6.3.1 Strategy analysis... 33
6.3.2 Strategy errors ... 36
7 Discussion... 36
7.1 Influence of freedom on robust learning and motivation... 37
7.2 Best trade-‐off ... 37
7.3 Just offering freedom is not enough for more flexibility knowledge... 38
7.4 Difference in results between 7
thgraders log data vs pre/post–test... 39
7.5 No improvement conceptual knowledge... 41
7.6 Future research ... 42
7.7 Conclusion... 42
9 References ... 43
Appendix A Pre-‐Test... 45
Appendix B Post-‐Test ... 46
Appendix C Motivation Questionnaire... 48
1 Introduction
Over the last decades there has been growing interest in research into intelligent tutoring systems (ITSs).
In an ITS, a computer program takes on the role of a human tutor. ITSs are effective in improving the learning of students (Anderson, Corbett, Koedinger, & Pelletier, 1995). They are not as effective as the best human tutors, though are possibly more effective than average human tutors. Moreover it is a cheaper (and more achievable) alternative for real human tutors. There are several research projects that show very positive learning results by integrating tutoring systems in classroom settings (Koedinger, Anderson, Hadley, & Mark, M.A, 1997; VanLehn, 2006)
ITSs are notoriously difficult to build; it takes a lot of time and effort to build effective and robust tutors.
One feature that makes an ITS hard to build is the level of strategy freedom for the student. Strategy freedom refers to the freedom to solve problems with the method or strategy of your choice. When there are several solutions possible for a given problem the solution space increases. The development time and complexity of the ITS architecture increase enormously when all these solution methods are made possible for the students. With this in mind, the influence of tutor freedom on learning is important; it is interesting to know if the extra time and effort pays off in terms of better student learning.
There is a correlation between the level of freedom in ITSs and in educational principles. The issue of whether greater freedom or more structured (or direct) instruction is more educationally effective is being hotly debated in the educational psychology literature (Dean & Kuhn, 2006). Several researchers claim that students learn with greater understanding when they discover their own procedures instead of only adopting instructed procedures (Von Glasersfeld, 1995). Offering freedom with respect to strategies may offer cognitive benefits (e.g., students get to try out different strategies, and perhaps experience when each strategy is help). It may also be motivating; conversely, not offering such freedom may be frustrating. Strategy freedom gives students options to choose from and different studies show that
“choice” is important for a higher intrinsic motivation and learning gain (Cordova & Lepper, 1996; Patall, Cooper & Susan, 2010).
Others claim that direct instruction is better, partly because discovery learning can overload working memory, or because discovery learning is inefficient, or does not lead to good solutions in the first place (Klahr & Nigam, 2004). Much empirical research is still needed to settle this question (Kirschner, Seweller,
& Clark, 2006) and to the best of our knowledge, there is no such empirical research yet in the field of ITSs.
ITSs may be viewed as providing rather direct instruction, but they can be designed in many different ways, to be more strict and provide more structure, or to be more open (and thus more related to discovery learning). They can be designed to allow many solution variants, or allow few.
The current study investigates the value of allowing multiple solution variants, and thus the value of more complex tutoring architectures capable of supporting them.
Implicitly, researchers and developers so far assumed that freedom in ITSs is important and beneficial for learning results and motivation. Generally, more complex tutors with a lot of freedom are seen as good ITSs. Moreover it seems counterintuitive to restrict students when, in the given task domain, many solution paths are possible (Mitrovic, Mayo, Suraweera, & Martin, 2001; VanLehn, 2006)
The influence of tutor freedom on learning will be measured with tutors for one important algebra skill:
solving linear equations. Algebra is a difficult topic in middle and high-‐school math and previous studies did show advantages of ITSs for algebra lessons (Koedinger et al., 1997). It is a suitable task domain for research into freedom and restrictions, because a linear equation can often be solved in multiple ways.
On the other hand, there is a widely used standard strategy that can solve many linear equations in an optimal way (i.e., with a minimum number of solution steps), but occasionally, yields a solution path that is slightly longer than strictly necessary.
The goal of this study is to investigate how much freedom and what restrictions within tutored problem solving are optimal for student learning and motivation. Three versions of an ITS for solving linear equations were developed. These versions differed only in the amount of freedom offered within each problem (or equivalently, the range of solution paths that the tutor recognized as valid solutions). Middle school students will work with one of the three versions and the learning gains and motivation will be measured. The tutoring system will log the students steps and will measure to what extend the students will use the freedom offered by the ITS. With these answers we are not only looking how much freedom is optimal for learning, but also if the extra freedom is worth the extra effort.
2 Theory
In the next chapter we discuss the theoretical background of robust learning, the important knowledge types that can cause robust learning and information about algebra learning. Moreover, we discuss the different types of ITS (and if/how these different types provide strategy freedom).
2.1 Robust Learning
The subject of this study is to test what influence a varying degree of structural freedom within an ITS has upon the effectiveness of student learning with the assistance of such an ITS. In order to be able to make statements regarding what is “effective learning” we need a qualitative concept of learning. What, actually, is “learning” and what constitutes “good learning?” We need these statements in order to put together a meaningful pre-‐ and a post-‐test for our subjects. What criteria do we use to determine whether they made progress and learned “effectively”? Besides this, we need to grasp the concept of learning to make a tutor that is sophisticated enough to fit in with the way the human brain handles knowledge.
ACT-‐R, a theory for simulating and understanding human cognition, can give insights into what learning actually is (Anderson, 1993). ACT-‐R distinguishes between two types of knowledge, declarative knowledge and procedural knowledge. The learning process can be represented by the process of acquiring these two types of knowledge. Declarative knowledge is the knowledge of facts, like ‘1+2 equals 3’. In ACT-‐R, declarative knowledge is represented by a network of knowledge pieces, called chunks. Procedural knowledge is the knowledge of how things do things or cognitive tasks, like solving a linear equation.
Procedural knowledge is represented by a lot of units called production rules, these are condition-‐action units which respond to various problem-‐solving conditions.
Declarative knowledge can be acquired in two ways: by encoding information from the environment and by storing the results of past mental computations. Procedural knowledge is learned in ACT-‐R by a process called analogy, where skills are acquired by making references to past problem solutions while actively trying to solve new problems. This means that procedural knowledge can only be learned by doing (Anderson et al., 1995). Now we know what “learning” actually is, we can go further with answering the question about what “good” learning is.
The Pittsburgh Science of Learning Center (PSLC) has created a theoretical framework for learning (Koedinger, Corbett & Perfetti, 2010). “Good learning” is referred to as “robust learning”. Learning, as in
“acquiring knowledge” is robust when it meets at least one of the following criteria:
1) Retention: If the knowledge or skill is retained for long periods of time, at least for days and even for years.
2) Transfer: It transfers, that is, it can be used in situations that differ significantly from the situations present during instruction.
3) Future Learning: It accelerates future learning. That is, when related instruction is presented in the future, this knowledge allows them to learn more quickly and effectively (Koedinger, Corbett
& Perfetti, 2010).
But that does not answer the question what “acquiring knowledge” is. According to the PSLC there are three systemic elements in learning: “sense making”, “fluency” and “refinement.” All three concepts are characteristic elements of learning that can increase retention, transfer and accelerating of future learning. Sense making is a process where students try to understand the instruction in higher-‐level thinking in accordance with conceptual knowledge. Fluency and refinement is the ability to master a skill, similar to procedural knowledge.
Sense making is a robust learning process where students try to understand the instruction in a higher-‐
level thinking. In the current study for example the knowledge about the different features in an equation (variable terms, like terms, negative signs), the function of these features and how they relate to each other. This usually refers to the early stages of instruction; students try to understand the instructions and construct coherent and meaningful knowledge components. Sense making can be seen as conceptual understanding or conceptual knowledge (Koedinger, Corbett & Perfetti, 2010). A definition of conceptual knowledge is the understanding of both the principles that govern the domain of and of the interrelations between pieces of knowledge in the domain. It is the comprehension of mathematical concepts, operations and relations (Booth & Koedinger, 2008).
Fluency is the skill in carrying out procedures flexibly, accurately, efficiently and appropriately. In the current study, it is the ability to solve linear equations without mistakes in a reasonable amount of time.
According to the ACT-‐R theory, procedural knowledge can only be improved by doing. This skill is mastered in later stages of instruction, when the students are consulting their knowledge by practice.
Procedural fluency and refinement can be seen as procedural knowledge; the knowledge of action sequences for solving problems. Refinement is a non-‐verbal learning process that improve the accuracy of knowledge (Koedinger, Corbett & Perfetti, 2010).
Sense making, fluency and refinement are the characteristics that cause robust learning. The level of sense making can be measured foremost in conceptual knowledge gain. The level of fluency and refinement can mainly be measured as procedural knowledge gain.
For robust learning students need to gain both conceptual and procedural knowledge. Moreover, previous research shows that these kinds of knowledge are intertwined and mutually influential (Booth &
Koedinger, 2008; Rittle-‐Johnson & Alibali, 1999). Practicing equations is not the only way to improve procedural knowledge in the algebra-‐domain. Rittle-‐Johnson & Alibali (1999) showed that there is causal evidence that conceptual and procedural knowledge influence each other. Conceptual instruction can improve procedural gain and procedural instruction can improve conceptual knowledge. However,
conceptual knowledge may have a greater influence on procedural knowledge than the reverse. (Rittle-‐
Johnson & Alibali, 1999)
Conceptual knowledge of problem features affects students’ performance as well as students’ learning. In a previous algebra study prior conceptual knowledge, as well as gain in conceptual knowledge predict procedural knowledge gain (Booth & Koedinger, 2008). Conceptual knowledge is also important for procedural knowledge gain. Students can practice their procedural knowledge by teacher demonstration or hint and error messages in a tutor. However, without conceptual knowledge they are not able to align the presented information with the information in the problem that they want to solve. Students without sufficient conceptual knowledge will likely only be able to make shallow analogies; for improving procedural knowledge deeper analogies are needed (Booth & Koedinger, 2008).
In short, the definition of “effective learning” or “robust learning” used by the PSLC is learning that increases retention, transfer and accelerating of future learning. The two key elements that cause this robust learning are sense making, in accordance with conceptual knowledge, and fluency and refinement, in accordance with procedural knowledge. Procedural knowledge according to ACT-‐R can only be acquired by doing (Anderson, 1993).
2.1.1 Flexibility
Procedural and conceptual knowledge are not the only important kinds of knowledge for robust learning.
Flexibility knowledge is also important. This is the knowledge of different strategies and the knowledge of how to use them adaptively in a range of situations (Star & Rittle-‐Johnson, 2008). In accordance with the PSLC framework, flexibility is most closely related to transfer learning.
Flexibility is important for the current study because the opportunity to develop flexibility seems to differ between the tutor versions that were developed for the study (described in greater detail below). In total there are three tutor versions. In the two strictest versions a standard strategy for solving equations is required. The standard strategy for solving linear equation is widely used by teachers and textbooks and is a defined order of operations that can solve almost all linear equations (Holt, Rinehart & Winston, 2007;
Benson et al., 1991). This strategy is described in greater detail below. Developing flexibility is impossible in the two strictest tutors, because, using different strategies (and improving flexibility knowledge) is impossible. In the tutor version that offers the greatest freedom, the students do have the option to develop flexibility because they are free to choose their solving strategy. Importantly, the hints and instructions in all tutor versions are the same. The free version of the tutor therefore does not encourage students in using different strategies; rather, they have the option to use different strategies.
Flexible knowledge has several advantages for learning and performance (Star & Rittle-‐Johnson, 2008;
Star & Seifert, 2006). Students with knowledge about multiple strategies are more likely to learn from instructional interventions (Alibali, 1999). Moreover flexibility is related to transfer and conceptual knowledge. Students with flexible knowledge are more likely to adapt existing strategies when they must solve unfamiliar transfer problems. By contrast, students without flexible knowledge have great difficulty on both near and far transfer problems. In the case of algebra (specifically, linear equation solving), there are several ways to solve a problem, optimal and less optimal ways (to be discussed in greater detail below). Choosing an optimal solution method is preferred because it is the most efficient way to solve a problem. Moreover, preventing students from using different solving methods can restrict their flexibility.
Prior-‐knowledge is important in the development of strategy flexibility. In estimation problems for multi-‐
digit multiplication problems (such as 17*41), students with a high fluency on the pre-‐test were more likely to utilize estimation strategies that led to more accurate estimates (Star, Rittle-‐Johnson, Lynch &
Perova, 2009). Students with less fluency adopted the easiest strategies (Star et al., 2009). In the case of algebra solving, it is possible that students with more procedural knowledge (more experience in algebra) would prefer an ITS that offers them the freedom to choose the most efficient solving strategy.
Conceptual and procedural knowledge are mutually interactive preconditions for learning gain. Improving one has a positive influence on the other. This is not so with flexibility. Star (2005) mentioned the possible trade-‐off in initial stages of learning between the goal of flexible use of multiple strategies and the goal of mastery of a standard algorithm. If one’s instructional goal is quick learning of a standard and efficient algorithm, then Star (2005) suggests that repeated practice on collections of similar problems might be more effective.
Several studies showed that students improve the skill that they practice (Star & Seifert 2006; Star &
Rittle-‐Johnson, 2008). A study into the influence of prompts to solve a problem a second time, but with a different strategy, showed that these prompts improved the flexibility. However, procedural knowledge did not improve as much as the condition where the students were not prompted to solve an equation with a different strategy.
National and international assessments commonly show that many US students lack procedural flexibility;
since many students only know algorithms, they have difficulty when faced with novel or unfamiliar problems (Schmidt, McKnight, Cogan, Jakwerth & Houang, 1999). The algebra lessons in most American schools have a focus on learning to solve equations fluently by teaching them a successful solving strategy by root, instead of teaching them flexibility.
As demonstrated above, flexibility has several learning advantages. However there is a possible trade-‐off in initial stages of learning between the goal of flexible use of multiple strategies and the goal of mastery of a standard algorithm. The American school system is more in favor of learning the standard system (National Council of Teachers of Mathematics, 2000). In this study, developing flexibility is possible in the free version of the tutor, but does not seem possible in the stricter versions. Although the free ITS does not prompt the students to develop strategy flexibility, as for example in the studies of Star & Rittle-‐
Johnson (2008), we are interested in the influence of this “possibility” on the procedural, conceptual and flexibility knowledge.
2.2 Algebra
The ITS in this study targets one algebra topic: solving linear equations. Algebra is an important topic in middle and high school mathematics. Learning algebra is an entirely different experience from learning arithmetic (Mathematics Learning Study Committee, 2001). This transition is difficult for most students.
School algebra consists of three main activities: Representational, transformational and generalizing activities (Mathematics Learning Study Committee, 2001).
1) Representational activities of algebra involve translating verbal information into symbolic expressions and equations. For example generating an equation that represents a particular problem situation.
2) The transformational or rule based activities. For instance, collecting like terms, factoring, expanding, substituting, solving equations and simplifying expressions; in other words, solving equations.
3) Generalizing and justifying activities. These include problem solving, modeling, noting structure, justifying, proving and predicting. These activities are not only algebra, but the language or tools of algebra is used. For example showing that the sum of two consecutive numbers is always an odd number (Mathematics Learning Study Committee, 2001).
The ITS developed for this study is designed for improving one of these activities, the transformable ones in the form of solving linear equations. One of the great strengths of algebra, according to the Mathematics Learning Study Committee, is that a great deal of its transformational activity can be carried out in what appears to be rather an automated manner. The ITS is sufficient for training this transformational activity, so that it can be automated in the (far) future.
Algebra is a suitable topic for research into freedom and restrictions in ITSs, because linear equations can be solved in different ways. Although it is possible to solve many if not all linear equations using a single, standard strategy, it is not necessary to adhere to a single strategy, and in fact sometimes using a more flexible, more efficient method is preferred. There is a standard strategy that can solve almost all linear equations. This strategy is widely used in several studies, by teachers and in math textbooks (Holt Rinehart & Winston, 2007; Benson et al., 1991). For many linear equations the standard strategy is optimal, this means that this strategy solves the equation with the minimal number of operations.
However, for some equations a different strategy is better. The standard strategy is the basis of the more restricted versions of the ITS that was developed for the current study.
The standard strategy is to solve an equation with the following (fixed) order of operations:
-‐Distribute any parenthetical terms
-‐Combine like constant terms and like variable terms on each side of the equation -‐Move variable terms to one side of the equation and constant terms to the other side -‐Divide both sides by the coefficient of the variable term
.
There are examples of three equations in Table 1. They are solved with the standard strategy and with a more efficient strategy (if possible)
TABLE 1: EQUATIONS SOLVED WITH THE STANDARD STRATEGY AND (IF POSSIBLE) WITH A MORE EFFICIENT STRATEGY. Equation type Solution standard
Strategy
Solution more efficient strategy ax + b = c 3x + 2 = 8
3x = 6 x = 2
Standard strategy is most efficient
a(x+b) =c 3(x+2) = 12 3x + 6 = 12 3x = 6 x = 2
3(x+2) = 12 x + 2 = 4 x = 2
a(x+b) + c = d 2(x+1) + 3 = 11 2x + 2 + 3 = 11 2x + 5 = 11 2x = 6 x = 3
2(x+1) + 3 = 11 2(x+1) = 8 x + 1 = 4 x = 3
The exact meaning of “strategy” is a definition question. There are different definitions that are used. Star
& Rittle-‐Johnson (2008) distinguish between three different strategies:
1) Standard strategy, a fixed order of operations (Distribute, Combine, Move, Divide).
2) Random, choose a random operation out of the possible operations.
3) Optimal, Decide which transformation to apply based on features of the particular problem.
In this thesis, every different order of operations is called a different strategy. For example the equation 2(x+1)+3 = 11. The operations Distribute-‐Move-‐Move-‐Divide or Distribute–Move-‐Divide-‐Move can solve this strategy and would be both classified as “random strategy” by Star & Rittle-‐Johnson (2008). In this study we classify them both as different strategies, because they consist of different orders of steps. This is more specified so we can get a better overview of all the differences in problem solving strategies.
Moreover, the ITS can only log the different operations and not the “intended” strategy of the students (standard, random or optimal) of the students.
Star and colleagues investigated whether students without knowledge of algebra would find the standard strategy by themselves or not when they must solve linear equations (Star et al., 2005). They discovered 25% of the students found the standard strategy mostly on their own. Finding this strategy was beneficial in that those who used this strategy on the post-‐test had more items correct than those who did not. Two manipulations were tested to examine if they would improve the finding of the standard strategy.
Demonstrated worked-‐out examples did not increase the use of the standard strategy. Furthermore, prompts to solve an equation in two different ways decreased the probability that students would use the standard strategy (Star et al., 2005).
We did a pilot study for getting more insight in solving linear equations. Five students who were enrolled in or had completed Algebra 1 participated in a pilot study for insight in strategy use and insight in common difficulties solving linear equations. Students with experience were chosen because they could give information about the way algebra is taught. Students were asked to solve ~25 equations that were analyzed in terms of strategy use. In half of the equations they were asked to think aloud while solving. In some cases they were asked to solve the same equation again, but with a different strategy. Moreover, they were given some questions about problem solving like “why do you use this strategy”, “which strategy did you learn in the algebra lessons?”, and “which strategy do you usually use?” etc. A strict and fluent use of the standard strategy by the students was noted. All students were aware of the standard strategy and most knew the order of the operations in this strategy by heart. Moreover, students could solve very difficult equations with this strategy but had problems with solving simpler equations with a different strategy.
2.3 Motivation
The main difference between the three versions of the ITS built for the current study is the amount of freedom and restrictions in the tutor. Research related to this topic is about choice in learning; the freedom to choose your own strategy can be seen as a kind of “choice” in learning. Choice and personalization techniques increase intrinsic motivation and learning results (Cordova & Lepper, 1996;
Patall, Cooper & Susan, 2010). This has been demonstrated in several topics: the choices for which kind of homework students want to do as well as choice within an educational game (Cordova & Lepper, 1996;
Patall, Cooper & Susan, 2010). In the latter motivation not only increased with choice, there was also
more learning gain. Choice in learning can be linked to the direct instruction/ discovery learning debate.
Discovery learning is related to “choice” while direct instruction is related to restrictions and “no choice”.
In the studies of the influence of “choice”, the students were aware of the choice that they had. In the current study this is probably not the case. The students are not aware of their “strategy choice” but more aware of the “strategy restrictions.” Strategy freedom can be seen as “normal.” If it is possible to solve a problem in different ways, strategy freedom in an ITS is probably taken for granted, because students do have this freedom while solving equations with pen and paper. Previous research showed that the perceived feeling of “choice” had positive effects on the intrinsic motivations. Therefore the expectation is that the opposite (the perceived feeling of “no-‐choice”) has a negative influence on the motivation.
2.4 Intelligent Tutoring systems
The amount of freedom that can be implemented in an ITS depends partly on the tutoring technique that is used. Simple tutors that support single solution paths within a given problem appear to be less complex, architecturally, than systems that support multiple solution paths. There are several, relatively simple ITS that support only one single solution path within a given problem, for instance ASSISTments (Heffernan & Koedinger, 2009), AnimalWatch (Arroyo, Woolf & Beal, 2006) and Wayang Outpost (Beal, Walles, Arroyo & Woolf, 2007). Single-‐path tutors can be built simply by associating correct and incorrect answers, as well as hint and error messages, with specific interface elements. Tutors that support multiple paths are more complex and often use some kind of knowledge representation scheme. For example cognitive tutors use rules, constraints based tutors use constrains and example tracing tutors use behavior-‐graphs. In the next section, two important tutoring architectures are described. In addition, we discuss how different solution paths are made possible in these architectures.
2.4.1 Cognitive Tutors
Cognitive Tutors were the first widely used tutoring systems. (Koedinger & Corbett, 2006) The basic assumption behind cognitive tutors is that learning is supported by doing. (Anderson et al., 1995) Cognitive tutors share two important aspects with human tutoring: monitoring students’ performance and providing context-‐specific instruction when the individual student needs it, as well as monitoring the student learning and select new problems just within the individual student’s reach. The cognitive tutor makes this aspect possible by two key algorithms, the model-‐tracing and the knowledge-‐tracing algorithm (Koedinger & Corbett, 2006). In model-‐tracing the system follows the student problem solving steps in the solution space. By knowledge-‐tracing the system keeps track of the students’ knowledge to select the right next problem.
The basis of model-‐tracing is the ACT-‐R theory of learning (Anderson, 1993). ACT-‐R distinguishes between implicit “procedural” knowledge and explicit “declarative” knowledge. In ACT-‐R, procedural knowledge can only develop through doing. The procedural knowledge in ACT-‐R is represented in a set of if-‐then rules. Cognitive tutors have a similar production system with if-‐then rules, which represent different strategies that students can use. The production system also contains common misconceptions. The students can follow their own individual path through the problem space. The cognitive model traces this path and gives specific feedback and hints, if needed. This kind of tutor is sufficient for complex solution spaces and when the developers want freedom for the users.
The cognitive tutor algebra is a successful tutor for learning algebra. Students that have used the cognitive Tutor in classroom settings performed much better on traditional math tests than students that followed the same curriculum without the ITS (Koedinger et al., 1997). The cognitive tutor algebra has a lot of
freedom; students can solve the equations with almost all possible strategies. The only restriction is that steps must be mathematically correct. For example, it is possible to add a random number to both sides of the equation. The tutor will allow the step, but will classify it as “suboptimal” and the student has the choice to delete this suboptimal step or work further with the equation after the suboptimal step.
2.4.2 Example tracing tutors
In recent years much effort has been put into making the ITSs easier to make, for example with the example tracing technique (Aleven, McLaren, Sewall & Koedinger, 2009). Example tracing tutors are much easier and faster to build than cognitive tutors even though they share key elements that make an intelligent tutor effective. It keeps track of the students problem state, it can give immediate feedback to the student, it can give hints to the next solution steps and it can assess the learning skills of the students (Aleven et al., 2009). Building a good tutor with this technique is 4-‐8 times faster than building a cognitive tutor with the same behavior (Aleven et al, 2009).
Example tracing tutors evaluate the problem solving steps of the students by comparing them to examples of the problem solutions. This kind of tutor can be built with the Cognitive Tutor Authoring Tools (CTAT) (Aleven et al., 2009). CTAT is a programming environment where an ITS can be built even by non-‐programmers, faster than by traditional environments. The builder must demonstrate the problem-‐
solving behavior and CTAT records the problem-‐solving steps in a behavior graph. When there are different solutions the builder must demonstrate (specify) the different solution paths. Even though this is time costly, in most cases it is easier than specifying the whole solution space in a formal language.
Example tracing tutor occupy the middle ground between the simple and complex (cognitive &
constraints based) tutors. They support multiple paths, but do not easily support large numbers of substantially different paths within a given problem.
The example tracing tutors are developed in four phases. The first phase is creating a tutor interface in Adobe Flash Player. The second phase is demonstrating how problems can be solved in the tutor interface. These steps are recorded by CTAT and the result is a behavior graph. The third phase is editing, annotating and generalizing the resulting behavior graph. In this phase hints and error messages are added. The last phase is mass production. With mass production several (similar) problems can be created without demonstrating the whole solution space for each. Mass production is not only faster, it also supports consistency and maintenance (Aleven et al., 2009). An example of a behavior graph is shown in Figure 1.
The ITS keeps track of the student’s problem solving behavior and can give the right feedback and hints.
The behavior graph of a certain equation contains one or more paths that lead to the right solution, as well as some common mistakes. When students work with the ITS, the “example tracer” keeps track of the students problem solving behavior by monitoring which states the student has visit (i.e. where student is in the behavior graph). The example tracer also keeps track of all viable paths (start to finish paths that are consistent with the students’ observed behavior so far). A student action is classified as correct if it is an unvisited and correct link on a viable path.
FIGURE 1 BEHAVIOR-‐GRAPH WITH TWO SOLUTION PATHS.
3 Research Question
This research project is a comparative evaluation of single and multipath tutors for solving equations.
Although researchers and teachers often have a preference for more freedom in the tutor if possible, the exact influence of this feature is not known. With this research we try to answer if the complex tutoring technology that is needed for more freedom supports more robust student learning.
The research questions the study attempts to answer are:
1) Within tutored problem solving, how much freedom and structure is optimal for robust student learning?
2) To what extend will the students use freedom that is offered?
With the answers of the previous questions a more subjective and broad research question can be answered:
3) Does the extra complexity and effort to make the multipath tutor pay off in terms of better learning results or motivation of the students.
We created three versions of an algebra tutor, all of which were designed with an empirical background and input from teachers, and contain features aimed at supporting deep understanding and robust learning. We built: (1) a very restricted ITS (strict standard strategy), (2) an ITS that offers a limited amount of freedom (flexible standard strategy), and (3) an ITS with a large amount of freedom (multi strategy). The very restricted version supported the standard strategy for equation solving and allowed no variations in the solution steps. The version with a limited amount of freedom supported the same
standard strategy with minor variations. The version with a large amount of freedom supported multiple strategies. We tested the three versions in a controlled experiment conducted in a middle school. The learning gains by working with the different systems were measured and compared. Several important aspects of learning were measured: procedural knowledge, conceptual knowledge and flexibility in strategy use. Students’ motivation and opinion of the system was measured with a questionnaire.
Moreover, the log data showed the variety of minor variations within the standard strategy and the different strategies that individual students used on problems in the ITS with a limited amount and with a large amount of freedom.
The second goal of this study is to test whether it is possible to build a robust ITS for solving linear equations with the example-‐tracing technique. Linear equations can be solved in many different ways, resulting in a large solution space. Using the example-‐tracing technique, ITS can be built without programming background and 4-‐8 times faster than with the cognitive tutoring technique (Aleven et al, 2009). There are several tools for building in freedom, however it is not clear if these techniques are sufficient for the large amount of freedom needed in this study. It is also not clear if the advantages of the technique, building tutors with the same behavior much faster, will apply. This technical research question will be discussed in the section on system design.
3.1 Hypotheses
As discussed above, there are several kinds of knowledge that indicate robust learning. In this study we use procedural knowledge, conceptual knowledge and flexibility knowledge as indicators for robust learning, so we will give the hypotheses for these items as well as for the motivation questionnaire.
Previous research lead to the hypothesis that more restricted tutors can help students to learn a well-‐
defined, optimal problem-‐solving strategy and freer tutors can be more helpful for deeper knowledge and transfer learning. Star (2005) suggests that there is a possible trade off in initial stages of learning between the goal of flexible use of multiple strategies and the goal of mastery of a standard algorithm.
Star & Rittle-‐Johnson (2008) showed that prompting students to solve the same equation in different ways provides better results on flexibility items. However, the procedural knowledge improved less.
So, if the goal is to quickly learn a standard and efficient algorithm, repeated practice of collections of similar problems with the same strategy might be more effective than learning different strategies.
Because the students in the two strictest conditions (strict standard strategy & flexible standard strategy) are forced to use a standard strategy they will practice this strategy extensively (probably more so than the students in the condition with the greatest freedom). Therefore, the hypothesis is that the two stricter tutors are better for improving familiar procedural knowledge.
In this study, learning a standard and efficient algorithm quickly is only one part of the learning goal. The main goal is robust learning (discussed above) where flexibility is also an important feature. Flexible knowledge is the ability of students to know multiple strategies and adaptively choose efficient strategies.
(Star & Rittle Johnson, 2008)
The second hypothesis is that the free tutor (multi strategy) is better for flexibility items, like solving equations in different ways and recognize alternative strategies. In more free tutors, students can try and practice different strategies, which is not possible in the more restricted tutors. So, tutors that allow more freedom may support flexible thinking and are possibly better for deeper understanding. This deeper
