Faculty of Social and Behavioural Sciences
Graduate School of Childhood Development and Education
The effect of modelling on learning
subject-specific content: an experimental
approach
Research Master Educational Sciences Thesis 2
Erika Schlatter Supervisors:
Dr. Jannet van Drie Prof. dr. Peter de Jong Dr. Bert Bredeweg 01-02-2017
The effect of modelling on learning subject-specific content: an experimental approach Erika Schlatter
Abstract
Modelling might help students to understand dynamic systems but learning how to model is a difficult task in itself. Therefore the question was asked whether modelling helps students learn about subject-specific content better than regular non-modelling lessons. In this study, 74 11th grade students from two schools participated in either a modelling or a control
condition during two biology lessons. The first lesson served as a modelling learning phase, in which students in the modelling condition learned how to create a model with the computer tool DynaLearn. The second lesson served as a modelling application phase, in which students in the modelling condition applied their modelling skills to the second topic. Students in the control condition did not learn how to model, but did learn about the same dynamic systems. Pre- and post-tests were administered to assess subject-specific content learning. In addition, students in the modelling condition did a short pre- and post-test on modelling and data on DynaLearn activity was gathered and analysed. Analyses showed that students in the
modelling condition did improve in their modelling abilities. However, with regard to subject-specific content students in the modelling condition did not improve more than students in the control condition. The main reasons for the absence of an effect of modelling were most likely a result of the short learning time and compromises with regard to model and content complexity. It is expected that modelling is more successful with more complex content and modelling techniques.
Keywords: modelling, content learning, learning time, learning by modelling, DynaLearn,
The Effect of Modelling on Learning Subject-specific Content
Many of the topics students learn about in school can be characterized as dynamic systems through which changes propagate dependent on various conditions that may or may not be met (Forrester, 1994; Liem, 2013). An example of such a dynamic system is an ecosystem, in which the organisms can exist only if certain conditions are met, and in which some organisms provide these conditions for others. As various elements depend on each other, changes in one element can propagate to other elements. Dynamic systems can not only be found in Biology but also in other parts of nature and society. As such, they are also
studied in the physical as well as in social sciences domains.
For students, dynamic systems can be difficult to understand. One way in which thinking about dynamic systems can be supported is through the use of models. Models provide users with a way to formalize their thinking about reality and are widely used in scientific practice (Clement, 2000). Models are used to express ideas and theories about complex real-world phenomena (Liem, 2013) and in order to do so they need to be
constructed in a parsimonious and focused manner. As such, a model does never represent the entire reality but rather a small and comprehensible portion of reality, just enough to
understand the dynamics of the system (Crawford & Cullin, 2004; de Jong et al., 1998). The question can be asked whether the use of models can help students understand dynamic systems.
There are a number of ways in which models can be used in education. Models can be used as an additional representation of the dynamic system, often supplementary to a text describing it (Liem, 2013). Another way of using models in education is the active act of modelling. We know that in general, models can help students understand subject content (Rutten, van Joolingen, & van der Veen, 2012). However, for active modelling such evidence is more scarce. A reason for this lack of evidence is that modelling takes time to learn and is
experienced as being difficult by students (Sins, Savelsbergh, & van Joolingen, 2005). This makes it difficult to assess whether students who model may perform better than students who do not model. Therefore, research often focusses on multiple modelling conditions. Although this type of research does show difficulties in modelling and may help in the development of better modelling education, it does not show whether modelling is actually to be favored over non-modelling tasks. In the current study the effect of active modeling with one of these tools, DynaLearn (Bredeweg et al., 2013), on the acquisition of subject-specific knowledge was evaluated by comparing a modeling and a non-modeling condition. To accommodate for modelling learning, the study was split up in a modelling learning phase in which the modelling group received instruction on using DynaLearn alongside the subject-specific instruction, and a modelling application phase in which students in the modelling condition were expected to be able to use DynaLearn.
Modelling in secondary education
Currently text-and-picture representations of dynamic systems are most prevalent in secondary education. Although such additional representations may help students understand subject matter better (van Someren, Boshuizen, de Jong, & Reimann, 1998), there is also a problem with this practice: as a textbook picture is by definition static, it is hard to show the dynamics of a system (Westra, 2008). A solution for this problem can be found in computer simulations. In a review of studies on computer simulations, Rutten et al. (2012) found that students using computer simulations generally improved their conceptual understanding as well as their reasoning for predicting experimental outcomes. However, effects differed for the types of simulations used. Understanding of dynamic systems appears to be what
computer models are most helpful for, as the system’s dynamics cannot be represented as well on paper.
Rutten et al. (2012) also discuss simulations and models that provide students with the possibility of changing parameters in order to understand changes in a system. Rutten et al. argue that such dynamic, manipulable representations of systems may help students to
understand system dynamics even better than ‘regular’ computer simulations. Manipulating a model by changing parameters might help students understand the ways in which changes propagate throughout a system. An example of a simulation often used in Dutch Biology education is ‘save the parrots on Bonaire’ (see http://www.betasimulaties.nl/papegaaien/), in which students manipulate parameters concerning the ecosystem on the island of Bonaire in order to discover under what conditions the parrots thrive. A disadvantage of such
manipulable simulations is that the underlying model remains the same. Seeing the benefits of being able to manipulate an existing model, the question can be asked whether actually
building a model from scratch may be even more beneficial for learning.
Creating models from scratch is seen as a higher order skill within the larger set of systems thinking skills (Hopper & Stave, 2008). In the literature various benefits are pointed out with regard to this active modelling as opposed to text-and-picture models or simulations in which only parameters can be manipulated. In an active modelling process, students do not only tinker with parameters but actually define, evaluate and revise models (Schwarz & White, 2005). It is argued that experiencing this interactive process of modelling leads to a better understanding of what a model is and how to interpret one (Gobert et al., 2011; Saari & Viiri, 2003). Students who go through the whole cycle of making a model experience that models are not absolute representations of reality but always subject to change, for example when new information comes available or when a process is looked at from a different perspective (Clement, 2000; Gericke & Hagberg, 2010). Furthermore, important aspects of modelling such as specification and revision are also an inherent part of scientific practice. As such, modeling is argued to improve students understanding of the nature of science (Gobert
et al., 2011; Schwarz & White, 2005). Lastly, as modelling makes students think their ideas through in a more thorough way in order to build their model it is also argued that modelling helps understanding specific school science content (Doerr, 1996; Louca & Zacharia, 2015; Zohar & Barzilai, 2013). As such, learning how to model is argued to be more than learning the skill of modelling: it might also help students understand science in general as well as subject-specific content. It is this subject specific understanding that is investigated in this study.
Although widely believed, evidence that active modelling can help students develop such understanding is scarce (VanLehn, Wetzel, Grover, & van de Sande, 2016). Research is often from a time that computers were slower and harder to operate then they are today (Doerr, 1996), focuses on modelling outcomes rather than outcomes on a post-test (Mulder, Lazonder, & de Jong, 2015) or on various levels of support between modelling conditions, rather than comparing a modelling condition to a non-modelling control condition (Mulder et al., 2015; VanLehn et al., 2016). Additionally, there are reasons to believe that modelling may not always help students understand subject-specific content. Students experience modelling as difficult (Sins et al., 2005) and often misunderstand modelling paradigms (VanLehn, 2013). Modelling tools may also tempt students to focus on achieving a well-fitting model through mindless manipulation of parameters and relations, rather than through the use of prior knowledge come to conceptual understanding (Sins et al., 2005). These difficulties imply that there is an inherent need for modelling support in order for modelling to be beneficial for understanding.
Furthermore, learning how to model is not only experienced as difficult but also takes time. As an extreme example, Hashem and Mioduser (2011) had students in the modelling condition take 48 hours of extra coursework in order to learn working with NetLogo before starting on the three-hour experimental period, whereas control conditions in which students
either explored or manipulated the model did not receive extra instruction. This learning time is an especially important factor: the time spent on learning how to model cannot be spent on learning subject-specific content. This problem is reported by researchers (VanLehn et al., 2016) and likely also experienced by teachers. Learning time problems are all the more relevant in subject areas in which modelling is not part of the obligatory curriculum, as is the case with Biology in the Netherlands: the extra costs of learning how to model should not be too high in relation to the learning gains from the modelling.
Modelling tools and intervention characteristics
From the above it is clear that modelling is not necessarily an easy fix to improve conceptual understanding. In order to understand what difficulties arise and how they can be mitigated, it is important to understand how modelling tools and interventions are similar in some ways but also differ in others. In this, modelling paradigms and support should be considered as key factors.
The one aspect that is similar across most modelling tools used in secondary education, especially those rooted in system dynamics, is the visual interface. Using such tools students can relatively easily define the elements that are to be in the model, the properties of these elements and the relations between elements or properties (e.g. Co-Lab (van Joolingen, de Jong, Lazonder, Savelsbergh, & Manlove, 2005), Dragoon (VanLehn et al., 2016), DynaLearn (Bredeweg et al., 2013), PowerSim (Westra, 2008), and STELLA (Doerr, 1996)). However, tools using visual interfaces can be distinguished on two
dimensions. First, a distinction can be made between qualitative and quantitative modelling (VanLehn, 2013). Second, tools and interventions differ as to how and how much support is offered. Like the similarity in interface, it is important to keep the differences in mind when implementing modelling tools in secondary education.
The first dimension, qualitative as opposed to quantitative modelling, concerns the extent to which a model’s properties and relations are expressed in a numerical manner. In some tools (e.g.Co-Lab, Dragoon, PowerSim, STELLA) students can define the exact quantities of a property and the formulae that describes the relations between properties. These tools often use the ‘stocks and flows’ metaphor, the stocks describing current values of properties and the flows the change rates. In other tools, more emphasis is placed on the structure of the model, relations are represented more verbally and quantification placed on the background (e.g. Model-it) or the stocks and flows metaphor is less prominent (e.g. Dragoon). However, more advanced modellers are still able to add numeric descriptions of elements’ properties and relations. It is this numerical approach that is abandoned in purely qualitative tools. In qualitative modelling, knowing the exact value of a variable is no longer needed as long as a clear distinction is being made between value ranges (Forbus, 2008). For example, it is not necessary to know at what temperature water changes from fluid to frozen as long as it is known that frozen water is colder than fluid water. Change rates are described in sign algebra: a parameter can increase, decrease or stay steady, but never go from increase to decrease without going through a steady phase. The qualitative approach requires
abstraction (Forbus, 2008), but is also believed to be more intuitive for novice modellers (VanLehn et al., 2016). Qualitative modelling is also seen as a preparation step for
quantitative modelling (van Someren & Tabbers, 1998). However, as qualitative modelling requires abstraction and conceptual understanding, it is not to be seen as the step up to quantitative modelling but rather as a modelling paradigm in its own right.
The second dimension, modelling support offered, concerns the amount of support offered but also the way in which this support is delivered to students. Support of modelling can be offered as part of the modelling tool (e.g. Bredeweg et al., 2013), as part of the
most simple form, support is limited to defining how much of the modelling should be done by the student and how much is prepared beforehand. In some studies (e.g. Sins et al., 2005) students do not build a model from scratch but rather revise and complete a readily provided model. At the extreme opposite are free modelling assignments, such as described by
Bredeweg, Liem and Nicolaou (2016), in which students choose a system themselves about which they are to build a working and executable model.
However, the starting point or amount of modelling done by the student is not the only way in which modelling support can vary. Teachers and researchers can also build in
scaffolds in the learning materials and instructions. Such scaffolds can take many forms. Sins et al. (2005) suggest that helping students formulate modelling sub-goals and reviewing the model one revision at a time might lead to better modelling outcomes. Another way to support modelling is to provide worked-out examples with the modelling assignment (e.g. Mulder, Lazonder, & De Jong, 2014) or in a repository coming with the modelling tool (Bredeweg et al., 2013). Students can use these examples to compare their own work with. Alternatively, software can be used to compare a student’s model with a model from a repository and provide feedback. Lastly, Sins et al. (2005) stress the importance of activating prior knowledge, as they observed that students using their prior knowledge arrived at better models than students who did not. Scaffolds, such as the ones named here, are essential for a modelling task to be successful. In a more general sense it can be said that modelling tasks should be implemented with great care: the modelling process should be sufficiently
supported, time for learning how to model should be allowed generously and there should be a focus on conceptual understanding.
Aims and research questions
Although learning how to model still poses difficulties for students and teachers, research comparing different modelling conditions is helpful for designing lessons in which
students model. However, it is still unclear whether active modelling actually helps students understand subject-specific content better than traditional methods. In the current study, the effect of qualitative modelling on subject-specific understanding is investigated. As little is yet known about this possible effect, it was decided to focus on having a controlled
environment rather than a high ecological validity. An experimental study was set up in which participants were assigned to a modelling or control condition. In order to keep circumstances as equal as possible between conditions and between schools, it was decided that the study would take place at the Science Park of the University of Amsterdam, rather than at the participating schools.
All students participated in two lessons on two separate topics within the domain of Ecology. In the modelling condition, students use DynaLearn to create their models. As the participants did not model in DynaLearn before, there was a need for modelling instruction. This modelling instruction was integrated in the first lesson, which served as a modelling learning phase for students in the modelling condition. In the second lesson, the modelling application phase, these students were expected to be able to apply their modelling knowledge on the second topic. In the control condition the same Ecology topics were treated, making it possible to assess progress of both groups for the modelling learning phase and for the modelling application phase. To examine the effects of qualitative modeling, we asked two specific questions:
1. What is the effect of the modelling in the learning phase on the learning of subject-specific content?
2. What is the effect of modelling in the application phase on the learning of subject-specific content?
Taking into account the difficulties students experience while learning to model, it was expected that during the modelling learning phase students in the control condition would
improve as much or more than students in the modelling condition with regard to
subject-specific content knowledge. In the control condition, students would only have to concentrate on the subject-specific content, as opposed to students in the modelling condition who would have to concentrate on both modelling and subject-specific content. As students were expected to have some modelling knowledge after the modelling learning phase, in the modelling application phase students in the modelling condition were expected to improve more with regard to subject-specific content knowledge than students in the control condition, as the modelling was expected to be helpful for understanding this subject-specific content.
Methods Participants
Twenty four schools in greater Amsterdam were e-mailed to inform them about the study and ask whether they were interested in participating. Two schools agreed to participate in the study with their Biology classes as part of their regular program. The first school participated with 44 students (mean age = 16.99, SD = 0.56, 45% girls) who were normally taught in two separate classrooms by a different teacher, and the second school participated with 30 students (mean age = 16.67, SD = 0.72, 83% girls) who were normally taught in one classroom. As such, 74 students aged 15 to 18 years old, M = 16.86, SD = 0.65, from two schools participated in this study. Of the participants, 45 (60.8%) were female and 29 (39.2%) were male. All students were in their fifth year of pre-university secondary education (grade 11) and chose Biology as part of their exam program. The two schools used a different Biology method: School 1 used Biologie voor Jou (Biology for You) whereas School 2 used
Nectar. As the schools made the study part of their program, all students were expected to
Although recruitment took place via schools, all data was collected at the Science Park of the University of Amsterdam. Students and schools were not compensated for their
participation, but following the last measurement occasion students did get a tour of the Science Park. The study was approved by the ethics committee of the department of Child Development and Education at the University of Amsterdam. Passive informed consent was asked from the parents of participating students through a letter and information brochure e-mailed directly to the parents. Three participants were over 18 years of age and were asked to sign an active informed consent form before the first measurement occasion.
Research design
For each school, half of the students were randomly assigned to the experimental modelling condition whereas the other half was assigned to the control condition, resulting in groups of 37 students per condition. Each of the schools visited the Science Park twice within the same week and during each of these visits students followed a lesson on a specific topic within the domain of Ecology. The same learning goals with regard to content knowledge were covered in modelling and control group lessons. In the modelling learning phase, the first lesson, modelling learning goals were also covered in the modelling condition. As such, students in the modelling condition did not only learn about Ecology but also about
modelling. In de modelling application phase, the second lesson, no separate modelling learning goals were formulated and lessons for both groups covered the same content
knowledge goals. A description of the learning goals can be found in the intervention section.
Intervention
Students in both conditions followed two lessons on Ecology, a sub-domain of Biology. Learning goals were addressed in the same order in both conditions. All of the lessons were worksheet-based, thus minimizing the amount of instruction given by the teachers. Students were instructed to work alone, although they could ask fellow students,
their teacher or the researcher for help if needed. Apart from the paper worksheet, students in both conditions also worked on the computer for some assignments during the lesson, and all students made the final writing assignment for each lesson on the computer.
In both conditions the lessons were similarly structured: students were given a small amount of information at the beginning of each lesson, on which they were asked to formulate an initial hypothesis. As students gathered more information during the lesson, either through texts provided in the assignment, modelling or looking up information on the internet,
students were asked to revise their previous answers or models. As such, students
understanding of the subject matter was built up slowly, adding more aspects of the system as they proceeded through the lesson. In modelling research this approach has been named Modelling Elaboration Progression (MEP; Mulder, Lazonder, & De Jong, 2011), in which students start out by making a small model and add more and more elements throughout the lesson. In the current study a discovery learning element was added to MEP. In discovery learning students explore their initial ideas about a certain phenomenon, but also gather new information that challenges their ideas (Vosniadou, 2001). In the current study this meant that students were not only asked to elaborate on their model as they got further, but also to sometimes revisit their previous model as new information became available. This could mean addition of elements, but also pruning of the model if a certain element became obsolete. This was expected to aid students’ conceptual understanding of the system they were studying.
Although in the control condition of the current study students did not model, the system was introduced to them in the same manner: starting with a small portion of the system and elaborating on it throughout the lesson. Students in both conditions were furthermore encouraged to go back and forward between their own hypotheses, the model, and the simulation outcomes in order to promote conceptual change (Vosniadou, 2001).
Additionally, this might help students get acquainted with the scientific principle of revision (Gobert et al., 2011; Schwarz & White, 2005).
The first lesson in both conditions covered the topic of trophic cascades. The lesson started with a newspaper article from a Dutch national newspaper describing a plan from the Norwegian government to drastically reduce the wolf population. Students are asked to form a hypothesis on the effect of a decline of the wolf population on other, mostly smaller, animals. The students then did a case study of the Yellowstone ecosystem, in which wolves became extinct in the early 20th century and were re-introduced in the late 20th century. Various animals and plants were introduced throughout the lesson, and students were asked how these organisms react to one another’s presence. Texts were used to convey this information, and students were throughout the lesson asked to identify the different elements in the ecosystem, their properties and the way in which they thought the properties of organisms may influence other organisms. As new information was conveyed, students were not only asked to think about the new information but also about their previous hypotheses, and whether these were still feasible. In addition to these small assignments, the students were asked to write a letter to the editor of the newspaper, in which they expressed their opinion about the plan of the Norwegian government to reduce the wolf population.
The second lesson in both conditions covered the topic of eutrophication. The lesson started with a transcript of a phone call in which a woman complains about the bad smell of the waterbody behind her house. From this transcript students learned that there is a possible connection between a rapid increase in algae and increasingly bad smell coming from the water. Students receive information about algae and possible sources of smell, and learned that living algae themselves do not smell excessively. As they found out that the smell is most probably caused by dead plants and animals, they gradually learned how an increase in algae may cause this death even if the algae are not the direct cause of death. In order to do so, they
were asked to read texts on what various organisms in the water need to survive, and how these may be connected to properties of other organisms. As with the first lesson, this lesson was also concluded with a writing assignment: students were asked to write an advice to the municipality on how to solve the smell issue.
Modelling condition. In the modelling condition, students would as part of the lesson
make a model from the system that was the principal subject of the lesson. For this, an updated version of the modelling tool DynaLearn was used (Bredeweg et al., 2013). DynaLearn is a qualitative modelling tool that in its current version can be used in Google Chrome without any further installations. In DynaLearn, the user starts by defining entities. These entities can be connected as in a concept map, but the user can also define any number of quantities for each entity. A quantity is a measurable property of an entity, and can be related to any other quantity. These relationships can be positive or negative. Furthermore, each quantity has a change rate that can be set to increase, decrease or steady. It is, however, not necessary or sensible to set a change rate for every quantity.
Figure 1
Simple DynaLearn model A: Model
B: Simulated model
A simple example of a DynaLearn model regarding eutrophication can be found in Figure 1. In this model, part of the eutrophication process in which underwater sunlight is blocked by algae is depicted. The entity ‘waterbody’ is connected to three other entities:
algae, sunlight and water plants. These all have quantities related to the available amount of the entity. The amount of algae is negatively related to the amount of sunlight, as algae float at the water surface and thus block sunlight under water. The amount of sunlight is positively related to the amount of water plants, as these plants need sunlight to grow. All three
quantities have a change rate, depicted by . Only the change rates at the beginning of a causal chain need to be defined, in this case that of algae. It is in this example defined as increasing. If the model is simulated, as can be seen in Figure 1 B, arrows indicate that sunlight would decrease and thus water plants would decrease. This example shows only one entity and a small number of quantities and relations. The students were asked to make more elaborate and complex models, such as the one found in Figure 2.
Figure 2
Model of trophic cascade in the Yellowstone ecosystem
In the current study, students had to learn how to use five out of the fifteen possible DynaLearn ingredients, notably entities, quantities, change rates and negative and positive proportionalities. In the first lesson students were guided through the whole process of modelling. In the second lesson, students were expected to know how to make a model in DynaLearn but were still supported in the decision processes as to what entities, quantities and relations to include. This was done through assignments, in which students for example
read a text and were asked what relevant entities or quantities they could find in this text, or an assignment in which they were asked whether they thought a relation they had formulated earlier was still correct and relevant with the knowledge they had gained since. In both phases the models were relatively simple, with a limited number of elements in each model and without feedback loops or other complicated causality structures.
Control condition. In the control condition, students were guided through the systems
and taught about them in the same order as in the modelling condition. The same texts were presented to these students and students were asked similar questions on the ecosystems as students in the modelling conditions: what are the elements in this ecosystem, what properties do they have and how are these properties related? However, these system elements were not referred to in terms of entities, quantities and change rated but rather as elements and their properties. Furthermore, students were not asked to show these in a model but in a textual answer to questions. As students in the modelling condition had to work with the computer, additional computer assignments were added to the control condition lessons in order to control for possible effects of computer usage. In the first lesson the additional computer assignment consisted of making a PowerPoint on all animals in that lesson, and in the second lesson the additional computer assignment consisted of looking up additional information on the internet.
Measures
Content knowledge. In order to assess content knowledge on the topics of trophic
cascades and eutrophication, pre- and post-tests were administered. For both topics five questions were asked: two knowledge reproduction questions, two questions with regard to comprehension and one application questions. An example of a reproduction question is: ‘what properties of algae contribute to the process of eutrophication?’. An example of a comprehension question is: ‘what are the differences and similarities between changes and
disturbances in an ecosystem?’. An example of an application question is: ‘In Chile, a lot of salmon aquaculture can be found. These fish farms have regular algal bloom problems, caused by El Niño. According to some, fish farmers worsen the problem by throwing fish that has died as a result of algal bloom back into the ocean. Explain how these dead fish reinforce the problem of algal bloom’. Questions were constructed in such a manner that for each question on the trophic cascade there would be a similar question on eutrophication in order to allow for comparability between the topics.
The questions were identical for the pre- and post-test and scores were determined as in a school test: reproduction questions were assigned the smallest maximum score (2.5 and 3 points), comprehension questions were awarded an intermediate amount of points (3 and 4 points) and application questions were awarded the highest maximum score (5 points). For each question it was also possible to score less than the maximum but more than 0 points, if the question was answered partially correct. For each test (pre-test 1, pre-test 2, post-test 1 and post-test 2) scores could range between 0 and 17.5 points. Reliability was found to be very low on pre-test 1, Cronbach’s α = .074, and low on pre-test 2, Cronbach’s α = .409, post-test 1, Cronbach’s α = .483, and post-post-test 2, Cronbach’s α = .435.
As each of the tests consisted of reproduction questions and comprehension and application questions, separate analyses could be carried with regard to improvement on reproduction on the one hand and comprehension and application on the other hand. For the reproduction questions a total of 5.5 points per test could be scored, whereas for the
application and comprehension questions a total of 12 points per test could be scored. It must be noted that reproduction and comprehension questions would not appear sorted in the test.
Systems thinking. In addition to the subject-specific parts of the test three systems
thinking questions were developed. These questions regarded a system that was not discussed during the lesson. Two of the questions had to be answered in text, whereas for the third
question the students were asked to draw out the system. These questions were not used in the analyses. The two textual questions proved to be too simple to answer correctly whereas the drawing question posed scoring difficulties: it was impossible to unambiguously determine whether an answer was wrong or right.
Modelling knowledge. In order to assess students modelling ability four questions
from a larger systems thinking/modelling questionnaire were administered during the pre- and post-tests in the modelling condition.
Modelling activity. For students in the modelling condition data was available as to
how often they performed certain actions in DynaLearn: create, modify, delete, undo, redo and new model. The last three were rarely performed and were therefore not used. Create, modify and delete on the other hand were performed more often and were used to gain insight as to whether more or less activity in DynaLearn may have some correlation with subject-specific understanding.
Interest. Students from the second participating school filled out a questionnaire on
triggered situational interest after the second lesson. This questionnaire was based on a situational interest questionnaire reported by (Linnenbrink-Garcia et al., 2010) and consisted of 8 questions with regard to triggered situational interest, focused on the lessons that were followed as part of the study rather than general interest in Biology or Ecology topics. Examples of the questions used are ‘I think I learned new things during these lessons’ and ‘After the first lesson I did not feel like doing the second lesson at al’. Reliability for this questionnaire was found to be acceptable (α = .763).
General ability. In order to assess students’ general ability in Biology final grades for
Worksheet completion. In order to asses to what extent students participated in the
lesson worksheet completion was examined. As each of the worksheets had a different number of questions, the percentage of questions completed was calculated.
Procedure
Each of the schools visited the Science Park twice within the same week for three hours per visit. Before the visit, students were randomly assigned to either of the conditions and upon their arrival students were assigned to one of two computer rooms depending on the condition they were assigned to. When settled in the room the study was introduced shortly and students took a pre-test for approximately 30 minutes. After taking the pre-test, students worked on the lesson individually. Halfway through the lesson students had a short break, and the first visit ended with a short post-test on the topic of the first lesson, taking approximately 10 minutes to finish. During the second visit, students started immediately with the lesson. After two hours, in which a short break was included, students took the test. This post-test took 20 to 30 minutes. After the post-post-test, the study was discussed in more detail with the students, after which the students were taken on a short tour of the Science Park.
During the visit of School 1, two teachers were present at both days. During the visit of School 2, a different teacher accompanied the students on each of the days. At all times, one teacher or researcher was present in the room to answer questions. At times, a teacher as well as a researcher were present in the room. If a teacher was present, he or she would answer questions students had.
Results
From a total of 74 students, data was gathered and then entered in and analysed with IBM SPSS statistics (IBM Corp., 2013). Treatment integrity and exclusion criteria are discussed first. Then analyses regarding content learning, for the entire test as well as for the
separate reproduction and comprehension segments, are discussed for the modelling learning phase and subsequently for the modelling application phase. In order to gather information with regard to how successful modellers differ from less successful modellers, DynaLearn action data is analysed next. Lastly, students’ situational interest is discussed. For all analyses assumptions were checked. Normality was inspected visually, for homogeneity of variances Levene’s test was used and for homogeneity of covariance matrices Box’s M was used. Usually all assumptions were met. Where assumptions were not met, this is mentioned in the text.
Treatment integrity
Worksheet completion was assessed as a measure of treatment integrity. On average, between 85% and 99% of the worksheet was completed. However, there were students who got substantially less far with their worksheets, the lowest completion rate being 25%. Completion was slightly higher for control group students in both phases, and in the
modelling application phase worksheet completion was higher than in the modelling learning phase.
For students in the modelling condition DynaLearn activity was examined as well. From the 36 students in the modelling condition, one never logged in on DynaLearn despite being present in during the lesson. Three students who were present during the first lesson did not show activity on that day. Seven students did not show any activity on the day of the second lesson. Five of these students did attend the second lesson and it is therefore believed that they collaborated with another student while making the model.
Students performed between 31 and 253 actions per lesson, with an average of 92.38 actions in the modelling learning phase (SD = 19.09) and 102.64 actions during the modelling application phase (SD = 48.22). Although these averages are skewed due to two highly active students, it can be said that students in the modelling condition did actively use DynaLearn.
In addition to the formal, measured criteria of treatment integrity informal
observations were also made. In general, students in the modelling group needed more time to finish the lesson than students in the control group. Experimental circumstances were
observed more strictly during the second measurement occasion: students from School 1 collaborated more often and sometimes left early, especially in the modelling condition during the modelling learning phase. Some modelling students from School 2 did take a long break during the modelling learning phase, resulting in a shorter possible learning period.
Inclusion criteria
Minimum levels of worksheet completion and DynaLearn activity were required for students to be included in the analyses. Although informal observations also showed that some students worked on the lessons less seriously than others, these observations are incomplete and subjective and could therefore not be used to determine whether students should be in- or excluded.
Students who finished less than 60% of their worksheet were excluded from analyses on content learning and DynaLearn activity, as it was believed that these students did not participate seriously enough. Students who were present but did not show DynaLearn activity were included in the analyses on content learning, as they were believed to have collaborated with classmates and therefore worked on the assignment seriously as long as they finished at least 60% of their own worksheet. However, only students for whom DynaLearn data was available on a certain day could be included in the DynaLearn activity analyses for that day. Therefore, having performed at least one action in each of the categories (create, modify and delete) served as an additional inclusion criterion for DynaLearn activity analyses.
In addition to the minimum requirements for inclusion it was also examined whether there were outliers on the other side of the spectrum. With regard to the data extracted from DynaLearn two students showed exceptionally high levels of activity. Although it was
observed during the lesson that these students took the assignment very seriously, their high activity disturbed the normal distribution as their number of actions was three standard
deviations above the average. These students were excluded from the analyses for this reason. High outliers were not found for worksheet completion or test scores. In Table 1 the total number of students included in the different phases can be found.
Table 1
In- and exclusion for all groups, phases and analyses
Modelling group Control group Total
Modelling learning phase
Pre- and post-test available 33 36 69
Content learning1 Students included 33 33 66 Students excluded 0 3 3 DynaLearn activity2 Students included 29 - 29 Students excluded 4 - 4
Modelling application phase
Pre- and post-test available 32 32 64
Content learning1 Students included 28 31 59 Students excluded 4 1 5 DynaLearn activity2 Students included 21 - 21 Students excluded 11 - 11
1 inclusion criterion: at least 60% of the worksheet completed
2 inclusion criterion: at least 60% of the worksheet completed and at least one action on create, modify and
delete. Students with extreme high activity were also excluded.
Progress in modelling ability
As modelling was the subject of this intervention, a short multiple-choice test was administered in the modelling group only in order to assess whether modelling students actually learned about modelling. Means and standard deviations for this modelling knowledge test can be found in Table 2.
Table 2
Means and standard deviations of test scores1 on modelling by school
Pre-test Post-test
N Mean SD Mean SD
School 1 17 2.00 0.71 2.35 0.79
School 2 11 2.18 0.98 2.64 1.03
1 a total of 4 points could be scored for this test
In order to assess the differences between pre- and post-test and potential differences between schools a mixed ANOVA was performed with measurement occasions being the within-subject factor and the school a student attended as between-subject factor. There was a trend towards improvement over time, F(1, 26) = 3.70, p = .065. The interaction found
between measurement occasion and school was not significant, F(1, 26) = 0.059, p = .811. As such, it can be concluded that students from different schools gained as similar amount of modelling knowledge.
Effect of modelling in the learning phase on learning of subject-specific content
In order to assess the acquisition of subject-specific content knowledge during the modelling learning phase, a pre- and post-test were administered right before and right after the first lesson. Means and standard deviations for these tests can be found in Table 3. Means are also graphically displayed in Figure 3. From the mean scores, it appears that there may have been an effect of condition, but that this effect might not have been consistent across schools.
Table 3
Means and standard deviations of test scores1 for the modelling learning phase by
condition Pre-test Post-test N Mean SD Mean SD School 1 Modelling 20 7.18 2.41 5.93 1.59 Control 19 6.37 1.23 7.63 2.29 School 2 Modelling 13 6.88 2.22 9.00 2.18 Control 14 7.89 2.67 8.89 3.27
Figure 3
Means on pre- and post-test in the modelling learning phase A: School 1 B: School 2
To examine the differences between the conditions, a mixed ANOVA was conducted with measurement occasion (pre- or post-test) as the within-subject factor and school and condition as between-subject factors. No interaction between measurement occasion and condition was found, F(1, 62) = 1.18, p = .282. However, there was an interaction of measurement occasion and school, F(1, 62) = 580, p = .019 and, more notably, a three-way interaction between measurement occasion, school and condition, F(1, 62) = 7.93, p = .007. As this three-way interaction indicated that the school participants attended played a role in how they reacted to the intervention, post-hoc comparisons were carried out. The results of the post-hoc comparisons are presented in Table 4.
From this table it can be seen that students from School 1in the control condition show significant improvement whereas students from School 2 do not. In the modelling condition, mean scores of students from School 2 improved significantly whereas scores of students from School 1 declined significantly over time. Thus, the modelling appeared to have helped understand subject content for students of the second school only.
Table 4
Pairwise comparisons of pre- and post-test scores1 in the modelling learning phase per school
Average improvement (SE)2 p 95% CI
School 1 Modelling -1.250 (0.575) .034 -2.40 – -0.10 Control 1.263 (0.590) .036 0.08 – 2.44 School 2 Modelling 2.12 (0.713) .004 0.69 – 3.54 Control 1.00 (0.687) .151 -0.37 – 2.37
1 a total of 17.5 points could be scored for this test 2 pre-test score subtracted from post-test score
Modelling is believed to particularly support conceptual understanding. Therefore, the reproduction and comprehension segments of the test were also analysed separately. The means and standard deviations of these test segments for the modelling learning phase can be found in Table 5.
Table 5
Means and standard deviations of pre- and post-test scores for comprehension1 and reproduction2 questions in the modelling learning phase.
Pre-test Post-test N Mean SD Mean SD Comprehension School 1 Modelling 20 3.70 1.90 2.20 1.39 Control 19 3.39 1.49 3.45 1.80 School 2 Modelling 13 3.77 1.58 4.73 1.69 Control 14 4.39 2.25 4.64 2.63 Reproduction School 1 Modelling 20 3.48 0.98 3.73 0.63 Control 19 2.97 1.01 4.18 0.95 School 2 Modelling 13 3.12 1.08 4.27 0.95 Control 14 3.50 1.04 4.25 0.89
1 a total of 12 points could be scored on this test segment 2 a total of 5.5 points was could be scored on this test segment
A mixed ANOVA with scores on the reproduction segment and the comprehension segment of the test as within subject factors and school and condition as between subject factors was carried out. Again, a three-way interaction between measurement occasion, school
and condition was found for both reproduction, F(2, 62) = 5.35, p = .024, and comprehension,
F(2, 62) = 4.68, p = .031. However, as Levene’s test was significant for the post-test segment
on comprehension, F(3, 62) = 3.04, p = .035, it may be that this effect is overestimated. Post-hoc comparisons for this analysis can be found in Table 6. It can be seen that the decrease in score that was earlier found for modelling students from School 1 can be
explained by a decrease specifically on comprehension questions, whereas on reproduction questions the score of these students does not decrease. A similar pattern is found in School 2: the improvement on comprehension questions in the modelling condition is not significant (although a trend towards improvement can be seen), whereas the improvement on
reproduction questions is significant. In the control condition students from both schools improved on reproduction but not on comprehension.
Table 6
Pairwise comparisons of pre- and post-test scores for comprehension1 and reproduction2 questions in the modelling learning phase.
Average improvement (SE)3 p 95% CI
Comprehension Modelling School 1 -1.50 (0.46) .002 -2.42 – -0.58 School 2 0.96 (0.57) .096 -0.18 – 2.10 Control School 1 0.05 (0.47) .911 -0.99 – 0.89 School 2 0.25 (0.55) .650 -0.85 – 1.35 Reproduction Modelling School 1 0.25 (0.26) .346 -0.28 – 0.78 School 2 1.15 (0.33) .001 0.50 – 1.81 Control School 1 1.21 (0.27) <.001 0.670 – 1.75 School 2 0.75 (0.32) .020 0.12 – 1.38
1 a total of 12 points could be scored on this test segment 2 a total of 5.5 points was could be scored on this test segment 3 post-test score – pre-test score
In sum, no separate effect of condition was found in the modelling learning phase. An interaction between measurement occasion, school and intervention was found. All observed improvement was found on the reproduction segment of the test.
Effect of modelling in the application phase on learning of subject-specific content
In Table 7 and Figure 4, it can be seen that there are rather large differences between conditions on the pre-test scores for School 1. Students in the modelling condition score higher on the post-test, but considering the differences in pre-test scores further analysis was needed to test these differences.
Table 7
Means and standard deviations of test scores1 in the modelling application phase by condition Pre-test Post-test N Mean SD Mean SD School 1 Modelling 17 8.32 1.83 9.35 2.66 Control 19 5.70 2.74 7.66 2.66 School 2 Modelling 11 6.50 2.47 10.55 2.69 Control 12 6.31 2.09 9.10 3.03
1 a total of 17.5 points could be scored for this test
Figure 4
Means on pre- and post-test in the modelling application phase A: School 1 B: School 2
A mixed ANOVA with measurement occasion (pre- or post-test) as the within-subject factor and school and condition as between-subject factors was conducted. No interaction between measurement occasion and condition was found, F(1, 55) = 0.02, p = .903. Thus, improvement over time could not be explained by the condition students were assigned to.
Unlike in the modelling learning phase, an interaction effect between measurement occasion, school and condition was not found, F(1, 55) = 2.02, p = .161. The only factor that interacted with measurement occasion was the school students attended, F(1, 55) = 7.73, p = .007: students from School 2 on average improved more than students from School 1.
Post-hoc pairwise comparisons of pre- and post-test were carried out and as can be seen in Table 8 it was found that students from School 2 all improved significantly regardless of the condition they were in. For School 1 significant improvement was only found for students in the control condition.
Table 8
Pairwise comparisons of pre- and post-test scores1 on the modelling application phase per school
Average improvement (SE)2 p 95% CI
School 1 1.50 (0.45) .002 0.60 – 2.39
School 2 3.49 (0.56) <.001 2.37 – 4.62
1 a total of 17.5 points could be scored for this test 2 post-test score – pre-test score
Again, additional analyses of test segments measuring subject-specific comprehension and reproduction were carried out. Means and standard deviations of these test segments can be found in Table 9.
Table 9
Means and standard deviations of pre- and post-test scores for comprehension1 and reproduction2 questions in the modelling application phase.
Pre-test Post-test N Mean SD Mean SD Comprehension School 1 Modelling 17 4.18 1.63 5.50 1.63 Control 19 2.17 2.25 3.79 2.36 School 2 Modelling 11 3.48 2.21 6.36 2.29 Control 12 2.75 2.08 5.35 2.76 Reproduction School 1 Modelling 17 4.15 0.63 3.85 0.61 Control 19 3.53 1.05 3.87 0.78 School 2 Modelling 11 3.02 0.53 4.18 0.81 Control 12 3.42 0.97 3.75 0.54
1 a total of 12 points could be scored on this test segment 2 a total of 5.5 points was could be scored on this test segment
A mixed ANOVA with scores on both of the test segments as within subject factors and school and condition as between subject factors was used to this end. For reproduction, a three-way interaction was found between measurement occasion, school and condition, F(1, 55) = 7.99, p = .007: on average, only modelling students from School 2 improved on reproduction. For comprehension this interaction was not significant, F(1, 55) = 0.183, p = .670. A trend towards a two-way interaction between measurement occasion and school was found for the comprehension segment of the test, F(1, 55) = 3.57, p = .064, but as Levene’s test was significant for the comprehension segment of the post-test, F(3, 55) = 4.24, p = .009, this trend may be detected erroneously.
In order to investigate the differences between schools and conditions for the separate test segments, post-hoc comparisons were carried out. In Table 10, however, it can be found that students from both schools improved significantly on comprehension. In Table 11 it can
be found that for reproduction there is only a significant improvement for students in the modelling condition from School 2.
Table 10
Pairwise comparisons of pre- and post-test scores for comprehension questions1 by school in the modelling application phase.
Average improvement (SE)2 p 95% CI
School 1 1.47 (0.42) .001 0.63 – 2.32
School 2 2.75 (0.53) <.001 1.69 – 3.80
1 a total of 12 points could be scored on this test segment 2 pre-test score subtracted from post-test score
Table 11
Pairwise comparisons of pre- and post-test scores for reproduction questions1 by condition and school in the modelling application phase.
Average improvement (SE)2 p 95% CI
Modelling School 1 0.29 (0.24) .215 -0.76 – 0.18 School 2 1.16 (0.29) <.001 0.57 – 1.74 Control School 1 0.34 (0.22) .129 -0.10 – 0.79 School 2 0.33 (0.28) .238 -0.23 – 0.89
1 a total of 5.5 points was could be scored on this test segment 2 pre-test score subtracted from post-test score
Thus, in the modelling application phase, no effect of condition was found but
students from School 2 did improve more between measurement occasions than students from School 1. Again, improvement was only found in the reproduction segment of the test.
Specific outcomes for students in the modelling condition
Dynalearn activity: differences between schools. As school appeared to play a role
in students’ improvement, it was examined whether students from the two schools differed as to how often they created, modified and deleted DynaLearn elements. It is important to note that differences between the various types of actions are not of interest as these are bound to be different: students who delete or modify an element must have created that element before, and students are not expected to finish with an empty model. It is, however, of interest
and standard deviations for the number of DynaLearn actions in the modelling learning phase can be found in Table 12.
Table 12
Means and standard deviations of the number of modelling actions in the modelling learning phase by school
Create Modify Delete
N Mean SD Mean SD Mean SD
School 1 18 42.33 15.08 26.06 7.35 26.06 7.35 School 2 13 42.54 10.15 28.69 5.81 15.08 4.15
In order to assess the differences between schools, a MANOVA was carried out with school as independent variable and create, modify and delete as dependent variables.
Multivariate test results show that school does not play a role in the amount of actions a student performed, Λ = 0.850, F(3, 27) = 1.59, p = .215.
Table 13
Means and standard deviations of the number of modelling actions in the modelling application phase by school
Create Modify Delete
N Mean SD Mean SD Mean SD
School 1 13 47.15 14.92 30.31 10.60 9.00 4.20
School 2 8 54.63 11.03 33.00 7.75 15.50 7.31
A MANOVA was carried out to assess differences between schools on creation, modification and deletion of DynaLearn elements. Multivariate tests show no significant effect of school as a predictor on the amount of actions performed, Λ = 0.703, F(3, 17) = 2.40,
p = .104.
DynaLearn activity: high and low improvement. Lastly, it was investigated whether
the amount of improvement between pre- and post-test was related to the amount of actions performed in DynaLearn. Improvement in each of the phases was determined by the
difference between pre- and post-test in that particular phase. For each of the phases, the median value was determined and used as a cut-off point: students scoring below that value formed the low improvement group and students scoring above that value were assigned to
the high improvement group. Means and standard deviations of the DynaLearn activities for both groups in the modelling learning phase can be found in Table 14. A MANOVA was carried out and multivariate tests showed no significant effect of improvement level on amount of actions performed, Λ = 0.981, F(3,24) = 0.16, p = .924.
Table 14
Means and standard deviations for modelling actions by improvement level in the modelling learning phase
Create Modify Delete
N Mean SD Mean SD Mean SD
Low 15 43.67 13.58 27.13 8.27 14.27 6.70
High 13 41.77 14.17 27.31 5.66 14.00 6.26
Means and standard deviations for both groups in the modelling application phase can be found in Table 15. For this phase an effect was found, Λ = 0.637, F(3,17) = 3.24, p = .048. subsequent univariate tests show that the groups differ on create, F(1, 19) = 5.75, p = .027), and modify, F(1,19) = 9.27, p = .007, but not on delete, F(1,19) = 1.12, p = .303.
Table 15
Means and standard deviations for modelling actions by improvement level in the modelling application phase
Create Modify Delete
N Mean SD Mean SD Mean SD
Low 11 43.82 11.60 26.27 4.80 10.09 5.82
High 10 56.80 13.21 36.90 10.45 13.00 6.77
Thus, no difference in DynaLearn actions was found between schools in either of the phases. However, students who improved more on the subject-specific content for the modelling application phase were found to have created and modified significantly more DynaLearn elements. This effect was not found for the modelling learning phase.
Situational interest
As after the first data collection it appeared that some students lost their interest, either as a result of working too long on the lesson or finishing very early, it was hypothesized that students in different conditions might differ in their interest. Therefore, in the second school a
questionnaire on situational interest was administered at the last measurement occasion. The questionnaire was not filled out entirely by all of the students, resulting in a total of 22 questionnaires being available for analysis. Students on average scored 2.64 (sd = 0.62) on this 5-point scale and no difference was found between students in the experimental and control condition, F(1, 20) = 1.20, p = .286.
Discussion
The current study investigated whether modelling aids students understanding of specific content. As very little is yet known about the effects of modelling on specific content learning, the study was set up as an experimental comparison of subject-specific content learning in a modelling and non-modelling conditions. It was hypothesized that as students learned how to model, they would have less cognitive capacity available for the learning of subject-specific content and as a result subject-specific learning would be less for this group than for the control group. After this initial learning phase, however, it was expected that modelling would aid students in gaining subject-specific knowledge. It is this modelling application phase that is of specific interest, as results in this phase reveal whether the learned modelling skills support the learning of subject-specific content.
In the modelling learning phase a three-way interaction between measurement occasion, school and condition was found, indicating that students from one school reacted differently to the intervention than students of the other school: modelling students from School 2 improved with regard to subject-specific content whereas scores for modelling students from School 1 declined.
In the modelling application phase students improved regardless of the condition they were in. The modelling intervention did not aid to subject-specific understanding for students
who had already learned how to model. This lack of effect may be explained in a number of ways. First, there may have been implementation differences between conditions and between schools. Second, the three most important instruments were developed especially for the current study and although these were constructed with great care, they may not have been reliable enough to detect differences. Third, the learning time allocated for this study was relatively low and the subject-specific content and modelling task was relatively simple, which may have influenced the results. These issues will be discussed below, followed by suggestions for future research.
Implementation
Differences between conditions. Overall, the experiment was demanding for
students: regardless of the condition they were assigned to, students were expected to work individually on their tests and worksheets for almost three hours during each of the sessions. This is considerably longer than the duration of most experimental studies (Hoyle, Harris, & Judd, 2002) and may have asked a lot of students’ attention span. Although it was tried to keep lessons fairly similar in terms of workload, students in the modelling condition did work towards a larger number of learning goals and had a larger number of questions on their worksheets. Still, no considerable differences were found between worksheet completion for modelling and control group students and completion was around 90% on average.
Furthermore, the data gathered in DynaLearn shows that students in the modelling condition all performed a considerable number of actions in DynaLearn. Although students who
improved more on subject-specific content did perform more actions than students with a low improvement, most students did actively use the modelling tool. Lastly, students in both conditions did not differ with regard to situational interest. However, more informal observations did show that students in the control condition often finished quicker than students in the modelling condition. Although formally there were minimal implementation
differences, the informally observed differences suggest that that students in the modelling condition needed more time to attain the same level of subject-specific improvement as students in the control condition.
Differences between schools. In both the modelling learning phase and the modelling
application phase differences between schools were found. The participating groups were fairly similar: no differences were found in prior scores on biology and age, and gender did not prove to be a significant control variable. It was plausible that students attending School 1 would have a slightly higher score on the pre-test on eutrophication as this topic was treated more recently in the Biology method used at their school, but such a difference was not found. It is therefore likely that the differences found are not rooted in actual differences between the schools themselves but in implementation differences.
No formal measurements to this regard were taken, but informal observations showed differences as to how orderly the experiment was conducted. In general, implementation was more orderly with School 2 than it was with School 1. In School 1, the control condition was more orderly than the modelling condition. The informal implementation of the experiment in the modelling condition at School 1 may have prompted students to take the experiment less seriously, which may explain the decline in scores for these students during the modelling learning phase. Although no decline was found for this group in the modelling application phase, the poor learning during the modelling learning phase may have had an effect on students modelling abilities resulting in no improvement whereas all other groups improved.
Measurements and limitations
Tests on subject-specific content. For the two tests on subject-specific content a low
reliability was observed (Cronbach’s α between .074 and .483). It is possible that this
imprecision resulted in not finding a difference between conditions. A likely cause for the low reliability is the diversity in questions treated in these tests. Test questions differed on three
dimensions: the learning goal covered, level of understanding assessed and cost of filling out the question. Although diversity on all three dimensions was sought, it was later noticed that comprehension questions had a higher cost of filling out than reproduction questions: for comprehension questions, students had to write multiple lines and were expected to show their reasoning steps, whereas reproduction questions were all multiple choice. Therefore the comprehension questions were more sensitive for students’ ability to express their knowledge (Drenth & Sijtsma, 2006). This imbalance may explain why students improved only on the reproduction segments of the tests, and not on the comprehension segments: these were either too costly to seriously fill out or not sensitive enough to measure actual understanding.
Another probable cause of imprecision is related to the pre- and post-test being identical. This may have prompted students to focus on tested content more than on other content. In addition, it appeared to demotivate students, who often pointed out that they had already made these questions. Often the researcher present reminded the students that their progress between the two tests was of interest, and students then continued filling out the test.
Tests on systems thinking. Although the subject-specific content learning was of
principal interest in the current study, the extent to which students are able to apply systems thinking may in large part influence how successful a modelling intervention is in terms of subject-specific content learning. Therefore, it was attempted to also measure students’ systems thinking abilities. As no existing instruments were found that could either be used or adapted for the current study, questions with regard to systems thinking were developed for this study specifically. However, this test was not used in final analyses for two reasons. First, some of the questions failed to discriminate as many students scored the maximum amount of points. Second, another question posed scoring difficulties. For this question students were asked to draw the system they had been reading about. As the answers differed in a large number of ways, it was complicated to determine a definition of what was a good and what
was a less good answer, which is a requirement for quantitative assessment of progress (Drenth & Sijtsma, 2006, pp. 77). As a result, no measurement of systems thinking skills was available.
Learning time
Learning time is a persistent challenge in modelling education and modelling research alike. Modelling is believed to support learning, but before it can be a tool to accomplish subject-specific learning goals students have to spend time on learning how to model. During this time modelling is to be seen as a learning goal on its own (Jacobson & Wilensky, 2006; VanLehn et al., 2016). In the current study, it was tried to separate learning how to model and modelling for learning with the introduction of separate modelling learning and modelling application phases. Students were in the modelling application phase able to create models without further instruction on the use of DynaLearn. However, it is possible that learning time was too short for students to develop their modelling abilities to a level that aids subject-specific understanding. If that is the case, students would still need a lot of cognitive capacity for learning how to model even though specific modelling learning goals were not defined for this phase. The fact that students improved only on the reproduction segment of the test and not on the comprehension segment may support this idea, although it was not assessed whether students actually mastered the modelling learning. As such it is not possible to determine whether students’ modelling skills were sufficient for modelling to serve as a learning tool rather than a learning goal.
The modelling assignments in the current study were relatively large for the amount of time given: students were to create one model from scratch in each lesson, whereas in other short-term modelling studies often modelling elaboration or modification tasks are used (e.g. Mulder et al., 2015; Sins et al., 2005). However, compromises were made in terms of subject content and model complexity. The models in the current study consisted of a small number
