Faculty of Social and Behavioural Sciences
Graduate School of Childhood Development and Education
Effects of digital practicing systems on
math performance in primary education
Research Master Child Development and Education Master Thesis
Daan Farjon, BSc
Supervisors:
dr. N.M. (Nienke) Ruijs & prof. dr. P.F. (Peter) de Jong Thursday 7th of July, 2016
2 Abstract
Digital practicing systems with elements of gamification seem to be a promising tool for enhancing learning performance. There is, however, a lack of evidence to support the beneficial effects of ICT on student performance. This study investigated the impact of practicing with Math Garden on math performance in primary education. The difference-in-differences approach was used to compare the learning gains of 30 intervention groups that practiced with Math Garden for one school year to 24 control groups that did not practice with Math Garden yet. These age groups were comprised of more than 1,500 students. Data was aggregated at the group level. Results of the analyses indicated that learning gains in the 2012/2013 school year were similar for both the intervention and the control groups. Several explanations were discussed as to why Math Garden did not have an impact. Additional research is required to support that digital practice systems are effective tools for enhancing math performance.
Key words: digital practice systems, gamification, difference-in-differences, math performance, primary education
3 Effects of digital practicing systems on math performance in primary education The use of ICT for educational purposes is rapidly increasing to the extent that ICT seems to be indispensable, as indicated by several worldwide studies. Almost 75% of the 15-year-old students used a computer or laptop at school in 2012 (OECD, 2015). Although this percentage has not changed since 2009, the frequency of ICT use is increasing in most countries. The European Commission and ICILS indicated how often – over 50% of eighth graders used ICT at least once a week in 2013 (European Commission, 2013; Fraillon, Ainley, Schulz,
Friedman, & Gebhardt, 2013). The amount of computers that students can use varies: there is on average one computer available per 3-7 students in the EU (European Commission, 2013). ICT is also used by many teachers. For instance, an extensive study on the use of ICT in Dutch education indicated that 90% of teachers have been using ICT in teaching for half a decade (Kennisnet, 2010, 2015). But although the use of ICT in education is widespread, the manner in which it is generally used varies.
In Dutch primary education, almost all primary teachers used ICT to track students’ results, communicate, prepare lessons and give instructions (Kennisnet, 2015). They used ICT for these purposes quite often: 50% of the primary teachers spend more than 10 hours using ICT each week and 35% of the primary teachers even more than 15 hours. Other common purposes of ICT for students consisted of the use of ICT in order to let them practice subject matters and play (educational) games. There are therefore varying possibilities when
educational ICT is used either for or by students.
The combination of ICT use for practicing subjects and for playing educational games is quite interesting. ICTs which help to increase study time and practice, such as digital practicing systems, were linked by the OECD (2015) to better student performance. Children enjoy playing games, which therefore acts as a catalyst to practice and learn (Boyle, Connolly, Hainey, & Boyle, 2012). The use of games and non-game ICTs with game elements
(gamification) is furthermore one of the upcoming trends in primary education (e.g. Johnson et al., 2014; Voss, 2015). The current study aims to explore this ICT use and attempts to investigate the effect of digital practicing systems with elements of gamification on learning performance. More specifically, the impact of practicing with Math Garden (Van der Maas, Klinkenberg, & Straatemeier, 2010) was examined using a difference-in-differences (DiD) approach (e.g. Schlotter, Schwerdt, & Woessmann, 2011). The concept of gamification and research on the impact of gamification will be discussed below, after which the choice for the DiD approach will be explained.
4 1.1 Using gamification to enhance learning performance
Digital practice systems are computerized applications which provide opportunities to practice subject matters. These systems have various advantages compared to traditional paper-and-pencil methods. For instance, answers are checked automatically and instantly, students receive rich and varied exercises, and results are easily tracked and reported. Besides these advantages, many practice systems arguably contain elements from the gaming world. Game technologies are often used to make learning more pleasant for students (Punie, Zinnbauer, & Cabrera, 2006). Digital practicing systems might therefore be perceived by students as a game.
The use of gaming concepts in non-game contexts, such as education, is referred as gamification (e.g., Kim & Lee, 2013; Simões, Redondo, & Vilas, 2013). The main purpose of gamification is to motive the users (Nussbaum, 2007). Elements of gamification are, for example, the use of pleasantly frustrating tasks, trial and error, control over your
development, adjustment of level to user’s ability, and progress reports such as badges and leaderboards. The goal is usually to maximize enjoyment and engagement by capturing the interest of learners and inspiring them to continue learning and increase learning performance (Boyle et al., 2012).
The driving force of gamification can be identified in psychological theories including those of Vygotsky, and Ryan and Deci (Boyle, Connolly & Hainey, 2011). In digital practice systems, the balance between the exercise level and students’ ability is similar to Vygotsky’s zone of proximal development. According to Vygotsky (1978), learning progress is made when students receive exercises that are just beyond their current ability, but can be
completed with some help of the practice system. Ryan and Deci’s self-determination theory could also be used to explain students’ motivation for and engagement in playing games. Ryan and Deci (2000) indicated that students have intrinsic needs for competence, autonomy and relatedness and by fulfilling these needs they become motivated and will learn. These basic needs of students can be fulfilled by playing (educational) games (Przyblyski, Ryan, & Rigby, 2009). So digital practice systems with elements of gamification are able to support learning progress and fulfill intrinsic needs of students according to these theories.
An example of Vygotsky’s zone of proximal development is a digital practice system that adjust its difficulty to the ability of the learner. The adaptability can also be argued as an element of gamification. This kind of practice systems is promising for learning, because adaptive education has beneficial effects on the development of students due to differentiation and practice at the level of students’ ability (Blok, 2004). Adaptive practicing systems are
5 able to provide tasks or challenges that are always matched to the ability of the learner. An experiment on the impact of such a practice system on reading engagement showed that students enjoyed the system but were less motivated to learn reading than expected (Ronimus, Kujala, Tolvanen, & Lyytinen, 2014). Another study on an adaptive practice system showed that primary students gained an extra 1.5 months of learning experience compared to a control group after both groups had practiced math with this system for 6 months (Faber & Visscher, 2016). This quasi-experiment also showed that the top 20% students in math performance benefitted the most. In contrast, significant effects on reading performance were not found. The effects of adaptive practice systems are varied and it seems that the effects differ across subject domains.
1.2 Methodological considerations in estimating causal relations between ICT and learning Various studies have examined the general impact of educational games and applications of gamification on motivation and learning performance. A systematic literature review on empirical studies suggested that the benefits of educational games are mostly linked to cognitive (knowledge acquisition and content understanding) and motivational outcomes (Connolly, Boyle, MacArthur, Hainey, & Boyle, 2012). A meta-analysis on (quasi-) experiments, which studied the effects of educational games on learning, found a medium effect with respect to learning (ES. = +0.29) and retention (ES. = +0.36) (Wouters, Van Nimwegen, Van Oostendorp, & Van der Spek, 2013). The results also showed a medium learning effect (ES. = +0.30) for students under 12 years. But remarkably, no effects of
educational games on learning were found when only studies with a randomized control group were examined. The effects of educational games and gamification on educational outcomes seem to be depend on the research design.
That studies with a randomized control group show different results than studies without is not typical for these kinds of ICT.In general, consistent and convincing evidence of an impact of ICT on students’ performance is hardly found in randomized experiments (De Witte, Haelermans, & Rogge, 2015; Biagi & Loi, 2013; Livingstone, 2012). Various studies demonstrated that studies with comparable intervention and control groups, such as
randomized controlled trials, showed lower effect sizes than studies with less comparable intervention and control groups, such as non-randomized experiments (Cheung & Slavin, 2013; Wouters et al., 2013). Different effects were furthermore found in randomized studies on the impact of ICT in educational settings. Studies showed a positive effect (Cheung & Slavin, 2013; Li & Ma, 2010), no effect (Dynarski et al., 2007), or even a negative effect
6 (Campuzano, Dynarski, Agodini, & Rall, 2009). The impact of ICT on learning performance is therefore unclear and questionable (Biagi & Loi, 2013; Campuzano et al., 2009).
The decreased effect sizes and the mixed results in randomized studies, as described above, are to some extent due to the complexity of the relationship between ICT and learning performance. Estimating a causal relation between the use of educational technology and changes in learning performance is difficult (De Witte et al., 2015; Biagi & Loi, 2013). The main reason is that schools that used these tools cannot be directly compared with schools that did not use these tools. For instance, differences in performance gains might be caused by differences in the characteristics or perspectives of the school, instead of the use of ICT. The use of randomized controlled trials is the golden standard to minimize bias in outcome relating to differences between intervention and comparison groups in unmeasured
characteristics (Ginsburg & Smith, 2016). In order to investigate the complex causal relation between practice systems and learning gains, randomized control groups are therefore preferred.
However, randomized control groups are not always an option in studying the impact of ICT on learning performance. The difference-in-differences (DiD) approach could be used as an alternative to randomized controlled trials in order to obtain a comparable control group (e.g. Schlotter et al., 2011; Angrist & Pischke, 2014). In short, learning gains of a group which used ICT are compared with the learning gains of a control group which have not used ICT yet. The specification of this approach will be discussed in the methods section (see section 2.4). This econometric technique could be seen as a quasi-experiment with a multiple baseline, but is not often applied to studies on ICT in education. With the DiD approach, a control group is therefore as similar as possible for a non-randomized experimental study.
Although studies with the DiD approach lack the preferred randomized control group, one major benefit arises that is hardly possible in randomized controlled trials. In
experimental studies, participants may behave differently than they would usually do, for instance because of the Hawthorne effect (Kocakaya, 2011) and the John Henry effect (Adair, 1984). The former indicates a type of reactivity in which participants modify or improve their behavior in response to their awareness of being observed and the latter is a reactive behavior by the control group that may actively work harder to overcome the disadvantage of being in a control group. These biases are avoided in the current study, since observational data collected from school administration systems were used. Studies with the DiD approach are therefore not positioned as an experiment but rather as an investigation of a naturalistic setting.
7 1.3 The current study
In order to explore the effect of digital practicing systems, the current study assessed a large practicing system: Math Garden (Van der Maas et al., 2010). Almost 1,600 Dutch schools have a license for this practice system and more than 100,000 Dutch users are currently active. Math Garden is an example of a practicing system with gamification, since this tool contains several aspects aiming to enhance learning such as adaptability, progress reports, and trial and error. Meijer and Karssen (2014) examined the effect of Math Garden on math performance and indicated that students showed greater math gains compared to the non-randomized control group. These findings were unfortunately based on students of only three different schools and on specific tests for rote learning and automatizing basic math skills. Convincing effects of Math Garden on the standardized tests for math performance in general were not found.
The current study aimed to find the potential impact of Math Garden on math performance using more schools (more power) and a comparable control group. Since historical data were used, the participants could not be randomly assigned to conditions. The DiD approach was therefore used to examine the effects of Math Garden. The intervention groups consisted of schools that introduced Math Garden at the start of the 2012/2013 school year. The control groups consisted of schools that introduced Math Garden one year later and had not used Math Garden yet in the 2012/2013 school year. The impact of Math Garden was assessed by estimating the difference in math development of these two groups over the 2012/2013 school year. The gains in math of the first-mentioned group were expected to be bigger due to the practice with Math Garden.
The DiD approach was not only used to assess the main effect of practicing with Math Garden, but also to examine three kinds of differential effects. First, the current study
examined the effect of Math Garden on the math performance of students of different ages. Students of 7-9 year olds were expected to benefit most from Math Garden, since they are in the middle of learning basic math skills (Van den Heuvel-Panhuizen, Buys, & Treffers, 2001). Second, the current study examined whether many and/or long practice sessions influence learning gains. The students of the groups that practiced intensively and regularly with Math Garden were expected to have a more substantial performance gain than similar-aged groups that practiced occasionally. Third, the current study examined whether students with high or low math ability benefit from Math Garden. Faber and Visscher (2016) found that students who belong to the top 20% with respect to math performance benefitted the most from practicing with a practice system. However, another study found that students with low math
8 skills benefitted the most (Haelermans, Ghysels, Stals, & Weeda, 2013). This means there was a dual expectation, since the former students could gain more resulting from practice at their own level and the later students had more to gain compared to other students.
Methods
In this study, Math Garden was examined as an intervention for practicing and enhancing math performance in primary education (see section 2.2). Test of Cito (Janssen, Verhelst, Engelen, Scheltens, 2010) were used in order to measure math performance (see section 2.3). Data of Math Garden had to be combined with the student achievement data of Cito. The impact of Math Garden on math performance was estimated with the difference-in-differences research design (see section 2.4).
2.1 Sample
Data from 30 Dutch primary schools were used. The flowchart in Appendix A summarizes the selection process. A total of 82 schools were contacted that 1) introduced Math Garden in 2012 or 2013 in the months ranging from August to October, 2) used Math Garden for at least a full school year, and 3) offered regular Dutch education. These criteria were used in order to obtain similar starting points and a comparable duration of the intervention across schools. A total of 30 schools gave permission to use their data in this study. These schools were also asked whether they used a digital practice system before the introduction: 18 schools responded positively.
Math Garden and Cito separately provided the data of the selected schools. Since it was not feasible to connect data at the student-level due to privacy regulations, the data were aggregated at the group-level. Data of Math Garden contained inconsistent grade divisions. The groups were therefore defined by the age of students for each data sets. Those who were born from October till September of a particular school year were placed in the same group, similar to the grade division of the Dutch Ministry of Education, Culture and Science. Grade retention therefore had no effect on the composition of age groups. A sufficient correlation of .95 was found in the Cito data between the groups of a student defined by age and defined by grade. As a result, the definition of which student is in which group was close to the actual grouping and similar in Math Garden data and Cito data.
All age groups that represented grade 2 (7 years old) to grade 5 (10 years old) in 2013 were included in this study. However, not all groups of the schools that gave permission could be used in the analyses: there were additional inclusion criteria for the age groups. In order to
9 have a representative mean of the test scores, one of the inclusion criteria for test scores stated that at least 60% of the age group participated in the math test. The score was marked as missing if less than 60% of the students participated in the test. Moreover, not all groups had test scores available for the same time span, because students in the lower grades were too young at some of the earlier measurement moments. Table 1 shows an overview of how many test scores were available for which groups and in which school year.
The aggregated scores of 54 groups were included in the analyses. The sample consisted of 30 intervention groups that were introduced to Math Garden at the start of the 2012/2013 school year and 24 control groups that were introduced to Math Garden one year later. This data set comprised 18 schools and represented over 1,500 students. Table 2 shows the age group descriptive statistics. Public school data of the Dutch Ministry of Education, Culture and Science were not only used to assess the group size and gender composition, but also to estimate the proportion of students with a weight in an age group, indicating the share of students with a disadvantaged background.
2.2 Math intervention
The practice system Math Garden was used as an intervention for practicing and enhancing math ability (Van der Maas et al, 2010). Central in this web-based system was the users’ personal garden, see left side of Figure 1. The garden contained several flower-beds that corresponded to separate operations sections where users could practice, for example addition, subtraction, multiplication, and division. A single practice session consisted of ten to fifteen exercises presented one by one, see right side of Figure 1. The computer screen showed a problem, response options, a question mark as a wild card, a green progress bar, and a row of coins which indicated the time left (max. 20 seconds). The student won or lost virtual coins for every second that was left if their answer is respectively correct or incorrect. Students were motivated by two reward systems, which are examples of gamification. The first type of
Table 1.
Overview of availability of math test scores for each age group.
School year N Age 2008/2009 2009/2010 2010/2011 2011/2012 2012/2013 Total Represented grade of age group in 2012/2013 Grade 2 16 7-8 - - - 30 32 62 Grade 3 15 8-9 - - 24 28 30 82 Grade 4 13 9-10 - 22 22 24 26 94 Grade 5 10 10-11 16 16 16 20 19 87 Total 54 16 38 62 102 107 325
10
Table 2.
Descriptive statistics of intervention groups (started Math Garden in 2012) and control groups (started Math Garden in 2013)
Represented grade of age group in 2012/2013
2 3 4 5 All age groups
Start Math Garden 2012 2013 2012 2013 2012 2013 2012 2013 2012 2013 pa
N groups 8 8 9 6 7 6 6 4 30 24 Group size 28.00 24.25 30.67 23.17 29.29 24.33 34.67 28.25 30.43 24.67 .15 (13.91) (14.23) (16.22) (16.36) (14.28) (11.81) (15.24) (21.53) (14.40) (14.61) Gender (% Boy) 42% 48% 45% 42% 43% 35% 39% 45% 42% 43% .80 % Students with weightb 7% 4% 4% 5% 8% 6% 8% 9% 6% 6% .69
Average math test scores of the groups in the indicated school year
2011/2012 40.58 43.43 58.05 60.81 75.78 74.92 87.68 85.35 63.46 63.05 .91 (8.75) (8.90) (7.17) (8.27) (5.76) (6.28) (6.07) (5.46) (18.83) (17.83) 2012/2013 60.50 61.67 75.89 75.63 88.22 88.23 100.86 98.32 79.29 77.91 .64
(6.98) (7.06) (5.80) (5.58) (5.33) (4.49) (6.96) (3.90) (15.67) (14.81) Denomination (ABZ = Neutral; OPB = Public; PC = Protestant; RK = Catholic)
ABZ 0 0 1 0 0 1 1 0 2 1
OPB 4 5 4 3 4 2 2 2 14 12
PC 1 0 1 0 1 0 1 0 4 0
RK 3 3 3 3 2 3 2 2 10 11
Note. Data were aggregated at age group level. Standard deviation in parentheses. aIndicates the difference between all
intervention and all control groups. bIndicates the percentage of students with a disadvantaged background.
was described above, i.e. good performance was rewarded by virtual coins for virtual prizes and growing flowers. The students were also motivated to continue practicing with Math Garden because if they failed to do so their flowerbeds would wither.
This system can adjust math problems and exercises to the level of the student
(Klinkenberg, Straatemeier, & Van der Maas, 2011). The selection of problems was regulated by an adaptive algorithm and depended on the estimated ability of the student and estimated problem difficulties (Maris & Van der Maas, 2012). So the students did not have to do math exercises that are too easy or too difficult (Jansen et al., 2013). Research on Math Garden
11 showed that math scores of Math Garden correlate strongly with student achievement on standardized tests (r ≈ .80; Van der Maas, Klinkenberg, & Straatemeier, 2010). Strong correlations were also found between the separate kinds of mathematical operations such as addition and subtraction, indicating sufficient reliability.
Data on Math Garden were used to estimate different practice profiles relating to the use of Math Garden by the age groups. The dataset consists of administrations of each
exercise the students made, and includes information on time stamps, response time, and type of exercise. The average time and frequency a student practiced with Math Garden each week was computed with this information. Table 3 shows the descriptive statistics of the use of Math Garden by the age groups.
Table 3.
Descriptive statistics of Math Garden for both intervention (started Math Garden in 2012/2013) and control groups (started Math Garden in 2013/2014)
Represented grade of age group in 2012/2013
2 3 4 5 All groups
Start Math Garden 2012 2013 2012 2013 2012 2013 2012 2013 2012 2013 pa
N groups 8 8 9 6 7 6 6 4 30 24
Group size 28.00 24.25 30.67 23.17 29.29 24.33 34.67 28.25 30.43 24.67 .15 (13.91) (14.23) (16.22) (16.36) (14.28) (11.81) (15.24) (21.53) (14.40) (14.61) Number of students which practices with Math Garden at least once
Active users 16.63 14.50 19.78 17.83 17.71 14.00 20.17 17.75 18.53 15.75 (15.34) (11.28) (13.52) (19.50) (14.60) (12.13) (12.25) (14.73) (13.40) (13.59) % of total 54% 61% 70% 82% 59% 56% 63% 62% 62% 65% .70 Exercises made by students inside (09:00-12:00 and 13:00-15:30) or outside school during one school year
Inside 234 274 290 320 213 214 253 205 250 259 .90 (299) (361) (301) (323) (214) (260) (216) (243) (255) (295) Outside 171 82 115 39 70 43 100 33 117 53 .03
(187) (98) (103) (31) (93) (41) (150) (25) (136) (64) Descriptive statistics of practice sessions with Math Garden (students’ averages)
Practice frequency 7.15 6.48 6.79 5.43 5.68 5.20 6.53 4.45 6.55 5.56 .33 (3.96) (2.76) (3.91) (3.08) (2.88) (3.30) (4.15) (3.36) (3.59) (2.96) Practice time 8.05 7.42 8.00 7.63 8.87 7.62 8.61 7.48 8.35 7.53 .26
(1.58) (0.87) (2.37) (1.48) (2.59) (1.27) (2.64) (1.78) (2.23) (1.21) Practice profiles (see section 2.2)
# Freq. Time
1. low low 2 2 3 0 2 2 3 1 10 5
2. low high 1 1 1 3 2 2 1 2 5 8
3. high low 1 0 2 2 0 0 0 0 3 2
4. high high 4 5 3 1 3 2 2 1 12 9
Note. Data were aggregated at group level. Standard deviation in parentheses. aIndicates the difference between all
12 In order to investigate different practice volumes, all groups were categorized into four different practice profiles. In order to compare the first year of Math Garden usage by both groups, these profiles were estimated in the 2012/2013 school year for the intervention groups and in the 2013/2014 school year for the control groups. The practice profiles were based on 30 non-holiday school weeks. Each group was classified into either below or above the average frequency of practice sessions (students practiced 6.1 of the 30 weeks on average) and the average duration of practice sessions (students practiced 8.0 minutes each session on average). This resulted into four different practice profiles:
1) Below average practice frequency and below average practice time; 2) Below average practice frequency and above average practice time; 3) Above average practice frequency and below average practice time; 4) Above average practice frequency and above average practice time.
2.3 Math performance
Math performance was measured by the Arithmetic and Math test of a pupil monitoring and evaluation system (Janssen et al., 2010). This math test contained both open-ended and forced-choice items and context math exercises in order to measure if students can apply their skills in practice. Material components included numbers, mental calculation, and more complex applications. A comprehensive study by Cito (Janssen et al., 2010) showed
satisfactory reliability scores across groups, ranging from 0.92 to 0.97. Students made this test twice a year: in the middle (January) and at the end of the school year (June).
Cito included a norm-referenced scale score of the tests (hereinafter referred to test score) based on all Dutch schools that used this math test (Janssen et al., 2010). The test scores across different grades could be compared and aggregated as a result. The aggregated test scores of Cito data consisted of the mean or percentile (10th, 25th, 50th, 75th, and 90th) of all scores of one age group at a particular test moment. Test scores ranged from 0 to 120. Cito data up to four years before the introduction of Math Garden at the school were used (see Table 1) in order to establish the development of math performance prior to the intervention.
2.4 Research design and statistical analyses
The difference-in-differences (DiD) approach (e.g. Schlotter et al., 2011 and Angrist & Pischke, 2014) was used to analyze the impact of practicing with Math Garden on math performance in the 2012/2013 school year. The effect of Math Garden was studied with this research design by comparing the average change in the test score of the intervention groups
13 with the average change over time of the control groups. The intervention and control groups consisted of age groups that started practicing with Math Garden at their school in the
2012/2013 and 2013/2014 school years respectively.
The general regression estimate of DiD was used in the analyses: 𝑌𝑔𝑡 = 𝛽0+ 𝜆𝑔+𝛾𝑡+ 𝛽𝑋𝑔𝑡′ + 𝛽
1(𝐼𝑛𝑡𝑒𝑟𝑔) + 𝛽2(𝑃𝑜𝑠𝑡) + 𝛿(𝐼𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛𝑔× 𝑃𝑜𝑠𝑡) + 𝜖𝑔𝑡
In this formula, 𝑌𝑔𝑡 indicates the aggregated math test score of age group g at time t. 𝛽0, 𝜆𝑔, γ𝑡, and 𝑋𝑔𝑡′ respectively indicate the intercept, group fixed effects, time fixed effects, and a
vector of control variables (group size, gender, and student weight). 𝐼𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛𝑔 indicates whether the age group was an intervention group and 𝑃𝑜𝑠𝑡 indicates whether the moment of test administration was in the 2012/2013 school year. The coefficient 𝛿, which measured the interaction of 𝐼𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛𝑔 with 𝑃𝑜𝑠𝑡, was of particular interest. This coefficient indicates whether the test score belonged to an intervention group and if the test took place in the 2012/2013 school year. This enabled capturing the additional gain in math performance of the intervention group compared to the control group. Finally, 𝜖𝑔𝑡 is the error term, which is assumed to be independent from 𝐼𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛𝑔 and 𝑃𝑜𝑠𝑡. In words, the DiD model aimed to investigate the average gains in math performance of groups that started using Math Garden in 2012/2013 (intervention groups) compared to the gain in math performance of groups that introduced Math Garden one year later (control groups). See Figure 2 for a graphical
explanation of a DiD model.
Figure 2. Difference-in-difference estimation, graphical explanation1.
In addition, the DiD model needed to be corrected for correlated groups, because test scores were nested within age groups and within schools. Both hierarchical linear models (e.g. Kreft & De Leeuw, 1998) and clustered standard errors were used in the analyses. The
14 corrections were made and compared at group level, at school level and hierarchical at both levels. The results were expected to be robust for correlated groups based on these
assumptions. The regression analyses of the DiD model was performed with R (R Core Team, 2013) which includes ‘nlme’ (Version 3.1-127; Pinheiro, Bates, DebRoy, Sarkar, & R Core Team, 2016) and ‘rddtools’ packages (Version 0.4.0; Stigler & Quast, 2015).
Analyses of DiD models also required the parallel trend assumption, i.e. the
development in math scores of intervention and control groups should be similar before the intervention. Similar to Dynarski (2003), Angrist and Pischke (2014), and Oosterbeek, Van Praag, & IJsselstein (2010), the assumption was checked firstly by comparing the descriptive statistics and trends of the intervention and control groups. Second, various placebo DiD effects were examined for each point in time before the intervention (e.g. Slusky, 2015 and Bertrand, Duflo, & Mullainathan, 2004). Only at the moment of intervention and not at earlier points in time should DiD effects be visible in DiD analyses, because the development of both intervention and control groups should be similar when the intervention has not started yet. The following model was tested, which includes an interaction of the intervention variable with eight time dummies:
𝑦𝑔𝑡 = 𝛽0+ 𝜆𝑔 + γ𝑡+ 𝛽𝑋𝑔𝑡′ + 𝛿
−7𝐷𝑔𝑡 + 𝛿−6𝐷𝑔𝑡+ ⋯ + 𝛿−2𝐷𝑔𝑡+ 𝛿−1𝐷𝑔𝑡+ 𝜖𝑔𝑡
In the above formula, 𝑦𝑔𝑡 is the math score for age group g at time t. 𝛽0, 𝜆𝑔, γ𝑡, and 𝑋𝑔𝑡′ are
respectively the intercept, group fixed effects, time fixed effects, and a vector of control variables (group size, gender, and student weight). Interaction term 𝐷𝑔𝑡 indicates whether the
test score belonged to an intervention group and at which particular point in time the test took place. The interaction 𝛿−8𝐷𝑔𝑡, which indicates the placebo test of the 2008/2009 school year
in January, was omitted and served as point of reference.
Results 3.1 Preliminary results
Analyses of the difference-in-difference (DiD) model require the parallel trend assumption, i.e. the development in math scores of both groups should be similar before the intervention. Two steps were taken in order to check the parallel trend assumption: comparing descriptive statistics and trends, and placebo DiD effects. Table 2 shows all descriptive statistics of the groups and the simple t-tests indicated that the intervention and control groups were similar in the 2012/2013 school year. Table 3 shows the first year of use of Math Garden and the simple t-tests indicated that the intervention and control groups were similar with respect to their
15 usage. They differed only in the number of exercises made outside school, which was two times larger for the intervention groups. Furthermore, Figure 3 shows the development in math scores of both intervention and control groups. The visual inspection of this figure suggested that the trend before intervention (white area) was similar for both groups. Finally, results of the placebo DiD regression did not show significant interactions (see Appendix B) and confirmed the visual inspection that the trend of the intervention and control group were similar. Since the math development of both groups were similar in pre-intervention period, the trend of both groups was therefore assumed to be parallel.
3.2 Results of the main difference-in-differences analysis
The main interest of this study was the impact of Math Garden on math performance. The math gains of the intervention and control groups were compared after the moment the intervention groups started to use Math Garden (September 2012). The math development of all intervention and all control groups are shown in Figure 3. A visual inspection suggested a slightly greater math development of the intervention group compared to the control group in the 2012/2013 school year. However, Table 4 shows that for all different specifications of the DiD model the coefficient δ did not significantly differ from zero. The learning gains of math were therefore similar for both intervention and control groups.
Table 4.
Results of the main difference-in-differences model analysis with different specifications (54 groups). No correction for clustered data Clustered standard error at1 Multilevel with random intercept at1 Without covariates With covariates1,2 Group level School level Group levela School levelb Both levelsc Interaction effect of Post & Intervention
δ 1.98 1.61 1.61 1.61 2.87 4.56 2.87 (S.E.) (4.07) (4.06) (2.33) (1.70) (3.53) (4.21) (3.53) p .63 .69 .49 .34 .42 .28 .42 Main effect Intervention 𝛽1 -0.59 -0.31 -0.31 -0.31 -1.22 -2.65 -1.22 (S.E.) (2.38) (2.42) (2.77) (1.75) (3.50) (2.91) (3.50) p .80 .90 .91 .86 .73 .37 .73 Main effect Post 𝛽2 20.72 21.31 21.31 21.31 23.66 15.97 23.66 (S.E.) (3.10) (3.10) (1.73) (1.54) (2.72) (3.23) (2.72) p .00 .00 .00 .00 .00 .00 .00
Note. Total of 325 observations of aggregated (mean) test scores was included in the analyses (see Table 1).
Standard errors are in parenthesis.
1Included the control variables students’ weight, group size, and gender. 2Proportion explained variance was .28
and F(6,318) = 20.83, p < .001.
aRandom effects: intercept 9.54 (14.03); bRandom effects: intercept 2.78 (17.24); cRandom effects: school level
16
Figure 3. Development in math scores for intervention and control groups of separate grades.
3.3 Results of analyses on the different grades
In order to find differential effects of Math Garden for different ages, the DiD model was tested for different age groups. Separate analyses were performed for different corresponding grades, from grade 2 (7 years old) up to grade 5 (10 years old). Figure 3 shows the
development in math scores for intervention and control groups of the different grades separately and Table 5 shows the results of the analyses. All four analyses of the DiD regression on the different age groups indicated that the coefficient δ did not significantly differ from zero. Math gains were therefore similar for the intervention and control groups across the different ages.
Table 5.
Difference-in-differences analyses for separate age groups. Represented grade in 2012/2013 Grade 2 Grade 3 Grade 4 Grade 5 Interaction effect of
Post & Intervention
δ 1.29 4.03 0.88 3.25 (S.E.) (1.77) (2.64) (2.22) (4.01) p .47 .13 .69 .42 Main effect Intervention 𝛽1 -1.15 -2.72 -1.35 -.41 (S.E.) (1.94) (3.01) (1.90) (3.25) p .56 .37 .48 .90
Main effect Post
𝛽2 18.35 22.08 29.18 31.62 (S.E.) (1.22) (2.07) (1.91) (2.99)
p .00 .00 .00 .00
Intervention groups 8 9 7 6
Control groups 8 6 6 4
Note. Total of 325 observations of aggregated (mean) test scores was included
in the analyses (see Table 1). Standard errors are in parenthesis. Controlled for students’ weight, group size, and gender. Clustered standard errors at school level were used.
17 3.4 Results of analyses on the different practice profiles of Math Garden
In order to find differential effects of Math Garden for groups with different Math Gardens practice profiles, the DiD model was tested for the four different profiles. These profiles are described in section 2.2. Groups varied in the average frequency and in the average time that the group members have practiced with Math Garden. The intervention groups with a
particular practice profile in the 2012/2013 school year were compared with control groups of the same practice profile in the 2013/2014 school year. So separate analyses were held for intervention and control groups with the different practice profiles.
The development in math scores for intervention and control groups are shown in Figure 4. Table 6 shows the results of the analyses. Results of the analysis of groups that practiced above average (Profile 4) suggested that the coefficient δ differed marginally from zero (p = .06). Intervention groups with Profile 4 gained 5.46 points (S.E. = 2.87) on their math score by practicing with Math Garden compared to equivalent control groups. Results of the analyses of groups with the other 3 profiles showed that the coefficient δ did not
significantly differ from zero for any of these profiles.
Table 6.
Difference-in-differences analyses for different Math Garden profiles. Profile 1 Profile 2 Profile 3 Profile 4 Practice frequency low low high high Practice time of a session low high low high Interaction effect of
Post & Intervention
δ 0.22 2.90 -9.27 5.46 (S.E.) (5.34) (4.53) (18.96) (2.87) p .97 .53 .63 .06 Main effect Intervention 𝛽1 9.57 -.37 -27.84 2.62 (S.E.) (3.00) (5.67) (84.58) (2.83) p .00 .95 .75 .36
Main effect Post
𝛽2 23.35 23.53 28.64 17.60 (S.E.) (5.51) (2.89) (9.37) (2.42)
p .00 .00 .01 .00
Intervention groups 10 5 2 12
Control groups 5 8 2 9
Note. Total of 325 observations of aggregated (mean) test scores was included in
the analyses (see Table 1). Standard errors are in parenthesis. Controlled for students’ weight, group size, and gender. Clustered standard errors at school level were used.
18
Figure 4. Math gains of intervention and control groups with different practice profiles (see section 2.2)
3.5 Results of analyses with different percentile math scores
In order to find differential effects of Math Garden on students with a different math ability, the DiD model was tested with different percentile scores as outcome variable. Separate analyses were held with the 10th, 25th, 50th, 75th and 90th percentile scores. Several percentile scores were compared, such as the bottom 10% or the top 25% students of the intervention and control groups. A visual inspection of the different developments in math scores for intervention and control groups (see Figure 5) suggested that the intervention and control groups differed more in math gains over the 2012/2013 school year when a higher percentile was used. Table 7 shows the actual results of the analyses. The analyses with the different outcome variables indicated that that no coefficient δ differed significantly from zero. In words, the math development of the intervention groups was similar as the control groups, regardless of which percentile score was used.
Table 7.
Difference in differences analyses for different percentile scores (54 groups). Percentile
10th 25th 50th 75th 90th
Interaction effect of Post & Intervention
δ .87 -.38 2.32 1.86 1.77 (S.E.) (1.96) (1.70) (1.59) (2.03) (2.00) p .66 .82 .15 .36 .38 Main effect Intervention 𝛽1 -1.30 .21 .06 .39 .21 (S.E.) (2.39) (1.95) (1.84) (1.89) (1.90) p .59 .92 .97 .83 .91
Main effect Post
𝛽2 22.56 23.58 21.77 21.35 19.32 (S.E.) (1.65) (1.59) (1.48) (1.72) (1.46) p .00 .00 .00 .00 .00
Note. Total of 325 observations of aggregated (percentile) test scores was included in
the analyses (see Table 1). Standard errors are in parenthesis. Controlled for students’ weight, group size, and gender. Clustered standard errors at school level were used.
19
Figure 5. Development in math scores for intervention and control groups according to different percentiles.
Discussion
The current study used the difference-in-differences approach to investigate the impact of Math Garden on math performance. The results indicated that the math development of the intervention groups after the introduction of Math Garden was similar to the math
development of control groups that started to use Math Garden one year later. There were no indications for differential effects for different age groups (grade 4 to grade 7) and for students with either high or low math abilities. Altogether, this study did not find evidence that indicates practicing with Math Garden enhances math performance in primary education. Two explanations for Math Garden having no impact will be discussed: 1) the impact of Math Garden was underestimated and 2) Math Garden was not an appropriate system to enhance math performance.
An indication that the impact of Math Garden was underestimated is the lack of power. If Math Garden had any effect on math performance, it would be a small one. The power of .30 (see Appendix C) was not sufficient enough to find a small effect. A condition that undermined the power was the small number of age groups. Although data of over 1,500 students were used – which is reasonable for small effects – the data had to be aggregated at group level due to privacy regulations. Even though student level data were obviously
preferred, a combination of both data sets at student level was not possible. Active consent of each student had to be obtained due to the terms of use of Math Garden and Cito, which
20 unfortunately was not feasible for this study. In order to find small effects adequately with aggregated data and the current model, more than 200 groups are needed (see Appendix C).
A second indication is that the current study did not include information on how the groups practiced with Math Garden. Although this study included Math Garden data from actual class settings, additional information on how teachers and schools deployed Math Garden was not obtained. De Witte et al. (2015) indicated that teachers determine if and how ICTs are used in the classroom. Practice systems could be used in educational settings in various ways, for instance as a tool to practice subject matter or as a reward to play on the computer or tablet. It could furthermore be used by different kinds of students, such as students with a low (math) ability. A different use of these systems might result in different impacts on learning. It is important that such information is included in a study on such a tool.
A third indication is that the current study could only investigate the impact of Math Garden of the first year, because that is when the control groups started to practice with it. The effect of Math Garden on math performance could consequently be underestimated, since teachers had to get used to this practice system. However, the effect of Math Garden could also be overestimated, since teachers were enthusiastic about this system in the first year, although this enthusiasm did fade over time. If, to what extent and which of these two conditions were applicable in the current study was not investigated.
A fourth indication is that 60% of the schools have used other practice systems before the introduction of Math Garden. Even though Math Garden had small effect on the math performance, this impact would not be found with the current approach if the former practice system had a similar or greater impact. This would lead to the conclusion that the math development would then be similar before and after the introduction, regardless of the positive impact of Math Garden. The impact of Math Garden might consequently be underestimated.
The last indication is that students practiced insufficiently with Math Garden in order to have an effect on their math performance. The current study indicated that the intervention groups that had practiced above average had a slight increase (p = .06) in math performance compared to equivalent control groups. Chueng and Slavin (2013) indicated that practice systems that were used at least 30 minutes a week have a bigger impact than those used less than 30 minutes a week. Math Garden was used less intensively: 4.5 minutes a week
maximally2. This might be an indication of a threshold, implying that students had to practice
2 Optimistically, students made at average 375 exercises over 30 weeks inside and outside school (see Table 1),
which each took 20 seconds maximum. The students practiced thus 375 ∙ 20 ∙ 1 60∙
1
21 with a minimum frequency or for a minimum period of time before Math Garden has an impact on math performance. Unfortunately, both Faber and Visscher (2016) and Meijer and Karssen (2014) did not mention the practice intensity that would allow comparison with the current study and with Cheung and Slavin (2013).
The other explanation of the absence of an effect is that Math Garden was not an appropriate practice system to enhance math performance with. Cox and Marshall (2007) indicated that the contribution of ICT to students’ learning was dependent on the type of ICT and the particular subject. Even though Math Garden has the ability to adjust the math problems and exercises to the level of the student, it can be seen as a drill and practice application (Selwyn, 2011). Math Garden does not offer other feedback than whether the exercise was correctly done. In-depth explanations of how to do the exercise or what aspect in particular was incorrect were not provided. Math Garden might therefore be a simple tool that has no impact on math learning, other than providing practice time.
Furthermore, Li and Ma (2010) suggested that the used learning method affects the effect of technology on math development. Math gains are greater in a constructivist
approach, i.e. student-centered instruction and emphasizes learning strategies, compared to a traditional approach, i.e. teacher-centered and emphasizes whole-class instruction (Li & Ma, 2010). Whether Math Garden represented the more favorable constructivist approach is questionable, since the instructions and feedback within this system are not extensive. A follow-up study could include a practice system that does possess additional and adaptive feedback on and explanation of the exercises.
An advantage of the current study is the use of the difference-in-differences approach. The intervention groups were compared to an arguably similar control group. A comparable control group is important in finding effects of ICT in education, since (pre-existing)
differences between the groups might cause the effect. Another advantage of this approach is that students, teachers, and schools were not aware of being a subject of a study. Participants of the current study were therefore not affected by biases such as the Hawthorne effect and John Henry effect, which indicate that participants behave differently when they are aware that they participate in a study.
Further studies on the effects of digital practice systems on math performance in primary education should include more groups or measurements at student level, information of context factors, several school years, and the effects of comprehensive feedback and explanation within the game, as suggested above. In order to establish the impact of a digital practice systems with more certainty, a study could also include a third group that stops
22 practicing after a certain period of time. If an effect of the practice system was found after the moment of introduction, it would flat out as a kind of reverse DiD effect. If the increased growth in math decreases after this point, conclusions of the effects of digital practice systems on learning are more convincing.
Further studies should also include tests that measure specific math skills. As suggested above, Math Garden was a practice system that did not enhanced math performance. Meijer and Karssen (2014) stated that Math Garden had an impact on rote learning and automatizing basic math skills. In general, such adaptive practice systems are mainly focused on automatizing basic math skills and less on enhancing math comprehension and understanding, as suggested by Heemskerk et al. (2011). However, the Cito math tests (Janssen et al., 2010) were mainly aimed to measure math comprehension and understanding. Other math tests, such as measurements for rote learning and automatizing basic math skills, could be used in further studies as an addition to math tests that measure math performance in general.
A final suggestion for further studies is an investigation into the impact of digital practice systems, in particular those including gamification, on motivation. The motivation of students is important, since motivation produces learning (Ryan & Deci, 2000). It would therefore be interesting to include measurements on motivational outcomes, since the major aim of educational games is to motivate the user. Are students more motivated to learn when they use digital practice systems? Whether students are intrinsic or extrinsic motivated is another interesting question, since Ryan and Deci (2000) stated that extrinsic motivation undermine the more valuable intrinsic motivation. A study that investigates the intrinsic motivation to do math, while students play on a practice system, might add interesting
findings to the study of Ronimus et al. (2014). They found that their practice system enhanced engagement but not the intrinsic motivation to learn. Digital practice systems are perhaps the tool to enhance motivation to learn math.
Altogether, the current study did not find any evidence indicating that practicing with Math Garden enhances math performance in primary education. Even though several
indications were discussed that suggest that the impact of Math Garden was underestimated, it is more likely that Math Garden were not able to have a significant impact on the math
development of students in primary education. Additional research is required in order to qualify or disqualify digital practice systems as an effective tool in enhancing math
performance and in order to specify what kind practice systems are most effective, including those with gamification. As shown in the current study, the difference-and-differences is an
23 approach that could be used when randomized controlled trials are not an option, but
comparable control groups are desired. After all, when digital practice systems do not appear to be more effective than regular methods, they might still be valuable if they enable students to engage with learning.
Appendix A Summarize of the selection process in the current study.
24 Appendix B
Table B.
Analysis the placebo difference-in-difference effects (54 groups). School year Month δ (S.E.) p
Placebo interaction effect of time by intervention 2008/2009 January - - - October -1.83 (16.21) .91 2009/2010 January -2.99 (13.83) .83 October -3.24 (13.83) .81 2010/2011 January -0.82 (12.95) .95 October -4.29 (12.95) .74 2011/2012 January -2.12 (12.30) .86 October -2.60 (12.30) .83 𝛽1 S.E. p
Main effect intervention 2.77 11.46 .81 School year Month 𝛽2 S.E. p
Main effect time
2008/2009 January - - - October 11.08 (12.82) .39 2009/2010 January 9.00 (11.10) .42 October 19.82 (11.10) .08 2010/2011 January 16.97 (10.33) .10 October 30.12 (10.33) .00 2011/2012 January 25.49 (9.69) .01 October 34.84 (9.69) .00 𝛽0 S.E. p Intercept 32.88 9.06 .00
Note. Total of 218 observations of aggregated (mean) test scores were included in the analyses (see Table 1). See section 1.4 for the used model. Placebo interaction effect of school year 2008/2009 month January was omitted and served as point of reference. Clustered standard errors at school level are in parenthesis. Controlled for students’ weight, group size, and gender.
Appendix C
The power vs. number of groups figure was computed with Optimal Design.
25 References
Adair, J. G. (1984). The Hawthorne effect: A reconsideration of the methodological artefact. Journal of Applied Psychology, 69(2), 334-345. doi:10.1037/0021-9010.69.2.334 Angrist J. D., & Pischke J. S. (2014). Chapter 5. Difference-in-Differences. In J. D. Angrist &
J. S. Pischke (eds.), Mastering Metrics. The path from cause to effect (pp. 178-208). Princeton, NJ: Princeton University Press.
Bertrand, M., Duflo, E., & Mullainathan, S. (2004). How much should we trust Differences-in-Differences estimates? The Quarterly Journal of Economics, 119(1), 249-275. doi:10.1162/003355304772839588
Biagi, F., & Loi, M. (2013). Measuring ICT use and learning outcomes: Evidence from recent econometric studies. European Journal of Education, 48(1), 28-42.
doi:10.1111/ejed.12016
Blok, H. (2004). Adaptief onderwijs: Betekenis en dffectiviteit [Adaptive education: Meaning and effectiveness]. Pedagogische Studiën, 81, 5-27.
Boyle, E. A., Connolly, T. M., Hainey, T., & Boyle, J. M. (2012). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28(3), 771-780. doi:10.1016/j.chb.2011.11.020
Boyle, E. A., Connolly, T. M., & Hainey, T. (2011). The role of psychology in understanding the impact of computer games. Entertainment Computing, 2(2), 69-74.
doi:10.1016/j.entcom.2010.12.002
Campuzano, L., Dynarski, M., Agodini, R., & Rall, K. (2009). Effectiveneses of reading and mathematics software producs: Findings from two student cohort. Institute of
Education Sciences, National Center for Educaiton Evaluation and Regional Assistence, Washington, DC.
Cheung, A., & Slavin, R. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis.
Educational Research Review, 9, 88-113. doi:10.1016/j.edurev.2013.01.001 Janssen, J., Verhelst, N., Engelen, R., Scheltens, F. (2010). Wetenschappelijke
verantwoording van de toetsen LOVS Rekenen-Wiskunde voor groep 3 tot en met 8 [Scientific justification of the LOVS arithmetic and mathematics tests for grade 3 to 8]. Arnhem: Cito. Retrieved from www.toetswijzer.nl/html/tg/14.pdf
Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., & Boyle, J. M. (2012). A systematic literature review of empirical evidence on computer games and serious games. Computers & Education, 59(2), 661-686. doi:10.1016/j.compedu.2012.03.004
26 Cox, M.J., & Marshall, G. (2007). Effects of ICT: Do we know what we should know?
Educational Information Technology, 12(2), 59-70. doi:10.1007/s10639-007-9032-x De Witte, K., Haelermans, C., & Rogge, N. (2015). The effectiveness of a computer-assisted
math learning program. Journal of Computer Assisted Learning, 31(4), 314-329. doi:10.1111/jcal.12090
Dynarski, S.M. (2003). Does aid matter? Measuring the effect of student aid on college attendance and completion. American Economic Review, 93(1), 279-288. doi:10.3386/w7422
Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., … Sussex, W. (2007). Effectiveneses of reading and mathematics software producs: Findings from the first student cohort. Institute of Education Sciences, National Center for Educaiton Evaluation and Regional Assistence, Washington, DC.
European Commission (2013). Survey of schools: ICT in education benchmarking access, use and attitudes to technology in Europe’s schools. doi:10.2759/94499
Faber, J. M., & Visscher, A.J. (2016). De effecten van Snappet. Effecten van een adaptief onderwijsplatform op leerresultaten en motivatie van leerlingen [The effects of Snappet. Effects of an adaptive education platform on learning outcomes and student motivation]. Zoetermeer: Kennisnet. Retrieved from www.kennisnet.nl/fileadmin/ kennisnet/leren_ict/leren_op_maat/bijlagen/De_effecten_van_Snappet_Universiteit_T wente.pdf
Fraillon, J., Ainley, J., Schulz, W., Friedman, T., & Gebhardt, E. (2014). Preparing for life in a digital age: The IEA international computer and information literacy study
international report. Cham: Springer. doi:10.1007/978-3-319-14222-7
Ginsburg, A., & Smith, M.S. (2016). Do randomized controlled trials meet the “Gold standard”? A study of the usefulness of RCTs in the What Works Clearinghouse. American Enterprise Institute
Haelermans, C., Ghysels, J., Stals, D., & Weeda, F. (2013). Het effect van online oefenen op rekenprestaties [The impact of online practice on math performance]. Onderwijs & Wetenschap, 98(4671), 650–635.
Heemskerk, I., Meijer, J., Van Eck, E., Volman, M., Karssen, M., & Kuiper, E. (2011). EXPOII: Experimenteren met ict in het PO tweede tranche: Onderzoeksrapportage [EXPOII: Experimenting with ict in primary education second round: Research rapport]. Amsterdam: Kohnstamm Instituut. Retrieved from
27 Johnson, L., Adams Becker, S., Estrada, V., Freeman, A., Kampylis, P., Vuorikari, R., and
Punie, Y. (2014). Horizon report Europe: 2014 schools edition. Luxembourg: Publications Office of the European Union, & Austin, Texas: The New Media Consortium. doi:10.2791/83258
Kennisnet (2010). Vier in balans monitor 2010 [Four in balance monitor 2010]. Zoetermeer: Kennisnet. Retrieved from archief.kennisnet.nl/fileadmin/contentelementen/kennisnet/ Over.kennisnet/Vier_in_balans/Vier_in_Balans_Monitor_2010.pdf
Kennisnet (2015). Vier in balans-monitor 2015 [Four in balance monitor 2015]. Zoetermeer: Kennisnet. Retrieved from www.kennisnet.nl/publicaties/vier-in-balans-monitor Klinkenberg, S., Straatemeier, M., & Van der Maas, H. (2011). Computer adaptive practice of
Maths ability using a new item response model for on the fly ability and difficulty estimation. Computers & Education, 57(2), 1813-1824.
doi:10.1016/j.compedu.2011.02.003
Kim, J. T., & Lee, W. (2013). Dynamical model and simulations for gamification of learning. International Journal of Multimedia and Ubiquitous Engineering, 8(4), 180-190. Kocakaya, S. (2011). An educational dilemma: Are educational experiments working?
Educational Research and Reviews, 6(1), 110-123.
Kreft, G.G. and De Leeuw, J. Introducing Multilevel Modeling. Sage Publications, London: 1998.
Jansen, B., Louwerse, J., Straatemeier, M., Van der Ven, S., Klinkenberg, S., & Van der Maas, H. (2013). The influence of experiencing success in math on math anxiety, perceived math competence, and math performance. Learning and Individual Differences, 24, 190-197. doi:10.1016/j.lindif.2012.12.014
Li, Q., & Ma., X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215-243. doi:10.1007/s10648-010-9125-8
Livingstone, S. (2012). Critical reflections on the benefits of ICT in education. Oxford Review of Education, 38(1), 9-12. doi:10.1080/03054985.2011.577938
Maris, G., & Van der Maas, H. (2012). Speed-accuracy response models: Scoring rules based on response time and accuracy. Psychometrika, 77(4), 615-633. doi:10.1007/s11336-012-9288-y
Meijer, J., & Karssen, M. (2014). Effecten van het oefenen met Rekentuin, technisch eindrapport [Effects of practicing with Math Garden, final technical report].
28 Amsterdam: Kohnstamm Instituut. Received from
www.kohnstamminstituut.uva.nl/rapporten/beschrijving/ki925.htm
Nussbaum, M. (2007). Games, learning, collaboration and cognitive divide. OECD. Retrieved from www.oecd.org/dataoecd/43/39/39414787.pdf.
Oosterbeek, H., Van Praag, M., & IJsselstein, A. (2010). The impact of entrepreneurship education on entrepreneurship skills and motivation. European Economic Review, 54, 442-454.
Organization for Economic Cooperation and Development (OECD) (2015). Students, Computers and Learning: Making the Connection. Paris: OECD Publishing. doi:10.1787/9789264239555-en
Pinheiro, J., Bates, D., DebRoy, S., Sarkar, D. and R Core Team (2016). nlme: Linear and Nonlinear Mixed Effects Models [computer software].
URL:cran.r-project.org/web/packages/nlme
Przybylski, A.K., Ryan, R.M., & Rigby, C.S. (2009). The motivating role of violence in video games. Personality and Social Psychology Bulletin, 35(2), 243–259.
doi:10.1177/0146167208327216
Punie, Y., Zinnbauer, D., & Cabrera, M. (2006). A review of the impact of ICT on learning. Luxembourg: European Commission. Retrieved from
ftp.jrc.es/EURdoc/JRC47246.TN.pdf.
R Core Team (2013). R: A language and environment for statistical computing [computer software]. Vienna, Austria: R Foundation for Statistical Computing. URL: www.R-project.org
Ronimus, M., Kujala, J., Tolvanen, A., Lyytinen, H. (2014). Children’s engagement during digital game-based learning of reading: The effects of time, rewards and challenge. Computers & Education, 71, 237-246. doi:10.1016/j.compedu.2013.10.008
Ryan, R.M., & Deci, E.L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development and well-being. American Psychologist, 55(1), 68–78. doi:10.1037/0003-066X.55.1.68
Selwyn, N. (2011). Education and technology. Key issues and debates. London/New York: Continuum.
Schlotter, M., Schwerdt, G., & Woessmann, L. (2011). Econometric methods for causal evaluation of education policies and practices: a non-technical guide. Education Economics, 19(2), 109-137. doi:10.1080/09645292.2010.511821
29 Slusky, D. (2015). Significant placebo results in Difference-in-Differences analysis: The case
of the ACA’s parental mandate. Eastern Economic Journal. doi:10.1057/eej.2015.49 Simões, J., Redondo, R. D., & Vilas, A. F. (2013). A social gamification framework for a K-6
learning platform. Computers in Human Behavior, 29, 345-353. doi:10.1016/j.chb.2012.06.007
Stigler, M., & Quast, B. (2015). rddtools: Toolbox for Regression Discontinuity Design ('RDD') [computer software]. URL:cran.r-project.org/web/packages/rddtools Van den Heuvel-Panhuizen, M., Buys, K., & Treffers, A. (2001). Kinderen leren rekenen.
Tussendoelen annex leerlijnen. Hele getallen bovenbouw basisschool [Children learn to do math. Targets and curriculum. Integers for middle school]. Groningen: Wolters-Noordhoff.
Van der Maas, H., Klinkenberg, S., & Straatemeier, M. (2010). Rekentuin.nl: Combinatie van oefenen en toetsen [Math Garden: Combination of exercise and tests]. Examens, 4, 10–14.
Voss, K. (2015, November 5). The growth of Gamification: What it means for schools and districts. Getting Smart. Retrieved from gettingsmart.com/2015/11/the-growth-of-gamification-what-it-means-for-schools-and-districts
Vygotsky, L. S., (1978). Mind in society: The development of higher psychological processes. M. Cole, J.S. Steiner, S. Scribner & E. Souberman (Eds.). Harvard. University Press. Wouters, P., Van Nimwegen, C., Van Oostendorp, H., & Van der Spek, E. D. (2013). A
meta-analysis of the cognitive and motivational effects of serious games. Journal of Educational Psychology, 105(2), 249-266. doi:10.1037/a0031311
