Citizenship for all? : analyzing the presence of a Matthew effect in citizenship education

(1)

Citizenship for all?

Analyzing the presence of a Matthew effect in citizenship education.

Rosa Stikkelman, 6089682

(2)

1

Abstract

Key words: Citizenship, education, orientations, Matthew effect.

Increased social and political tensions have given rise to a call for citizenship education. This education should teach children competences to be an active and engaged citizen. By teaching these competences to all children, citizenship education should counteract fragmentation in society.

Because prior experiences influence the way in which people learn citizenship competences in new situations, and children differ in their prior experiences, citizenship education will not have the same effect of the development of citizenship competences for all children

This study aimed to answer the question whether a Matthew effect is present in citizenship education. Data from the COOL5-18 measurements of the Citizenship Competences Questionnaire from 2007/2008 and 2010/2011 were used (N=965).The analyses were organized by distinguishing four citizenship orientations (societal interest, reflective thinking, prosocial ability, and assertiveness) and two knowledge scales (societal knowledge and interpersonal knowledge). For each scale, the difference between the first and the second measurement was added as a variable. It was analyzed whether and how this difference score could be predicted by the initial score (2007/2008).

The analyses indicated that for some orientations and scales the initial scores can significantly predict the difference score. However, unlike what was hypothesized, the direction of the effect is negative. This means that differences do not increase but they become smaller. A below average initial score corresponds to a positive difference score (growth). An above average initial score corresponds to a negative difference score (decrease). Therefore it can be concluded that no Matthew effect is present in citizenship education. However, the effect that is found cannot easily be

interpreted, because the regression to the mean that was found, can be a statistical effect rather than an actual effect in the sample.

If the pattern that is found does in fact indicate a true effect, this wouldone the one hand be desirable, as differences decrease. On the other hand, it is not favourable that above average students fall back to average scores in the course of three years of education.

(4)

3

1. Introduction

Since the turn of the century, citizenship has become a topic of interest in many Western countries. Social and political tensions have given rise to debates about fragmentation and erosion of a shared society. Increasingly, schools have become considered to be a context in which young generations can be reached and socialized(Geijsel et al., 2012; Schulz, Ainley, Fraillon, Kerr, &Losito, 2010). Here, children are to be taught skills and knowledge which are necessary to be an engaged and participating citizen of modern society (Isac, Maslowski, & van der Werf, 2011).

Education, and citizenship education in particular, have a great potential to enhance social cohesion through the development of social capital (Print & Coleman, 2010). This form of capital incorporates such things as trust, understanding, and networks which all help to coordinate social action and thus stimulate the efficiency of society. Citizenship education has been shown to have a positive relationship with the development of social capital(Print & Coleman, 2010).

However, social capital on the level of individual students, which is part of what is stimulated by education, does not guarantee social cohesion. When citizenship education is biased towards certain groups in society this could in fact cause existing tension in society to worsen (Print &

Coleman, 2010). For example, when education is structurally formed around the point of view of men, thereby ignoring the significance of women in society, existing inequalities between men and women could grow and cause increasing tension between those group. This potential for ill effects is

especially potent and problematic in divided societies where the overt and hidden curriculum can reinforce existing prejudice. In such situations, citizenship education might still increase social capital, but only for select groups in society, thus increasing inequalities and social friction .

Although citizenship education is universal in the sense that it is focused on the general school-going population, this does not mean that the effects of citizenship education are the same for everyone. Citizenship education that is biased could benefit some more than others. When it is the case that those that benefit more were in a privileged position to begin with, this would mean that differences increase, and consequently social cohesion might diminish.

Research has shown (Geboers, 2012; Geijsel et al., 2012) that children vary widely in their citizenship competences and the development of these competences. This study will examine whether the development of citizenship competences during three years of education can be predicted by prior acquired competences. Such an effect could show itself in a pattern where previous attainment and development have a negative relationship. This would mean that children with low prior proficiency develop more than children with high prior competences. However, because contexts in which citizenship competences are developed can strengthen each other (Biesta, Lawy & Kelly; 2009) and no systematic approach exists to counter that effect, it is more likely that previous attainment and development have a positive relationship. This would indicate that existing differences increase while the aim of citizenship education is to decrease fragmentation and increase of social cohesion. If this

(5)

4

effect is found the goals of citizenship education would need to be re-evaluated as an increase in differences would not counteract fragmentation and social tensions.

1.1 Citizenship

Citizenship is a concept, thedefinition of which is widely debated. In its simplest and most technical form, citizenship refers to ‘membership of individuals in a nation-state with certain (universalistic) rights and obligations’ (Levanon&Lewin-Epstein, 2009). However, developments in modern society have made this definition troublesome. Due to increased globalization, both economic and cultural, societies have become more diverse and societal boundaries have become unclear. In addition, subgroups within society increasinglyaim to differentiate themselves from society and create their own understanding of citizenship to strengthen their separate identity (Levanon&Lewin-Epstein, 2009). Together, these developments make that this technical definition of citizenship does not do justice to the complexity of social reality and call for a debate about a new definition of citizenship which goes beyond membership of a nation-state. It should be able to merge the political-legal element with the social-cultural aspect (Geijsel et al. 2012).

On the other hand, the concept should be broad enough to include the wide variety of cultures which coexist in one society and to leave room for autonomy and individuality(Geijsel et al., 2012; Geboers, 2014). Therefore, the concept of citizenship should find a balance between community and autonomy. It should incorporate a set of shared moral values which is narrow enough to be able to counteract the fragmentation of modern society. This tension complicates the definition of the concept ofcitizenship.

However, one thing is generally agreed upon, which is that an essential aspect of citizenship is democracy. In this context democracy does not refer solely to a political construct but also to a social and cultural ‘mode of associated living’ (Dewey, 1966, in Geijsel et al., 2012). In such a mode of living, citizens must be able to coexist and interact regardless of differences. In addition, citizens should take an active part in creating and reshaping the civil society (Ten Dam et al., 2011). This requires people to be able to ‘critically evaluate different perspectives, explore strategies for change, and reflect on issues of justice, (in)equality and democratic engagement’ (Westheimer, 2008, in Ten Dam et al. 2011).

The development of such ability does not come about spontaneously but needs to be taught and happens gradually (Ten Dam &Volman, 2004). This means that children and adolescents do not possess citizenship abilities to the same extent as adults do. In addition, young people are formally not politically active. Therefore, the concept of citizenship needs to be adjusted to be meaningful in the context of lives of children and adolescents. It should be focused less on political rights and duties but more on situations in which these groups can gain experience with citizenship. Citizenship of young people can be studied in the form of ‘the capacity of young people to act as citizens or- in other words- their citizenship competences’ (Geijsel et al., 2012). These competences are made up of

(6)

5

knowledge, skills, attitudes and reflection, which are necessary to function in daily social situations that children or adolescents encounter (Rychen & Salganik, 2003).

1.2 Citizenship competences

Ten Dam et al. (2011) extended the model by Rychen and Salganik (2003) by distinguishing four social tasks that young people have to deal with (acting democratically, acting in a socially

responsible manner, dealing with differences and dealing with conflicts). By combining these with the four components knowledge, skills, attitudes and reflection, a comprehensive model is created which consists of 17 scales which measure citizenship competences (see appendix 1 for details).

Geboers(2014) has restructured the knowledge, skills, attitudes and reflection scales into a compact model consisting of four citizenship orientations and two knowledge scales, as is visualized in table 1 and 2. This designis better aligned with educational goals and daily citizenship practices of young people and will therefore be used in the current study.

The conceptualization of citizenship by Geboers (2014)is based on exploratory and

confirmatory factor analyses of data gathered with the Citizenship Competences Questionnaire (Ten Dam, Geijsel, Reumerman, &Ledoux, 2011). The four citizenship orientations and two types of citizenship knowledge that are thus created will each be discussed shortly.

Table 1 Citizenship orientations

Citizenship orientation Original scale

Societal interest Attitude acting democratically Attitude dealing with differences Prosocial ability Attitude dealing with conflicts

Skills acting democratically

Skills acting in a social responsible manner and dealing with conflicts

Skills dealing with differences Reflective thinking Reflection acting democratically

Reflection acting in a responsible manner Reflection dealing with differences Assertiveness Skills acting democratically

(7)

6 Table 2 Knowledge scales

Knowledge scale Original scale

Societal knowledge Knowledge acting democratically Knowledge dealing with differences Interpersonal knowledge Knowledge acting in a social responsible

manner

Knowledge dealing with conflicts

The four orientations are: societal interest, prosocialability, reflective thinking and

assertiveness. Societal interest refers to the ‘willingness to be part of a community and willingness to take responsibility for other people in the community’ (Geboers, 2014). This involves respect for and an interest in other people and their differences. Prosocial ability refers to ‘skills needed for proper communication and for adaptation to practices and habits of people in society, familiarity with social rules, and the ability to empathize with others’ (Geboers, 2014). Reflective thinking consists of the ability and willingness to reflect on social issues in society. Assertiveness relates to ‘the skills needed to stand up for your own ideas and clearly formulate these’ (Geboers, 2014). Correlations between the orientations range from low (r=.205) to high (r=.736).

The two knowledge scales are: societal knowledge and interpersonal knowledge. The first refers to knowledge of the organization and structure in society. This can relate to governmental and democratic structures and ‘norms that are at issue in society’. The interpersonal knowledge domain concerns ‘ knowledge of prevailing social values, behavioural rules, and social everyday manners’ (Geboers, 2014). The correlation between the two knowledge scales is high (r=.838), but the

knowledge scales are not significantly correlated with the citizenship orientations. This indicates that knowledge about citizenship and abilities and willingness to show behaviour related to citizenship are two distinct entities.

1.3 Differences in citizenship proficiency and development

These and other models have been used in both international and Dutch studies analyzing differences in citizenship proficiency and its development. Isac et al. (2011) studied both cognitive and non-cognitive citizenship outcomes of students in 31 countries on the student level, classroom level, and national level. On student level, several effects were found for variables such as gender, exposure to television, and the number of books at home, but combined these explained only 14 % of the variance at student level. On the classroom level, instructional quality and opportunity to learn showed to have an impact on citizenship knowledge. An open group climate and a homogeneous classroom seemed to foster the best results. On the national level, investment in citizenship education had an effect on knowledge acquisition. However, unlike what would seem likely, this relationship was negative. In addition, freedom to divide and spend funds had a positive relationship with students’ knowledge.

(8)

7

However, autonomy in assessment, disciplining, and admittance had a negative relationship with knowledge. The authors were not able to explain this finding. Lastly, the level of teacher training showed to have an impact on the development of citizenship knowledge (Isac et al., 2011).

Schulz et al. (2010) also made an international comparison of citizenship development in students. They found effects for gender, parents occupational status, parents educational level, home literacy, parental interest in politics, and SES-background. In general they found a complex picture of factors on the level of the individual student and factors on the level of the school that impacted students’ citizenship development on several scales. The findings suggest even more complexity when comparing nations, because they showed great differences in effects and hence no consistent patterns could be found.

Cleaver et al. (2005) and Keating et al. (2010) studied the effects of citizenship education in schools in Great Brittan. Cleaver et al. (2005) found effects of factors on three levels; individual, family context and community context. Examples of such factors are gender, home literacy, and ethnic background. The study also suggests that above and beyond the complex cluster of factors that influence students citizenship development, the school can also have an impact. In a follow up study Keating et al. (2012) add to these conclusions that citizenship education can only really have a substantial impact when students perceive their citizenship education as extensive. In addition, their findings suggest that the effect of citizenship education does not last over time once students no longer receive such an education. In other words, in order to generate sustainable effects, a citizenship program in school needs to be thorough and long lasting. The most important factor for predicting students development was their previous scores on citizenship scales. Background variables such as gender, ethnic background, parental educational level and home literacy do play a role according to their findings, but less so than previous attainment. To summarise, citizenship education can have a positive effect on students’ citizenship development, but it needs to be understood in the complex context of their daily lives.

Geboers (2014) did a similar study in the Dutch context. Differences between schools were too minimal to be able to draw conclusions on the effects of school context, which signals a difference compared to the outcomes of the studies by Cleaver et al. (2005) and Keating et al. (2010). Student background variables did show meaningful effects on the development of citizenship competences, however. Students with an ethnic minority background developed more reflective thinking and societal interest than other students. Gender also was of importance. Boys showed less development in societal interest and reflective thinking than girls. The level or track of education was also relevant. Children in general educational tracks developed slightly more reflective thinking and knowledge than students in prevocational education.

Geijsel et al. (2012) also studied differences in the Dutch context. They used the 17 competences as scales. Girls showed higher scores on most attitude and skill scales and the

(9)

8

citizenship knowledge. Cognitive level also had a small positive effect on reflection and attitudes, and less on skills. In this study, socio-economic status was measured as parental education level, which had a slight relationship with knowledge. Ethnicity had a mixed effect. Overall, minority children scored lower on knowledge and higher on attitude, skills and reflection.

Apart from individual background variables, the effect of time, or age, was included by both Geijselet al. (2012) and Geboers (2014). In both studiesdifferences in developmental patterns between the used scales or orientations were found. Geboers (2014) reports a decline in societal interest, reflective thinking, an increase in the two knowledge scales, and stability in prosocial ability and assertiveness. Geijsel et al. (2012) also reports an increase on all knowledge scales. For both studies, the size of the effect of age on knowledge is small. Geijsel et al. (2012) found a slight negative effect on attitude and reflection. The citizenship skills showed no effect with age in that study.

As an explanation of a ‘dip’ of certain scales at ages around 14-16, Geijsel et al. (2012) proposed a ‘puberty effect’. This effect might be caused by hormonal changes (which make teenagers more rebellious) or the social structure of a typical school. This effect might be reinforced by the fact that teenagers usually do not have much to say about the organisation of a school day, which indicates that school might not be the ideal place to learn about democracy and citizenship.

This overview of international and national studies suggest a complicated pattern of factors that influence the development of citizenship competences in students. Several factors come up in most of these studies; gender, SES, parental education level (or occupational level), and ethnic background. The literature does not give a consistent picture of what the precise effects of these factors are. With these descriptions of effects of citizenship education in mind, we now turn to the hypothesized effect; the Matthew effect.

1.4 Matthew effect

Citizenship education by definition includes all students (Ten Dam &Volman, 2004). The call for this type of education was a response to an increase in differences and fragmentation in society. Schools are legally required to equally educate all students, regardless of differences, ethnicity and

background, to be democratic citizens (Onderwijsraad, 2012). In general, however, education which is provided to all children (universal education) will not affect all students equally, because children are not equally responsive to educational style or context. Students differ in their educational needs and when this is not explicitly considered, education will not affect all students in the same way. Of course, even when differences in needs are considered, students will still differ in their development, but at least part of the variance can be countered or controlled. Education that does not consider differences among students is often subject to a Matthew effect (Ceci & Papierno, 2005) which can be described as ‘the rich get richer/ the poor get poorer’ principle (Scarborough & Walker, 2003). It indicates that a gap between groups on a certain area will increase over time.

(10)

9

More specifically,in the context of education a Matthew effectwould indicate that children who are more able at the start of school will improve more than children who are less able. For example, children who can read relatively well when they start school will become increasingly better at reading compared to children who start with poor reading skills. An example of such an effect was shown by Luyten and Bruggencare (2011), who studied the presence of a Matthew effects in Dutch language skills and found several. The concept of the Matthew effect is a useful tool to investigate the development of different groups in general, but it provides little information on what factors

contribute to this effect in specific situations. To formulate a grounded hypothesis about the presence of a Matthew effect in citizenship education, a more specific theory will be needed.

Biesta et al. (2009) provide some insight in how a Matthew effect might manifest itself in citizenship education. They describe how context, relationships, and dispositions influence the effects of citizenship education. Citizenship skills are not only learned in school but also in contexts outside of it. In those contexts, the relationships people have can influence the way those context are

perceived. Through personal characteristics, experiences in contexts and with relationships, people form dispositions in citizenship. These dispositions influence the way peopleperceive new

situations,such as citizenship education, and affect how people learn from those situations.

Children who have had democratic experiences at home, in leisure time, and other contexts, with relationships built on mutual trust, may have formed dispositions that enable them to see other situations in the light of citizenship and democracy. An example of a democratic experience at home might be the interaction with a parent about making rules. When children have experienced that expressing their opinion can have an effect on their surroundings, it might have an influence on how they understand other democratic situations. As a result, these children can learn new aspects of citizenship and reflect on it.

In contrast, children who have not had those experiences, relationships, and dispositions will have more trouble understanding new situations as related to citizenship because they have not been able to develop a thorough understanding of what citizenship means. Therefore, these students will be less able to understand a citizenship curriculum in the context of their own live and will be less likely to learn from such a curriculum (Biesta et al., 2009).

This discussion of citizenship development indicates that previous experiences and current understanding of citizenship influence the extent to which citizenship education can be utilized by students to develop their citizenship further. A lack of previous experience with citizenship and a limited understanding of that concept, will make it more difficult for a student to learn from formal citizenship education. On the other hand, students who have had experiences in which they

recognized elements of citizenship and who do have a sense of what citizenship entails, will be better able to learn from education that targets citizenship development.

(11)

10

1.6 Current study

The above discussion has given us clues that citizenship development might be influenced by prior experiences. This effect might take the form of a negative relationship, where higher previous attainment is related to little development and low previous attainment with high development. However, as was mentioned earlier, the discussion by Biesta et al. (2009) indicates that the opposite effect is also likely, or in other words, that a Matthew effect might be present in citizenship education. This would mean that children who start school with a sense of democracy and citizenship, might benefit more from a citizenship curriculum than children who have not developed such an

understanding. Such an effect could result in the widening of the gap of existing differences between children. However, a possible Matthew effect in citizenship education has never been investigated. This study will therefore analyze the presence of a Matthew effect in citizenship education using the COOL 15-18 cohort study data (Driessen, Mulder, Ledoux, Roeleveld & van der Veen; 2009).

It is hypothesized that a higher initial score will be followed by a greater, or more positive, development and that a lower initial score will be followed by a smaller development. The

consequence of such a pattern would be that an initial gap between scores would increase, or a Matthew effect would be present.

Apart from analyzing the presence of a Matthew effect, the effects of background variables that have been found to play a part in the outcomes of citizenship education will be investigated. Background characteristics that have shown to be of importance in several studies are gender,

ethnicity, educational level, and SES (parental education level). No specific hypotheses will be added because the current literature does not provide consistent result to warrant such hypotheses. These variables will therefore be included in this study in order to assess whether they function as mediators in the development of the citizenship scales and knowledge.

In summary, the main question that will be addressed in this study is:

Is a Matthew effect present in citizenship education?

The sub questions that will provide an answer to the main question are the following:

- Can the development of citizenship competences be predicted by prior acquired competences, and if so, how?

- Are ethnicity, gender, SES, and cognitive level function relevant for the development of citizenship competences, and if so, how?

(12)

11

2. Method

2.1 Participants

The dataset that was used for this study is a subsample from the COOL5-18 data, which was gathered for a cohort study that collected a wide range of data from a sample of students that is representative for the Dutch student population. The participants were followed starting from the age five until they were 18 years old. The subsample that was used for this study was selected on several criteria. First the students had to have participated in at least two measurement cycles. Second, on both these measurement moments, the Citizenship Competences Questionnaire had to have been filled in. Only one cohort met these criteria, which consisted of students who were in the last grade of primary education in 2007, and in the third grade of secondary education in 2010. However, only a part of this cohort was measured on citizenship competences. From the original COOL5-18 dataset, only 965 participants remained after the selection procedure, and were used for this study.

The complete COOL5-18 dataset has been shown to be representative of the Dutch population of students in the relevant age range and schools (Zijsling, Keuning, Naayer& Kuyper, 2011). The subsample of 965 students from 184 schools has been compared with the total sample and showed to be comparable on IQ, social ethnic background and education and income of the parents (appendix 2). In addition, the primary education scores on the four citizenship orientations and two knowledge scales of the students who participated in both the first and the second measurement cycle were compared to the scores of the total group who filled in the Citizenship Competences Questionnaire in primary education. The distributions of all orientations and scales were similar for both groups. Therefore it is assumed that the subsample that was used for this study is also representative for the Dutch population of students.

2.2 Instruments

The Cool5-18 study used a wide range of instruments(Zijsling et al., 2011), of which only the Citizenship Competences Questionnaire was used for this study. The 94 items in this instrument wereorganized into four components ( knowledge, reflection, skills, and attitudes) and four social tasks (acting democratically, acting in a democratic responsible manner, dealing with differences, and dealing with conflicts), resulting in 17 scales.

As mentioned before, the framework that was utilized here has been created by Geboers (2014). The four citizenship orientations and two knowledge scales are made up of the averages of combinations of the 17 original scales, as is indicated in table 1 and table 2. These tables include the reliability coefficients for each new scale.

2.4 Analysis

For each orientations and knowledge scale, a new variable was created which represented the difference between the 2009-2010 score and the 2007-2008 score. This variable represents the

(13)

12

development that the participants have made during this period. This variable and the initial score were centred to facilitate the interpretation of the results. In addition, aggregates of the initial scores were created in order to be able to check whether these have a significant effect on the development in multilevel models.

Six subsets were created, one for each orientation and scale, including the centred initial score, centred difference score, aggregate of initial score, school ID, and students ID. In each subset, students with missing scores on any of these variables were deleted to ensure that all subsequent models were analyzed using the same amount of participants.

The normality was checked for all initial scores and all difference scores. Most differences scores were normally distributed, apart from assertiveness, which is positively skewed. The initial scores of societal interest and prosocial ability were slightly negatively skewed and the initial scores of assertiveness, social knowledge, and interpersonal knowledge clearly negatively skewed. This might indicated that a ceiling effect is at work for these variables. The initial score for critical

reflection is slightly positively skewed, indicating a high portion of low scores. This description of the distribution of the data does give an impression of what the pattern of scores looks like, but non-normality of the scores does not need to be a problem for this analysis, as the assumption of non-normality considers the normality of residuals, not of scores.

Next, the linear relationships between the initial scores and the corresponding difference scores were analyzed in order to get a preliminary understanding of the patterns in the data. On all six orientations and scales a clear linear relationship is visible. However, these relationships are not in line with the hypotheses. Contrary to the expected effect, a low initial score is connected with a positive development, and a high initial score is associated with a negative effect.

In order to draw conclusions about these relationship, six separate models were created, one for each orientation and knowledge scales, with the difference score as predicted variable, and the 2007-2008 score as predictive score. With these models, it was analyzed whether the citizenship development could be predicted by the prior acquired competences. With this analysis it was possible to determine whether initial differences increase, decrease or are stable over a three year time period.

The first step in the analysis was to determine which orientations and scales were to be analyzed with a multilevel technique. Calculations of the intra-class correlations indicated that the school context explained a significant part of the variation for the societal knowledge, interpersonal

knowledge and prosocial ability. Therefore these orientations were analyzed multilevel. For societal interest, reflective thinking and assertiveness the school level showed no effect, and therefore these orientations were analyzed with a generalized least squares model (GLS).

The model specification process for the multilevel models will be shortly discussed here. The first models created in this process were empty random intercept models, which indicatedwhether the difference scores was predicted only by the intercepts (Model 1). This model was extended by adding

(14)

13

the first level predictor that was hypothesized, namely the initial score (Model 2). By comparing deviances it wasanalyzed whether this new model better fit the data than the empty model. When it did, the subsequent steps are based on Model 2. The second model was extended by adding the aggregate of the initial score, in other words, the mean initial score for each school (Model 3). The deviance of this model was compared to the deviance of Model 2 in order to establish whether Model 3 better fit the data. If this was the case, the processcontinued with Model 3. Next, the model was extended by analyzing whether the initial score has a random slope (Model 4). Again, this model was compared to the last in order to see if it better predicted the variance in the difference score. Lastly, the model that was produced during this process was compared to a new GLS model that had the same fixed effects included. With this, it was analyzed whether the multilevel analysis was necessary considering the structure of the data. When the multilevel model fit the data best, itwas concluded to be the final model. When the multilevel model did not significantly better fit the data than the GLS model, the later was concluded to be the final model.

For the second research question, the subsets that were used for the first were altered by adding the background variables and again deleting all participants with missing scores. This greatly diminished the sample sizes, because quite large amount of information was missing on the

background variables. The final multi-level models and the GLS models for societal interest,

reflective thinking and assertiveness that were created in the first analysis, were extended by adding the hypothesized background effects. Because these models rendered very few significant effects, exploratory models were created with all four background variables and their interaction effects with the slope between the initial score and the difference score. Because of the large amount of variables in these models, the interpretation of individual effects became difficult. Therefore, final models were created which included only the effect that showed significant or near significant effects in the total models. Because these final models were not created according to any hypotheses, they should be regarded and interpreted as exploratory and more research is needed in order to draw conclusions about these effects.

3. Results

The results will be organized according to the theoretical structure of four citizenship orientations and two knowledge scales. As such, the analysis of the presence of a Matthew effect and the effects of the background variables will be presented together. Because of this structure, the results can be easily interpreted for each orientation and scale and the theoretical implications are emphasized over the technical execution of this study.

(15)

14

3.1 Citizenship orientation

The regression analyses indicated that for the three orientations societal interest, reflective thinking, and assertiveness the initial score (2007-2008) explained a significant part in the variation of the development of those orientations (see table 3). The direction of these effects is visualized in figure 2 to 4. These three figures all show that a participant who scored above average, tended to score a negative difference score and a participant who scored a below average initial score, tended to have a positive difference score.

The analysis of the effects of the background variables indicated that for reflective thinking, assertiveness, and prosocial ability no variables made a significant addition to the simple regression models. Therefore, the results as presented in table 3 and table 5 form the final models for these orientations. For the orientation societal interest, however, the analysis of background variables indicated that IQ made a significant addition to the predictive power of the simple regression model as presented in table 3. Therefore, the final model for societal interest includes the effect of IQ, as is presented in table 4. An ANOVA indicated that this model provided a significantly better explanation of the variance of the differences scores than the model without IQ (χ2

(1, n=416)= 593.0272– 588.0762=4.951,p=.013). This new model states that both the initial score and IQ are significant predictors of the differences score. Given the variables in this model, an average student has a difference score on societal interest of -0,16. A one unit increase on the initial score corresponds to a decrease of 0,62 on the difference score. An increase of one unit on IQ is associated with an increase of 0,004 on differences score.

Table 3 Results regression analyses societal interest, reflective thinking, assertiveness.

β t-test (Df) Significance Societal interest -0.570 -17.002(833) .000 Reflective thinking (Final model) -0.614 -18.594(833) .000 Assertiveness (Final model) -0.636 -19.883(833) .000

Table 4 Results analysis societal interest

Societal interest (Final model)

Fixed effects Coefficient S.E. Intercept -0.156* 0.025 Initial score -0.621* 0.047 IQ 0.004* 0.002 Deviance: 588.0762*

(16)

15 Table 5 Results analysis prosocial ability

Prosocial ability (Final Model) Fixed effects Coefficient Intercept -0.043* Initial score -0.617* Aggregate 0.174

Random effects Variance component Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) 0.117 0.003 Deviance: 614.0224*

Figure 1 Relationship between initial score and difference score societal interest

(17)

16

Figure 3 Relationship between initial score and difference score assertiveness

For all citizenship orientations the necessary assumptions were checked. In the case of societal interest, reflective thinking and assertiveness, all assumptions held, indicating that the results of the analysis can be interpreted. For prosocial ability most assumptions proved to hold, except for the homoscedasticity of the second order residuals. Transformation of the data did not improve this situation. Therefore the model should be assumed to not meet the assumptions and should therefore not be interpreted.

Societal and interpersonal knowledge

Next, the analysis of societal knowledge and interpersonal knowledge will be discussed. The societal knowledge model was a multilevel model which included both the initial score and the aggregate of the initial score as fixed effects and the initial score as a random effect. Given all the variables in this model, an average student had a difference score on societal knowledge of 0.097, which indicates a slight increase. Furthermore, an increase of one unit on the initial score, corresponds to a decrease in the difference score of 0.759. Over and above this effect, an increase of one unit on the average of the initial score on the school level is associated with an increase of 0.302 on the difference score. These results indicate a negative relationship between the difference score and the initial score. This indicates that students who score below average on their initial score, tended to have a positive difference score and participants who have an above average initial score, tended to have a negative difference score.

The model for interpersonal knowledge incorporates the initial score and the aggregate of the initial score as predictors of the intercept and the initial score as a random effect. The results indicate that an average student has a difference score of zero. In addition, an increase of one unit on the initial score corresponds to a decrease of 0.660 on the difference score. Over and above this effect, an increase of one unit on the school average of the initial score is associated with an increase of 0.369 on the difference score. These results again indicate a negative relationship between the initial score

(18)

17

and the difference score and are summarized in table 6 Like the previous models, a participant who scored an above average initial score, tended to have a negative difference score and a participant who scored a below average initial score, tended to have a positive initial score.

Table 6 Models Societal Knowledge and Interpersonal knowledge

Societal knowledge Interpersonal knowledge

Fixed effects Coefficient S.E Coefficient S.E. Intercept 0.097* 0.009 -0.018 0.009 SOCKNOW_PO_C -0.759* 0.048 -0.660* 0.047 SOCKNOW_PO_AVE 0.302* 0.072 0.369* 0.106 Random effects Variance component Variance component

Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) τ11=Var(U1j) 0.022 0.033 0.002 0.052 0.002 0.038

These models were extended with background variables. The analysis indicated several additional effects for both knowledge scales (table 6) The final model for societal knowledge extended the model of the first analysis with IQ as a predictor, IQ as an interaction effect with the slope between initial score and differences score, and gender. These variables significantly enhanced the model for societal knowledge (χ2

(1, n=416)= (-454.366)–( -507.689)= 53.323,p<.0001/2). Table 7 presents the effects of this model.

Given the variables in this model, an average student has a difference score on societal knowledge of zero. An increase of one unit on the initial score, is associated with a decease on the difference score of 0,81. Over and above this effect, an increase of one unit on the school average initial score, corresponds to an increase on the difference score of 0,27. An increase of one unit on IQ corresponds to an increase on the difference score of 0,003. The difference score of an average girl is 0,06 points higher than the difference score of boys. The significant interaction effect indicated that an increase of one unit on IQ is associated with a lower slope between initial score and difference score of 0,01. In other words, a higher IQ decreases the relationship between the initial score and the difference score.

The analysis of interpersonal knowledge indicated that only IQ as a predictor for the intercept of the difference score was significant. The model including this variable was compared to the final model of the first analysis (χ2

(1, n=416)= (-286.543)–( -301.141)= 14.598,p=.0001/2). This ANOVA indicated that the new model is significantly better than the old model. This new model indicates that an average student has a difference score on interpersonal knowledge of zero. An increase of one unit on the initial score is associated with a decrease of the difference score of 0,57 . The effect of the

(19)

18

school average is no longer significant in this model. An increase of one unit on IQ is associated with an increase of 0,002 on the difference score.

Together these analyses indicate that some background variables can predict some difference scores, but the effects are small.

Table 7 Final model societal knowledge and interpersonal knowledge

Societal knowledge (Final model)

Interpersonal knowledge (Final model)

Fixed effects Coefficient S.E. Coefficient S.E. Intercept 0.020 0.022 -0.010 0.013 Initial score -0.807* 0.064 -0.574* 0.061 Aggregate 0.269* 0.098 0.105 0.143 IQ 0.003* 0.000 0.002* 0.001 Gender 0.055* 0.013 IQ : Initial score -0.010 0.003

Random effects Variance component Variance component Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) τ11=Var(U1j) 0.051 0.026 0.001 0.016 0.026 0.025 Deviance: -765.0896* -765.0896*

The assumptions for the final models of the knowledge scales were checked. Unfortunately, the final model for the societal knowledge showed many issues. Basically none of the assumptions held, even after several attempts to transform the data in different ways. As a result, the outcomes of this analysis are not trustworthy and should not be interpreted. The interpersonal knowledge model had less issues. One outlier and hetreoscedasticity (both on the first and the second level) were detected. Deletion of the outlier did not correct for the hetreoscedasticity, nor did transformation of the data. Therefore, this model should also be assumed not to be trustworthy. Effect sizes were calculated for these models, but as these models cannot be interpreted, this measure has no practical meaning.

4. Conclusion and discussion

This study aimed to answer the question whether a Matthew effect is present in citizenship education. Growth of students on four citizenship orientations (societal interest, reflective thinking, prosocial ability, and assertiveness) and twoknowledge scales (societal knowledge and interpersonal

knowledge) of citizenship (Geboers, 2014) was analyzedwith and without background variables. For this analysis, the COOL5-18 measurements of the Citizenship Competences Questionnaire from

(20)

2007-19

2008 and 2010-2011 were used. For each orientation and scale, the difference between the first and the second measurement was uses as a dependent variable. It was analyzed whether and how this difference score could be predicted by the initial score (2007-2008). In addition, the role of the four background variables SES, IQ, social ethnic status and gender in this relationship were analyzed.

For each orientation and scale a model was created to analyze whether and how initial scores could predict the difference score. Where necessary multilevel models were used in order to account for school level effects.These models indicated that for the orientations societal interest, reflective thinking, and assertiveness school context had no significant effect. However, for the orientation prosocial ability and the two knowledge scales the school context did have an impact. These results are interesting because previous studies found inconsistent results about the influence of the school context (Cleaver et al., 2005; Geboers, 2014; Keating et al., 2010; Schulz et al., 2010). It might be that the organisation of citizenship ability in orientations and scales such as used in this study, accurately differentiates between those elements that are influenced by school context and those that are not, while other methods of organising the elements of citizenship have more trouble making that distinction.

With regard to the hypothesized effect, the results indicated that the initial scores do explain a significant part of the variation in the difference score. However, the direction of the found effects is opposed to the direction of a Matthew effect (Scarborough & Walker, 2003): a below average initial score corresponds to a positive difference score (growth) and an above average initial score

corresponds to a negative difference score (decrease). Therefore it can be concluded that no Matthew effect is present in citizenship education.

Another interesting effect might be at play: citizenship scores on the orientations and scales that can be interpreted,show a regression to the average score. This would seem to indicate that children who start with a below average score show a positive development, but children who start with an above average score portray a negative development. On the one hand, this might indicate that citizenship education does bring all students to a minimal level of citizenship competences and

decreases differences in those competences which is part of the aim of citizenship education. On the other hand, children who score high on the citizenship orientations and citizenship knowledge at the first measurement, seem to show a decline of citizenship competences, which cannot be considered desirable, because this would indicate a loss of citizenship abilities. However, it is not possible to draw conclusions about this effect because it could be due to the fact that statistical analyses with a structure such as the one in this study, tend to show regression to the mean, even when this is in fact not present in the real population (Barnett, van der Plos, & Dobson, 2004).

The only conclusion that can be drawn from this study is that no Matthew effect can be found for the citizenship scales societal interest, reflective thinking, and assertiveness.The pattern that was found is difficult to interpret because no effect sizes were calculated and because of the issues

(21)

20

regarding regression to the mean. For the scale of prosocial ability and the two knowledge scales, no conclusions can be drawn, because the models did not meet the necessary statistical assumptions.

In addition to the analysis of the general pattern of each scale and orientation, four background variables to the models were added to the models: IQ, gender, SES, and social ethnic status. These variables were added both as predictors of the intercept of the difference score and as predictors of the slope between the initial score and the difference score. Final models were created with those effects that showed to be significant predictors of either the intercepts or the slopes.

The models for reflective thinking, assertiveness, and prosocial ability indicated that none of the four variables had any added predictive value for either the intercept or the slope. In order words, this means that apart from the initial score, no other predictive variables were found. This in itself is an interesting finding, because according to previous research, these variables had the most potential of predicting the difference score (Cleaver et al., 2005; Geboers, 2014; Geijsel et al., 2012; Keating et al., 2010; Schulz et al., 2010).

The final model for societal interest, interpersonal knowledge and societal knowledge

indicated that gender and IQ had effects on some of the intercepts, and IQ had an effect of some of the slopes. Although it was hypothesized, SES and social ethnic background did not have any effects in these six models. However, the effects that were found were very small, which means their practical value is small as well. In addition, the practical relevance of these findings is very limited because of the exploratory nature of the analyses, the number of significant effects and the inconsistency of the effects across orientations and scales.

Apart from the statistical limitations that were already discussed in the results section and the conclusion, several limitationsof this study should be mentioned. Firstly, the data that was used for these analysis consists completely of questionnaire data. Although this is a useful tool for gathering information about large groups of people, it could be argued that the subject of citizenship ability and its development require a more qualitative aspect as well. The complexity of any type of education, but particularly of education that aims to influence the development of non-cognitive, social element of development, might make that we cannot do justice to reality when we try to understand it purely quantitatively.

In addition, the author of this study had no part in the data collection process, and could therefore not take into consideration any issues or insights that might have come up in that process otherwise.

Despite the limitations of this study, it does provide some valuable insights. First and foremost, no indications of a Matthew effect were found. Although it remains important to stay vigilant of unintended effects, it seems unlikely that a Matthew effect is present in citizenship education. Furthermore, although statistical issues complicate the conclusions that can be drawn from these

(22)

21

results, there is a clear indication that a relationship between previous attainment and further development in citizenship abilities might exist.

A practical implication of the results of this study is that designers of educational materials and teachers who work with those, should be conscious of such a relationship. Possible responses to such a relationship might be to differentiate in citizenship education based on previous attainment, in order to ensure that all students gain as much as possible from their education. A possible explanation of these results is that teachers might already invest more time and attention in those children that seem to be lacking in citizenship experience in order to compensate for the differences in level of ability with more advanced students. This would explain why students with lower initial scores have a positive development while students with high initial score seem to show a decline in scores.

Apart from these practical implications, this study shows the need for more extensive statistical analysis, where more moments of measurement are included (as this would partially solve the complications of regression to the mean) but also of more extensive qualitative measurements. This could help with the interpretation of the results of this study, as it might provide insight into whether the pattern that was found is simply due to the statistical phenomenon of regression to the mean or whether this pattern is truly present in the population. Future research should include both qualitative and more extensive quantitative data and combine these methods of analysis in order to better understand the development of citizenship in children and its relationship to prior achievement on citizenship measures.

(23)

22

Literature

Barnett, A. G., van der Pols, J, C., Dobson, A. J. (2004). Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology. 34(1) , 215-220.

Biesta, G., Law, R., & Kelly, N.(2009). Understanding young people’s citizenship learning in everyday life: The role of contexts, relationships and dispositions. Education, Citizenship and Social Justice, 4 (1), 5–24. DOI: 10.1177/1746197908099374

Ceci, S.J., &Papierno, P.B. (2005). The Rhetoric and Reality of Gap Closing: When the “Have-Nots” Gain but the “Haves” Gain Even More. American Psychologist, 60(2), 149-160.

Cleaver, E. Ireland, E., Kerr, D.,& Lopes, J. (2005). Citizenship Education Longitudinal Study: Second cross-sectional survey 2004. Listening to young people: Citizenship education in England .(DfES Research Report 262). London, England: DfES.

Driessen, G. Mulder, L., Ledoux, G., Roeleveld, J. & Veen, H. van der (2009). Cohortonderzoek COOL5-18. Basisrapport basisonderwijs, eerste meting 2007/08. Nijmegen/Amsterdam: ITS/SCO-KI.

Geboers, E., Geijsel, F., Admiraal, W., & Dam, G. ten. (2012). Review of effects of citizenship education. Educational Research Review, 9, 158-173.

Geboers, E. (2014). Citizenship of young people (dissertation). Amsterdam: University of Amsterdam

Geijsel, F., Ledoux, G., Reumerman, R. & ten Dam, G. (2012). Citizenship in Young people’s daily lives: Differences in citizenship competences of adolescents in the Netherlands. Journal of Youth Studies, 2012, 1-19. DOI:10.1080/13676261.2012.671932

Isac, M. M., Maslowski, R., & van der Werf, G. (2011). Effective civic education: an educational effectiveness model for explaining students’ civic knowledge. School effectiveness and school improvement: An international journal of research, policy and practice. 22(3), 313-333. DOI: 10.1080/09243453.2011.571542

Levanon, A., &Lewin-Epstein, N. (2010). Grounds for citizenship: Public attitudes in comparative perspective. Social Science Research. 39(?), 419-431. DOI:10.1016/j.ssresearch.2009.12.001

(24)

23

Keating, A., Kerr, D., Benton, T., Mundy, E. & Lopes, J. (2010). Citizenship education in England 2001-2010: Young people’s practices and prospects for the future.

Onderwijsraad (2012). Advies verder met burgerschap in het onderwijs. The Hague: Author. Retrieved from http://www.onderwijsraad.nl/upload/publicaties/725/documenten/verder-met-burgerschap-in-het-onderwijs.pdf

Rychen, D.S. &Salganik, L.H.(2003). Key competencies for a successful life and wellfunctioning society. Gottingen, Germany: Hogrefe& Huber.

Print, M. & Coleman, D. (2010). Toward understanding of social capital and citizenship education. Cambrige Journal of Education, 33(1), 123-149.

Ten Dam, G., Geijsel, F., Reumerman, R. &Ledoux, G. (2011). Measuring young people’s citizenship competences. European Journal of Education, 46(3), 354-372.

Ten Dam, G. &Volman, M. (2004). Critical thinking as a citizenship competence: teaching strategies. Learning and Instruction, 14(?), 359-379. DOI :10.1016/j.learninstruc.2004.01.005

Zijsling, D., Keuning, J., Naayer, H., & Kuyper, H. (2011). Cohortonderzoek COOL5-18. Groningen: GION.

(25)

24

Appendix 1

(26)

25

Appendix 2

Details on comparison complete sample and subsample. Table 8 Descriptives total sample CCQ 2007/2008

Descriptive Statistics

N Range Minimum Maximum Mean Std. Deviation Variance Skewness Kurtosis

Statistic Statistic Statistic Statistic Statistic Statistic Statistic Statistic Std. Error Statistic Std. Error

iq_2011 15111 111.42 40.95 152.38 99.9943 14.97503 224.251 -.006 .020 -.021 .040 Soc.Etnischeachtergrond 13381 5 1 6 4.86 1.360 1.849 -.930 .021 .081 .042 Hoogste diploma invuller 13366 6 1 7 4.59 1.606 2.581 -.635 .021 -.507 .042 Hoogste diploma partner 12101 6 1 7 4.49 1.761 3.100 -.492 .022 -.917 .045 Betaaldebaan>12 uur (invuller)? 13334 1 1 2 1.78 .417 .174 -1.327 .021 -.239 .042 Betaaldebaan>12 uur (partner)? 12120 1 1 2 1.89 .312 .097 -2.504 .022 4.272 .044 Valid N (listwise) 9035

Table 9 Descriptives final sample

N Range Minimum Maximum Mean Std. Deviation Variance Skewness Kurtosis

Statistic Statistic Statistic Statistic Statistic Statistic Statistic Statistic Std. Error Statistic Std. Error

iq_2011 732 89.54 52.10 141.63 97.5049 15.94513 254.247 .161 .090 -.322 .180 Soc.Etnischeachtergrond 552 5 1 6 4.72 1.375 1.891 -.759 .104 -.132 .208 Hoogste diploma invuller 549 6 1 7 4.46 1.584 2.508 -.524 .104 -.578 .208 Hoogste diploma partner 497 6 1 7 4.33 1.777 3.157 -.448 .110 -1.006 .219 Betaaldebaan>12 uur (invuller)? 546 1 1 2 1.74 .441 .195 -1.075 .105 -.847 .209 Betaaldebaan>12 uur (partner)? 499 1 1 2 1.89 .311 .097 -2.530 .109 4.418 .218 Valid N (listwise) 390

(27)

26

Appendix 3

Model selection process for multilevel models. Table 10 Model selection process prosocial ability

Model 0 (GLS) Model 1 (Empty model) Model 2 Model 3

Fixed effects Coefficient S.E. Coefficient S.E. Coefficient S.E. Coefficient S.E. Intercept -0.043* 0.015 -0.044* 0.018 -0.042* 0.014 -0.043* 0.014 PROSKILL_PO_C -0.601* 0.028 -0.617* 0.029

AVE_PROSKILL_PO 0.174 0.100

Random effects Variance component Variance component Variance component Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) 0.179 0.118 0.117 0.005 0.002 0.003 Deviance: 991.2206 984.8884* 617.126* 614.0224* Model 4 Model 5 (GLS)

Fixed effects Coefficient S.E. Coefficient S.E. Intercept -0.043* 0.014 -0.043* 0.017 PROSKILL_PO_C -0.0619 0.033 -0.617* 0.030 AVE_PROSKILL_PO 0.174 0.100 0.129 0.086 Random effects Variance component

Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) τ11=Var(U1j) 0.115 0.003 0.007 Deviance: 614.0224 616.962

(28)

27 Table 11 Model selection process societal knowledge

Fixed effects Coefficient S.E. Coefficient S.E. Coefficient S.E. Coefficient S.E. Intercept -0.093* 0.007 0.093* 0.012 0.097* 0.010 0.096* 0.008 SOCKNOW_PO_C -0.727* 0.029 -0.759* 0.032

SOCKNOW_PO_AVE 0.248* 0.081

Random effects Variance component Variance component Variance component Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) τ11=Var(U1j) 0.041 0.024 0.024 0.005 0.003 0.002 Deviance: -222.933 -249.5202* -711.8594* -720.5182* Model 4 Model 5 (GLS)

Fixed effects Coefficient S.E Coefficient S.E. Intercept 0.097* 0.009 0.094 0.006 SOCKNOW_PO_C -0.759* 0.048 -0.759 0.033 SOCKNOW_PO_AVE 0.302* 0.072 0.285 0.058 Random effects Variance component

Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) τ11=Var(U1j) 0.022 0.002 0.052 Deviance: -765.0896* -696.4572

(29)

28 Table 12 Model specification process interpersonal knowledge

Fixed effects Coefficient S.E. Coefficient S.E. Coefficient S.E. Coefficient S.E. Intercept -0.015* 0.008 -0.016* 0.010 -0.013 0.010 -0.015 0.009 INTKNOW_PO_C -0.651* 0.033 -0.680* 0.035

INTKNOW_PO_AVE 0.328* 0.108

Random effects Variance component Variance component Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) 0.051 0.034 0.034 0.002 0.002 0.003 τ11=Var(U1j) Deviance: -89.65982 -94.55058* -413.84252* -422.35* Model 4 Model 5

Fixed effects Coefficient S.E. Coefficient S.E. Intercept -0.018 0.009 -0.015* 0.007 INTKNOW_PO_C -0.660* 0.047 -0.680* 0.036 INTKNOW_PO_AVE 0.369* 0.106 0.364* 0.081 Random effects Variance component

Level-1 variance: σ2 e =Var(Eij) Level-2 variance: τ00=Var(U0j) 0.033 0.002 τ11=Var(U1j) 0.038 Deviance: -436.7896* -408.505

Citizenship for all? : analyzing the presence of a Matthew effect in citizenship education