An explorative study to the possibility of repeating writing assignments to measure the writing development of pupils in Dutch grades six to eight

The power of repetition

An explorative study to the possibility of repeating writing assignments to measure the writing development of pupils in Dutch grades six to eight

Noortje Hemmen  University of Groningen  1/26/2015

Student number: s1768948

Supervisor: Dr. J. F. van Kruiningen Co-supervisor: Prof. Dr. C. M. de Glopper

Research Master Linguistics

Language and Cognition

3

Table of contents

Table of contents ... 3

Abstract ... 7

1. Introduction ... 9

2. Theoretical framework ...12

2.1 Writing development ...12

2.2 Task specificity in writing ...14

2.2.1 Motivation to write ...14

2.2.2 Writing knowledge ...16

2.2.2.1 Genre knowledge ...16

2.2.2.2 Topic knowledge ... 18

2.2.2.3 Knowledge of audiences ...19

2.2.2.4 Knowledge of language ...21

2.3 Writing assessment ... 22

2.4 Writing in Dutch education ... 24

2.4.1 Reference levels ... 26

2.4.2 Schrijfmeters maken ... 27

2.5 Repetition effects ... 28

2.5.1 Repetition effects on motivation to write ... 29

2.5.2 Repetition effects on text quality ... 29

2.5.3 Repetition effects on linguistic complexity ... 30

2.5.4 Repetition effects on content production ... 30

3. Method ... 32

3.1 Participants ... 32

3.2 Data ... 33

3.3 Procedure ... 35

3.4 Assignments ... 35

3.5 Questionnaires ... 36

3.5.1 Motivation to write ... 36

3.5.2 Recognition ... 37

3.5.3 Self-reported similarity in approach and content ... 37

3.6 Text analyses ... 37

3.6.1 Text quality ... 38

3.6.2 Linguistic analysis ... 38

3.6.3 Content analyses ... 40

3.6.3.1 Elaboration of themes and ideas ... 40

4

3.6.3.2 Overlap analysis ... 43

3.7 Statistical analyses ... 44

3.7.1 Questionnaires ... 44

3.7.1.1 Motivation to write ... 44

3.7.1.2 Recognition of the assignments ... 45

3.7.1.3 Self-reported similarity in approach and content ... 46

3.7.2 Text analyses... 47

3.7.2.1 Text quality ... 47

3.7.2.2 Linguistic analyses ... 49

3.7.2.3 Content analyses ... 49

4. Results ... 50

4.1 Questionnaires ... 50

4.1.1 Motivation to write ... 50

4.1.2 Recognition of the assignments ... 53

4.1.3 Relationship between average motivation to write and recognition ... 56

4.1.4 Self-reported similarity in approach and content ... 59

4.1.5 Relationship between average self-reported similarity in approach and content and average recognition ... 60

4.1.6 Relationship between average self-reported similarity in approach and content and average motivation to write ... 63

4.2 Text analyses ... 65

4.2.1 Text quality ... 65

4.2.2 Correlation between average motivation to write and writing quality per assignment in the experiment group ... 68

4.2.3 Linguistic analyses ... 70

4.2.3.1 Number of t-units per 100 words ... 70

4.2.3.2 Number of words per t-unit ... 71

4.2.3.3 Number of clauses per t-unit ... 72

4.2.4 Content analyses ... 73

4.2.4.1 Elaboration analysis ... 74

4.2.4.2 Overlap analysis ... 76

5. Conclusion ... 78

5.1 Questionnaire constructs ... 78

5.2 Text analyses ... 81

6. Discussion ... 84

7. References ... 87

Appendices ... 92

5

I. Assignments ... 93

II. Questionnaires ... 96

III. Text assessment ... 103

IV. Structural Analysis of Written Language form (White, 2007) ... 110

V. Instructions content analysis second assessor ... 111

VI. Reliability analysis of overlapping ideas ... 119

7

Abstract

The assessment of writing is known to be extremely difficult and in The Netherlands, and there are no instruments developed with which teachers can easily and rapidly measure and follow their pupils’ writing development (De Glopper & Prenger, 2013). In the current study, it was explored whether it is possible to repeat writing assignments to measure the development of writing ability, or whether repetition would negatively affect motivation to write, text quality, linguistic complexity and content production. This study follows up on the Schrijfmeters maken project, in which three assignments were developed with accompanying rating scales. During that project, these assignments were performed by 344 pupils in Dutch grade six to eight. In this study, 107 pupils repeated two of these three assignments roughly one year later. For 134 pupils in the control group, all assignments were new. In addition to the assignments, all pupils filled in questionnaires containing questions about motivation, recognition and memorization of the assignments and similarity in content and approach.

Overall results of this study indicated that recognition of the assignments was relatively low,

and that there was no effect of repetition on motivation to write and self-reported similarity

in approach and content. However, there appeared to be effects of repetition on linguistic

complexity, content production, and text quality.

9

1. Introduction

Globally, writing seems to be integrated into all aspects of society, and it has become a fundamental part of engaging in both professional, educational, and social activities (Zimmerman & Risemberg, 1997). For years, people have kept long distance contacts through writing letters, and with the rise of the internet and social media, it is even easier to communicate with people all around the world through writing. Moreover, writing plays a vital role in the workplace as the administration of most organizations depends on written memo’s, emails and letters. It is in school that students learn to write well, which can be viewed as a critical component of being able to communicate effectively to a variety of audiences. Also, students can explore and analyse ideas through writing and so, writing not only serves to communicate or for self-expression, it also stimulates critical thinking and problem-solving skills. Furthermore, writing enables students to continue learning in academic areas, providing a solid basis for future (professional) development. The fact that writing can be used for many purposes, such as communicating, learning and self-expression, and the fact that it enables people to develop themselves in both professional and academic areas, and to actively participate in society makes it a crucial ability to develop.

Since early literacy development provides the foundation for later academic success (Ritchey, 2008; Snow, Burn, & Griffin, 1998), the majority of studies into writing development have focused on the early years of infancy and childhood, while only a few studies focused on writing development from late childhood into adolescence (Christie & Derewianka, 2008).

Although the foundation of writing development may be laid during the preschool years, writing development mostly takes place during the school years. For teachers to be able to foster writing development, it is necessary to gather information about the writing process and the development of students’ writing ability throughout the years. Studies into writing development have been conducted since the 1970’s, and although these studies have provided useful insights into several stages of writing development, the majority of these studies solely focused on one phase in writing development. Thereby, studies in which writing development is followed over time are scarce. The choice of studying a certain phase in writing development can be explained by the fact that “(…) the construct of writing ability is broad, multifaceted, situated, contextual, and resistant to a monolithic, stable definition” (Slomp, 2012, p. 81). Also, Applebee (2000) in his overview of writing models describes the development of writing ability as being non-linear, individual and context-dependent. He furthermore states that writing ability and writing development are hard constructs to define, which makes it hard constructs to explore. As with literacy development in general, there are numerous sociocultural and cognitive factors (e.g. motivation to write, writing knowledge, processing speed, working memory load, etc.) that influence writing development. These problems not only are addressed in writing development studies, but are also experienced by teachers who struggle with assessing the writing development of their students.

A study performed by Krom and colleagues (2004) to writing education in the Netherlands

revealed that writing ability of students in the highest grade of Dutch elementary schools

(grade eight in The Netherlands, equivalent to grade six in the U.S. school system) was below

desired levels. Moreover, text quality from students in grade six (equivalent to grade four in

the U.S. school system) appeared not to differ significantly from text quality in texts written

by pupils in grade eight. Also, from the total time devoted to language education in primary

10

schools, only ten percent was dedicated to writing. Last, their study showed that between 1999 and 2004, writing education in The Netherlands had not improved. Another study performed by the Dutch Centraal Instituut voor Toetsontwikkeling (CITO) in 2009, showed that although now 17% of the total time was dedicated to writing, writing proficiency of pupils in grades five and eight had not improved in 2009 compared to 1999 either. Therefore, the Dutch government decided in 2010 that Dutch writing education should be improved and the level of language proficiency pupils in different phases of education have to achieve was determined by law (Nederlandse overheid, 2010). In this law, two levels of both language proficiency and numeracy were determined. The fundamental levels have to be achieved by at least 75% of the students and the target levels have to be achieved by at least 50% of the students. For example, at least 75% of pupils at the end of primary school have to comply to reference level 1F, meaning that they have to be able to write short texts about everyday topics or topics directly related to the writer (Inspectie van het onderwijs, 2010).

Although the reference levels are developed to support writing education, there appear to be several problems in the use of these reference levels. For example, although they should help educators to foster and measure the writing ability of their students, only the end goals are defined and there are no intermediate goals with which teachers can determine whether a student’s development is on track or not. More importantly, although there are tests developed for the other reference levels that were determined (e.g. numeracy), there are no standardized instruments with which educators can easily assess, record and track the writing development of their students (De Glopper & Prenger, 2013). One problem both encountered by educators and researchers in writing assessment is the effect of task specificity on writing performance, which refers to inconsistencies in performance by the writer. These inconsistencies are caused by numerous factors other than the writing abilities of the writer self, such as variance in topic knowledge, genre knowledge, and motivation to write (DeGroff, 1987; Haslett, 1986; Hidi & Boscolo, 2006; Huot, 1990; Olinghouse &

Graham, 2009; Schoonen, 2005; Wollmann-Bonilla, 2001).Therefore, if only one task is used to examine students’ writing ability, the performance probably only reflects students’ skills on that particular task, making it hard to generalize this to the broader domain of writing ability. However, when using multiple writing assignments to measure writing development, task-specific performance can make it hard to compare the writing performance of students over time, unless the same writing assignments can be repeatedly used.

The purpose of this study was to explore the possibility of repeating writing assignments to measure the development of writing ability of pupils in Dutch sixth to eighth grade in primary school. When repeating writing assignments, teachers can easily compare students’

performance on a particular task over time, making it easier to assess and follow students’

writing development. Up to this day in writing research, however, assignments have never been repeated. In all these studies the reasons for not repeating assignments stay implicit, but it may be because of the manifestation of what I will call repetition effects. It can be argued that when assignments are repeated, participants may recognize the assignment and will use the same strategies or produce identical content during the repeated assignment, which may cancel out possible effects from an intervention. Also, motivation to write may decrease as participants do not consider the writing task profitable the second time they perform the task. As several studies have shown, the performance and persistence of performing a task is dependent on intrinsic motivation and value of the writing task (Hidi &

Boscolo, 2006; Troia, Shankland & Wolbers, 2012). A decrease in intrinsic motivation may

11

lead to lower text quality. However, cognitive studies into the writing process have shown that writing is a problem-solving process that requires the constant monitoring of progress toward task goals. The difficulty of this process may create motivational problems and these motivational problems may reduce when repeating assignments, since writers have gone through the problem-solving process before. Therefore, they already have encountered and solved many problems and can rely on this process the second time they perform the assignment. Last, it is argued that with age, children develop their writing knowledge and skills, including genre knowledge, topic knowledge, audience knowledge and language knowledge. This is likely to result in the production of more ideas and linguistically more complex texts, which eventually lead to higher text quality. The development of these variables may rule out the negative effects repetition may have on content production and linguistic complexity of the produced texts.

This study followed up on the Schrijfmeters maken project, which aimed at encouraging writing education in The Netherlands and providing educators with tools with which the development of pupils’ writing ability could be easily followed and stimulated (De Glopper &

Prenger, 2013). To do so, three writing assignments and accompanying rating scales were developed. Two of these assignments were repeated in the current study by 107 pupils roughly one year after pupils had performed the last assignment for Schrijfmeters maken.

For 134 pupils in the control group, these assignments were completely new. Based on the performance of pupils in the control and experiment group, it could be determined whether repetition effects appeared when repeating assignments. The research question central to this study was as follows:

Does repeated use of writing assignments to measure the development of writing ability of pupils in grades six to eight in Dutch elementary schools cause the manifestation of repetition effects on motivation to write, text quality, linguistic complexity and content production?

To formulate an answer to this question, a combination of quantitative and qualitative

analyses was performed in which possible repetition effects on motivation to write, text

quality, content production and linguistic variation were analysed.

12

2. Theoretical framework 2.1 Writing development

Often it is assumed that children learn to read and write in the first three to four years of primary school, and that after this learning period, these ‘basic skills’ are repeated in an unproblematic way (Christie & Derewianka, 2008). However, mastering the writing process is an ongoing developmental process that typically takes more than two decades (Kellogg, 2008). Moreover, studies that have focused on the development of emergent literacy have shown that writing development starts well before children attend school. For example, Gombert and Fayol (1992) after reviewing several studies about children’s early conceptions of the writing process conclude that from the age of three, children start to scribble wavy lines and non-figural graphics which represent letters. The scribbles often show some principles of writing, such as linearity and separate units and are not confused with drawings.

At the age of four, when children enter kindergarten, they will receive formal instruction and start to develop their formal literacy skills through reading and writing different text types.

During primary school years, the foundation for the development of topic knowledge, genre knowledge, audience knowledge and language knowledge is laid as pupils are prepared for the complex academic writing they have to perform in subsequent school levels (Boscolo, 2007).

Research into the writing process has been performed since the 1970’s and was first mainly

performed from a cognitive perspective in which writing ability is viewed as the result of

(successfully) employing strategies such as planning, translating, reviewing, monitoring,

generating ideas, organizing, goal-setting, evaluating and revising (Bereiter & Scardamalia,

1987; Flower & Hayes 1980, 1981; Hayes, 1996; 2012). It is assumed that before writing,

writers create rhetorical goals containing text type, purpose, and intended audience and that

this representation guides text production. For a writer to fulfil these rhetorical goals, (s)he

has to constantly reflect whether the ideas retrieved from long-term memory and the text

produced so far meet the predetermined goals. According to Bereiter and Scardamalia (1987),

the writing processes from novice and expert writers differ in the extent to which a writer is

able to move between the rhetorical space (i.e. one’s representation of the text type, purpose,

audience, etc.) and the content space (i.e. the text produced so far). According to these

researchers, novice writers employ a knowledge-telling strategy, in which thoughts are

directly formulated into language without taking the rhetorical goals into account. By

contrast, expert writers are assumed to have elaborate rhetorical goals and they use this

functional representation to guide their text production. This writing strategy is characterized

as the knowledge-transforming strategy, as this process requires active transformation of

content in order to satisfy rhetorical goals. In contrast to novice writers, expert writers

constantly move between the content produced so far and their rhetorical goals to check

whether the content is in accordance with their predetermined rhetorical goals. These

strategies seem in line with the findings of Van der Pool (1995). In her study to differences in

text structure in texts written by children (aged ten to twelve), adolescents (from the age of

15) and adults, she found that children only select information they need and put this in the

correct order, while adolescents and adults also evaluate the information they select and the

order in which they present it in their texts. These findings show that there is an evolution in

writing with age from describing to evaluating.

13

According to Kellogg (2008), it takes twenty years of maturation, instruction and training to become an expert writer. He argues that the development of writing skills progresses through three stages. It starts with knowledge-telling, a stage that has already been discussed above and in which writing is solely used to communicate what one knows. After ten years of training, one shifts to knowledge-transforming. While Bereiter and Scardamalia (1987) viewed this stage as the expert phase, Kellogg considers it as an intermediate stage in which the writer reviews his/her own composition only to check if it represents what (s)he wants it to represent. After twenty years of training, the knowledge-crafting stage can be achieved, in which the writer both evaluates the text to check whether it represents what (s)he wants it to represent and whether it meets the imagined reader’s understanding of the text. This last stage only is achieved by mature adults who want to become skilled professional writers.

Moreover, Kellogg (2008) explains that the development from the first (novice) stage to the third (professional) stage of writing proceeds through practicing and learning within specific writing tasks. Through practice, the basic writing processes, such as planning, language generation, and reviewing are constantly developed, just as the mental representations that must be generated and stored into working memory. With this explanation, Kellogg implicitly states that different writing tasks require different writing skills that can be developed by practicing the different text types over and over. Therefore, a writer will operate at a more advanced level in more practiced text types compared to less practiced genres, as it is more knowledgeable in the one text type than the other. The fact that writing ability is not a single ability that can be employed in order to compose all kinds of texts causes writers to perform very inconsistent on different writing tasks.

In addition to the valuable insights about the writing process that were provided by the cognitive writing theories, studies performed from a sociocultural view on writing provided important contributions to understanding the social aspects of writing (e.g. Burns &

Casbergue, 1992; Brock & Green, 1992; Boscolo, 1991; Rogoff, 1990; Russell, 1997). In

sociocultural writing research, the emphasis is on the social and cultural dimensions of

writing and the ways in which a writer creates meaning from its experiences in the social

world. According to this approach, cognition is situated in and thus influenced by the social

and physical context. The writing process is not easy to define from a sociocultural approach,

since this process not only is formed by the thoughts that exist about writing in a particular

culture in which is written, but the writing process is also influenced by a range of contextual

variables (e.g. task-objectives, motivation, interest, media, setting etc.). While the cognitive

and sociocultural approaches to writing first were strongly opposite and kept separated,

nowadays parts of both perspectives are integrated to come up with a complete as possible

theory of the writing process. Boscolo (1991), for example, argues that during writing, a

writer needs to have both a set of cognitive and linguistic skills (such as planning for the

reader’s understanding, revision of the text in order to check gaps of meaning, and constantly

choosing the appropriate rhetorical and pragmatic strategies to keep the text understandable

for the reader). However, these processes will in all probability be formed and influenced by

the (cultural) environment of and contextual factors surrounding the writer, such as the

medium used (e.g. pen and paper or a computer), topic knowledge, writing knowledge,

interest in the topic and motivation to write. Each of these factors can influence the writing

process and performance in its own way, and together with the expertise a writer has in a

certain text type this will result in task-specific performance.

14

2.2 Task specificity in writing

Measuring a productive language skill such as writing is considered to be tremendously difficult and numerous articles have been written in which problems with validity and reliability in assessing writing ability are stressed (Blok & Hoeksema, 1984; Huot & Perry, 2009; Knoch, 2011; Linn, Baker & Dunbar, 1991; Linn & Burton, 1994; Parkes, Suen, Zimmaro & Zappe, 1999; Schoonen & De Glopper, 1992; Wesdorp, 1981). The problem in assessing writing ability is often not interrater agreement, but task generalizability due to task specificity (Linn & Burton, 1994). Task specificity refers to an inconsistency in performance on a test, caused by factors other than the abilities of the tested candidate (Schoonen, 2005). Before the implications of task specificity on the assessment of writing are discussed, first two variables that have been identified in psychological research that can account for the variation in writing performance will be explained: motivational factors and writing knowledge stored in long term memory.

2.2.1 Motivation to write

Several studies have shown that positive motivation is associated with task persistence (Zimmerman & Ringle, 1981), and academic achievement (Kurz & Borkowski, 1984).

Motivation started as a subject of research in the behavioural sciences and the variety of motivational aspects that were identified in these studies have been applied in writing studies as well. Boscolo and Hidi (2007) propose that it is useful to organize these aspects in three main areas: 1) the intrinsic motives that activate a person’s behaviour, 2) a person’s perceptions of his or her writing ability, and 3) the strategies individuals use to manage a demanding task. Although these motivational factors are divided into three areas, they are rarely separated from one another. For example, the will to write is closely connected to a writer’s self-perception of ability, as well as to the ways and tools (s)he is able to use for self- regulating.

One very important intrinsic motive that has been identified in motivation studies, is goal- orientation (Troia, Shankland & Wolbers, 2012). It is argued there are three types of goals that can be set: mastery, performance or avoidance goals. The type of goal that is set, depends on the way the goal is displayed in one’s cognition (i.e. an individual’s mental representations of desired outcomes), behaviour (i.e. particular actions that should be undertaken to achieve goals), and affect (i.e. goals may be associated with either positive or negative feelings, resulting in approach or avoidance of an action). For example, a mastery goal can be set when there is a focus on developing knowledge and skills to improve a certain competence. A performance goal will be set when a person has the intention to show one’s abilities to others and to receive recognition of an audience. The third type of goal, avoidance, is the opposite of a performance goal, as a person can choose to avoid displaying incompetence and not undertake any action. To fulfil these goals, one has to have an idea of the desired outcomes (cognition), the actions (s)he has to execute to fulfil the desired outcomes (behaviour) and a positive association with the goals (affect). Depending on the goal that is set and depending on the way the goal is displayed in cognition, behaviour and affect, a person is likely to be motivated to undertake action or to avoid performance.

The second area of motivational aspects Boscolo and Hidi (2007) distinguished, concerns a

person’s perceptions of his or her own writing ability, including self-efficacy, self-concept,

and self-perception of one’s abilities. Several studies have shown that the extent of self-

efficacy is positively related to the amount of effort that is put in the execution of a task, task-

15

persistence, the employment of strategies to complete a task, and actual task performance, regardless of one’s age, gender or ethnicity (Pajares, 1996; Pintrich, 1999; Pintrich & Schunk, 2002). Often, these perceptions are related to the difficulty of the task that needs to be performed, as well as to the third area of motivational aspects: the strategies individuals use to manage a demanding task. With increasing age and practice, writers are assumed to become better able to self-regulate essential literary processes and so become better writers (Zimmerman & Risemberg, 1997). During the course of practicing, writers will change their strategies when they receive negative feedback, and will continue to use the self-regulation strategies when these increase their perceptions of self-efficacy. It is likely that a person struggles with a certain task because (s)he has not developed the right strategies yet to complete the task. Consequently, an individual will have low expectations and self-efficacy levels. In contrast, when an individual has the right strategies developed for a particular task, (s)he will encounter little problems in completing the task and will have higher self-efficacy levels and expectations of the writing outcome. According to Boscolo and Hidi (2007), this marks the difference between expert and novice writers, as expert writers are able to recognize the problems and to come up with solutions, while novice writers may recognize a problem but are unable to come up with a solution themselves. Novice writers need feedback to improve their texts and therefore see problems as obstacles, making writing an unattractive activity for them.

Another explanation Boscolo and Hidi (2007) provide for the lack of writing motivation of students is the limited function of writing in school and its limited power to foster reasoning and the exploration of ideas. These limitations were first listed by Britton (1975) and can be seen in a developmental perspective, as through school levels and grades, writing tends to be reduced to a very limited number of academic genres. Moreover, since students often have to process given ideas in a correct form, and since writing in education solely focuses on assessment, students do not use writing as a process through which they can approach, understand and analyse problems in a personal manner. These factors together are expected to lower writing motivation. Moreover, since writing tasks in education gradually become more demanding, the execution of these tasks will become more and more demanding and the possibility exists that students will struggle more. Consequently, it is likely that students develop negative associations with the activity of writing and this will influence the writing performance. This theory seems to be confirmed in a survey from 2009 to writing proficiency that is periodically performed in the Netherlands (Cito, 2009). Pupils in fifth grade appeared to engage themselves in writing outside school settings more frequently than the eighth- grade pupils. Furthermore, these pupils appeared to have a far more positive attitude towards writing than pupils in the upper grade of primary school. It is argued that when a writer views the action of writing profitable and thus has positive associations with the task, a writer will be intrinsically motivated to write and likely to set mastery or performance goals. By contrast, when the task is associated with negative feelings, an individual is less likely to undertake any action and so will set avoidance goals.

In short, there are three main aspects that play a role in motivation: intrinsic motives,

individual perceptions of writing ability, and the strategies of an individual to manage a

demanding task. As mentioned before, these aspects are interrelated: a writer that has high

levels of self-efficacy probably also has the ability to employ the correct strategies to manage

a demanding task. Moreover, it is argued that a writer develops these strategies through

practice, and adopts certain strategies when (s)he receives positive feedback. A writer that is

16

more practiced in different types of writing has adopted more strategies and will therefore struggle less with a writing task. Consequently, this writer will be more motivated and perform better than a less practiced writer. Accordingly, a writer that is more practiced in one genre than another genre will have developed more strategies that can be employed in the one genre than in the other. Therefore, this writer will struggle less with this task and will be more motivated to perform this task than to perform a task in another less practiced text genre, resulting in task-specific performance.

2.2.2 Writing knowledge

For a writer to be able to produce well written texts for several purposes, (s)he not only needs to be motivated to perform and persist in the action of writing, but also has to have sufficient writing knowledge and skills (Byrnes, 2001). Studies into the needed knowledge and skills in writing have come up with four types of writing knowledge and skills that need to be developed: genre knowledge, topic knowledge, audience knowledge, and knowledge of language (Lindgren, Leijten & Van Waes, 2011). It is argued that differences in the extent to which each of these knowledge types are present in individuals’ long term memory will result in writing output that differs in quality (Byrnes, 2001). To avoid inconsistencies in task performance by pupils in this study, these factors were accounted for as much as possible.

Although these variables are independently discussed in this chapter, they seem to be interrelated. For example, McCutchen (1986) showed that genre knowledge can compensate for a lack of topic knowledge, resulting in a well-structured text. However, a lack of topic knowledge seems always to lead to less detailed descriptions of the topic and so, differences in the amount of topic knowledge held by individuals will result in differences in performance. Also, knowledge of an audience alone is not sufficient to write a qualitative good text, as a writer also needs to have sufficient knowledge about vocabulary and grammar (Byrnes, 2001). For the sake of clarity, the development and influence on the written outcome of genre knowledge, topic knowledge, audience knowledge, and knowledge of language will be discussed independently of each other in the following paragraphs.

2.2.2.1 Genre knowledge

According to Hyland (2003) genre refers to “abstract, socially recognised ways of using language” (p. 21), and is based on

“the assumptions that the features of a similar group of texts depend on the social context of their creation and use, and that those features can be described in a way that relates a text to others like it and to the choices and constraints acting on text producers.” (p. 21)

Thus, genre knowledge is the ability to classify text types and the capability to use different

suitable strategies when writing in different types of text genres. For developing writers,

knowledge about diverse writing tasks and purposes appears to be critical as their written

compositions are shaped by the knowledge they possess about the type of text they are

expected to write (Gillespie, Olinghouse & Graham, 2013). Early studies have shown that one

method to acquire genre knowledge is through extensive reading, as this helps children to

develop schemata about different types of texts (Byrnes, 2001). However, studies into the

actual writing of stories have found that there is a gap between recognizing a good story and

being able to produce one (Applebee, Langer, Mullis, Latham & Gentile, 1994; Stein, 1982).

17

Hence, it may be that the development of genre knowledge alone is not sufficient to produce a text that fits the requirements of a certain genre. Indeed, in a study to the narrative structures of stories written by 4-year-olds to 7-year-olds, Haslett (1986) concludes that children’s’

narrative skills are limited by both their cognitive and linguistic skills and found that the older participants appeared to be better able to produce a clear and coherent representation of a narrative than the younger participants. In combination with improved linguistic skills, this resulted in higher text quality.

Although the results found by Haslett (1986) were quite clear, the difference in genre knowledge possessed by younger and older participants was not measured in her study and the question arose whether improved writing performance was due to the combination of increasing genre knowledge and linguistic skills, or whether an increase in genre knowledge alone can result in improved text quality too. This was studied by Olinghouse and Graham (2009), who explored whether there is a difference between second-grade students’ (7- and 8-year-olds) and fourth-grade students’ (9- and 10-year-olds) depth of genre knowledge, declarative knowledge (concerning audience, purpose and text structure) and procedural knowledge (concerning steps in the writing process) about the characteristics of good writing in general and stories in particular. Their results indicated that indeed, there was a positive relationship between the depth of genre knowledge and writing performance, and that overall, fourth grade students possessed more genre knowledge and were more knowledgeable about the characteristics of good writing in general than second grade students. However, when comparing the declarative knowledge about the characteristics of a story, there appeared to be no significant difference between the younger and older participants as all students’ descriptions of the elements of a story were quite vague. It was very unusual for students to describe elements as plot, setting, characters, goals and actions.

Instead, students were more likely to talk about the beginning, middle and/or end when describing the basic features of a story and thereby did not show very refined knowledge of the narrative text genre.

Consistent to the findings of Olinghouse and Graham (2009), Klein and Rose (2010) argue that the amount of declarative knowledge possessed by students may be inherent to the text genre. In their study to genre knowledge about persuasive and explanatory texts of students in grades five and six (10- to 12-year-olds), all students showed more knowledge about persuasive writing than explanatory writing. However, these results may be induced by the fact that in this study, the development of genre knowledge was measured over a very small timespan. A study in which the development of genre knowledge was explored for a longer period of time, was performed by Lin, Monroe and Troia (2007). They studied the development of genre knowledge in grades two to eight and found that middle-school students had much more complete and refined concepts of stories than elementary students.

The older students in this study did mention specific elements of a story (e.g. setting, plot,

resolution) and these elements were also connected one to another. Moreover, they found

that while elementary school students gave similar descriptions of a story, persuasive and

explanatory text, middle-school students in their descriptions made a clear distinction

between the three text types. Last, middle-school students possessed more knowledge about

good writing than students from elementary school. In short, this study showed that

somewhere between elementary school and middle school, students’ knowledge about

specific genres becomes more complete and differentiated.

18

With increasing age, genre knowledge and knowledge about the characteristics of good writing develop and linguistic complexity increases. However, the depth of declarative knowledge possessed by students appears to differ per text genre. This is consistent to the statement from Kellogg (2008) that writing skills develop through practice and learning within specific writing tasks. These findings not only have implications for the type of writings children can and should perform at different stages in their life, it also implies that writing performance of pupils in elementary school is highly task specific due to limited cognitive development, linguistic development and the ongoing development of genre knowledge.

2.2.2.2 Topic knowledge

The majority of the empirical work examining the role of knowledge in writing has focused on the relationship between topic knowledge and writing performance. Research has consistently shown that topic knowledge develops with age, as younger children generate less ideas than older children and adults (Bereiter & Scardamalia, 1987). Moreover, several studies have shown that students in fourth grade and higher who possess higher degrees of topic knowledge compose better texts than younger students (Benton et al., 1995; DeGroff, 1987; Langer, 1984; McCutchen, 1986; Mosenthall, Conley, Colella, & Davidson-Mosenthall, 1985). One explanation for the influence of the amount of topic knowledge held by writers on the writing process is provided by Kellogg (1988), who stated that the more knowledgeable a writer is in a certain topic, the less effort (s)he has to make to retrieve and use relevant knowledge during writing. The less effort a writer has to pay to these processes, the more energy can be put in producing the actual content of a text.

In an early study by McCutchen (1986), it is explored whether topic knowledge helps to construct coherent and well-structured texts, and similarly whether a lack of topic knowledge will result in ill-structured texts. In this study, participants (ten each from grades four, six and eight) were placed in a high-knowledge group or a low-knowledge group based on a questionnaire containing questions about football (a subject that already had been successfully used in expert-novice writing studies), and were asked to write about football and about their school or people they knew. The writings on the latter topics served as control texts, since it was assumed that all participants would have equal knowledge of these topics.

Although results indicated a clear developmental trend in local coherence, children produced more coherent texts on topics in which they were more knowledgeable, regardless of grade.

Moreover, subjects with low knowledge levels of football appeared to have tried to elaborate their texts as much as their control texts. Thus, despite their limited knowledge about the topic, they still attempted to create well-formed and elaborated texts. This lead the author to conclude that although the content component was impoverished, the genre component still made a contribution to text production. Second, high-knowledgeable subjects generally elaborated more on main arguments and major events than the low-knowledgeable subjects, who elaborated more on the general aspects of football. The researcher concludes that topic knowledge plays a definite role in content production, and that with an increase in knowledge, texts will become more coherent. However, children acquire linguistic and genre skills with age and these skills can be used to still produce a well-structured text, even when their knowledge about a certain topic is sparse.

While McCutchen only studied the influence of topic knowledge in the composing stage of the

writing process, DeGroff (1987) explored the influence of topic knowledge on the whole

19

writing process. In this study, fourth-grade students’ drafted, participated in peer conferences and then revised texts they had written about baseball. Results of this study showed that prior knowledge is a significant factor in all phases of the writing process. While high-knowledgeable students referred more often to detailed events in their first drafts, the drafts of low-knowledgeable students were characterized by non-game relevant information.

Moreover, it appeared that low-knowledgeable writers were not able to use writing, conferencing, and revising to acquire additional valued information about the game of baseball, while students with high knowledge levels after the conference reconsidered making specific information about baseball explicit. Last, the writings from high-knowledgeable students were found to be syntactically more complex than writings from low-knowledgeable students. This result implied that topic knowledge not only accounts for the thematic content of text, but also for the syntactic complexity in texts.

Although the above discussed studies showed that the amount of topic knowledge by itself can explain differences in writing performance, variables such as genre knowledge and linguistic development also play a role in the quality of the writing that is produced. While linguistic skills and genre knowledge are developed with age, in the ‘90’s of the former century, the question arose what could cause the differences in amount of topic knowledge held by individuals. If it would be the case that topic knowledge is developed by age, this would imply that children from the same age possess the same amount of knowledge about a topic. However, as the above discussed studies also showed: that is not the case. To answer the question, studies to the role of topic knowledge in writing started to focus on the relationship between topic knowledge and (task) interest. Interest has been defined both as a motivational variable, as well as a psychological state and is composed both by affection and cognition (Hidi and Boscolo, 2006). Researchers suggested that interest influences students’

writing because it combines what students know about a topic with what they value (Hidi &

McLaren, 1991). In a study performed by Renninger, Ewen and Lasher (2002), for example, students had to write either about baseball (which was considered a high-interest topic) or soccer (the low-interest topic). The results of their study indicated that the high-interest topic

‘baseball’ appeared to be significantly related to writing measures such as the production of topic-relevant ideas, discourse and topic knowledge, and that students wrote more topic- relevant information on the high-interest topic than on the low-interest topic, resulting in higher text quality. Thus, these results indicated that interest in a topic is positively related to the amount of topic knowledge, and that differences in interest will result in differences in topic knowledge, which will eventually result in inconsistent (i.e. task specific) performance on different writing tasks with different topics.

2.2.2.3 Knowledge of audiences

Broadly defined, an audience is the person or people addressed by a speech or a piece of writing (Magnifico, 2010). However, the role of the audience in the writing process is special, in the sense that writers mostly write individually, without any direct interaction with their intended audience. In writing, knowledge of audiences (or: audience awareness) involves both understanding the expectations and beliefs of the intended audience and also being able to provide cues in the text about the role the writer envisions for the audience (Ede &

Lunsford, 1984). According to Kellogg (2008), only experienced writers are able to keep the

reader in mind during writing since it requires a lot from a writer’s cognitive skills. On the

one hand, the writer must be able to evaluate whether the text expresses what (s)he intended

and on the other hand whether its content and form communicate properly with the intended

20

reader. Therefore, it is argued that taking the audience into account during writing is only possible when a certain stage of cognitive development is achieved and when the physical task of writing becomes automatic (Carvalho, 2002).

Since children’s cognitive skills are not yet fully developed, one would expect young children to be unable to be aware of an audience during writing. However, different studies appear to show different findings. A study that confirms this expectation was performed by Aalderink (1988). In his study to what he called contextual dependence, he found that pupils in Dutch fourth grade (7- and 8-year-olds) struggled with decontextualizing their writing and that they were unable to provide informative cues for the audience in their texts. In this study, pupils had to write a story based on a picture depicting two children looking troubled on a branch high up in a tree. Analyses of the texts showed that more than half of the children solely wrote about the depicted situation, that a quarter of the participants described both the situation and a solution, and that only ten percent wrote about a possible motive to climb the three, described the situation and came up with a solution. Moreover, half of the pupils used reference words without naming the antecedent. In short, these results indicated that children this age are having problems decontextualizing their writing for an invisible audience.

A study that disproved the expectation that children have no audience awareness during writing was performed by Wollman-Bonilla (2001). She showed that children in first grade (6- and 7-year-olds) are able to show signs of attention to an audience under the right circumstances. Pupils in this study had to write persuasive letters for real purposes in a context characterized by an authentic and responsive audience, and in which the instruction given by the teacher included an explicit focus on audience awareness. The concreteness of the situation appeared to be an influential factor in writing for an audience, as was also confirmed in a study by Carvalho (2002) to audience awareness in texts written by fifth- grade (10- and 11-year-olds) and ninth-grade students (14- and 15-year-olds). She found that although the ninth-grade students showed a higher degree of audience awareness in their texts than fifth-grade students, the main difference in audience awareness between the younger and older students was caused by dependence on the concreteness of the situation.

The younger students were able to perform better on audience awareness when the situation in which they wrote and the intended audience became more concrete. Not only provided these studies important implications for the teaching of audience awareness of writing, the results also implied that in the years between the fifth and ninth grade, and thus between the end of elementary school and the first three years of middle school, there is a development that allows students to become more independent of context and to better implement audience awareness in the writing process.

As with topic knowledge, genre knowledge and audience knowledge to a certain degree

appear to be related. Boscolo and Ascorti (2004), for example stated that narratives and

expository texts not only differ in text structure and content, but also have different

requirements for the audience. When reading a narrative, a reader is expected to constantly

make inferences because the writer will only provide information (s)he considers to be

relevant and/or necessary for the reader’s understanding. Also, when composing a narrative,

the writer is allowed to implicitly refer to the reader’s prior schematic knowledge (of human

feelings or experiences and interactions usually described in narratives) and genre knowledge

(about structure and features of narratives). In order for the reader to build a coherent text

21

representation, (s)he often has to interpret what stays implicit in the text (e.g. why a character performed a certain action or how other characters reacted to it). By contrast, when writing an expository text, the writer is expected to provide as much relevant information as possible and to write a well-organized text without any (knowledge) gaps, since an audience is not expected to make inferences when reading an expository text. Thus, for a writer to successfully write in different genres, (s)he both has to have sufficient genre knowledge and has to know what (s)he can expect from an audience too. When either one of these knowledge types is lacking, a writer will not be able to perform consistently on different types of texts.

2.2.2.4 Knowledge of language

Although knowledge of language here is the last discussed type of writing knowledge, it is certainly not the least important. In order to convey the right meaning, a writer not only needs to have certain degrees of genre knowledge, topic knowledge and audience awareness, a writer also needs to know how to place specific words in specific grammatical constructions. More specifically, for a writer to formulate the intended message correctly, (s)he must possess a good vocabulary and command of syntax (Byrnes, 2001). Although there are numerous studies that have focused on lexical development (e.g. the stages in vocabulary development), for this study linguistic development is more relevant and therefore, only studies to this part of language development will be discussed.

Studies to syntactic development have been performed since the 1960’s, and a pioneer in this field of research is Kellogg Hunt, who in 1965 came up with the t-unit as measurement for linguistic maturity. Hunt discovered that many children do not use periods in their writing.

Instead, they chain their clauses together with the conjunctive ‘and’. When the language of children was analysed on sentence level, it would appear as if the language of children who did not use periods in their texts was more mature than the language of children who (correctly) broke their sentences with periods, since the former group produced longer sentences. Moreover, Hunt found that editing the texts on sentence level was unreliable, as different people put periods in different places. Therefore, he introduced the t-unit as measuring unit, which was defined as an independent clause with all attached or embedded dependent clauses. This measurement enabled the exploration of written language in an objective and reliable manner, and in a number of studies, significant differences in linguistic complexity were linked to significant differences in writing quality (Combs, 1976; Morenberg, Daiker & Kerek, 1978; Stewart, 1978).

In a study by Wagner and colleagues (2011), the structure of texts written by 208 first- and

fourth-grade students was analysed on, among other things, syntactic complexity. Therefore,

the mean length of t-unit and clause density (the ratio of the total number of clauses (main

and subordinate) divided by the number of t-units) per text were measured. Their results

showed significant differences in both complexity measures, as average length of the t-unit

was significantly shorter in texts written by first grade students compared to the texts written

by fourth grade students. Also, clause density appeared to significantly increase between first

to fourth grade. This effect of age was also found in a study by Nippold, Ward-Lonergan and

Fanning (2005). They explored the syntactic, semantic and pragmatic development of

children (from the age of eleven), adolescents (from the age of seventeen) and adults (from

the age of 24) and found that the production of relative clauses and the use of literate words

(including adverbial conjuncts, abstract nouns and metalinguistic and metacognitive verbs)

increased with age and that together, these changes resulted in higher writing quality.

22

Although nobody would argue against the idea of linguistic complexity increasing with age, earlier studies have found an effect of genre on linguistic complexity of a text too. Crowhurst and Piche (1979), for example, explored the syntactical complexity of sixth- and tenth- graders argumentative texts, descriptive texts, and narratives and found an effect of both age and genre. In their study, there appeared to be significant differences in syntactic complexity between the argumentative and descriptive texts of sixth- and tenth-graders. However, these differences were not found when the syntactic complexity of solely the narratives written by the younger and older students were compared. Moreover, the researchers found that genre caused a greater difference in length of t-units than did age, as there was a greater difference in t-unit length between the argumentative texts and narratives written by pupils in grade ten than between the descriptive texts and narratives or the descriptive texts and argumentative texts written by the same pupils. Last, no significant differences were found in t-unit length between the texts written by pupils in grade six and grade ten. Based on these results, the researchers drew two conclusions: 1) while syntactic complexity increases with age, it may be the case that it does not continue to increase in all types of writing and 2) when assessing the language of students, one has to take the type of task into account given the differences in syntactic complexity caused by genre. Another, more extended, study by Crowhurst (1980) to the syntactic complexity of texts written in two different genres (narrative and argumentative text) at three different grade levels provided evidence for the above first conclusion, as she found that there were significant differences in syntactical complexity of the argumentative writings between grades six and ten and between grades ten and twelve. Moreover, there appeared to be a significant difference between grade six and ten in the syntactic complexity of the narratives, however this difference was not found between the narratives written by students in grade ten and twelve. Also, Crowhurst found that there was a significant effect of genre on syntactic complexity at each grade level as the argumentative texts appeared to be syntactically more complex than the narratives. By confirming the first conclusion, the second was also confirmed, being that when assessing a writer’s writing ability based on the syntactic complexity of a text, one should also take the genre into account as this influences syntactic complexity as well.

In short, the above discussed studies showed that linguistic complexity increases with age.

However, another factor was also found that appeared to influence the linguistic complexity of a text: the type of text that has to be written. Hence, concerning the linguistic complexity of a text, genre can cause task-specific performance as a writer may be better able to produce a syntactically more complex argumentative text than to produce a syntactically complex narrative.

2.3 Writing assessment

Generally, writing ability is assessed through written products in which the performance on a

particular writing task is demonstrated. However, as shown in numerous studies, the

reliability and validity of writing assessment is frequently questioned (Blok & Hoeksema,

1984; Knoch, 2011; Schoonen & De Glopper, 1992; Wesdorp, 1981). For instance, raters often

seem to differ in their judgements of the same texts and they do not agree with themselves at

different points of time (Cooper, 1984). Through the course of time, different methods of

assessment have been developed with the purpose of improving the evaluation of writing

ability. These methods are roughly dividable into two types of assessment: indirect and direct

assessment of writing.

23

In the 1960’s, indirect writing tests were developed to retrieve information about individual’s writing ability. These tests aimed at eliminating possible rater inconsistencies by offering multiple-choice tests questioning different aspects of writing ability, such as spelling or grammatical fluency. These type of tests were based on the assumption that writing ability could be measured through other skills than the actual production of a text. The majority of research to the validity and reliability of indirect assessment has focused on the relationship between test scores on both indirect and direct measures of writing. It was claimed that a high correlation between the two scores justified the use of indirect measures instead of direct measures. The opposite method of indirect assessment is direct assessment of writing:

the assessment of an actual written composition. Although rater reliability may be a problem in direct assessment, it appears that the writing of an actual text covers a wider range of composition skills than the skills that can be measured through indirect assessment. These skills are very well measured through direct assessment of written texts and since this increases the construct validity, the direct assessment still is an often used method in measuring writing ability (Wesdorp, 1981).

There are several different rating procedures developed for the direct assessment of texts.

The most commonly used procedure by teachers in the classroom is holistic scoring, in which the assessment is based on a global impression of the written composition (Huot, 1990). In this analysis, the various aspects of a text are not scrutinized independently and this makes the assessment of texts fast and easy to perform. However, this also causes problems with reliability, since it remains unclear to what extent various aspects of the text played a role in the final judgement (Schoonen & De Glopper, 1992). This especially becomes a problem when several different writing tasks are assessed in which different aspects contribute to higher text quality. A slightly more detailed form of this rating procedure is the primary trait approach (Lloyd-Jones, 1977; Schoonen & De Glopper, 1992; Van Gelderen, Oostdam & Van Schooten, 2010). Within this approach, the assessor focuses on one main aspect of a text (often the intended communicative goal) and performs the assessment of the complete text based on this aspect. Although this is a quick method, it is not very informative since the texts are rated based on one aspect and all other aspects of writing ability are not included in the assessment. A third and more informative procedure is the analytical rating procedure, in which the judgment of texts is based on several different aspects of writing ability. Main advantage of this method is that it provides a detailed report on writing ability, and this method for example has been successfully used in the Dutch National Assessment in Education (Krom et al., 2004).

Besides the flaws of several assessment methods, there are a number of other factors that can influence the rating process. Several of these factors have been described as rating effects (Schoonen & De Glopper, 1992), which refer to inconsistencies in assessment caused by subjectivity of the assessor. These inconsistencies can be caused by the fact that it is not clear for the assessor on what aspects (s)he has to focus in the assessment (called a signific effect).

This becomes even more problematic when several assessors assess texts with different foci

and thus base their judgement on different aspects of the texts. These differences can result

in very different and unreliable judgements of the same texts. The assessment of texts can

also be influenced by the order in which they are assessed (called a sequence effect), resulting

in a relatively lower score on a text that is rated after a relatively good text compared to a text

score that was assigned to a text after a series of low quality texts had been assessed. A halo

effect occurs when certain aspects of a written text influence the assessment of other aspects

24

of the text. Often mentioned aspects that provoke a halo effect are handwriting and spelling errors. Another rating effect is the standard shift, which means that a rater (unconsciously) adapts its judgment to the texts instead of basing the assessment on a fixed reference level.

One last rating effect that can occur is the contamination effect, which occurs when there is a conflict of interest that blurs the assessment. This can occur when the assessment has a goal other than the determination of writing competence (e.g. confirmation or disproval of a particular theory), or when a teacher wants to make a certain point with the assessment, for example to punish some students for their misbehaviour. One last problem with the objectivity of the assessment by an educator lies in the fact that the assessor knows the person that wrote the text. More importantly, (s)he is likely to have a certain opinion of this person and it is very likely that this opinion will influence the total assessment as well.

A rating procedure in which these rating effects are minimized, is the procedure in which a rating scale with anchor texts is used (Blok & Hoeksema, 1984). In this assessment method, the quality of written texts is assessed based on a comparison with a set of predetermined anchor texts. These anchor texts each represent a specific reference point, for example the difference between adequate and inadequate text quality and are ascending in text quality.

With this method, rating effects such as the signific effect and standard shift are certainly excluded, as the anchor texts form the frame of reference in which the written texts are assessed. Furthermore, since the texts are assessed based on a comparison with the anchor texts, a sequence effect is also excluded. This method was first used in a periodic survey to the language proficiency of Dutch pupils in primary education that was commissioned by the Ministry of education, culture and science in The Netherlands (Blok & Hoeksema, 1984).

Also, it was employed in Schrijfmeters maken, a project that will be discussed later, and it will be used in the current study too.

2.4 Writing in Dutch education

Since 1987, the Periodieke Peiling van het Onderwijsniveau (‘Periodical Survey of the educational levels’) is performed in The Netherlands to systematically explore the level of writing proficiency of pupils at different stages of education (Blok & Hoeksma, 1984). More specifically, the methods that are used in Dutch writing education and the results of these methods are studied at a number of Dutch primary schools. Participating pupils often have to perform a number of assignments that are specially designed for the survey and that require pupils to fully deploy their writing skills. Since these surveys are performed periodically, a clear image is provided of pupils’ writing proficiency in The Netherlands over a longer period of time.

The most recent periodical survey stems from 2009 and the overall result of this survey was that writing education in The Netherlands had not improved between 1999 and 2009 (Cito, 2009). In this survey, factors such as writing frequency, writing curriculum, writing assessment, and writing ability of pupils in Dutch grades five and eight were inventoried.

This survey showed that writing received far less attention than reading and spelling in the

curriculum of primary schools, and that only 17% of the total time dedicated to language

education was spend on writing. When this was compared to 1999, there appeared to be no

increase in writing frequency. Moreover, there appeared to be no differences between both

the curriculum and the writing goals of 1999 and 2009, as in both surveys the most

performed assignments appeared to be narratives, reports and poems. Also, in both surveys it

was found that the writing goals often are to provide information, to report on pupils’ own