Language Development Abroad
Development of complexity, accuracy, and fluency in Spanish writing skills
of international students abroad
Karin van der Worp
S1542230
MA in Applied Linguistics
Faculty of Liberal Arts
University of Groningen
Supervisor:
Marjolijn Verspoor
1
Index
Acknowledgement ... 2 Abstract ... 3 1. Introduction ... 4 2. Literature review ... 82.1. Dynamic systems theory and second language development ... 8
2.2. Research on SLD in a study abroad context ... 9
2.3. Complexity, Accuracy and Fluency in second language development ... 10
2.3.1. Definitions of CAF ... 11
2.3.2. Measurements of CAF ... 11
2.3.3. Expanding accuracy by an error analysis ... 13
2.4. Summary ... 13
3. Methodology ... 14
3.1. Variables to be measured ... 14
3.1.1. Complexity, Accuracy and Fluency ... 14
3.1.2. Proficiency levels ... 17
3.1.3. Interacting variables on L2 development ... 17
3.2. Data collection ... 18
3.2.1. Sample ... 18
3.2.2. Written picture description task ... 19
3.2.3. Questionnaires ... 20
3.3. Data analysis ... 21
3.4. Summary ... 23
4. Results ... 24
4. Results ... 24
4.1. Development in Complexity, Accuracy, and Fluency ... 24
4.2. Differences between the groups ... 27
4.3. Factors involved in development of SSL ... 29
4.4. Summary ... 31
5. Discussion ... 33
5.1. Development in Complexity, Accuracy, and Fluency ... 33
5.2. Differences between the groups ... 34
5.3. Factors involved in development of SSL ... 35
5.4. Summary ... 38
6. Conclusion ... 39
6.1. Key findings and implications ... 39
6.2. Limitations and recommendations ... 40
6.3. Summary ... 40
References ... 42
Appendix 1a. Writing task for intermediate and advanced groups ... 46
Appendix 1b. Writing task adapted for beginning group ... 47
Appendix 2a. Questionnaire pre-test (Spanish) ... 48
Appendix 2b. Questionnaire pre-test (English) ... 50
Appendix 3a. Questionnaire post-test (Spanish) ... 52
2
Acknowledgement
3
Abstract
International students‟ exchanges are very popular nowadays. One of the aims of the ERASMUS exchange program is learning another language. Although it is easily assumed that immersion in the native speech community combined with formal classroom learning creates the best environment for learning a second language, there is very little research to document changes in language proficiency. Moreover, the extent to which language develops is not clear, and especially writing proficiency is paid little attention to.
This multidimensional study aimed to examine the development of writing skills of international students learning Spanish as a second language in a study abroad context. 32 International students in San Sebastian, at three different proficiency levels, took part in this investigation. The research questions addressed were how complexity, accuracy and fluency in writing skills of this group develop, if there are differences in development between the groups, and which factors are related with the differences in development. Data was gathered at the beginning and at the end of the ERASMUS language course, by means of a written picture description task and a questionnaire survey. The CHILDES program was used to process the written texts, and subsequently the data was statistically analyzed using SPSS.
Findings from the study revealed that on average the general writing proficiency of the students did develop. By looking at the different elements of complexity, accuracy, and fluency, only lexical complexity demonstrated a significant growth. Differentiating between the groups, only the intermediate group showed development, whereas the other groups did not. Some variables were found that were related with general writing proficiency development and development of lexical complexity. The development in self-evaluation of vocabulary knowledge, the positive development of attitude towards Spanish language and culture, and a high immersion by receptive input were positively related with a progress in lexical complexity. The time spent in the L2 context was positively related with both lexical complexity and general writing proficiency.
4
1. Introduction
Since 1987, the ERASMUS program provides grants for students‟ mobility within Europe. Up till now, two million students have gratefully taken this opportunity and students‟ exchanges seem to be more the norm than exception. The ERASMUS organization (2011) states that many studies have shown that spending a period abroad not only contributes to the students‟ academic and professional knowledge but, among others, also to language learning, which is the focus in the current study.
For a long time it has been assumed that immersion in the native speech community combined with formal classroom learning creates the best environment for learning a Second Language (L2). However, Freed (1998) has indicated that there has been very little research to document changes in language proficiency. Moreover, it is not clear to which extent the language develops, as it strongly depends on numerous variables like motivation, aptitude and the degree of immersion. In addition, according to Brown (2011) and Taillefer (2005), the studies on second language acquisition in a SA context that have been undertaken in the past focused predominantly on competencies related to listening and speaking. Besides, most of those studies were exclusively based on test-scores and therefore somewhat limited (Freed 1998). Only a few studies have examined the writing proficiency but most of them only concentrated on a few features. Arguably, there remains a need to study writing proficiency, in order to contribute to a more detailed description of language development in an SA context.
5
In this study three related questions are answered. The first research question is how the writing skills in Spanish as a Second Language (SSL) of students in an SA context develop in terms of complexity, accuracy and fluency. The hypothesis is that fluency and lexical complexity will develop most, based on findings of Möhle (1984), Raupach (1984, 1987), and Möhle and Raupach (1983) on development of speaking skills of German learners of French. In their study students gained most on fluency and vocabulary, while accuracy of grammar and grammar complexity did not change significantly. The same was argued by DeKeyser (1986, 1991), who found that American students in Spain did gain in fluency and vocabulary but not significantly in grammar.
The second research question is if there are differences in development by the three different levels: beginner, intermediate, and advanced. The hypothesis is that the lower the level, the more the students gain in writing skills. According to Freed (1998) there is support for the impression that greatest gains in an immersion environment are made by students with beginningly lower language proficiency. In addition, Thompson (1996) indicates that in an SA context there are not that many possibilities to practice language writing skills outside the classroom, and therefore development beyond an intermediate level will not increase significantly.
The third research question is which variables are related to the individual differences in development, as the Dynamic System Theory (DST) states that systems are embedded in other systems and systems and variables influence each other‟s change over time (Jordan 2004). Leki et al. (2008) made an inventory of factors that have been proven to play a role in L2 writing in English. Based on their work, the variables taken into account in this study are gender, L1, months in L2 context, previous knowledge of Spanish, attitude towards Spanish language and culture, self-evaluation of Spanish language skills and the extent of immersion.
6
and Davidson (1991: 115) state, the higher the scores on pre-grammar tests, the more likely students are to gain in all other skills. A positive attitude and a higher self-esteem on language skills will also positively influence language development (Leki et al., 2008). The last variable, the extent of immersion, is also supposed to play a role. From a Usage Based perspective, frequency of input leads to automatization and is thus the most important factor of language learning (Larsen Freeman 1976; MacWhinney 2008). Hernández (2010) found that students‟ amount of language contact appears to be significantly related to the development of speaking skills of Spanish in an SA context. Freed (1998) mentions that interaction of students with native speakers may be far less intense and frequent than was thought and that immersion in the SA environment might not be that uniform.
In general, the current study has employed a multi-dimensional research approach in order to provide a detailed view on the development of the writing skills of SSL in an SA context. The study involved 32 international ERASMUS students participating in an Intensive Spanish Language Course in San Sebastian on three different levels: beginning, intermediate and advanced. Data on their writing skills was collected by means of a written picture description task at the beginning and at the end of the course. The students‟ background, self-evaluation, attitude, and language use were identified by means of a questionnaire at the beginning and at the end of the program.
Given the scope of the present study, only writing skills have been analyzed. No attention was paid to the other skills like reading, speaking, and listening as those have already been studied extensively by other researchers. Unfortunately, the corpus is quite limited as many students were not present at one or both moments of testing. Only students who did both the pre and post test were included in the analysis in order to obtain a sound corpus.
7
8
2. Literature review
This chapter reviews the literature associated with the main areas of interest in this study. Section 2.1 describes the theoretical framework of this study, including the Dynamic Systems Theory and its application to Second Language Development. Section 2.2 identifies the research on Second Language Development during Study Abroad. In the last part of that section, attention is paid to a specific matter within this field: the development of L2 writing skills. Section 2.3 defines the constructs of CAF and indicates the most important studies in this area.
2.1. Dynamic systems theory and second language development
The major theoretical framework of this study is Dynamics Systems Theory (DST): a theory of change, which came into vogue in the 1960s. Originally, it was an application of mathematics, but since the late 1990s it has also been used to explain Second Language
Acquisition (SLA) (De Bot, Lowie, and Verspoor 2007). The three main characteristics of
DST are the existence of interacting dynamic subsystems, the tendency to self-organization, and the occurrence of non-linear chaotic patterns of development (Lowie 2011).
Systems are embedded in other systems and systems and variables influence each other‟s change over time. Interaction with the environment takes place, as well as internal self-organization, which eventually makes the system settle in a certain attractor state. The process of growth of a system is dependent on resources. Those resources comprise the learner‟s internal capabilities, motivational resources, but also external resources, like the learner‟s environment, time, and material resources.
9
As language acquisition is a nonlinear process, showing a lot of variation, DST usually does not incorporate traditional statistics, as those only indicate monocausal relationships. However, in the current study statistics are used. The reason therefore is that there were only two moments of measurements which does not allow for an exclusive dynamic analysis.
The above mentioned term acquisition was introduced by Krashen (1985) as counterpart of learning. As he states, acquisition occurs by means of an unconscious process, comparable to the way a child learns an L1, resulting in implicit knowledge. By contrast, learning implies a conscious process, leading to explicit knowledge about a language. Krashen also argues that those two processes are distinct and, more important, that what is learned cannot be transformed to something acquired. The case of the students in the current investigation is in that sense complicated. They attend an intensive language course, implying language learning, while they are living in the country of the target language, making them acquire language also outside the course, in daily life. It is impossible to distinguish what is learned from what is acquired (Fernández 1997). Therefore the neutral term Second Language Development (SLD) will be used.
Another syntagma that has to be mentioned is the difference between foreign
language and second language. The difference is that the former is learned outside the
country of the target language, whereas the latter is acquired in the country of the target language itself (Fernández 1997). Based on these definitions, the term used in this study is second language (L2) as it concerns a study abroad context.
2.2. Research on SLD in a study abroad context
10
German, Italian and Russian. Willis et al. (1977) examined the speaking, listening and reading skills of 88 British students in France or Germany and Dyson (1988) assessed the listening and speaking skills of 229 British students in France, Germany or Spain. The results of all these studies indicated that studying abroad does indeed contribute to greater proficiency. But they rely only on test scores to document the linguistic advantage, which makes the results somewhat limited.
Later on in the 1980s, the oral proficiency test came into vogue and much research was carried out focusing exclusively on oral proficiency, like the studies of Veguez (1984), Liskin-Gasparro (1984), Magnan (1986), O‟Connor (1988), Milleret (1990), and Foltz (1991). All those studies indicated that the oral L2 proficiency of students studying abroad developed significantly more than that of the control group that did not go abroad. However, the major shortcoming of these studies is again that they are mainly based on test scores and provide only very general results (Freed 1998). Therefore, multidimensional studies should be realized to study SLD in SA from various perspectives. Very recently, a good example of such a multidimensional study on the development of Spanish by American students in an SA context has been published by DeKeyser (2010). His study is multidimensional in the design of data collection, as different instruments were used: questionnaires, interviews, written proficiency tests, and aptitude tests.
Although not situated in a study abroad context, the study of Verspoor (2011) is also an excellent example of multidimensional research of L2 writing development. It is not multidimensional in the design of data collection, as only a writing task was used to gather data, but it is dimensional in the number of variables that was taken into account.
The current study therefore aims to adapt a multidimensional approach in both aspects: data collection and variables. It further tries to fill a gap in SLD research by focusing on L2 writing development, as L2 writing has not been studied sufficiently so far, despite its importance. Especially in today‟s world of digital media students more and more use written language in mail, chat, digital social networks and text messages.
2.3. Complexity, Accuracy and Fluency in second language development
11
developmental level (Wolfe-Quintero et al. 1998), filling the gap of the general studies exclusively based on pre- and posttest scores.
Complexity reveals the scope of L2 knowledge, while accuracy shows the conformity of this knowledge to target language norms. Fluency represents the accessing of that knowledge. CAF studies can be carried out on many linguistic levels, including lexicon, morphology, syntax, phonology, semantics, discourse or pragmatics, the first three of which are studied most. Often, the L2 development is related to proficiency which can be conceptualized in different ways like program levels, rating scales, standardized test, school levels, classroom grades, short term changes in classes and comparisons with native speakers. The former is used in this study, as recommended by Wolfe-Quintero et al. (1998), because it represents multi-trait assessment procedures focused on overall language proficiency, of which writing is a part.
2.3.1. Definitions of CAF
It is important to define precisely what is understood by the three terms of CAF.
Complexity is defined as the use of a wide array of vocabulary items and syntactic
structures (Caspi 2009). Accuracy is the comparison of the produced L2 writings with the target like use and represents therefore the degree of conformity to certain norms (Pallotti 2009). Fluency is defined as the number of words and structures used in a limited time (Wolfe-Quintero et al. 1998).
Another component proposed by Palotti (2009) is Adequacy. He is surprised about how few CAF studies report data about the degree to which learners‟ performances are more or less successful in achieving the task goals efficiently. Furthermore, adequacy should be considered in the interpretation of the CAF measures themselves. Many studies seem to assume implicitly that higher levels of CAF are „better‟, and that less CAF depends on limitations in language processing capacities. However, both assumptions are questionable. Despite the convincing reasons to report on adequacy, this is not examined in this study, due to the limited scope of it.
2.3.2. Measurements of CAF
12
studies. They compared the results across all the studies that used different measures, with the aim of identifying the most adequate ones. Their valuable conclusions have served as a guideline for this research.
Complexity is divided in grammatical complexity and lexical complexity. The former can be best analyzed by looking at clauses per T-unit and dependent clauses per clause or T-unit. T-units are defined by Ramírez (2000: 213) as a main clause with a subordinate clause attached to it. Norris and Ortega (2009) however commend that in the measurement of syntactic complexity, there are some metrics that are redundant if used together, because they tap the same measurable dimension of the construct and, conversely, there are other measures that are distinct and complementary and thus can be best used and interpreted together, because they tap different dimensions of complexity. They propose a multidimensional approach to lexical complexity, by including an analysis on sub clausal complexity (561-562). A prime example of a study on L2 writing, using this additional complexity measurement, is undertaken by Verspoor (2011).
Lexical complexity is about the range and size of vocabulary. Although Wolfe-Quintero et al. (1998) present evidence for the adequacy of the Type Token Ratio adapted for sentence length (W/√2W), invented by Carroll (1967), much critique on it is uttered in other studies. Therefore, Guoxing Yu (2009) and Skehan (2009) propose the measurement of D, a single parameter of a mathematical function that models the falling TTR curve, without being a function of the sample size. It is beyond the scope of this article to discuss D mathematically. In essence, the higher the D, the greater the diversity of the text is.
Accuracy could be analyzed by counting errors. The best measures are error free T units per T unit and Errors per T-unit. Shaw (1998) argues that improved correctness in the written texts is often balanced by errors in the students‟ attempt to use new structures. That is why accuracy on its own does not provide any information on what kind of errors are made. Therefore, an additional in-depth error analysis could be done, as will be discussed in the next section.
Fluency should count the number, rate and length of units. Adequate measures are words per T-unit, words per clause or words per error-free T-unit.
13
by Ramírez (2000: 213) to count predicates where each new construction with a finite verb is considered a T-unit, regardless of whether the verb has its own overt subject or not. The second option is proposed by Wolfe-Quintero et al. (1998) to use clauses as measure-unit. Both seem to be good options, but in this study the latter is adapted.
2.3.3. Expanding accuracy by an error analysis
An error analysis (EA) would provide more information on the nature of the errors made by the students. Error analysis had its peak in the late 1960s but soon received a lot of criticism later on. Hammarberg (1974) for example states that is too one-sided to analyze the errors and neglecting the careful description of the non-errors. To overcome this problem, in this study the error analysis is part of a larger analysis that does indeed take into account the non-errors.
Errors are defined as an involuntary transgression of the norm (Fernández 1997), or an unsuccessful bit of language (James 1998). In error analysis the basics are not a comparison between L1 and L2 but the real productions of the learners. In this way it is also suited for this study, where the researcher is not familiar with all of the L1s of the different students.
2.4. Summary
14
3. Methodology
This chapter introduces, and contains a discussion of, the methodological approach and research design best suited to examine the research questions set out in chapter one. A multi-method design is proposed in order to arrive to the answers of the research questions. An overview of the research design is given beginning with an outline of the key methods employed, namely, a written picture description task and a questionnaire survey. Given the importance of design and validity in the choice of research instruments, justification of each method used is provided. Section 3.1 includes a presentation of the variables involved in the analysis. In section 3.2 the specific process of data collection is described, followed by an overview of methods used for data analysis in section 3.3.
3.1. Variables to be measured
It is important to describe clearly which are the variables involved in this study. Therefore in this section definitions and operationalizations of the following variables are given; CAF, proficiency levels and the interacting intern and extern factors.
3.1.1. Complexity, Accuracy and Fluency
The first research question focuses on the development of writing skills in terms of CAF. In table 1, an outline of the operationalizations of each of the terms is provided.
CAF Operationalizations
Grammatical Complexity - Number of dependent clauses per clauses - Types of dependent clauses per clauses
(relative, nominal, adverbial, nonfinite)
- Types of sentences per sentences
(simple, compound, complex, compound-complex)
Lexical Complexity - Type Token Ratio corrected for text length (D) Accuracy - Number of error free clauses per clauses
- Number of errors per clause
Fluency - Number of words per clause
15
16
Main categories Subcategories
Verbs - Omitted verb
- Erroneous verb conjunction
- Erroneous use of reflexive pronoun with verb - Lack of agreement in number with the subject
Articles - Omitted article
- Superfluous article
- Wrong article (e.g. uno (undetermined) instead of el
(determined)
Gender agreement - Lack of agreement with gender in adjective - Lack of agreement with gender in article - Lack of agreement with gender in pronoun Number agreement - Lack of agreement with number in adjective
- Lack of agreement with number in article - Lack of agreement with number in pronoun
Pronouns - Erroneous pronoun
- Omitted pronoun Prepositions - Omitted preposition
- Erroneous preposition - Superfluous preposition
- Lack of subtraction of preposition and article Conjunctions - Omitted conjunction between clauses
- Erroneous conjunction - Superfluous conjunction Spelling - Erroneous use of accent
- Misspelling
- Erroneous use of space between words
- Erroneous use of masculine or feminine ending for noun - Erroneous use of singular or plural form for noun
Copula - Erroneous use of ser / estar / hay
Table 2. Types of errors coded for in the Error- Analysis
17
3.1.2. Proficiency levels
For the second research question, a comparison is made between the three groups. The independent variables involved are beginning, intermediate and advanced learners. The definition of these three proficiency levels is based on the division by program level which was realized by means of an intake-test consisting in a grammar cloze test combined with a written essay. For example, with fewer than 20 points obtained, a student was placed in the beginner‟s group. The Italians were corrected because of their similar L1, and should in this example obtain fewer than 25 points to be located in the same beginner‟s group.
3.1.3. Interacting variables on L2 development
In order to answer the third research question, both internal and external factors are taken into account in order to establish relationships between those variables and the L2 development. In table 3 below, an outline of the operationalization of each of the variables is presented.
Variable Operationalizations
Gender - Masculine / Feminine
L1 - Mother tongue of the student
Time in L2 context - Number of months in San Sebastian at the time of the pre-test Previous knowledge - Previous stay in a Spanish speaking country where the target
language was needed
- Attendance to previous Spanish course
Attitude - Score on 5-point scale on seven assumptions about Spanish language and culture:
Spanish is a beautiful language
Spanish is a difficult language
Learning Spanish is fun
Learning Spanish is hard work
I like the Spanish culture
I would like to become part of the Spanish society
I think it is important to know Spanish
Self-evaluation - Score on 5-point scale on knowledge of the four L2 skills Immersion - Score on frequency of reading and listening on a 4-point scale
- Score on frequency of writing and speaking on a 4-point scale
Table 3. Operationalizations of internal and external interacting variables
18
variables are straightforward to code for. The four last variables are operationalized as the total number of points awarded to each item. For example, for attitude, the total number of points awarded to the seven assumptions represents the attitude.
3.2. Data collection
In this section information will be provided on the data collection. First, the sample will be presented. Afterwards, the instruments for data collection will be discussed as well as the procedure of data collection. As already mentioned, Ferris (2004) has advocated that a multidimensional approach should be used to study written discourse. Therefore, this study adopted a multiple research approach, including an analysis of written texts in terms of CAF and an analysis of other interacting variables, by means of a questionnaire. Both instruments will be presented.
3.2.1. Sample
The participants were a group of international students (N=32) enrolled in a Spanish language course in San Sebastian during their study abroad. These learners (13 males, 19 females) were from a range of L1 backgrounds. Among them, 9 were Italian, 9 were German, 4 were Greek, 4 were Polish, 3 were Persian, 1 was English, 1 was Czech and 1 was Russian. 24 were between 20 and 25 years old and 8 were over 25. They were classified at three different proficiency levels. N=13 at beginning level, N=14 at intermediate level and N=5 at advanced level. These three groups were attending different classes according to their level. 16 of them had been for 2 or less than 2 months in San Sebastian at the time of the pretest and 6 of them had spent between 2 and 5 months in San Sebastian and 10 had been for 6 or more than 6 months there. 12 of the students had already been in a Spanish speaking country before, for varying reasons like work or holidays, while for the other it was their first time. 15 of the students already had some basic previous knowledge of Spanish, while the other 17 did not have any previous knowledge at all.
19
available: one for the beginning group, two for the two different intermediate groups and one for the advanced group.
3.2.2. Written picture description task
According to Verspoor (2011), writing samples show the use of active language in all its facets. In contrast with speaking, the learner can show better what he is capable of, because writing allows for more reflection and is therefore usually somewhat more complex. Another advantage is that it is easier to collect. Therefore, this study has opted for a written task.
A picture description task was chosen in order to ensure that the writings of the students would not differ too much in their contents, which would make a comparison less reliable. A successful example of data collection by means of a written picture description task is the study of Ramírez (2000).
A composition test is suitable for many reasons, according to Jacobs (1981: 3). The main reason is that it emphasizes the communicative feature of language, instead of mechanical or meaningless language-like behavior. However, there are some important basic criterions for the writing task. The task must be realistic (it should mirror the kind of composition tasks the students normally perform), appropriate (in terms of the topic), understandable, personal (the students must know something about the topic), feasible (writable), reliable (representative) and fair (not too difficult, nor too easy). The choice for the picture in this study (see appendix 1) is based on those requirements.
Ideally the test would not be timed, so that even the slowest writers get the maximum opportunity to show their abilities (Jacobs 1981), as is the case in this study. The students were asked to write a minimal of 25 lines. The task for de beginning students was adapted to their level by putting circles with numbers around the different activities. This would make it easier to refer to the different situations as no conjunctions and pronouns were needed. The students were asked to describe the place, the people, and the things happening in the picture. They were also free to write a short story about the characters in the picture.
20
The same picture description task was given twice, after the students filled in the questionnaire. As Thelen and Corbetta (2002: 61) state, using exactly the same task several times is one way of dealing with the fact that even subtle differences in a task can affect performance profoundly. An example of successful repeated task design is the study of Larsen Freeman (2006) on CAF development by Chinese learners of English.
3.2.3. Questionnaires
The main aim of the questionnaire was to obtain data about the students‟ gender, L1, time abroad, previous knowledge of Spanish, attitude towards Spanish, self-evaluation of Spanish skills and the extent of immersion in Spanish language (see appendix 2 and 3 for the questionnaires). The use of a questionnaire to collect biographical information on students and quantifiable data on language abilities, practices, and attitudes is proposed by Li Wei et al. (2008: 158). The disadvantage is that it does not provide data on performance and that the directive mode of questioning and the close-ended nature of the answers constrains the depth of the acquired knowledge (Li Wei et al. 2008: 171). This is remedied in this study by combining it with additional data collection by written essays. Two other downsides, mentioned by Li Wei et al. (2008) are, among others, the literacy difficulties, the official status of a written questionnaire which will make the students reluctant to operate, and that the researcher does not know who completed the questionnaires, if they are not filled in on site. These disadvantages are solved in this study by translating the questionnaire to English as well, to take away literacy difficulties and enhance participation. Furthermore, the researcher was present at the time the students individually filled in the questionnaires, which were returned immediately after completion, so it is known who filled them in.
The most important thing in questionnaire design is that they are short (four pages as a maximum). They should not take more than 30 minutes to be completed (Dörnyei 2003), because otherwise the informants will get tired and start answering randomly. Because of this argument, the questionnaire used is not longer than two pages, as they also had to complete the writing task at the same day.
21
Including some open-ended questions is also advised as they may provide extra insights. In this study most questions are close-ended with a few additional open questions. The answers on the open question about the students‟ experience on language learning will be used in the discussion chapter to support certain outcomes of the analysis. Multiple choice scales and ranked scales are both used in this survey. A five point scale is considered broad enough by Li Wei (2008: 173). For each scale it has to be indicated what the numbers stand for. Moreover, a variation of formulations is recommended. Therefore this survey contains interrogative sentences, directives, which are used to obtain factual information, and declaratives, which are used to gather data on values, attitudes and beliefs, as these are better asked indirectly.
The questionnaire was filled in two times, once at the pretest at the beginning of the course and once at the posttest 10 weeks later in the last week of the course. The questionnaires were roughly the same except for two things: the posttest did not contain questions on unaltered background data and in the posttest some questions were added on the students‟ experience of Spanish language learning and their evaluation of the change of their Spanish skills during this course.
3.3. Data analysis
As a first step, the handwritten texts of the students were typed in CHAT-format according to the conventions proposed by MacWhinney (2008). Then each sentence was coded for sentence-type, clause-type, and errors. An example of the first part of a coded text of an intermediate student is presented in figure 1 below.
Figure 1. Sample of first coded sentence of an intermediate student.
@Begin
@Languages: spa
@Participants: STU 1 Student
@ID: spa|pretest|STU|30;|male|intermediate||Student|| @Font: Courier New
@L1 of #: per
*STU: creo [^c] que las numb mujeres, una rubia y una negra, en el parque hablan sobre sus maridos [^c] en cambio [*] de tener [*] cuidado para [*] los niños [^c] , tal vez hablan sobre la peloqueria [*] [*] [^c] [*] que fueron la ultima [*] vez [^c] .
%syn: complex
%xcla: nominal, nonfinite, adverbial,
22
First, background information on the student was provided in the header lines, preceded by @. Then each sentence the student wrote was presented in a main line, preceded by *STU. In that sentence, any clause was identified by [^c] and the errors by [*]. Then three dependent tiers were added; in %syn, information on the type of sentence was entered, in %xcla, information on the types of dependent clauses was entered and in %err, the types of errors were coded. Digits were replaced by the code „numb‟ and words that were unreadable were replaced by the code „xxx‟, and both were excluded from analysis. Then, the CLAN-program was used to perform analyses on the different variables. Frequency analyses were carried out for the types of clauses, sentences, and errors. The variable D was calculated by CLAN as well. All the outcomes of the CLAN-analysis were entered in SPSS. Also the answers of the questionnaires were added to SPSS.
All that data together formed a large amount of variables to be examined. First it had to be made sure that the measurements of CAF are reliable and that the measurements indeed reveal something about the level of Spanish L2 writing skills and are able to differentiate between the groups. Therefore a correlation analysis between the pre-test, post-test and group was used. If the measurement is reliable, it should show a high correlation on all these items with a Spearman Coefficient of ≥0,3. For all the reliable measurements, a progress variable was created by subtracting the post-test minus the pre-test. All the progress variables were tested with a one-sample T-test for significance. The alpha-level was set on 0,05 for all tests in this investigation.
Next, one general measurement for Writing Proficiency needed to be created by combining the separate linguistic variables. The Cronbach‟s Alpha was taken into account to make sure that there was an internal consistency in the variables. Negative item correlations were corrected by reversing them. Then, the mean scores for each variable, divided by their standard deviation of the pre-test, are summed up separately for the pre-test and the post-test. Again a progress variable was created by taking the subtraction of the post-test minus the pre-test. A one sample T-test was then used to show significant progresses.
23
In the last step, factors that might be related with language development were studied. If necessary, variables of the questionnaires, representing the same factor, were combined after checking internal consistency with Cronbach‟s Alpha. In order to find correlations between the different factors and the progress in writing proficiency, Pearson Correlation tests were carried out.
3.4. Summary
24
4. Results
An analysis of research data gathered by means of the written picture description tasks and the questionnaires is presented in this chapter and the three research questions, as posed in chapter one, are reiterated and addressed separately. First, the analysis of the L2 writing skills development in terms of CAF will be presented in section 4.1. After this, a comparison of the progresses in terms of CAF between the three groups will be made in section 4.2. Finally, the results of the analysis on the interaction between different factors and CAF development will be presented in section 4.3.
4.1. Development in Complexity, Accuracy, and Fluency
As stated in chapter one, the first research question was directed to an examination of the development of writing skills in SSL of students in an SA context, in terms of complexity, accuracy and fluency
As a first step, the reliability of the CAF measurements for progress in writing skills was tested with the Pearson Correlation Coefficient to identify relationships for each variable between pre-test scores, post-test scores and the variable „groups‟. Only the variables that showed a strong correlation of R ≥ 0,05, on all of these three were used for analysis. Then a T-test was carried out for the reliable measurements in order to discover significant progresses in the distinct variables. Additionally, a coordinating variable was created by adding up the different reliable measurements. Therefore, internal consistency was guaranteed by running a Factor-Analysis to examine the Cronbach‟s Alpha. To balance the items, these were all divided by their Standard Deviation (SD) of the pre- test and then added up in SPSS.
First, the Pearson Coefficients were calculated for each variable on pre-test, post-test and group. The H0 that there is no correlation between them was tested. The results are
25 CAF R (pretest-posttest) R (group-pretest) R (group-posttest)
Dependent clauses per clause 0,715 0,486 0,352
Relative clauses per clause 0,456 -0,086 -0,70
Nominal clauses per clause 0,781 0,610 0,535
Adverbial clauses per clause 0,559 0,611 0,650
Nonfinite clauses per clause 0,159 0,462 0,072
Simple sentences per sentence 0,413 -0,413 -0,134
Compound sentences per sentence 0,793 -0,035 -0,017
Complex sentences per sentence 0,491 0,355 0,082
Compound-Complex sentences per sentence 0,487 0,190 0,103
D 0,726 0,766 0,598
TTR 0,278 -0,046 -0,060
Error free sentences per sentence 0,525 0,269 0,395
Errors per clause 0,671 -0,490 -0,610
Verb errors per clause 0,046 -0,104 -0,230
Article errors per clause 0,392 -0,136 -0,016
Pronoun errors per clause -0,168 -0,233 -0,306
Preposition errors per clause 0,540 -0,169 -0,195
Conjunction errors per clause 0,581 -0,002 -0,160
Spelling errors per clause 0,605 -0,631 -0,644
Word errors per clause 0,251 -0,346 -0,461
Copula errors per clause 0,150 -0,217 -0,021
Agreement errors per clause 0,268 -0,180 -0,324
Words per clause 0,231 0,172 0,244
MLU 0,677 0,346 0,403
Table 4. Strength of correlations with pre-test, post-test, and group for the different CAF-variables
The Pearson test revealed that there was a significant positive relationship between the pre-test, post-test and group for the next variables; dependent clauses per clause, nominal
clauses per clause, adverbial clauses per clause, D, errors per clause, and spelling errors per clause, at p<0,01 (two-tailed). These are the variables that apparently are reliable
measurements for CAF and will therefore be further examined. It has to be noticed that TTR, as was expected, in this case indeed was an unreliable measurement and that the correction for sentence length, as realized in the variable D, is necessary for a reliable measurement of lexical diversity.
26
variable was created as the subtraction of the post-test score minus de pre-test score for each variable. The H0 that there is no progress in each of these variables was tested.
CAF Mean Progress
Dependent clauses per clause 0,0418
Nominal clauses per clause 0,0014
Adverbial clauses per clause 0,0096
D 8,4219
Errors per clause -0,0564
Spelling-errors per clause -0,0144
Table 5. Average progress between pre-test and post-test for each of the variables.
A one sample T-test revealed that there was only a significant difference in pre-test and post-test for D with M= 8,42, SD= 16,84; t(31)= 2,83, p=0,008. That means that on average D increased during the course for all students.
As there are many linguistic variables involved, it would be useful to combine them into one general measurement of writing proficiency. The measurements that appeared to be adequate and reliable for measuring development of language skills between groups were therefore condensed to one general measurement by means of a factor-analysis. The factor-analysis revealed that the error variables show a negative item rest correlation. That is legitimate as the higher the level, the fewer the errors the students are supposed to make. Those error variables were therefore reversed in order to obtain positive correlations, and the analysis was performed again. The general measurement, which will be referred to as
Writing Proficiency, appeared to have a good internal consistency as the standardized
Cronbach‟s Alpha Coefficient reported equals 0,91. The variable that showed the highest correlation with Writing Proficiency is D with a correlation of 0,73 for the pre-test and 0,74 for the post-test. To create a more balanced variable, the incorporated variables were all divided by their Standard Deviation of the pre-test. Two new variables were created:
Writing Proficiency Pre-test and Writing Proficiency Post-test. The Cronbach‟s Alpha of
27
correlations were 0,84 and 0,74 for Writing Proficiency. Writing Proficiency thus turned out to be even better than D.
To make sure that Writing Proficiency Pre-test and Writing Proficiency Post-test are reliable measurements, a Pearson Correlation test was used. Respectively the standardized Cronbach‟s Alpha was 0,81 for the pre-test and 0,86 for the post-test. These alphas were considered reliable. The correlation between pre-test and post-test was 0,86.
Once the variable Writing Proficiency was proven to be reliable, the progress was operationalized in a variable as Writing Proficiency Post-test minus Writing Proficiency Pre-test. Then, a one sample T-test was conducted to evaluate the development for the general writing skills. There was a statistically significant increase in score between pre-test and post-test with M= 1,04, SD=2,46 which was significant at t(31)=2,4, p=0,02 (two-tailed). This means that on average the writing proficiency of the students increased during the course.
4.2. Differences between the groups
The second research question considered whether there are differences in development of writing skills between the three groups. This question was analyzed using one sample T-tests per group. The variables that in the previous section appeared to develop significantly, which are D and writing proficiency, were examined in order to assess the statistical significance of differences in development among the three groups.
28
Figure 2. Average development of D per group
From figure 2 it is apparent that all groups show development in terms of D. The advanced group has the highest starting level, followed by the intermediate group and the beginning group starts with the lowest level of lexical variety. The same goes for the level at the end of the course. Nevertheless, there is a difference in the amount of growth. The group that demonstrates the greatest development is the intermediate group, as they advanced from 53,30 at the pre-test to 64,11 at the post-test (growth of 10,81). A one sample T-test on the progress of D for the intermediate group revealed that this progress was significant at t(13) = 2,22; p= 0,045. Then, the beginning group follows with a development from 37,74 to 46,46 (growth of 8,72). However, a one-sample T-test showed that this growth was not significant at p<0,05. The advanced group shows the smallest growth from 86,47 at the pre-test to 87,43 at the post-pre-test (growth of 0,96). This development was not significant at p<0,05, as revealed by a one sample T-test. It can be thus concluded, that despite of the apparent growth of D for all groups, only the growth in the intermediate group was significant.
Then, for writing proficiency the differences in development between the three groups were examined. First, the development in writing proficiency is visualized in the graph in figure 3 on the next page.
29 -2,10 -1,40 3,52 5,34 7,83 7,60 -4 -2 0 2 4 6 8 10
Writing Proficiency
Initial Intermediate AdvancedFigure 3. Average development of writing proficiency per group
From figure 3 it is apparent that only the beginning and intermediate groups show an increase of writing proficiency whereas the advanced groups shows a slight decrease. The advanced group has the highest starting level, followed by the intermediate group and the beginning group starts with the lowest level of lexical variety. The same goes for the level at the end of the course. The intermediate group again shows the highest development of writing proficiency from 3,52 at the pre-test to 5,34 at the post-test (an increase of 1, 82). A paired sample T-test on writing proficiency for this group of the pre-test and post-test scores revealed that this increase is significant with t(13)=2,9, p=0,01. The beginning group shows an increase from -2,10 to -1,40 (an increase of 0,70). A paired sample T-test showed that this growth was not significant at p<0,05. The advanced group showed a minimal decrease of writing proficiency from 7,83 at the pre-test to 7,60 at the post-test (a decrease of 0,23). A paired sample T-test showed that this decrease was not significant at p<0,05. It can therefore be concluded that only the intermediate group showed a significant development in writing proficiency.
4.3. Factors involved in development of SSL
30
stay in a Spanish speaking country (yes/no), attitude towards Spanish, self-evaluation of Spanish language skills and the degree of immersion. The correlation of those factors with D and writing proficiency was examined by means of a Pearson Correlation test.
First, the Pearson test revealed that there was no significant relationship between gender and writing proficiency nor between gender and D at p<0,05. Neither was there a significant relation found for age, L1, and previous knowledge at p<0,05. This means that gender, age, L1 and previous knowledge in this case cannot account for the individual differences in development.
The number of months the student already had been in San Sebastian at the time of the pre-test did seem to correlate with both writing proficiency and D. A Pearson Correlation Test revealed that there was a significant positive relationship between the number of months the student had stayed in the L2 country at the moment of the pre-test and the development of writing proficiency, r=0,44; p=0,01. The longer the students already spent in San Sebastian, the more their writing proficiency developed during the course. The same is true for the development in lexical complexity. A Pearson Test revealed a significant positive relationship between the number of months the student had already been in San Sebastian and the development of D, r= 0,40; p=0,02. The longer the students already spent in San Sebastian, the more their lexical complexity developed during the course.
31
In the questionnaire the students were also asked to evaluate their competencies for each of the four skills at a scale from one to five. A reliability test on all these items from the pre- and post-test revealed a standardized Cronbach‟s Alpha of 0,90 which indicates a high internal consistency. General self-evaluation was operationalized as the subtraction of the post-test variables divided by the pre-test SD minus the pre-test variables divided by their SD. A Pearson Correlation test showed no significant relationship at p<0,05. However, in the questionnaire another type of self-evaluation was realized by asking the students on how they thought they developed on the following items: attitude, motivation, confidence, grammar, vocabulary and orthography. A Pearson Correlation test revealed that there is a significant positive relationship between the self-evaluation of development of vocabulary knowledge and the development of D, r=0,393; p=0,03 (two-tailed). The more the students think they have increased their vocabulary knowledge, the higher the actual progress on D is.
The last factor to be analyzed was the amount of immersion in the Spanish language context. First there is the receptive immersion, that is the input the students receive by reading and listening. Four types of input were distinguished: reading books, reading the newspaper, listening to music and watching television in Spanish. The Cronbach‟s Alpha coefficient showed a reliable internal consistency between the variables of 0,80. The four types of input were combined into two variables; total input for pre-test and total input for post-test. There appeared to be a significant positive relationship between the amount of receptive input at the time of the post-test and the progress in Writing Proficiency, r=0,344; p=0,05. For productive input no significant relationship was found at p<0,05.
4.4. Summary
32
33
5. Discussion
This chapter provides a detailed analysis of key findings presented in chapter 4, with reference to each of the research questions. The results of the study are also discussed in relation to previous research studies. Section 5.1. discusses the development of complexity, accuracy and fluency in writing skills. In section 5.2. differences in this development between the groups are discussed. Finally, section 5.3. presents the factors that are related with individual differences in development..
5.1. Development in Complexity, Accuracy, and Fluency
The first research question was directed to an examination of the development of writing skills in Spanish as an L2 of students in an SA context, in terms of complexity, accuracy and fluency, as proposed by Wolfe-Quintero et al. (1998). The different terms of CAF were first examined for reliability and then tested for significant development between the pre-test and post-pre-test. Also a general measurement of writing proficiency was created by adding up the reliable measurements of CAF and was then tested for significant progress.
34
Also in the questionnaires answers were found supporting the outcomes:
“Thanks to the classes of the different assignments in Spanish, I read more articles in Spanish and learned more vocabulary.”
(Advanced student, Poland) “I‟ve got more problems with learning the subjunctive and the past tenses in Spanish.
The Spanish language course helped me a lot to improve my vocabulary.”
(Intermediate student, Poland) “Every day I learn new words and I like that, but I do have some difficulties with verb
tenses.”
(Beginning student, Italy)
These students all mention that they learned new vocabulary and comment on the fact that the difficulties they have, have got to do with grammar. It can thus be stated that lexical complexity of SSL on average increases significantly during the SA.
It should not be surprising that not all the elements of language development show a significant growth. From a DST point of view, language development is nonlinear and the growth of the subsystems is dependent on the various resources of the student. Perhaps the conditioning resources for lexical complexity development were more favorable than the others, for example more input of vocabulary than grammar. At the same time, it might be the case that the students focus on vocabulary and cannot pay attention to the other elements of language at the same time.
A T-test also revealed that writing proficiency in general developed significantly. During the study abroad the students did gain in writing skills. This was to be expected as Freed (1998) already stated that it is proven that general language proficiency develops in SA contexts. She additionally confirmed that unfortunately it is not clear to what extent this development takes place. In the present study it is demonstrated that although general proficiency develops, when zooming in to the different components of language development, only lexical complexity develops significantly.
5.2. Differences between the groups
35
environment is made by beginning students. Thompson (1996) also found that in an SA context, development of writing skills beyond the intermediate level will not increase significantly. In the current study, separate T-tests per group were used to identify progresses in the variables D and writing proficiency, as those had been proven reliable measurements. The T-test revealed that only the intermediate group showed significant progress on both variables, whereas the other two groups did not. So the intermediate group showed a higher lexical complexity and overall writing proficiency at the post-test compared to the pre-test, whereas the other groups did not show an increase. This fits Thompson‟s (1996) finding that beyond the intermediate level no significant development will take place. An advanced student even remarks the same in his questionnaire:
“If one doesn‟t study a lot on his own at home, after a certain level the knowledge won‟t change a lot”
(Advanced student, Germany)
However, according to Freed (1998) it should be expected that the beginning group also showed significant development. Nevertheless, Van Geert (1994) advocated that in the patterns of language development a lot of variability can be found, especially at the start of the development. This might account for the fact that in the beginning group, where a lot of variability is found, no significant progress is noticed as it is obscured by the amount of variability.
From a DST point of view, the growth of subsystems is supposed to reach a certain attractor state. Apparently in the intermediate level the writing skills reach that attractor state and that is where the result of progress becomes visible.
It can thus be concluded that the intermediate group shows significant progress in overall writing proficiency and on lexical complexity, whereas the other groups did not show significant progress.
5.3. Factors involved in development of SSL
36
knowledge of Spanish, time in L2 context, attitude, self-evaluation of skills and the extent of immersion. A Pearson Correlation Test was used to discover significant relationships between those variables on the one hand and general writing proficiency and lexical complexity on the other hand. The test revealed that some factors did indeed show a correlation. First of all, the progress in self-evaluation of vocabulary knowledge correlated with the progress of D. This means that the more a student thinks he/she improves on vocabulary knowledge, the more he/she in reality does and the other way round. This seems to imply that the students have rather good self-evaluation skills. It could be interpreted as a confirmation of the significant progress of D found by the statistical analysis.
Also the progress in attitude towards Spanish language and culture was related with a progress in D. The more the attitude of the students developed positively during the course, the more they gained on lexical complexity. This matches with the findings of Leki et al. (2008). Furthermore, the immersion of receptive input did show a positive and significant relationship with D. From a usage-based perspective this was to be expected as frequency of input is supposed to lead to automatization and is the most important factor of language learning. A student confirms this, writing:
“It helped me a lot to see movies on the television and listen to the radio.”
(Beginning student, Italy) Freed (1998) might have been right in stating that the interaction with native speakers may be far less intense and frequent than is thought. That could account for the fact that productive immersion is not related to SSL development. This is also confirmed by some students, like the following:
“I didn‟t learn a lot, or could have learned more, but I have got a lot of German friends with whom I speak German.”
37
“In the beginning, when I just arrived, I really wanted to know only Spanish speaking people and I really found it important to live with Spanish teenagers. At the end, this was not the case and I have been much time with other German people and ERASMUS students. I found that a little bit annoying but on the other hand I really love them and that‟s why in the end I wouldn‟t change it. But this is also the reason why I don‟t speak a lot of Spanish and don‟t really improve my Spanish level.”
(Intermediate student, Germany) This probably illustrates well what Freed stated. The immersion context is not that intense as it might seem, as many international students usually spend their time together with little context with native Spanish speakers.
The final factor that showed a correlation with both language proficiency and D was the number of months spent in San Sebastian at the time of the pre-test. It is assumed that these students have received more input over time and from a Usage Based perspective that supposes language automatization. Another thing that has to be taken into account is that some of the students that spent already a semester in San Sebastian, might have been participating already in a previous language course, like the next student:
“This semester my Spanish course looked different, compared to the language course of the first semester. The first semester we did more on grammar and this semester we talked more. I think this is the best method for me to learn Spanish. From the first course I obtained a good knowledge of grammar, and in this course I can talk about almost every subject.”
(Advanced student, Poland)
This student has done a previous course in his first semester in San Sebastian. He has received, apart from more input, also more instruction on SSL than the students that just arrived.
It can thus be concluded that the factors self-evaluation of vocabulary knowledge, progress in attitude towards Spanish, and the immersion of receptive input is positively related with lexical complexity development, and that the time in San Sebastian at the moment of the pre-test is positively related with both lexical complexity and general writing proficiency.
38
of the statistical significances, it should therefore be taken into account that the influences of the mentioned factors may exist only in combination with the other factors and that the influences may also change over time.
5.4. Summary
This chapter has summarized the present study‟s findings and discussed them with reference to each of the research questions. The results have also been considered in relation to relevant previous studies.
39
6. Conclusion
This chapter presents a summary of the research questions and the corresponding key results, followed with a consideration of practical and research implications in section 6.1. Subsequently, in section 6.2., the limitations of the study are assessed and recommendations for further research are provided.
6.1. Key findings and implications
The primary objective of this study was to investigate development in Spanish writing proficiency of ERASMUS students in San Sebastian. Therefore, three research questions were stated. The first research question examined the development of complexity, accuracy and fluency in SSL. The second question concerned the differences in development between the three groups beginning, intermediate, and advanced. The last question studied the factors involved in individual differences in language development.
The study was carried out in San Sebastian, in the Basque Autonomous Community, in beginning, intermediate, and advanced language classrooms for ERASMUS students. A multidimensional approach was adopted by using multiple instruments, a questionnaire survey and a written picture description task, and by involving several variables.
A preliminary finding was that on average all the students did develop their general writing proficiency and more specifically the lexical complexity. If a distinction was made between the groups, only the intermediate group appeared to develop significantly. The factors progress in self-evaluation of vocabulary knowledge, growth of attitude towards Spanish language and culture, immersion by receptive input and the time spent in San Sebastian at the moment of the pre-test, appeared to be correlated with lexical complexity and the latter also with general writing proficiency.
40
for the whole group, but grammatical complexity did not and neither did accuracy and fluency.
Therefore, teachers might consider paying more attention to grammar. Moreover, the ERASMUS program could consider extending the term of the exchange program, which is usually for only one semester, as this studied provided evidence that the amount of time was significant related to language development.
6.2. Limitations and recommendations
While the data from this research provides valuable insights on language development in an SA context, it has to be acknowledged that such findings are preliminary owing the relatively small sample size (32 participants). This limitation prevents a clear generalized statement about language development in a study abroad context. Moreover, the groups were very heterogeneous with students of different mother tongues. On the one hand, this does represent the actual situation of studying abroad, but on the other hand, it might obscure the outcomes of the analyses.
Another limitation that has to be mentioned is the number of measurements as in the present study only two moments of testing were realized. However, to consider language development from a DST perspective, it is recommended to collect data at more than two moments, in order to study the change of language over time.
In addition, it has to be noted that findings addressing writing do not necessarily apply equally to other language skills. Therefore, no generalizations should be made to the general language development of the students.
The multidimensional approach adopted in this study has facilitated the examination of language development from different perspectives. Further research incorporating a similar design and investigating a larger sample size would be of value. Besides, it would also be recommendable to holistically score the essays. The division by program level might be not sufficiently representative for the actual level of proficiency. An additional holistic evaluation could create a better division by proficiency levels.
6.3. Summary
41
abroad context. 32 International students of three different levels participated. Data was gathered by means of a written picture description task and a questionnaire survey.
The most useful finding from this study was that general proficiency did develop on average for all students. By looking at the different elements of complexity, accuracy and fluency, only lexical complexity appeared to develop. Differentiating between the groups, only the intermediate group showed development, whereas the other groups did not. There were also some variables that were related with general writing proficiency development and development of lexical complexity. The development in self-evaluation of vocabulary knowledge, the positive development of attitude towards Spanish language and culture, and a high immersion by receptive input were positively related with a progress in lexical complexity. The time spent in the L2 context was positively related with lexical complexity and general writing proficiency.
