• No results found

MA Thesis The Initial Validation of a Test of Emergent Literacy (TEL)

N/A
N/A
Protected

Academic year: 2021

Share "MA Thesis The Initial Validation of a Test of Emergent Literacy (TEL)"

Copied!
114
0
0

Bezig met laden.... (Bekijk nu de volledige tekst)

Hele tekst

(1)

MA Thesis

The Initial Validation of a Test of

Emergent Literacy (TEL)

Sophie Gruhn

Student number: S2518589

MA in Multilingualism

Departments of Applied Linguistics and Frisian Language and Culture

Faculty of Arts

Rijksuniversiteit Groningen

Supervisors:

Prof. Albert Weideman

Dr. Nanna Hilton

(2)

Acknowledgements

I would like to thank my supervisors, Prof. Albert Weideman and Dr. Nanna Hilton, for their supervision and support with this study. Further, I would like to thank the

Inter-institutional Centre for Language Development and Assessment (ICELDA), as well as the

Faculty of Arts of the Rijksuniversiteit Groningen for the opportunity to do an internship that preceded this study.

I would like to express my gratitude to the Marco Polo Grant programme and the Rijksuniversiteit Groningen for their scholarships that enabled me to do an internship with ICELDA on which I could gather the data for this study. Moreover, my gratitude goes to the PROMOS scholarship programme of the Leibniz Universität Hannover, which enabled me to follow the Master´s programme in Multilingualism at the Rijksuniversiteit Groningen.

My special gratitude goes to the experts in the field of language test design and early childhood education, as well as the pre-schools in Bloemfontein, South Africa, that participated in the evaluation and piloting of the Test of Emergent Literacy (TEL):

 Brebner School

 Christian Brothers´ College  Little Professor Nursery School  Eunice School

(3)

Abstract

(4)

Contents

1. Background and Context of the Problem ... 1

1.1 The Need for Equivalent Tests of Emergent Literacy in Multilingual Societies ... 1

1.2 Construct of Emergent Literacy ... 2

1.3 Language Test Design and Validation ... 4

1.4 Overview ... 5

2. Emergent Literacy ... 8

2.1 Literacy Development ... 8

2.2 Early Predictors of Prospective Literacy Underachievement ... 10

2.3 Existing Literacy Assessments ... 12

2.4 Conclusion ... 17

3. Phases and Principles of Language Test Design ... 19

3.1 Phases of Test Design ... 19

3.2 Principles of Language Test Design ... 20

3.3 Conclusion ... 25 4. Research Question ... 26 5. Methodology ... 28 5.1 Design Process ... 28 5.2 Initial Validation ... 29 5.2.1 Panel discussion……… ... 29 5.2.2 Pilot stage...………...31

6. The Design Process: From Construct to Task and Item Specification ... 36

6.1 Construct Articulation ... 36

6.2 The Task and Item Specifications ... 41

7. Results and Discussion ... 46

7.1 Overview of the Pilot Results ... 46

7.2 Evaluation of the Test in Terms of the Principles of Responsible Test Design ... 48

8. Refinements... 57

8.1 Refinements to the Overall Test ... 57

8.2 Refinements to Subtasks ... 58

8.3 Simulated Performance of the Refined Test Version ... 63

(5)

Appendix Q ... 96 Thesis agreement ... 97 References ... 98

List of figures and tables

Figure 1 Stages of literacy development 9

Figure 2 Test design cycle 19

Figure 3 Criteria of responsible test design 21

Figure 4 Histogram showing the frequency of the total scores for the Test of Emergent Literacy tasks 1 to 5

47 Figure 5 Scatterplot of inter-item correlations for the Test of Emergent Literacy

tasks 1 to 4

50 Figure 6 Estimated reaction of children during the assessment of the Test of

Emergent Literacy in an online evaluation by a panel of experts

55 Figure 7 Differential item functioning between female and male participants for

their responses to item 7 (task 1) and item 26 (task 3) of the Test of Emergent Literacy

56

Figure 8 Examples for the reading direction in order to find the wrong picture in the puzzles of the Test of Emergent Literacy task 2 Organising

information

61

Figure 9 Histogram showing the distribution of the Test of Emergent Literacy scores for the refined tasks 1 to 4

64

Table 1 Examples of literacy assessments for pre-school children (4- to 6-year-old)

13 Table 2 Participant information for the online evaluation of the Test of Emergent

Literacy

30 Table 3 Participant information from the first pilot of the Test of Emergent

Literacy

32 Table 4 Descriptive statistics of the total score from tasks 1 to 5 of the Test of

Emergent Literacy

46 Table 5 Summary statistics for the Test of Emergent Literacy tasks 1 to 5 47 Table 6 Reliability statistics of the Test of Emergent Literacy 49 Table 7 Subtask intercorrelations for Test of Emergent Literacy 50 Table 8 Desired ranges for the test item statistics per subtask of the Test of

Emergent Literacy

52 Table 9 Descriptive statistics for the tasks 1 to 4 of the Test of Emergent

Literacy

53 Table 10 Descriptive statistics for task 5 Emergent writing of the Test of

Emergent Literacy

53 Table 11 Descriptive statistics of the refined tasks 1 to 4 of the Test of Emergent

Literacy

(6)

1 1. Background and Context of the Problem

According to Naudé, Louw, and Weideman (2007), language and communicative skills are essential for the educational success of children. That is why developmental problems should be diagnosed as early as possible to introduce appropriate support, even at pre-school level (Naudé et al., 2007; Washington, 2001). Especially emergent literacy skills build the foundation for subsequent academic achievement (e.g. Catts, Fey, Zhang, and Tomblin, 2001; Duncan et al., 2007). Therefore, the identification of literacy problems prior to school entrance is highly relevant (Shanahan, 2008).

1.1 The Need for Equivalent Tests of Emergent Literacy in Multilingual Societies A strong need for the assessment of emergent literacy abilities at pre-school level exists particularly in multilingual societies, because children who speak a non-standard variety or have limited abilities in the language used in education, are at higher risk of literacy underachievement (Snow, Burns, and Griffin, 1998, pp. 15-40). For example, 51% of African American English speaking children and 49% of children with a Hispanic migrant background in the United States failed in 2011 to achieve basic reading levels in Standard American English, compared to 22% of their white peers (National Centre for Education Statistics, 2011, p. 15). To account for the so-called black-white gap, differences in socioeconomic status and early literacy practice in families are considered (Connor and Craig, 2006; Washington, 2001). Similarly, in 2011 12.5% of children without a migration background in Germany reached the highest reading proficiency level in the Progess in

International Reading Literacy Study (PIRLS), whereas only 4.0% of children with a

(7)

2 other European countries (Müller and Stanat, 2006; OECD, 2014; Schnepf, 2007). This illustrates why the early identification of emergent literacy abilities has wide relevance globally.

A multilingual setting for education leads to a need for language assessment that is equivalent in terms of the measurement across several languages (Hoff, 2013; Scharff Rethfeldt, 2013, pp. 133-134). This is relevant, for example, in South Africa where 11 official languages are recognised in the foundation phase of education (Republic of South Africa, 2012). Equivalence refers here to language assessments with equal levels of measurement in various languages, or a test that provides comparable results if administered to speakers of different languages, in other words is unbiased against different groups of examinees, which in turn increases the fairness of the language assessment (Koch, 2009; Kunnan, 2000).

1.2 Construct of Emergent Literacy

(8)

3 175). Thus, it is assumed that children start to acquire literacy directly after birth due to environmental experiences and that “their early language, their scribbles, their exploration of books, their interest in environmental print, their interactions with technology” are part of the literacy learning process (Makin and Whitehead, 2004, p. 10). Furthermore, literacy acquisition happens in the form of social practice and communication within the environment and communities of children (Makin and Whitehead, 2004, p. 10).

As stated by Suchodoletz (2005, p. 218) as well as by Lonigan, Allan, and Lerner (2011), recent emergent literacy tests focus on the assessment of pre-conditional skills for literacy acquisition. This seems to be aligned with the conservative ‘skill’ and ‘readiness’ approach on emergent literacy (Baker, 2011, p. 313; Makin and Whitehead, 2004, pp. 9-10). While these will be surveyed again in chapter 2, it seems that existing diagnostic instruments assess only a limited range of emergent literacy components, neglecting the influence of the environment on literacy development, and considering these abilities independently from each other, and in non-communicative settings.

(9)

4 by the context, with discourse types varying in different contexts, dependent on the purpose of communication and the different social relationships that exist between the communication partners (Weideman, 2011b, pp. 60-65).

There is therefore a need for a new definition of a construct for an emergent literacy test, which takes as its starting point an open and communicative perspective on what constitutes language, and considers from a more recent approach on emergent literacy also the experiences of children in their literate environment. The adjustment of the test design to align with a communicative perspective and the discourse of the pre-school context will enable us to draw inferences from a specific language use setting to language abilities (Bachman and Palmer, 1996, p. 78).

1.3 Language Test Design and Validation

If a new construct of emergent literacy can be articulated, it would be the first step towards the goal of developing tests that are equivalent in several languages. According to Weideman (2009b), the design of a language test starts with the identification of a language problem. This leads to technical imagination and knowledge being applied to develop a technically qualified design. This initial solution to the language problem is theoretically justified through the interaction of the qualifying technical and the analytical founding function of tests. The theoretical justification of the design might lead to further refinements of the instrument in order to finalise its blueprint (Weideman, 2009b).

(10)

5 reliability, which considers the consistency of the results delivered by the test independently from its testing conditions (Alderson, Clapham, and Wall, 1995, p. 170 ff.). However, McNamara and Roever (2006, p. 249) state that traditional techniques used to ensure reliability and validity are not sufficient in a postmodern and critical approach to ensure fairness, as well as appropriate and acceptable social consequences of tests. In order to include social considerations, a ‘social turn’ in validation theory led to a number of reinterpretations of the central validity argument (McNamara, 2005). For example, Messick (1980, 1989) states that the appropriateness and adequacy of inferences drawn from a test are constitutive of validity. Bachman and Palmer (1996, pp. 17-40) describe the usefulness and meaningfulness of a test as the central component of validity. Similarly, Shohamy (1997) and Kunnan (2000) stress the ethics and fairness of language assessment by considering the consequences drawn from test results, and therefore social justice depending on test design, but also define these concepts as conditions of validity. However, Weideman (2012) is critical of subsuming all test conditions under validity, and proposes a framework for responsible test design and for the validation of a language test which goes beyond validity in the conventional view. In order to shift the focus towards the quality of the design itself, validity is incorporated in a set of test design principles, while additionally taking social considerations into account, such as accountability, accessibility, transparency, alignment with language instruction, and utility (Weideman, 2009b). These concepts and ideas can also serve as a framework for the design and validation of a test of emergent literacy and its equivalents in several languages.

1.4 Overview

(11)

6 justification for the design of an English test of emergent literacy for pre-school children in South Africa, which can serve as the foundation for a set of equivalent tests of emergent literacy in the 11 official languages of South Africa (Republic of South Africa, 2012), and, as indicated above, possibly beyond South Africa.

The second chapter of this study gives an overview of the physiological literacy development of children and early risk factors which could be useful to identify prospective literacy underachievement. Furthermore, existing literacy assessments at pre-school level are reviewed, which reveals a need for the articulation of a new construct of emergent literacy. Next, the articulation of the new construct of emergent literacy, and the test which is based on it will be proposed.

The design process of the Test of Emergent Literacy (TEL) follows the framework of responsible test design formulated by Weideman (2009b) and the test design cycle proposed by Fulcher (2010), which will be specified in chapter 3. The requirements of a test to conform to the principles of responsible test design (Weideman, 2012) will be discussed, building the foundation for the initial validation of the TEL.

The aim and the central research question of this study are specified in chapter 4. Subsequently, in chapter 5 the methodology to be employed to help answer these research questions is outlined. The new construct of emergent literacy, for which a need was described in the second chapter, will be articulated in chapter 6. A solution for the problem of the lack of a language test that is aligned with the new construct of emergent literacy will be proposed by introducing a language test that conforms to the principles of responsible test design (Weideman, 2012).

(12)
(13)

8 2. Emergent Literacy

This chapter provides an overview of literacy development and factors which can predict prospective literacy underachievement. This could indicate what language abilities the underlying construct of the Test of Emergent Literacy (TEL) proposed in this study should acknowledge and include. The need for a new construct of emergent literacy will be identified by reviewing a number of existing literacy assessments at pre-school level.

2.1 Literacy Development

This section describes the pre-conditional skills children acquire and the developmental stages they go through to become proficient in reading and writing (Brandenburger and Klemenz, 2009, pp. 35-37; Frith, 1985), as illustrated in figure 1. Within the age range of 3 to 5 years children develop a natural interest in script and start at a preliterary-symbolic

stage to understand the function of script as a symbol of oral language. At about the age of

(14)

9 right, top to bottom on a page), and how to handle books (Foorman, Anthony, Seals, and Mouzaki, 2002, p. 177).

At school entrance, the children go over to the alphabetical or phonological stage where they acquire the phoneme-grapheme correspondence and start to write and read conventionally (Brandenburger and Klemenz, 2009, pp. 35-37). Simultaneously, an

explicit phonological awareness develops for smaller units of words, such as syllables and

phonemes (Schnitzler, 2008, pp. 32-35). This is fundamental in order to learn to read and to write, which are based on a segmental word processing, called phonetic decoding that requires the conversion of phonemes into graphemes, and vice versa (Schnitzler, 2008, pp. 13-14). Brandenburger and Klemenz (2009, pp. 26-34) declare that these decoding processes require sufficient auditory and visual discrimination skills, as well as the adequate short-term storage of phonemes and graphemes in the phonological and visual working memory, as assumed in the model of Gathercole and Baddeley (1993).

With rising experience in reading and writing, word forms get stored in the mental lexicon, which allows holistic and lexical access to them at the orthographic stage (Brandenburger and Klemenz, 2009, pp. 35-37). Finally, these processes get increasingly automatised (Brandenburger and Klemenz, 2009, pp. 35-37).

Figure 1. Stages of literacy development (Brandenburger and Klemenz, 2009, pp. 35-37).

Preliteral symbolic stage 3-4.5 years Logographic stage 3.4-5/6 years Alphabetic/ phonological stage Orthographic stage Integrative- automatic stage School entrance Automatisation of processes Emergent literacy

 pre-conditions for literacy

(15)

10 2.2 Early Predictors of Prospective Literacy Underachievement

Since at pre-school level conventional reading and writing abilities cannot be assessed, the risk of prospective literacy underachievement can only be identified in the form of early predictors (Suchodoletz, 2005, p. 198). According to Snow, Burns, and Griffin (1998), “many experiences contribute to reading development without being prerequisite to it” (p. vii) and it “is critical to distinguish predictors from causes or explanations of reading difficulties” (p. 100). Therefore, the authors differentiate between risk factors of literacy underachievement which are intrinsic to children, and factors which originate from the child´s environment (Snow et al., 1998, pp. 103-134), which will be further investigated in the following.

Factors which could explain or directly cause literacy underachievement are skills which are foundational for literacy acquisition, such as phonological awareness, phonetic decoding, or phonological working memory capacities (Brandenburger and Klemenz, 2009, pp. 7-37; Snow et al., 1998, pp. 103-134). A meta-analysis, based on 234 studies identifying emergent literacy skills of children within an age of 0 to 6 years, discovered moderate to strong correlations between the literacy pre-conditions print knowledge (e.g. alphabet knowledge, print concepts), phonological processing (phonological awareness, phonological working memory, phonological decoding speed), as well as oral language (vocabulary size, syntax, grammar, word knowledge) and the literacy abilities word decoding, spelling, and reading comprehension (Lonigan, Schatschneider, and Westberg, 2008). That is why print knowledge, phonological processing, and oral language are considered as key pre-conditional skills for prospective literacy achievement (e.g. Lonigan, 2006; Storch and Whitehurst, 2002).

(16)
(17)

12 2.3 Existing Literacy Assessments

In order to identify the need for the development of a new English test of emergent literacy and for the articulation of the construct it is based on, currently available literacy tests in English for children with an age range of between 5 to 6 years (pre-school level) are reviewed in this section. Table 1 provides an overview of the selected literacy assessments which are presented on several webpages on diagnostic material (Academic Therapy Publications, 2013; American Speech-Language-Hearing Association, 2015; Pro-Ed, 2012). Due to the large number of assessments and limited access to the tests themselves, this summary gives only a general impression. Where possible, additional information about the tests was retrieved from test samples provided online (Good and Kaminski, 2007; 2014; Pearson Education, 2010), and in several reviews of these tests (Invernizzi, Justice, Landrum, and Booker, 2004; Pool and Johnson, 2009; Rathvon, 2004; Turner, 2006; Wilson and Lonigan, 2010).

(18)

The information provided in the table below was derived from webpages (Academic Therapy Publications, 2013; American Speech-Language-Hearing Association, 2015; Pro-Ed, 2012), test reviews (Invernizzi, et al., 2004; Pool and Johnson, 2009; Rathvon, 2004; Turner, 2006; Wilson and Lonigan, 2010) and tests samples (Good and Kaminski, 2007; 2014; Pearson Education, 2010).

Table 1

Examples of literacy assessments for pre-school children (4- to 6-year-old)

Test Assessment type Administered subtasks Focus of assessment

BEAR: Basic Early Assessment of Reading (Riverside Publishing Company, 2002)

90 min screening, 30-40 min for single subtests,

Individual

1) Basic reading skills (print conventions, phonological awareness)

2) Listening comprehension

3) Language arts (knowledge of penmanship, print concept)

4) Letter recognition fluency

Reading readiness approach, non-communicative

DIBELS: Dynamic Indicators of Basic Early Literacy Skills (Good and Kaminski, 2002)

10-20 min, individual 1) Phonemic awareness 2) Pseudoword readings 3) Letter naming

4) Word use fluency

Reading readiness approach, non-communicative

ERA: Early Reading Assessment (Hammill, Pearson, Hresko, and Hoover, 2012)

10-15 min, individual 1) Written word vocabulary (print knowledge, matching of letters, words and pictures) 2) Rapid orthographic naming

3) Silent orthographic efficiency (speed of matching letters and words)

4) Phonological awareness (phoneme discrimination and manipulation)

5) Receptive vocabulary

Reading readiness approach, non-communicative

(19)

Table 1 (continued)

Test Assessment type Administered subtasks Focus of assessment

GRADE: Group Reading Assessment and Diagnostic Evaluation (Williams, 2001)

115 min, group assessment

1) Pre-reading (visual perception, verbal concepts, phoneme-grapheme conversion, phonological awareness)

2) Reading readiness (word and letter recognition) 3) Vocabulary (word definition and sentence

completion)

4) Listening comprehension

Readiness approach, communicative/non-communicative

GRTR-R: Revised Get Ready to Read! (Whitehurst and Lonigan, 2001)

10-15 min, individual

1) Print knowledge

2) Phonological awareness

Readiness approach, non-communicative

IGDIs: Individual Growth and Developmental Indicators (McConnell, 2002)

10 min individual

1) Phonological awareness: alliteration and rhyming 2) Oral language: picture naming

Readiness approach, non-communicative

K-SEALS: Kaufman Survey of Early Academic and Language Skills (Kaufman and Kaufman, 1993)

15-25 min, individual

1) Expressive and receptive language 2) Knowledge of numbers

3) Knowledge of letters and words 4) Vocabulary

5) Articulation of words

Readiness approach, non-communicative

(20)

Table 1 (continued)

Test Assessment type Administered subtasks Focus of assessment

PALS-PreK: Phonological Awareness Literacy Screening-Pre-Kindergarten (Invernizzi, Sullivan, Meier, 2001) 10-15 min, screening, individual or group assessment 1) Name writing 2) Letter naming

3) Phoneme synthesis from words 4) Identification of print and words 5) Rhyme identification and completion

Readiness approach, communicative/non-communicative

PALS-K: Phonological

Awareness Literacy Screening Kindergarten

(Invernizzi, Swank, Juel, and Meier, 2003) 20-25 min, individual or group assessment  Rhyme awareness  Sound awareness  Alphabet knowledge  Letter sound identification  Spelling

 Word concept (rhyme repetition)  Word recognition in isolation

Readiness approach, non-communicative

STRR: Slosson Test of

Reading Readiness (Perry and Vitali, 1991)

15 min, individual 1) Recognition of capital letters 2) Recognition of lower case letters 3) Matching capital and lower case letters 4) Visual discrimination: matching word forms 5) Auditory discrimination: rhyming words 6) Auditory discrimination and memory 7) Sequencing (correct order of stories) 8) Identification of antonyms

Readiness approach, non-communicative

(21)

Table 1 (continued)

Test Assessment type Administered subtasks Focus of assessment

TERA-3: Test of Early Reading Ability-3rd Edition (Reid, Hresko, and Hammill, 2001)

30 min, individual 1) Alphabet and its uses (identification and naming of letters in words, segmenting words, counting syllables and sounds in words, identify basic sight vocabulary) 2) Convention of print (book orientation, proceeding in

reading, meaning of punctuation and capitalisation, grammatical constraints of sentences, differentiating homophones and understand the symbolic function of print)

3) Meaning of print (identification of signs, logos, matching written words with pictures, differentiate text genres, reading comprehension/interpretation)

Recent emergent literacy approach,

communicative/non-communicative

TOPEL: Test of Preschool Early Literacy (Lonigan, Wagner, Torgesen, and Rashotte, 2007)

25-30 min, individual

1) Print knowledge (naming of letters and words) 2) Definitional vocabulary (naming and describing

pictures)

3) Phonological awareness (deletion of sounds from words, blend separate sounds to words)

Readiness approach, communicative/non-communicative

YCAT: Young Children's Achievement Test (Hresko, Peak, Herron, and Bridges 2000)

25-45 min, individual

 General information (word knowledge)  Reading (letter/word identification)

 Writing (copying of letters/words/sentences, completing of figures)

 Mathematics (numbers, counting, mathematical operations)

 Spoken language (repetition of numbers, describing and defining objects)

Readiness approach, communicative/non-communicative

(22)

17 Aligned with a previous literacy test summary provided by Lonigan, Allan and Lerner (2011), the literacy assessments presented in table 1 are based on the three pre-conditions of literacy development, which are oral language, phonological processing, and print knowledge, as described previously (e.g. Lonigan et al., 2008). It seems that most of the tests are based on a ‘readiness’ approach. The assessments presented here rarely take a more recent approach to emergent literacy into account. In these respects, only the Test of

Early Reading Ability (Reid, Hresko, and Hammill, 2001) stands out, because it considers

the literacy experiences of children in the form of their sight vocabulary, and their understanding of how to use literacy material, as well as their differentiation between text genres.

Moreover, it seems that most instruments take a restricted view on language, since the tests do not consider different language abilities in interaction with one another, but are separated into different sections for phonological knowledge, vocabulary, listening comprehension, and print knowledge (Van Dyk and Weideman, 2004; Weideman, 2009a; 2011b). Additionally, conventional tasks to measure literacy pre-conditions, such as “This is mouse [sic], flowers, pillow, letters (...). Mouse (...) begins with the sound /m/. Listen, /m/, mouse. Which one begins with the sounds /fl/?” (Good and Kaminski, 2007, p. 2), derived from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS, Good and Kaminski, 2002) for the assessment of phoneme awareness, are not presented in a communicative format. Only some tasks, such as story retelling and interpreting, seem communicative, and as activities refer to the context of pre-school children.

2.4 Conclusion

(23)
(24)

19 3. Phases and Principles of Language Test Design

Chapter 2 has concluded that there is a need for a new construct and test of emergent literacy. In the following, the steps are described which should be taken in order to articulate this new construct and to design and validate the Test of Emergent Literacy (TEL), which is the rationale of this study.

3.1 Phases of Test Design

As is evident from the test design cycle (figure 2) formulated by Fulcher (2010, pp. 94-102), the reason and purpose of a test should be clearly articulated from the start, because it leads to decisions about how the test will have to be designed. Next in the cycle is the construct definition, which describes all the abilities that the test is going to measure. In the following stage, the construct is operationalised through task and item specifications (Fulcher, 2010, p. 127 ff.). Afterwards, a critical evaluation of each task and item, prototyping, piloting, and field testing follow, which lead to further test refinements before the final blueprint of a test is developed (Fulcher, 2010, p. 159 ff.).

Figure 2. Test design cycle proposed by Fulcher (2010, p. 94).

Decisions Test criterion

(25)

20 3.2 Principles of Language Test Design

While there are different phases or stages for test design, the process is subject to certain conditions or requirements. In these respects, Weideman (2012) presents principles of responsible test design which can serve as a guideline for test development and validation. This framework is based on the approach that “applied linguistics is a discipline of design: it solves language problems by suggesting a plan, or blueprint, to handle them“ (Weideman, 2014, p. 1). Language tests are artefacts of applied linguistic design and their design process starts with a technical solution to the language problem which is followed and accompanied by the provision of an analytical foundation or rationale for the technically qualified design (Weideman, 2015, pp. 72-79). The latter is articulated in the theoretical justification of the test construct, leading to the development of a final blueprint for the test (Weideman, 2015, pp. 72-79).

(26)

21

Figure 3. Criteria of responsible test design (derived from Weideman, 2009b, pp. 19- 20).

The previously mentioned requirements for responsible test design, have been formulated in the form of principles by Weideman (2012, pp. 8-9). The principles enumerated in what follows below will build the framework for the design and initial validation of the TEL, on which this study focuses.

Systematically integrate multiple sets of evidence in arguing for the validity of a test.

Since validation is considered as a long-term process of collecting qualitative and quantitative evidence for the validation argument (Fulcher and Davidson, 2007, pp. 159-178), the framework of responsible test design requires several sources of evidence.

Specify clearly and to the public the appropriately limited scope of the test, and exercise humility in doing so. The transparency, ease of implementation, and accountability of the

test design are ensured by making the purpose of the test obvious, which restricts the interpretation and the uses of the test results to a responsibly delineated scope (Weideman, 2009b; 2011a).

Ensure that the measurements obtained are adequately consistent, also across time.

The consistency or internal reliability of the test expresses the independence of the test results from the test conditions, and the internal consistency of a test (Fulcher and

foundational direction anticipates/ is disclosed by/ in

validity (power) consistency unity/ multiplicity of evidence analytical rationale

constitutive concepts regulative ideas

(27)

22 Davidson, 2007, p. 104-108; Lowie and Seton, 2013, p. 78). A test which is consistent over time should deliver similar results if the test takers repeat the test within a reasonable period of time (Fulcher and Davidson, 2007, p. 105). Thus, consistency is fundamental for the justification of the technical design.

Ensure effective measurement by using a defensibly adequate instrument.

The effective and adequate measurement of an instrument is supported by the theoretical justification of the design (Weideman, 2009b; 2011a). Furthermore, the appropriate discrimination of tasks and items between participants with strong and weak abilities in the underlying test construct, as well as an appropriate difficulty level, ensure an effective measurement (Green, 2013, pp. 25-40).

Have an appropriately and adequately differentiated test.

According to Weideman (2012, p. 10), a test with various tasks and items has a higher potential to measure language abilities adequately in contrast to a monotone design. The use of various task types also captures students´ attention and serves to motivate them (Hughes, 2003, pp. 201-202).

Make the test intuitively appealing and acceptable.

(28)

23

Mount a theoretical defence of what is tested in the most current terms.

The theoretical defence refers to the theoretical justification of the design, conventionally known as construct validity, which is derived from the reciprocal relationship between the leading, technical function of tests and their analytical, and founding function (Weideman, 2009b).

Make sure that the test yields interpretable and meaningful results.

Since a test should deliver a meaningful and interpretable result in terms of the evaluation of a certain language ability, the scoring of tasks should be consistent with the construct and each component should be equally or rationally weighted (Weideman, 2014).

Make not only the test, but information about it, accessible to everyone.

The transparency of the test supports the fairness of the instrument, due to equal accessibility of information about the test to everyone (Kunnan, 2000; Weideman, 2014).

Obtain the test results efficiently and ensure that they are useful.

The conduct of the test should deliver useful results which have beneficial social consequences (Bachman and Palmer, 1996, pp. 17-40; Weideman, 2014). For example, the identification of a risk of literacy underachievement at pre-school level should support opportunities for the test taker to develop sufficient emergent literacy abilities prior to school entrance. The usefulness of a test is also linked to the efficiency of the technical design regarding the effort of its administration (Weideman, 2014).

Align the test with the instruction that will either follow or precede it, and as closely as possible with the learning.

(29)

24 and the ones assessed by the test are not aligned, it can lead to unwarranted outcomes for learners, and should be avoided in responsible test design (Hughes, 2003, pp. 2-3; Weideman, 2014).

Be prepared to give account to the public of how the test has been used.

The social consequences of the test should be controlled by providing information about the test use and aims. The transparency of the test design avoids misuse, and legitimises the instrument as a measure of certain language abilities (Weideman, 2014).

Value the integrity of the test; make no compromises of quality that will undermine its status as an instrument that is fair to everyone.

The social dimensions of fairness and ethics of test consequences should be ensured by caring for the adherence of the test to certain standards (Weideman, 2014), for example, that the test is not biased against certain groups of examinees in terms of their background, such as gender, race, or socioeconomic status (Kunnan, 2000; McNamara and Roever, 2006, p. 81). An item is biased if the participants respond differently due to “construct-irrelevant variance that distorts the test results and therefore makes conclusions based on scores less valid” (McNamara and Roever, 2006, p. 82).

Spare no effort to make the test appropriately trustworthy.

(30)

25 3.3 Conclusion

This chapter has characterised the test design process as a continuous evaluation of the technical instrument in terms of its alignment with the principles of responsible test design (Weideman, 2009b, 2014). Since validation is considered to be a long-term process of evidence collection, a question arises about when the evidence is sufficient to support a test design (Fulcher and Davidson, 2007, pp. 159-160). Commenting on this, Weideman (2009b, 2011a) declares that the constitutive test conditions, such as the theoretical justification of the design, the technical consistency or reliability of the instrument, and the effective measurement it achieves, or the appropriate interpretation of its results are

necessary for responsible test design and represent its key principles. However, it “takes

(31)

26 4. Research Question

This study takes as its point of departure the need for equivalent tests of emergent literacy in several languages in a multilingual setting. In order to take an initial step towards this goal, a construct of emergent literacy based on an open and communicative perspective on language will be developed, which considers from a more recent approach on emergent literacy also the experiences of children in their literate environment. Based on this construct, a Test of Emergent Literacy (TEL) will be designed to identify at the pre-school level (age 5-6 years) the risk of literacy underachievement at school. The design of the TEL will be a process including the formulation of the purpose of the test, the construct definition, and the specification of item and task types (Fulcher, 2010, p. 94).

Since validation is a long term process of evidence collection (Fulcher and Davidson, 2007, pp. 159-160), this study will evaluate only a selection of design principles (Weideman, 2012, pp. 8-9), as outlined in the previous chapter. The initial validation, that will be the outcome of this evaluation, will focus on the constitutive principles of responsible test design, as presented in chapter 3, as well as the utility and fairness of the test (Weideman, 2009b). After the results of this initial validation stage have been analysed, reasonable refinements of the first test draft will be discussed.

This study tries to answer the following questions:

 What is the construct of emergent literacy at pre-school level (age 5-6 years), based on a communicative perspective on language?

 How can emergent literacy be measured in a language test that takes a communicative perspective on language as its construct?

(32)

27 o Systematically integrate multiple sets of evidence in arguing for the validity of

a test.

o Ensure that the measurements obtained are adequately consistent (also across time).

o Ensure effective measurement by using a defensibly adequate instrument. o Have an appropriately and adequately differentiated test.

o Mount a theoretical defence of what is tested in the most current terms. o Make sure that the test yields interpretable and meaningful results. o Obtain the test results efficiently and ensure that they are useful.

o Value the integrity of the test; make no compromises of quality that will undermine its status as an instrument that is fair to everyone.

(33)

28 5. Methodology

This study consists of a description and analysis of the design process of the Test of

Emergent Literacy (TEL), and the initial validation of the first test draft, on the basis of

which refinements were made. The validation process was based on a set of panel discussions, and the results of a test pilot.

5.1 Design Process

The design process of the TEL was based on the test design cycle proposed by Fulcher (2010, p. 94), and the design phases and principles of responsible test design formulated by Weideman (2009b, 2012), that have been referred to above. The methodology followed the steps conducted in a previous study on the design of a Test of Early Academic Literacy (TEAL) for 8- to 9-year old children undertaken by Steyn (2014).

To formulate what the TEL is supposed to measure - the first stage of the process as envisaged by Fulcher (2010, p. 94) - the construct of emergent literacy had to be specified, because it builds the foundation for the specification of tasks and items. In order to adjust the construct to the context of 5- to 6-year-old children in South African pre-schools, the skills listed in the Curriculum and Assessment Policy Statement (CAPS, Department of Basic Education, 2011, pp. 23-30) and the list of accomplishments for successful pre-school learners defined by Snow, Burns, and Griffin (1998, p. 80) to identify the components of emergent literacy. Based on these lists, the definition of early

academic literacy, as articulated by Steyn (2014, pp. 23-24) for 8- to 9-year-old children,

(34)

29 Consequently, task and item specifications were designed which operationalise the construct of emergent literacy (Fulcher, 2010, p. 127 ff.). The initial test draft contained approximately 50% more items than the final test, because the refinement stage leads to the exclusion of certain tasks and items.

5.2 Initial Validation

The principles of responsible test design formulated by Weideman (2012, pp. 8-9) provided the framework for the initial validation of the TEL. The test was therefore evaluated in terms of its correspondence with selected principles of responsible test design (Weideman, 2012, pp. 8-9). As suggested by Alderson, Clapham, and Wall (1995, pp. 170-188), the validation argument was based on an evaluation by a panel of experts, qualitative intro- and retrospection of assessment, as well as a statistical analysis of the pilot results. A similar approach was taken in previous studies of test design (e.g. Steyn, 2012; 2014) and test validation (Rambiritch, 2013; Van der Walt and Steyn, 2007).

5.2.1 Panel discussion. The construct, the whole test, and single tasks and items were reviewed by a panel of experts. The test evaluation was conducted with the help of an online questionnaire designed with surveymonkey.com, and in two personal discussions, one prior and one after the pilot stage.

(35)

30 Table 2

Participant information (N = 8) for the online evaluation of the Test of Emergent Literacy

Criteria Range Mean (SD)

Age 23 to 67 years 34.6 years (17.4 years)

Education Postgraduate degree or higher (N = 6) Bachelor degree (N = 2)

Profession Language test design (N = 7)

Teaching, mainly higher education (N = 4)

Working experience 0 to 47 years 10.6 years (16.8 years)

The following questions were discussed to evaluate the construct of emergent literacy in order to refine each component:

1) How relevant is the following component for emergent literacy?

2) How appropriate are the following components of emergent literacy for 5- to 6-year-old children in terms of their proficiency level?

The overall design of the test as well as the single tasks and items were evaluated by considering the following questions:

1) Are the task types aligned with the specifications of the construct?

2) Do the task and each item measure the abilities making up the construct that it is supposed to measure?

3) Are the tasks and items relevant to, and representative of, the domain of pre-school discourse?

4) Are the tasks and items aligned with a communicative perspective on language? 5) Are the scoring criteria consistent with the construct definition?

(36)

31 7) How long do you estimate the test session will take?

8) How easy is the test administration compared to other literacy tests you know? 9) How clear are the manual and the task instructions of the test?

10) How will the children react during the assessment? 11) Are there any obvious flaws in the test items? 12) What are the suggested changes for refinements?

A panel of five experts reviewed the responses to the questionnaire in order to agree on refinements. Thus, after the first panel discussion, prior to the piloting, some test refinements were necessary. These referred to the construct specification, the clarification and simplification of some instructions and tasks, the replacement and addition of single items, as well as changes to the layout of the test. Where possible the scoring of some tasks was adjusted to multiple-choice format, because it makes the administration and data analysis easier, as well as the scoring more objective (Hughes, 2003, pp. 76-77). The refined test draft was pre-piloted on a group of three children, which revealed that some tasks required individual assessment and scoring during the test session. Therefore, the test was separated into two parts, one suitable for group assessments, and the other conductible only individually. The outcome of this first refinement process will be presented in chapter 6.

(37)

32 sample size. The test session took 1.5 h and was conducted by one or two administrators on groups of four to six children.

Participants. The participants with an age range from 5 to 6 years were attending an English pre-school. Further details are summarised in table 3. Since the language situation in South Africa is quite complex, also non-English first language speakers were included in the sample. Nevertheless, the sample can be considered as representative of the educational situation in South African cities, because there is a trend for middle class parents to send their children to English foundational education, even if they do not speak English as their home language (Weideman, Du Plessis, and Steyn, 2015, p. 6). However, the interpretation of the pilot results requires caution because competence in English might differ substantially between the participants.

Table 3

Participant information from the first pilot of the Test of Emergent Literacy

Criteria Value

Participants N = 54

Gender Male N = 31

Female N = 23

Mean age (SD) 5.7 years (4.2 month)

Spoken home languages Setswana English Afrikaans Xhosa Sesotho N = 9 N = 13 N = 5 N = 3 N = 35

Number of home languages per participant

One home language Two home languages

N = 43 N = 11

(38)

33 thoughts and impression of the assessment. Since the young target group of the TEL cannot adequately reflect on their thoughts and decisions, the test administrator observed the effect of the test design on the child’s behavior.

Statistical data analysis. The pilot results of tasks 1 to 5 were analysed to measure the performance of the test. The data were marked by hand, entered upon a spreadsheet, as well as analysed and cross-checked with the help of three statistical programmes. Tasks in multiple-choice format were analysed with Iteman 4.2, and TiaPlus separately from tasks with a scaled scoring, which were investigated with SPSS 21.0. The analysis was based on

Classical Test Theory, which is usually applied to small sample sizes, as in the present

study, and is therefore limited in the generalisability or estimation of the test taker´s performance (Green, 2013, pp. xii-xiii). Furthermore, the limited sample size in this study allows only a first insight into the test properties, because many statistical measures are conclusive only for samples with at least 150 or 200 test takers (CITO, 2005; Green, 2013, pp. xii-xiii).

The data analysis relied on parameters with recommended ranges, as discussed below, on which previous studies of test design and test validation have also based their conclusions (e.g. Du Plessis, 2012; Steyn, 2012, 2014; Van der Walt and Steyn, 2007).

To express the strength of discrimination and differentiation of an item between high and low participants´ abilities, discrimination measures such as the Pearson item

point-biseral correlation (rpbis), the corrected item-total correlation (CITC), and the item rest correlation (Rir) were utilised (CITO, 2005, p. 29; Green, 2013, p. 29; Guyer and

(39)

34 correctly (Guyer and Thompson, 2011, p. 30). Item discrimination measures above 0.2 were aimed for in this test (Steyn, 2014).

To measure item difficulty, their facility values (P) were considered, which range from 0 to 1 and express the percentage of participants answering the item correctly (Guyer and Thompson, 2011, p. 29). High P scores indicate easy items, while low values designate difficult items (Guyer and Thompson, 2011, p. 29). The desired range for facility values differed between the task types. To consider the chance of guessing correctly for the multiple-choice questions with three answer options, P should range between 0.15 and 0.62 for each item. Multiple-choice tasks with four answer options should show item difficulties above 0.15 and below 0.70 (freely adapted from Hingorjo and Jaleel, 2012; Steyn, 2014). For tasks with a scaled scoring, P should be above 0.15 and below 0.84 (Steyn, 2014).

Moreover, differential item functioning (DIF) was examined with the Mantel-Haenszel statistic (M-H), z-test statistic and p-score, to measure differences in the item performance across gender (Guyer and Thompson, 2011, p. 31). Items with DIF statistics of approximately 1 show no bias against male or female participants (Van der Slik and Weideman, 2010, p. 109). DIF scores lower than 1 point to a reduced probability in this case of female participants to have correct answers. A bias against the male participants is here indicated by DIF statistics of higher than 1 (Van der Slik and Weideman, 2010, p. 109). The z-scores and p-scores give information about the significance (alpha level 0.05) of the difference between group performances on a particular item (Guyer and Thompson, 2011, p. 31).

(40)

35 2014) to ensure that each subtask measures a slightly different component of the construct (Alderson et al., 1995, p. 184). A correlation above 0.7 between each subtask and the overall test indicates an alignment of each task with the underlying test construct (Alderson et al., 1995, p. 184; Van der Walt and Steyn, 2008, p. 196).

To analyse the internal reliability and consistency of the whole test, the reliability coefficients Cronbach’s Alpha (Alpha), using the Kuder-Richardson Formula 20, and the

Greatest Lower Bound (GLB) were calculated, which express the interconnection between

test items (CITO, 2005, p. 17; Guyer and Thompson, 2011, pp. 1, 23). Alpha values for the overall test above 0.7 express acceptable, and above 0.8 high reliability as well as internal consistency of a test (Lowie and Seton, 2013, pp. 58, 78; Pallant, 2001, p. 87). As measure of the subtasks´ reliability the GLB will be consulted, which is a better measure for tests of multidimensional abilities, such as in this study (e.g. CITO, 2005, p. 18; Sijtsma, 2009; Van der Slik and Weideman, 2005). Thus, a GLB around 0.7 indicates a high reliability for subtasks (Lowie and Seton, 2013, pp. 58, 78; Pallant, 2001, p. 87).

A factor analysis is a further method to measure the internal consistency of a test (Weideman, 2011a, p. 105) and its construct validity (Green, 2013, p. 123). This analysis gives an impression of the dimensionality of the data, and thus of the homogeneity of a test, by considering the association between items. The more outliers exist, the more heterogeneous a test is (CITO, 2005, p. 19; Weideman, 2011a, p. 105).

(41)

36 6. The Design Process: From Construct to Task and Item Specification

As stated in chapter 3, the beginning of test design is the identification of a language problem (Weideman, 2009b). Chapter 2 has concluded that the underlying constructs of existing tests of emergent literacy in a large part neglect the child´s experiences in a literate environment as influences on literacy development, and consider language from a non-communicative perspective. To remedy this, a new construct of emergent literacy is attempted in the following, as well as the design of tasks and items which operationalise this construct in a language test. The construct and the test draft presented here are the outcome of the initial refinement stage prior to the pilot.

6.1 Construct Articulation

At the beginning of language test design, according to Fulcher (2010, p. 94), stands the definition of the purpose of a test. The Test of Emergent Literacy (TEL) aims at the early identification of the risk of literacy underachievement at school, and targets pre-school children in an age range from 5 to 6 years. Early predictors of literacy underachievement therefore have to be assessed (Suchodoletz, 2005, p. 198).

(42)

37 Schatschneider, and Westberg, 2008). Moreover, sufficient tests to assess these skills already exist, as was noted in chapter 2. Additionally, these pre-conditions can be defined only to a limited extent as communicative abilities, because they are the beginnings and foundation of language development. In order to be aligned with the aim to identify the risk of underachievement in literacy development at school, the TEL will acknowledge the development of premises for early academic literacy, as defined by Steyn (2014, pp. 23-24) for 8- to 9-year-old children.

The steps taken by Steyn (2014) to articulate the underlying construct of the Test of

Early Academic Literacy (TEAL) served as a model for the formulation of the new

construct of emergent literacy in this study. As adapted from Steyn (2014), the skills from the Curriculum and Assessment Policy Statement (CAPS, Department of Basic Education, 2011, pp. 23-30) for grade R (pre-school in South Africa) and the list of accomplishments for successful pre-school learners (pre-school in the United States), as defined by Snow et al. (1998, p. 80), were used as a guide to reformulate the construct of early academic literacy (Steyn, 2014, pp. 23-24) in order to be aligned with a younger target group of 5- to 6-year-olds. The items of the CAPS (Department of Basic Education, 2011, pp. 23-30, Appendix A) and of the list of learner accomplishments (Snow et al., 1998, p. 80, Appendix B), which seemed to be relevant for the components of the definition of early academic literacy (Steyn, 2014, pp. 23-24), as well as for the development of emergent literacy, as described in chapter 2, were examined.

The construct of early academic literacy defined by Steyn (2014, pp. 23-24) includes the ability of a child to be able to:

(43)

38 expressing like or dislike), language patterns and slightly more complex words, and word families;

b) understand basic idioms and figures of speech, can identify connected words and text relations, and understand the basic structure of texts (particularly introductions and conclusions, beginnings and endings);

c) distinguish between different text types, such as instructions, reports and stories;

d) interpret, use and produce information presented in graphic or visual format -illustrations, graphs, charts and tables - at the appropriate level;

e) distinguish between essential and non-essential information, fact and opinion, propositions and arguments, cause and effect, and classify, categorise and handle data that make comparisons in texts of the appropriate level;

f) see sequence and order, recount events, instructions, retell a story and predict what will happen next;

g) know what counts as evidence for an argument, extrapolate from information by making inferences, and apply the information or its implications to other cases than the one at hand - in texts of the appropriate level; and

h) understand the communicative function of various ways of expression, in texts of the appropriate level, such as explaining or defining, providing examples, arguing;

i) understand basic grammar and spelling rules, can use degrees of comparison, can use basic verbs (in different tenses), prepositions, nouns and pronouns accurately. (Steyn, 2014, pp. 23-24)

(44)

39 early academic literacy and to define the new construct of emergent literacy. Certain components of the construct of emergent literacy are similar to those defined in the existing definition of early academic literacy. Other components were identified as precursors for the abilities defined in the construct of early academic literacy and were therefore transformed into a lower level of abilities appropriate for 5- to 6-year-old children.

Some components which were specified by the CAPS and the list of learner accomplishments did not correspond to the definition of early academic literacy (Steyn, 2014, pp. 23-24). However, these were added to the construct of emergent literacy, including emergent reading, emergent writing, concept of print and print motivation (Appendix D), because they are relevant for emergent literacy development, as previously explained in chapter 2, and might be related to the interaction of children with literacy in their environment (e.g. Foorman, Anthony, Seals, and Mouzaki, 2002, pp. 177-178; Makin and Whitehead, 2004).

The following definition of emergent literacy has been derived from this process and from the further refinement proposed by the panel of experts. Pre-school children in the age range of 5 to 6 years are able at an appropriate level to:

a) understand and use a range of vocabulary in context for various communicative functions, such as retelling, comparing, describing, and expressing like or dislike;

b) understand the information provided in a text, based on the meaning of words, relation and order among words, and understand the basic structure of a text;

(45)

40 d) interpret, use, link and produce information presented in graphic or visual format, as in pictures, illustrations;

e) distinguish between essential and non-essential information, cause and effect, and make predictions based on this information and prior experience;

f) see sequence and order, recount events, or instructions, retell a story and predict what will happen next;

g) know what counts as evidence for an argument, extrapolate from information by making inferences and conclusions, and apply the information or its implications to other cases than the one at hand – and apply the information to express an opinion;

h) understand the communicative function of written and printed language and understand the difference between pretended reading and writing and conventional reading and writing; understand how to use literacy material and how to proceed in reading and writing;

i) understand and use morphological and syntactic features, function words, nouns, verbs and adjectives to express temporal, local, causal, and modal relations;

j) mimic writing in different text types, invent own script to convey meaning, copy and write letters, words and names;

k) pretend to read different types of text, speak with a ʻreading voiceʼ and produce ʻbook languageʼ with the use of a typical register of written language, recognise the written form of frequently seen words and names; and

l) be inherently interested in literacy in various forms (playing with literacy in mimicking reading and writing with different text types, asking questions to extend own knowledge of literacy).

(46)

41 memory, visual and auditory perception, as well as motoric skills or eye-hand coordination (e.g. Lonigan et al., 2008; Suchodoletz, 2005), which are summarised in Appendix E. As explained above, these pre-conditions were excluded from the construct of emergent literacy because they do not refer to the purpose of the intended test. Nevertheless, the TEL could eventually give a first indication of problems in pre-conditions of literacy development. Since pre-conditions of literacy development are essential for the development of emergent literacy, their physiological development is considered to be foundational for each component in the articulated construct. Therefore, if indicated, further diagnostic assessment regarding the development of the literacy pre-conditions will be suggested.

6.2 The Task and Item Specifications

The articulation of the construct of emergent literacy gives a list of all abilities that the intended TEL is supposed to measure (Fulcher, 2010, p. 96 ff.). The TEL intends to identify if the children have sufficient command of emergent literacy abilities to develop literacy at school successfully. Thus, the TEL will be designed as a criterion-referenced proficiency test (Hughes, 2003, pp. 11, 21). The operationalisation of the construct into tasks and items that measure the proposed components of emergent literacy was the next step in the design cycle (Fulcher, 2010, p. 94), as explained in chapter 3. How this has been applied, is described in the following.

(47)

42 components of the construct of emergent literacy is presented in Appendix F. The TEL consists of the following subtests, from which tasks 1 to 5 can be administered in a group, and task 6 to 8 individually:

1. Scrambled picture story

The children have to listen to a story, sort a sequence of pictures in the correct order, and answer comprehension questions.

This section tests whether children are able to identify sequence and order, to connect visual and textual information, as well as to understand and interpret information. The task was derived from the TEAL section Scrambled story and was simplified by using only pictures.

2. Organising information

The students have to solve picture puzzles. They need to choose the correct picture from a collection of pictures that belongs in a certain place in a puzzle. Furthermore, they have to place a specific picture in a correct place in a puzzle, or have to identify the picture which does not fit in the presented puzzle.

With this section the abilities to draw inferences from visually presented information in order to make predictions, and to apply the information to other cases are assessed. To solve this task, pre-reading abilities are necessary in order to identify the meaning of picture arrangements in the puzzles. The task was derived from the game

(48)

43 3. Visual vocabulary

The students have to recognise signs and logos of different brands and products, which are frequently encountered in their environment. From a collection of four pictures per item, they have to find the odd one.

This section, which was particularly designed for the TEL, tests the students´ emergent reading abilities, in terms of their early sight vocabulary.

4. Text type and function of script

The pupils are introduced to different text types (e.g. note, menu, or advertising). They have to determine which text type is suitable to the communicative needs of certain situations in daily life.

This section tests the students´ knowledge of different text types and the function of written language. It also assesses their interest in literacy. This task was specifically configured for the TEL.

5. Emergent writing

The children have to pretend to write words which are considered important to them (e.g. their name, a friend’s name, the telephone number of their parents). The teacher has to explain to the children how they can pretend to write and that they do not need to know real writing yet. The task gives an insight into the literacy motivation and the understanding of the concept of print (e.g. writing from left to right).

(49)

44 6. Acting out

The children have to listen to tasks given by the teacher and have to act them out in terms of drawing shapes in specific colours in certain places of a grid. This section tests the students´ comprehension of vocabulary and grammar, as well as the ability to understand and interpret the information provided to them. Additionally, the pupils should conduct the tasks only if they hear a certain phrase, which assesses if they can distinguish between cause and effect.

The task is similar to the TEAL section Grammar but conducted at a lower level. The task type was derived from teaching material in Starting English (Weideman and Rousseau, 1992, pp. 54-57, 66-69).

7. Where does it belong?

The children have to identify in which room in a house certain items belong. They have to name the object and attempt to write down the word.

This assesses the ability to use and understand certain vocabulary regarding relations between words and semantic word fields. The children have to make predictions based on information, to link the information to their own experiences, and to apply the information to state an opinion. Furthermore, this task also includes the assessment of emergent writing abilities. This section is the counterpart to the task Which word works? in the TEAL. The task was developed with reference to similar tasks in the Starting

English course (Weideman and Rousseau, 1992, pp. 58-59).

8. Emergent reading

(50)

45 This section tests the emergent reading abilities of children and their knowledge of different text types. The production of ‘book language’ or typical ‘written language register’ (Makin and Whitehead, 2004, p. 73), such as ‘he said to the girl…’ is required. Furthermore, knowledge of the concept of print (reading from left to right and top to bottom) is assessed, as well as interest in literacy and motivation for dealing with print. This subtask was specifically designed for the TEL.

(51)

46 7. Results and Discussion

In the following the results of the initial validation of the Test of Emergent Literacy (TEL) are presented. A selection of the principles of responsible test design (Weideman, 2012, pp. 8-9), as outlined in chapters 3 and 5, will be utilised to evaluate the test, based on the data from the panel discussion and the pilot.

7.1 Overview of the Pilot Results

To give an insight into the characteristics of the sample, the descriptive statistics of the pilot results analysed with SPSS 21.0 are presented in table 4. The 54 participants in this study scored on average 35.8 (69%) out of a total of 52 marks (SD = 7.6). The distribution of the total scores, despite being slightly peaked and negatively skewed, seems normally distributed (Lowie and Seton, 2013, pp. 34-38), as visualised in figure 4.

Table 4

Descriptive statistics of the total score from tasks 1 to 5 of the Test of Emergent Literacy

Value Score

Total number of scores 52

N examinees 54

Mean 35.8

SD 7.6

Mean percent correct 69

(52)

47

Figure 4. Histogram showing the frequency of the total scores for the Test of Emergent Literacy tasks 1 to 5, with an inserted normal distribution curve.

Table 5 presents the results separately for tasks 1 to 5. The scores for each subtask seem normally distributed, except for task 5, which has a strong negative skewness and strong positive kurtosis (Lowie and Seton, 2013, pp. 34-38). The participants showed the best performance for task 5 with 89% correct responses on average.

Table 5

Summary statistics for the Test of Emergent Literacy tasks 1 to 5 (N = 54)

Score Marks Mean SD Median Mode Min

score Max score Skew-ness Kur- tosis Task 1 Scrambled picture story 16 7.8 2.2 7 7 4 14 0.33 -0.18 Task 2 Organising information 6 2.5 1.4 2 2 0 6 0.39 -0.03 Task 3 Visual vocabulary 10 4.6 2.9 4 2 0 10 0.52 -0.85 Task 4 Text type and function of script

7 4.1 1.7 4 3 1 7 0.01 -0.74

Task 5

Referenties

GERELATEERDE DOCUMENTEN

oude!banmolen!van!de!hertogen!van!Brabant,!ook!gekend!als!het!Spaans!huis.!In!de!periode!1625U

De vondsten die aangetroffen werden in sporen, zijn een oorfragment van een pan in rood geglazuurd aardewerk, uit WP4S7, een greppel; een wandfragment gedraaid fijn grijs aardewerk

In the proposed cross arm tip connection, all the main members are joined with one main connecting plate (figure 5.17) This type of end plate connection ensures that the nett effect

Figure 1. The principle of the machine-learning-based method [7] As the signature ORP is correlated with the dynamic performance of the ADCs, it is applied for the

consumption emotion, the motives for eWOM differ for social value orientation a multivariate. analysis of covariance or MANCOVA was conducted for both

Second, human capital is considered to be the most valuable asset of the firm at nascent ventures (Delmar & Shane, 2006). The effect of emotional conflict on performance

Het concept oordeel van de commissie is dat bij de behandeling van relapsing remitting multiple sclerose, teriflunomide een therapeutisch gelijke waarde heeft ten opzichte van

Wij hebben de in het Financieel Jaarverslag Fondsen 2017 opgenomen jaarrekening en financiële rechtmatigheidsverantwoording over 2017 van het Fonds langdurige zorg (hierna