Formative use of test results: A user’s perspective

(1)

Formative

use

of

test

results:

A

user

’s

perspective

Dorien

Hopster-den

Otter

a,

*,

Saskia

Wools

b

,

Theo

J.H.M.

Eggen

a

,

Bernard

P. Veldkamp

c

a

RCEC,Cito,UniversityofTwente,Postbus217,7500AEEnschede,TheNetherlands b

Cito,Amsterdamseweg13,6814CM,Arnhem,TheNetherlands c

RCEC,UniversityofTwente,Postbus217,7500AEEnschede,TheNetherlands

ARTICLE INFO Articlehistory: Received29June2016

Receivedinrevisedform10October2016 Accepted17November2016 Availableonlinexxx Keywords: Formativeassessment Testuse Testresults Informationneeds Needsassessment ABSTRACT

Despitethepotentialofusingtestdatatosupportstudentlearning,severalstudieshaveconcludedthat theactualuseoftestdataremainslimited.Thepresentstudyaddressesthisproblembyexamining(1)the typesofactionsforwhichteachers,internalcoaches,principalsandparentswithinprimaryeducation wanttousetestresultsand(2)theinformationneededtoperformtheseactions.Theresultsobtained fromthequestionnairesshowthatthevarioususerswanttousetestresultsforactionsthatsupport learning,whichamountstoadiscrepancyrelatingtoactualuse.Furthermore,thevarioususersperform actionsondifferentlevels,thusindicatingtheneedfortailoredreportsthatﬁttheinformationneedsof individual users.Theresultsof thefocusgroup method reveal theinformationneedsof teachers, suggestingimplicationsforthedevelopmentofnewscorereports.

1.Introduction

Researchpointstothepotentialofformativeassessmentsasa wayofsupportingstudentlearning(Baird,Hopfenbeck,Newton, Stobart,&Steen-Utheim,2014; Black&Wiliam,2009;Popham, 2009;Schildkamp &Lai, 2013).Formative assessments provide teacherswithdataaboutstudentperformance.Thisdatacanbe usedtomakedecisionsaboutthenextstepsininstruction,which are likely to be better, or better founded, than the decisions teacherswouldhavetakenintuitivelyintheabsenceofthatdata (Black&Wiliam,2009).

Tobeabletousetestdataforstudentlearning,teachersperform several cognitive steps (Davenport & Prusak, 1998; Ebbeler, Poortman,Schildkamp, &Pieters,2016; Marsh,2012).First, the collecteddatamustbeinterpretedbygivingmeaningtoscores. Thiscanbedonebysummarizingthedatainamoreconciseform. Subsequently,theinterpreteddatahastobecontextualizedby,for example,comparingtheinterpreteddatawithotherinformation. The combination of different sources of information results in usableknowledge,whichservesasabasisfordecisionsaboutan action,afterwhichtheactionisexecuted.Theimpactoftheaction onstudentlearningcanthenbeevaluatedusingnewdata.Assuch, aniterativeprocessiscreated(Mandinach&Jackson,2012).

Severalstudiesshowthatteachershavedifﬁcultycompleting thephasesofthisiterativeprocess(e.g.Hambleton&Slater,1997; Hellrung & Hartig, 2013; Meijer, Ledoux, & Elshof, 2011;

Schildkamp&Teddlie,2008;VanderKleij&Eggen,2013).They especially strugglewith(1)interpretingthetestresultsand(2) translatingthemintoactionsthatsupportlearning.Therearetwo possibleexplanationsfortheseproblems.First,thepresentation regardingtest resultsdoesnotcorrespondwiththeassessment literacyskilllevelofteachers,resultingindifﬁcultyinterpreting thedataandtherebymakinginappropriateuseofthetestresults, withallitsattendantconsequences(e.g.Popham,2009; Zapata-Rivera, VanWinkle,& Zwick, 2012). Second, thecontent of the presented data does not ﬁt the information needs of teachers, resultinginproblemstranslatingthedataintoactionsthatsupport learning(e.g.Wiliam,2011).

Aconsiderablenumberofstudiesaddresstheﬁrstexplanation by allowing teachers and other users to develop the required assessment literacy skills (e.g. Lukin, Bandalos, Eckhout, & Mickelson, 2004; Verhaeghe, Vanhoof, Valcke, &Van Petegem, 2011).Forexample,somestudiesshowapositiveeffectoftraining in terms of developing the required knowledge and skills to analyse and interpret data (e.g. Ebbeler et al., 2016; Van Geel, Keuning,Visscher,&Fox,2016;Zwicketal.,2008).Otherstudies address the interpretation problem by adjusting the data presentationtotheuser’s skilllevel(e.g.VanderKleij,Eggen,& Engelen,2014)sinceithasbeensuggestedthatthechosenmethod ofdatavisualizationcanreducetheassessmentliteracyneedsof users(Hattie&Brown,2008).

* Correspondingauthor.

E-mailaddresses:d.denotter@utwente.nl(D.Hopster-denOtter),

saskia.wools@cito.nl(S.Wools),theo.eggen@cito.nl(T.J.H.M. Eggen),

b.p.veldkamp@utwente.nl(B.P. Veldkamp).

http://dx.doi.org/10.1016/j.stueduc.2016.11.002

xxx–xxx

ContentslistsavailableatScienceDirect

Studies

in

Educational

Evaluation

(2)

The second explanationregardingthe problemofusing test dataforstudentlearningfocusesonthecontentofthepresented data. According to Zapata-Rivera and Katz (2014), everyone involvedin the learning process of a child uses test resultsas presentedthroughscorereports,yeteachaudiencehasitsunique typesof decisionsto bemadeontest results. If a scorereport designerdeﬁnestheneedsofthetargetaudience,heopensupthe possibility of tailoring the score report to meet the unique informationneedsofthataudience.Withinthetargetaudience, four groups of users are distinguished: teachers, who are responsible for instruction and teaching a group of students; internalcoaches,whocoachteachersandsupportstudentswith specialneedsacrossclasses;principals,whoareresponsibleforthe schoolorganizationandparents,whosupportthelearningoftheir ownchild.

Fittingthepresenteddatawiththeinformationneedsofusersis oftenoverlooked.AccordingtoWiliam(2011),assessmentdataare madeavailabletousersundertheassumptionthatthisdataare usefulinsomeway.Toolittleattentionhasbeenpaidtothetypes ofactionsthatintendedaudienceswanttoperformonthebasisof testdata.Thecurrentstudyaddressesthisprobleminthecontext ofDutchprimaryeducation.It seeks todetermine thetypesof actionsthatteachers,internalcoaches,principalsandparentsin primaryeducationwanttoperformwiththeuseoftestresultsand theinformationneededtoenabletheseactions.

1.1.Educationaldecision-making

Ineducation,decision-makingaboutinstructionalprocessesis aneverydayactivity.Thesedecisionsaretakenatindividual,group and school levels and can have important consequences for studentlearning.For example,onan individuallevel,decisions may pertain to whether a student should receive additional support.On a group level, decisions can relateto categorizing studentsintodifferentlevelsfordifferentiationofinstruction.Ona schoollevel,decisionsmay pertaintoselectinga newteaching method.Inordertoascertainwhetherthesekindsofdecisionsare correct,itisimportantthatdecisionsareinformedbyhigh-quality evidence(Brookhart&Nitko,2008).

Testresultsareonesourceofdatathatcanbeusedasevidence tosupporteducationaldecision-making(Zapata-Rivera&Zwick, 2011).A testcan bedescribed as“an instrumentor systematic procedureforobservinganddescribingoneormorecharacteristics of a student using either a numerical scale or a classiﬁcation scheme” (Brookhart &Nitko, 2008;p.5).Combinedwithother assessmentdata,suchasstudentobservations,oralquestionsand students’work,anaccuratepictureofthestudentcanbeobtained and decisions can be informed (Brookhart & Nitko, 2008; Mandinach,2012).

Despite the availability of test data meant to inform the didacticaldecisionsofteachers,variousstudiesconclude,however, that theactual useof test data for student learning is limited (Ledoux, Blok, Boogaard, & Krüger, 2009; Meijer et al., 2011;

Vanhoof,Verhaeghe, Verhaeghe, Valcke,& Van Petegem, 2011;

Verhaeghe et al., 2011). Instead, test data are used for other purposes,suchascommunicationandevaluation,which donot automaticallyresultinincreasedstudentlearning.Theuseofdata for communication has to do with informing parents about students’abilityorwithinforminginspectorate1_for_the_purpose_of

accountability(Ebbeleretal.,2016;VanderKleij&Eggen,2013) while the sole purposeof the useof data for evaluation is to

appraisestudents’performance.Theactionsthatcouldfollowfrom thesejudgmentsarenotcarriedout(Brookhart&Nitko,2008).

1.2.Presentationoftestresults

Testresultsarepresentedusingscorereports.Scorereportsare thevehiclefortranslatingthetestresultsintousefulactionsthat supportlearning.Itisaformofcommunication,withasender,a messageandanaudience.Thesenderofscorereportsisthetest developerortestagencypresentingtheresults.Themessagedeals withthecontentofthescorereport,andtheaudienceconsistsof thepeoplewhousethetestresults(Hattie,2009;Ryan,2006).

Tofostertheuseoftestresultsforeducationaldecision-making, thescorereportcontentshoulddirectlyinformtheaudienceabout theirdecisions(Aschbacher&Herman,1991;Hattie,2009; Zapata-Rivera&Katz,2014).Understandingthepurposeforreadingthe testresultsinascorereporthelpstopresenttherightmessage. Questionsillustratingthisstatementinclude:Whataretheusers’ goals?Whatdotheuserswanttoknow?Whatdecisionsshould the information inform, or what actions should it motivate or justify?If thescorereportpresentscontenttailoredtoauser’s desiredactionsordecisions,theuserwouldalwaysknowwhatto do withdata that have collectedand presented (Aschbacher & Herman,1991;Wiliam,2011).

1.3.Tailoringscorereportstovarioususers

Testresultsareoftenusedbymorethanoneintendedaudience, including teachers, parents,internal coaches and principals. As pointed out by Zapata-Rivera and Katz (2014) and Mandinach (2012),dependingonthepositionoftheuser,eachaudiencehasits uniquetypesofdecisionstobemadeonthebasisoftestresults.For example,teacherswouldbemoreinvolvedinthedecisionprocess of an individual student or group of students while principals wouldbemorefocusedonthedecisionprocessattheschoollevel (Schildkamp&Kuiper,2010).Internalcoacheswouldbeinterested intheperformanceofallstudentswhileparentswouldbemore interestedin theperformance of theirown child (NEGP,1998). Withvariousintendedaudiences,itislikelythatspeciallydesigned reportswouldbeneededforeach.Theneedfortailoredreports willthus bereinforceddependingonthevariationsamong the decisions and information needs of the different audiences (Bradshaw&Wheater,2009;Hambleton&Slater,1997).

1.4.Identifyingusers’needs

It is the responsibilityof test developers to ensurethat the contentofthescorereportﬁtstheinformationneedsoftheuser (Ryan,2006).Becauseofthisresponsibility,variousstudieshave calledfor the creationof score reports that meet theneedsof different audiences (Aschbacher & Herman, 1991; Goodman & Hambleton,2004;Hambleton&Slater,1997;Jaeger,1998;Wainer, Hambleton,&Meara,1999).HambletonandZenisky(2013)and

Zapata-Rivera et al. (2012) present a model for score report development auser-centredmodelwhichstartswithaneeds assessment. This needs assessment should establish common groundbetweenthetestdeveloperandthetestuser,bridgingthe gapbetweentheinformationthatresultsfromanassessmentand theactionstheuserwantstoperformfromtheinformation.The resultsfromtheneedsassessmentwillbethebasison“whichallof theotherstepsinreportdesignarelinked”(Hambleton&Zenisky, 2013;p.486).

The current study performed such a needs assessment. As mentionedearlier,itsaimwastodeterminethetypesofactions thatvarioususerswouldliketoperformwiththeuseoftestresults as well as the information needed to enable these actions.

1

The Dutch Inspectorate assesses and stimulates the quality of primary educationandreportsonthequalityofeachschooltothepublic

(3)

Regardingthisaim,welookedbeyondtheavailableinformation fromexistingtestsandscorereports.Instead,thefocalpointofthis studywasonstarting fromthedecisions oractionsthata user wouldideallyliketomake(Wiliam,2011)inordertosupportthe useoftestresultsforstudentlearning.

1.5.Researchquestions

Themainquestionsaddressedinthisstudyareasfollows:

1.Whichtypesofactionswoulduserschooseasdesiredusesof testresultsandhowdotheseactionsrelatetoactualuses? 2.What,ifany,istheextentofthedifferencesbetweenteachers,

internalcoaches,principalsandparentswithregardtodesired andactualusesandcorrespondingactions?

3.Whatinformationfromtestresults isneededtoperformthe desiredactions?

2.Method

Inordertoanswertheresearchquestions,datawerecollected fromdifferent usergroups within Dutchprimaryschools. Four different kinds of users were distinguished: teachers, internal coaches,principalsandparentsandguardians(hereafter,parents). Aquestionnairewasdevelopedfortheteachers,internalcoaches andprincipalstoidentifytheactionsforwhichtestresultsareused inthecontextofteaching.Asthefocuswasonactionsrelatedto teachingandwere,therefore,notapplicabletoparents,aseparate questionnaireforparentswasdeveloped.

Inaddition,qualitativedatafromfocusgroupsweregatheredto validatetheresultsofthequestionnairedataandtofurtherspecify theinformationneeds. Based ontheresultsfromthe question-naires,wedecidedtotargetthesefocusgroupsatteachers.Inorder tofacilitatethereadabilityofthisarticle,thesechoicesarefurther elaboratedintheresultssection.

Table1showstherelationbetweentheresearchinstruments andthedifferentusergroupsandresearchquestions.Inthenext session, the instruments, procedure, data analyses and sample characteristicsarediscussed.

2.1.Instrumentsandprocedure

2.1.1.Questionnaire1–teachers,internalcoachesandprincipals Theﬁrstquestionnairewasdevelopedtoinvestigatetheactions thatteachers,internalcoachesandprincipalsdeemeddesirablein relationtotestresultswithinthecontextofteachingaswellasthe actualuseofsuchresults.Testresultsweredeﬁnedasresultsfrom asystematicinstrument,suchaswrittenordigitaltests,excluding results from other assessment methods like observations and verbalresponsesfromstudents.Theactualusedepends onthe availability of information from current tests, which in the Netherlands,aremostlystandardized testsaimedatmonitoring studentsand writtenordigital testsfromteaching methods.In terms of desired use, respondents were asked to mention all actionsindependentofcurrentlyavailableinformationandtests.

Alongside the eleven items on the background of the respondents,thequestionnaireconsistedofthreeitemsshowing alistofpossibleactionsforwhichtestresultscouldbeused.The ﬁrstandseconditemsconsistedofmultipleresponsequestionsin which respondents were required to select actions relating to actualanddesireduse,respectively.Asrespondentscouldselectall actionsasdesireduse,thethirditemaskedrespondentstochoose the most important desired action from their selection. This providedgreaterinsightintothedegreeofinterestrelatingtothe different actions. The questionnaire is included inTable A.1 of

AppendixA.

The listof possibleactionsresulted fromthegrid shown in

Table 2.Thisgridconsistsofactionsonthree levels(individual level,grouplevelandschoollevel)andthreepurposes (communi-catinglearning,supportinglearningandevaluatinglearning).This enabledthepossibilitytodescribesomepatternsintheanswers. Thelevelswererelatedtotheprecisedatausedfortheaction.For example,theplacementofstudentsintogroupsfordifferentiation isagrouplevelactionbecausethedatafromthestudentgroupis usedtoperformthisaction.Purposesrefertowhatdataisusedfor. For example, determining individual students’ performance compared to the national performance is meant to appraise studentperformancewithoutsettingnewlearninggoals. There-fore, this action is labelled as having an evaluative purpose. Although this study is primarilyaimed atactions that support learning,thepurposesofcommunicatingandevaluatinglearning wereaddedinordertogainabetterunderstandingoftotalactual use.

Thegridcontainingpossibleactionsisbasedonthe question-naireofBlok,OtterandRoeleveld(2001).Theycollectedalistof actions for which data from various tests could be used. We validated the actions from the grid (Table 2) by asking two educationalconsultants,specializedinassessments,togenerateas manyactionsastheycouldthinkofforwhichteachers,internal coachesandprincipalswouldliketousetestresults.Thesetwo educational consultants have considerable contact with all the varioususergroupsabouttheirdesiredandactualusesoftests.The mentionedactionswerealreadyincludedinthequestionnaire.We alsoaskedthreeteacherstodescribewhatismeantbyeachaction ortogiveanexampleofanactionfromtheirownpractice.We concluded that the descriptions of the actions wereclear and appropriatetotheDutchcontext.Wepretestedthequestionnaire byaskingﬁveteacherstoﬁlloutthequestionnaireandtoindicate whether they were missing some actions or whether some questionswereunclear.Thisresultedinafewminoradaptations suchastheadditionoftheoption “nouseof testresults”.This option ensureda distinction betweenrespondentswho didnot answerthequestionbecausetheyskippeditandthosewhodidnot makeuseoftestresults.Furthermore,weaddedtheoption“other” sothattherespondentscouldmentionactionsoutsidethelist.

The teachers, internalcoaches and principalswere askedto complete the questionnaire through various channels. Those schools opting toparticipate asa focus group alsoreceived an e-mail witha linktothe questionnaire,which was distributed withintheschool.Thequestionnairewasﬁlledoutelectronically bytherespondents.

Table1

Relationbetweeninstruments,respondentsandresearchquestions.

Researchquestions Instruments Respondents 1 actions 2 userdifferences 3 informationneeds

Questionnaire1 Teachers,internalcoaches,principals x x

Questionnaire2 Parents x x

Focusgroup Teachers x x

(4)

2.1.2.Questionnaire2–parents

Thequestionnaireusedfortheparentsaddressedthe perspec-tiveofparentsontheuseoftestresults.Alongsidesixitemsabout parents’background, thequestionnaire consistedof fouritems. Thisarticleaddressesonlythetwoitemsrelatingtothepurposeof usingtestsandactionsaimedatsupportingthelearningprocess. Theﬁrstitemwasamultiplechoicequestioninwhichparentshad toselectthepurpose(communicating, supportingorevaluating learning)thatbestsuitedthereasonforwhichtheythoughttest resultswould bemostly used.Thesecond itemwas a question investigatingthesupportingactionsparentswouldliketotakein ordertodeterminetheextenttowhichtheseactionsdifferedfrom the actions of other users. The questionnaire is included in

TableA.2ofAppendixA.

Wepretestedthequestionnairebyaskingthreeparentstoﬁll out thequestionnaire and to indicatewhethersome questions wereunclear.Theparentsreportedthatallquestionswereclear, which was also demonstrated in the responsesthey provided. Therefore, the pretest did not result in adaptations to the questionnaire.

Parentswereaskedtocompletethequestionnaireusingvarious channels.Someschoolsagreedtoparticipateasafocusgroupand distributed the questionnaire to the parents of their school. Anotherexamplewasacallonaneducationalwebsiteforparents. Thequestionnairewasﬁlledoutelectronically.

2.1.3.Focusgroup teachers

Focusgroupmeetingswiththeteachersofsevenparticipating schoolswereheldtovalidatetheresultsofresearchquestion1and toidentifytheinformationneedsforquestion3.Thedesignofthe focus group method included the characteristics of a group interviewaswellasagroupdiscussion(Newby,2010).

Themeetingconsistedofthreeparts.Intheﬁrstpart,theresults fromthequestionnairewerevalidatedbyidentifyingthepurpose ofteachingandthecorrespondingactionsaimedatachievingthis purpose. While the questionnaire was a reactive taskwhereby respondentswereaskedtoselectactionsfromthegivenlist,the focus group was a generating taskwhereby respondents were askedtoidentifytheactionsbythemselves.In thesecondpart, theresearcher discussed someconceptual aspects of formative

assessmentsinorder toachieve thesameunderstandingof the concept.Inthethirdpart,theteachersweretoselecttheactions from the ﬁrst part for which they needed information from formativeassessments.Actionsthatdidnotrequireinformation werenotselected.Toillustrate,theteachersselectedtheaction “placement of students into different instruction groups for differentiation”and not theaction“using humor”becausethey neededsomeinformationfortheﬁrstactionbutnotforthesecond. Thereafter,theteachershadtothinkabouttheinformationneeded foreachaction.Allindividualanswerswererecordedonpaper.The teachers’ responses were then systematically grouped and validatedduringthefocusgroup.

Thestructureofthefocusgroupmeetingandtheformulationof thequestionswereﬁrstpretestedusingindividualinterviewswith tworespondentswhodidnotparticipateinthefocusgroups.These interviewsresultedinsomeadaptionsregardingtheformulation ofthequestions;forexample,wechangedthefollowingquestion “whichinformationfromtestsdoyouneedinordertocarryoutthis action?”into“whichinformationdoyouneedinordertocarryout thisaction?”Thisisbecausethepretestshowedthatrespondents onlygaveanswersaboutinformationthatknowntestreportsare able to give, a mindset deemed too limited for this study. In addition, the ﬁrst focus group was meant as a trial. Since no changes weremadeafterwards,thedatafromthis focus group wereincluded.

2.2.Dataanalysis

2.2.1.Questionnaire1–teachers,internalcoachesandprincipals Toanswertheﬁrstquestion,frequencyanalyseswereusedto showthenumberofoccurrencesofeachresponsechosenbythe respondents. Because the questionnaire included two multiple response questions,the numberof responsesdiffered fromthe number of respondents. We used McNemar’s test to ascertain statistical differences between actions in terms of actual and desired use. This test evaluates the difference between two correlatedproportions,whichmeansthatthetwoscoresarenot independent. In order todescribe thepatterns in theanswers, the number of actions was then summarized into the three purposes: communicating, supporting and evaluating learning.

Table2

Gridconsistingofthreelevelsandthreepurposesofpossibleactionsforwhichtestresultscanbeused.

Levels Purposes

Communicatinglearning Supportinglearning Evaluatinglearning

Individuallevel -Toinformparents/guardiansduringindividual meetingsorbymeansofscorereports

-Tocreateindividualactionplansforlow performingstudents

-Todetermineindividualstudents’ performancecomparedtothenational performance

-Toinformtheindividualstudentabouthis/her performance

-Tocreateindividualactionplansforhigh performingstudents

-Todetermineindividualstudents’progress regardinglearninggoalsorcontent -Toinformotherschoolsaboutanindividual

studentbymeansofaneducationalreport (schooltransitionofthestudent)

-Togivefeedbacktostudentsinorderto formulatetheirownlearninggoals

-Tomakedecisionsaboutstudents’ transitionyear

Grouplevel -Toinformparents/guardiansduringgroup meetings

-Tocreategroupactionplans -Todeterminethegroupperformance comparedtothenationalperformance -Toinformthestudentgroupabouttheir

performance

-Toadaptinstructiontoeducationalneeds -Todeterminegroupprogressregarding learninggoalsorcontent

-Toinformcolleaguesaboutthestudentgroup duringagroupdiscussionortransmission

-Toplacestudentsintodifferentgroupsfor differentiation

-Tocompareparallelgroupsregardingtheir progress

Schoollevel -Toinformpeopleviatheschoolprospectusor schoolwebsite

-Tocreateschoolorannualplans -Todeterminetheschool’sperformance comparedtothenationalperformance -Toinformtheschoolboardorparticipation

council

-Toformulatepolicyregardingtheselectionof newteachingmethods

-Todetermineprogressregardingschoolgoals -Toinformtheinspectorate -Tocreateprofessionalplans(performance

appraisals,careerdecisions)

-Tocomparetheperformanceofastudent groupwith(thoseof)previousyears. xxx–xxx

(5)

These “purpose” subscales all had high reliabilities relating to desireduse,withCronbrach’s

a

of0.80,0.81and0.86,respectively. Regarding actual use, the reliabilities of these subscales were moderate,withCronbrach’s

a

of0.59,0.65and0.74,respectively. Forthesecondquestion,thesameanalyseswereperformed,but wedividedtherespondentgroupintoteachers,internalcoaches andparents.Inaddition,thenumberandpercentagesofactions weresummarizedintothreelevels:individual,groupandschool. Sincetherewerenoadditionalactionsmentionedundertheoption “other” from outside the given list, we did not analyse these answers.

Thedata fromthe parentquestionnairewereanalysed both qualitatively and quantitatively. The quantitative analysis con-sisted of frequencyanalyses of those questions with a closed-answerformat.Theanswerstotheopen-answerquestionwere codedinaqualitativeway.Wecomparedtheanswersofthe open-answerquestionsandthengroupedrelatedpiecesofinformation intocategories.Wesubsequentlyusedthesecategoriestoclassify allanswers.Ifananswerdidnotﬁtintotheexistingcategories,the frameworkwasmodiﬁedandtheprocessrepeated.

2.2.3.Focusgroup–teachers

Theparticipants’responsestothequestionsinthefocusgroup meetingswerelisted,groupedanddocumentedduringthefocus group meeting. For the analyses, answers were considered irrelevant and were removed if they did not correspond with actions and needed information. For example, some teachers mentionedamethodoftesting(e.g.doinganobservation)orsome preconditions regarding teaching their students (to create an orderly group climate). Following the focus group method, all relevantdatawereregardedasvaluable,regardlessofhowmany teachers appointed the data. The resultsof the differentfocus groupsweresummarizedandcompared.

2.3.Samplecharacteristics

2.3.1.Questionnaire1–teachers,internalcoachesandprincipals A total of 140 teachers, 34 internal coaches and 14 school principalsﬁlledoutthequestionnaire.Ofthese188respondents, 30respondentsdidnotcompletethequestionnaire,whichmeans thattheresponsesof158respondentswereusedforanalysis.Some backgroundcharacteristicsofthe158respondentsarepresentedin

Table3.ThesamplecharacteristicsaretypicaloftheDutchprimary schoolteacherpopulation(www.onderwijscijfers.nl).

Altogether, 250 parents of students in primary education participatedinthisstudy.However,33parentsdidnotcomplete thequestionnaire,whichmeansthattheresponsesof217parents (48males,169females)wereusedforanalysis.Thedistribution

relating to the educational level of the respondents was 60% completinghighereducation,29%completingvocational educa-tion,and11%hadobtainedalowereducationallevel.Overall,the sampleincludedarelativelyhighproportionoffemaleandhighly educatedrespondentscomparedtothepopulationofparentsin theNetherlands.

2.3.3.Focusgroup–teachers

Focusgroupswereheldatsevendifferentschools.Allteachers withinaschoolparticipatedinthecorrespondingfocusgroup.We could therefore ensure that the data were gathered from enthusiastic teachers as well as those who were not very enthusiasticaboutusingtests.Tofurtherenhancethe representa-tiveness,weselectedschoolsofdifferentsizes.Theschoolteams varied between seven and 17 persons. In total, 84 teachers participated in the seven focus groups. We have no reason to believethatthissampledoesnotreﬂectthecharacteristicsofthe schoolpopulation.

3.Results

3.1.Question1:whichtypesofactionswoulduserschooseasdesired usesoftestresultsandhowdotheseactionsrelatetoactualuses?

The158respondentsindicated1922actionsasdesiredusesof test results (Table 4), representing, on average, more than 12 actionsperrespondent.Themostfrequentlychosenactionunder desiredusewas“toinformparentsduringindividualmeetingsor bymeansofscorereports”selectedby121respondents(76.6%). This action accounted for 6.3% of all the desired use answers. Informing parents was also the most frequently chosen action under actual use (91.1%). However, this action was selected signiﬁcantlylessfrequentlyasadesiredactionthanasanactual use(

x

2=14.7;p<0.001).Communicationstotheinspectorate,the creationofgroupplansandsomeactionsrelatingtotheevaluation of testresultswerealsoselectedsigniﬁcantlyless frequentlyas desiredusethanasactualuse.Thecreationofgroupplanswasstill thesecondmostfrequentlychosendesireduseaction(72.2%).

Someactionswerechosensigniﬁcantlymoreoftenasdesired usesthanasactualuses.Forexample,thefrequencyoftheaction “to give feedback to students in order toformulate their own learninggoals”doubled(

x

2=46.2;p<0.001)from19.6%to51.3%. Other examples included thecreation of action plans for high performing students (

x

2=5.8; p=0.02) and the formulation of policy regardingthe purchaseof teaching methods and instru-ments(

x

2=19.2;p<0.01).

Otherfrequentlymentionedactionsasdesireduses,although they were not chosen signiﬁcantly more frequently, were the creation ofindividualaction plansforlow performing students (70.3%)andtheplacementofstudentsintogroupsfor differentia-tion(68.4%).Theseactions,includingthecreationofgroupplans (72.2%),wereallexamplesofactionsrelatingtothecategoryof supportinglearning.

Table5presentsasummaryofthenumberandpercentagesof actionsintothethreepurposes:communicating,supportingand evaluating learning. The action “no use of test results” was a separatecategorythatdidnotbelongunderanyoftheotherthree purposes. Notwithstanding the fact that fewer actions were selectedasdesireduse(n=1922)incomparisonwithactualuse (n=1991),thenumberofactionsrelatingtosupportinglearning washigherfordesireduse(n=741)thanforactualuse(n=656). The opposite was truefor thepurposes of communicating and evaluatinglearning.Regardingtherelativedistributionofdesired use, respondents mostly chose actions relating to supporting learning (38.6%). This resultdiffered fromactual uses whereby actionsrelatingtothepurposeofevaluatinglearningweremost

Table3

Backgroundcharacteristicsofrespondents(N=158)fromquestionnaire1. Teachers Internalcoaches Principals Total Sex Male Female 19 100 1 26 6 6 26 132 Averageage(SD) 40.4(11.3) 45.9(8.8) 49.8(10.4) 49.8(10.4) Yearsoftotalexperience

0 5 5–10 10< 17 26 76 3 1 23 0 0 12 20 27 111 Total 119 27 12 158 xxx–xxx

(6)

commonlychosen(35.9%),andactionsrelatingtothepurposeof supportinglearningwerechosenlessfrequently(32.9%).

TheresultsshowninTable5wereconﬁrmedbytheanswerson the third questionnaire item, which required respondents to choosethemostimportantactionasadesireduseoftests.Intotal, 53.9%oftherespondentschoseanactionrelatingtothepurposeof supportinglearning. The most frequentlychosen action in this categorywas“toadaptinstructiontoeducationalneeds”(n=25), followedby“togivefeedbacktostudents inordertoformulate theirownlearninggoals”(n=18)and“tocreategroupactionplans” (n=16).Actionsrelatingtothepurposeofevaluatinglearningwere chosenby27.5%oftherespondents.Thisresultwasmainlydueto theaction“todetermineindividualstudents’progressregarding learninggoalsorcontent”(n=29,17.4%).

The viewofparents correspondedwiththis result;45.2% of themindicatedthattestresultsweremainlyusedtosupporttheir

child’s learning. This was followed by 40.1% of parents, who thoughtthatstudent-levelevaluationwasthemaingoal,and14.7% whosaidthatcommunicatingresultstotheparents,principalor inspectoratewasthecentralgoal.

Basedontheseresults,weconcludethatrespondentsmostly choseactionsrelatingtothepurposeofsupportinglearning,which amountstoadiscrepancyrelatingtoactualuse.Inordertocreate usefulscorereportsoftestresults,weinvestigatedwhetherthese actionswerethesameordifferentforthevariousaudiences.

3.2.Question2:what,ifany,istheextentofthedifferencesbetween teachers,internalcoaches,principalsandparentswithregardto desiredandactualusesandcorrespondingactions?

TableB.1inAppendixBpresentsthepercentagesofteachers, internalcoachesandprincipalschoosinganactionfordesiredand actualuse.No majordifferenceswerefoundwithregardtothe three purposes of communicating, supporting and evaluating learning(Table6).Teachersandprincipalsmostlychoseactionsas desired use under the purpose of supporting learning while internalcoacheschosealmostasmanyactionsforthepurposeof supportinglearningasforthepurposeofevaluatinglearning.All usergroupsindicatedthatcurrenttestresultswereprimarilyused toevaluatelearning.

Therewere,however,differencesbetweentheusergroupswith regard tothedifferentlevels ofactions (Table 7).Theteachers especiallyselectedactionsrelatingtotheindividuallevel(45.5%) andsubsequentlychosemanyactionsrelatingtothegrouplevel

Table4

Numberofresponsesandrespondents(N=158)choosinganactionasdesireduseandactualuse.

Actions Desireduse Actualuse

Responses Respondents Responses Respondents

n % % n % %

Communicatinglearning

*Toinformparents/guardiansduringindividualmeetingsorbymeansofscorereports 121 6.3 76.6 144 7.2 91.1

*Toinformtheindividualstudentabouthis/herperformance 83 4.3 52.5 68 3.4 43.0

Toinformotherschoolsaboutanindividualstudentbymeansofaneducationalreport (schooltransitionofthestudent)

89 4.6 56.3 102 5.1 64.6

Toinformparents/guardiansduringgroupmeetings 29 1.5 18.4 27 1.4 17.1

*Toinformthestudentgroupabouttheirperformance 39 2.0 24.7 22 1.1 13.9

Toinformcolleaguesaboutthestudentgroupduringagroupdiscussionortransmission 100 5.2 63.3 113 5.7 71.5

Toinformpeopleviatheschoolprospectusorschoolwebsite 15 0.8 9.5 13 0.7 8.2

Toinformtheschoolboardorparticipationcouncil 32 1.7 20.3 42 2.1 26.6

*Toinformtheinspectorate 46 2.4 29.1 88 4.4 55.7

Supportinglearning

Tocreateindividualactionplansforlowperformingstudents 111 5.8 70.3 115 5.8 72.8

*Tocreateindividualactionplansforhighperformingstudents 104 5.4 65.8 86 4.3 54.4

*Togivefeedbacktostudentsinordertoformulatetheirownlearninggoals 81 4.2 51.3 31 1.6 19.6

*Tocreategroupactionplans 114 5.9 72.2 137 6.9 86.7

Toadaptinstructiontoeducationalneeds 101 5.3 63.9 91 4.6 57.6

Toplacestudentsintodifferentgroupsfordifferentiation 108 5.6 68.4 118 5.9 74.7

Tocreateschoolorannualplans 39 2.0 24.7 32 1.6 20.3

*Toformulatepolicyregardingtheselectionofanewteachingmethod 63 3.3 39.9 32 1.6 20.3

Tocreateprofessionalplans(performanceappraisals,careerdecisions) 20 1.0 12.7 14 0.7 8.9

Evaluatinglearning

*Todeterminetheindividualstudents’performancecomparedtothenationalperformance 67 3.5 42.4 97 4.9 61.4 Todetermineindividualstudents’progressregardinglearninggoalsorcontent 107 5.6 67.7 118 5.9 74.7

*Tomakedecisionsaboutstudents’transitionyear 64 3.3 40.5 82 4.1 51.9

*Todeterminethegroupperformancecomparedtothenationalperformance 71 3.7 44.9 89 4.5 56.3

Todeterminegroupprogressregardinglearninggoalsorcontent 92 4.8 58.2 90 4.5 57.0

Tocompareparallelgroupsregardingtheirprogress 33 1.7 20.9 28 1.4 17.7

Todeterminetheschool’sperformancecomparedtothenationalperformance 58 3.0 36.7 55 2.8 34.8

Todetermineprogressregardingschoolgoals 62 3.2 39.2 66 3.3 41.8

*Tocomparetheperformanceofastudentgroupwith(thoseof)previousyears. 65 3.4 41.1 89 4.5 56.3

Nouseoftestresults 8 0.4 5.1 2 1.0 1.3

Total 1922 100 1991 100

Note:*p<0.05.

Table5

Numberandpercentageofactionsrelatingtodesiredandactualuseperpurpose.

Purpose Desireduse Actualuse

Count % Count %

Communicatinglearning 554 28.8 619 31.1

Supportinglearning 741 38.6 656 32.9

Evaluatinglearning 619 32.2 714 35.9

Nouseoftestresults 8 0.4 2 0.1

Total 1922 100 1991 100

(7)

(37.1%).Onlyasmallnumberofteachers’responsesrepresented actionsrelatingtotheschoollevel(16.9%).Theanswersfurnished bytheinternalcoachesshowedasimilarpatternalthoughtheyhad agreaterpreferencethanteacherstoperformsomeactionsatthe schoollevel(25.9%).Theprincipals’answersshowedtheopposite, withmostselectedactionsrelatingtotheschoollevel(35.9%).To illustratethisdifference,attheschoollevel,thedevelopmentof schoolplanswasselectedfarmorefrequentlybyprincipals(91.7%) thanbyteachers(13.4%)andinternalcoaches(44.4%).

Parents(N=217)mentionedalsoactionsrelatingtosupporting thelearningprocess.Helpingtheirchildwithhomeworkwas,for example,themostmentionedaction(19.0%).Furthermore,17.2%of theparentswouldliketopracticethelearningmaterialwiththeir child at home. Someparents would give theirchild additional support by providing learning material to remedy weaknesses (14.5%).Otherexamples ofactions mentioned includedreading books(9.3%),testingtheirchildonhis/herknowledgeforatest (7.9%),learninginaplayfulway(4.5%),helpingtodeveloplearning skillslikeplanningschoolwork(4.1%), givingsomeeducational games (3.4%) and visiting cultural organizations like museums (3.4%).Allsuchactionswereinrelationtotheirown individual child.

These differences in actions between the various users indicatedthatthereisaneedforscorereportstobetailoredto thespeciﬁcusergroups,correspondingtotheactionsthatthese kinds of users would like toundertake (Zapata-Rivera & Katz, 2014). This means that we should investigate the information needsofeachusergroupseparately.Basedonthepreviousﬁnding that test results would rather be usedto supportlearning, we decided to limit our focus on teachers for the third question. Teachers’ primary task was to support the learning process of students.Theywerealsotheuserswho actuallycommunicated theseresultstootheruserssuchasstudentsandtheirparents.

3.3.Question3:whatinformationfromtestresultsisneededto performthedesiredactions?

Theresultsofresearchquestion1 werevalidatedduringthe seven focus group meetings. The teachers in the focus groups underlinedthegeneralprinciplethattheywouldsupportstudent learning bydevelopingthe cognitiveand social knowledgeand skillsoftheirstudents.

Subsequently, the actions for achieving this purpose were generated.Nineactionsinthequestionnairewererelatedtothe

purpose of supporting learning. The most frequently chosen actionsinthequestionnairewerealsogeneratedbytheteachersin the focusgroups. Starting withthemost frequentlymentioned action,theseactionswere:(1)alignmentoflearningmaterialand learningobjectiveswiththeinitiallevelofstudents,(2)placement ofstudentsintodifferentinstructiongroupsfordifferentiation,(3) student-teacherconversationsaboutwell-beingandlearning,(4) development of group and individual action plans and (5) alignment of learning materials to learning objectives and preferences,withaction(4)fromthefocusgroupcoveringthree actionsfromthequestionnaire.Two actionsfromthe question-naire werenot mentioned in the focus group. However, these actions werealso chosen less frequently by teachers but more often by principals and internal coaches. As the actions were formulatedin thefocusgroups, theconceptualizationsof these actions were slightly different from the description in the questionnaire.Forexample,student-teacherconversationsabout well-beingandlearningwasrelatedtothequestionnaireaction“to givefeedbacktostudentsinordertoformulatetheirownlearning goals”.

Table 8 presents the ﬁve actions and the corresponding information needsof each action. The action mentioned byall sevenfocusgroupswasthealignmentoflearningmaterials and learning objectives with the initial level of students, which corresponds with the questionnaire item “to adapt instruction toeducationalneeds”.In ordertoperformthis action, teachers needinformationaboutthelearningobjectivesforeachyearand subjectaswellasinformationaboutstudents’masteryofthese learning objectives. Furthermore, information is needed with regardtothesequenceofacquiringlearningobjectives, realistic expectationsforthenextlearningobjectiveandlearningmaterial suggestionsofhowtoachievethisobjective.

Mostoftheinformationneedsmentionedforstudent-teacher conversationswereaboutwell-beingandlearning.Forthisaction, teachers need information about students’learning in orderto givestudentsfeedback,forexample,informationaboutstudents’ strategy to solve assignments. Furthermore, teachers need information about students’ personal aspects, like students’ well-being,workingattitudeandself-efﬁcacy.

The overall resultspresentedin Table8 showthatteachers havedifferentinformationrequirementsforperformingactions: on one hand, information about general teaching aspects like the learning objectives for eachyear and subject and, on the other hand, information about students suchas their learning

Table7

Responsepercentagesofactionschosenbyusersinrelationtothedifferentlevels.

Level Desireduse Actualuse

Teachers(n=119) Internalcoaches(n=27) Principals(n=12) Teachers(n=119) Internalcoaches(n=27) Principals(n=12)

Individual 45.5 39.9 33.1 44.5 40.7 31.2

Group 37.1 34.0 30.4 38.1 32.2 28.1

School 16.9 25.9 35,9 17.3 27.1 40.7

Nouseoftestresults 0.5 0.2 0.6 0.1 0 0

Total 100 100 100 100 100 100

Table6

Responsepercentagesofactionschosenbyusersinrelationtothedifferentpurposes.

Purpose Desireduse Actualuse

Teachers(n=119) Internalcoaches(n=27) Principals(n=12) Teachers(n=119) Internalcoaches(n=27) Principals(n=12)

Communicatinglearning 28.6 28.5 31.5 31.5 28.6 33.7

Supportinglearning 39.8 35.3 37.6 33.8 31.0 31.1

Evaluatinglearning 31.1 36.0 30.4 34.6 40.4 35.2

Nouseoftestresults 0.5 0.2 0.6 0.1 0.0 0.0

Total 100 100 100 100 100 100

(8)

progress. Furthermore, teachers need information about both thepersonal aspectsof students,liketheirinterestin subjects, andtheircognitiveaspects,suchastheirmasteryofalearning objective.

These results also indicate that teachers need the same information for different actions. For example, students’ mastery of a learning objective was needed to perform four actions. Realistic expectations regarding subsequent learning objectives were mentioned for three actions. Other kinds of information were only selected for one action, like students’ workingattitude.

Becausewestartedthefocusgroupbydiscussingtheactionsfor the purposeof teaching rather than the more speciﬁc actions relatingtotestresults,andbecauseweaskedthefocusgroupsfor actionsrequiringinformationfromallpossiblesources,including tests,teachersalsomentionedthreeactionsthatwerenotdirectly relatedtotheactionslistedinthequestionnaire:(1)placementof students into different groups for cooperative learning and collaboration,(2)connectiontostudents’perceptionoftheworld and(3)creationofownership.Cooperativelearningand collabo-rationmeansthatagroupofstudentshavetocooperateequallyon assignmentsin order toachieve learning goals. For this action, teachersneedinformationaboutthelearningobjectivesforeach yearand subjectand aboutstudents’ masteryof theselearning objectives.Thisisthesameinformationasthatmentionedearlier. However,teachersalsoneedinformationaboutstudents’behavior, their social and communicative skills and their willingness to collaborate. The second action concerned the connection to students’perceptionoftheworld,whichcontainedtheadoption ofthechosenexamples,thematictopicsandtheintroductionsof lessons relating to students’ experiences and interest. For this action, teachers also need additional information, such as the dynamicofastudentgroup,theproportionofboysandgirlsand students’ home situation. The third action concerned giving students responsibility to support their own learning. The information requirements included students’ persistence and abilitytoworkindependently.

4.Conclusionsanddiscussion

This study investigated the types of actions users want to performwiththeuseoftestresultsandtheinformationneededto enable these actions. By administering two questionnaires and conducting seven focus group meetings, both qualitative and quantitativedataweregathered.Intheanalyses,distinctionswere madeamongvarioususers,includingteachers,internalcoaches, principalsandparents.

Theresultsofthisstudysuggestthatinrelationtodesireduses, respondents mostly chose actions relating to the purpose of supportinglearning.Thestudyalsoshowedthatthisdesireduseof testresultswasnotthesameastheactualuse;testresultswere primarilyusedtoevaluatethelearningprocessbydeterminingthe student’sability.Theseresultscorroboratetheresultsofprevious studies,suggesting thelimited useof testresults for formative purposes(Ledouxetal.,2009;Meijeretal.,2011;Vanhoofetal., 2011;Verhaegheetal.,2011).

Furthermore, we conclude that the various users want to perform actions on different levels and in different contexts. Teachersandparentsreportedthattheywanttoperformactionsat the level of theindividual student whereby teachers act in an educational setting and parents perform in a more informal situation.Internal coachesandprincipalsselectedmore actions relatingtotheschoollevel.Thisresultisinaccordancewiththe expectation regarding the uniquedecisions of each user group (Zapata-Rivera&Katz,2014).

Based on the results of the first and second questions,we decidedtolimitourthirdquestiontoteachers.Theresultsfrom the first question were validated, and we gathered insights abouttheinformationneedsofteacherstoperformeachaction mentioned. The results show the need for different kinds of information,for instance,relatingtostudents’strategytosolve anassignment,students’motivationandtheirworkingattitude. This result confirms Brookhart and Nitko’s (2008) and

Mandinach’s (2012) argument that test data are only one source of information in supporting educational decision-making and that an accurate picture from the student could beobtainedwiththeuseofotherassessmentdata. Theresults alsoindicate thatteacherssometimes need thesame informa-tion for different actions; for example, information about students’ mastery of a learning objective was mentioned for the performance of fouractions.

Finally, the formulationof the question“what information do you need in order to carry out this action?” expanded the mindset of respondents but also resulted in information needs which might not arise from tests. For example, the information need “sequence of acquiring learning objectives” likely formed a greater part of the content knowledge of the teacher. This illustrates the view of Gummer and Mandinach (2015) that the process of using test data is complex and that for instructional decision-making, teachers need to combineanunderstandingofdatawith“standards,disciplinary knowledge and practices, curricular knowledge, pedagogical content knowledge, and an understanding of how children learn” (p. 2).

4.1.Limitationsofthestudy

Thisstudywaslimitedinseveralways.First,thesamplesize was limited (especially the number of principals and internal coaches),sotheresultscanonlybegeneralizedtoalimiteddegree. However,thefactthatmostofthefocusgroupsmentioned the same type of actions, which were also in the questionnaire, suggeststhatwehaveidentiﬁedthemostimportantactionsfor teachers.

Table8

Informationneedsmentionedbyfocusgroups(N=7)toperformactionstosupport learning.

(9)

The three additional actionsmentioned in thefocus groups suggestthattheactionslistinthequestionnairemighthavebeen incomplete.Thisisbecauseweaskedforactionsfromtwodifferent perspectives.Westartedthefocusgroupsbydiscussingtheactions for the purpose of teaching rather than the more speciﬁc informationneedsrelatingtotestresults.Furthermore,weasked thefocusgroupsforactionsrequiringinformationfromallpossible sources,includingtests.Thus,theanswersfromthefocusgroup includedactionsfromawiderperspective.Moreover,noadditional actionswere mentioned duringthe pretestingof the question-naire;the“other”optioninthequestionnairewasnotused;and twooftheadditionalactionswerementionedbyjusttwofocus groups.Forthisreason,weconsideredthisdifferenceofactionsto beofminimalimportance.

Finally,theuserschosetheactionsforwhichtheywanttouse testresults,butthischoicewasmadewithinanexistingframeof referenceconsistent withexisting tests in thecurrent national systemoftheNetherlands.Thisimaginationseemstobedifﬁcult andcontextualized,whichlimitsgeneralization.

4.2.Implicationsforpractice

This study provided insights into the actions and corre-spondinginformationneedsof teachers.The resultsshowthat teachersandothers wouldlike tousetest results for uses for whichcurrentmeasurementinstrumentsarenotvalidated.This mayresultinmisuseorlimiteduseofcurrenttestresults.The resultsare informativetoteachersandothers, interms ofthe use of different instruments, for their educational decisions about actions as theinformation needs of teachers cannot be obtainedfromonetest.Furthermore,testdeveloperscoulduse the insights herein for the development of tests and score reportsaimedat teachers.If thescore reportpresents content that is tailored to the actions that teachers would like to undertake,then teacherswouldalwaysbeabletoperformthe action once the data is collected and presented.For example, teachers can use data from test results to make up different groupsfor differentiation, toalign thelearning objective with the initial student level or to develop group action plans. Compared totheavailable information fromcurrenttests, test developers should develop tests that offer more detailed information like student strategies to solve assignments. In thisway,testresults willbeusedtomakedecisionsaboutthe

next stepsin instruction thatare likelytobe better, or better founded, than thedecisions theywouldhave taken intuitively in the absence of such data (Black & Wiliam, 2009). The development of tailored score reports can contribute to the potential of formative assessment as a way of supporting student learning.

4.3.Implicationsforfutureresearch

The results and limitations imply a future research agenda. First,itseemsworthwhiletoexaminetheactionsandinformation needsofstudentsasa usergroup.Sinceteachersindicatedthat theywouldformulatelearningobjectivestogetherwithstudents more often than theyactually did, it would be useful to also examinetheactionsandinformationneedsofstudents.Thisresult is in accordance with thetrend towards activating students as owners of their own learning as a key strategy of formative assessment(Wiliam&Thompson,2007).

Second,despitethegrowingbodyofresearchoneffectivescore reporting(Zenisky&Hambleton,2012),therehasbeenlittleefforton users’ actual use of developed score reports.Future research is neededonhowtodesigneffectivescorereportsforteachersthat visualize test results that are appropriate to the identiﬁedactionsand informationneedsreportedin thisstudy.Moreover,itwouldbe usefultostudytheextenttowhichthepresentationoftheidentiﬁed informationneedsresultinmoredatauseforstudentlearning.

Third,theresultsshowedthatteachersneeddetailed informa-tion from tests, suchas the extentto which each studenthas mastered a certain learning objective. This implies that score reports shouldvisualizesmallerlevelsofinformation(e.g.from total test scores tosubscores and items). Accuracy is however relatedtothelevelofreporting.Whenreportsaremoredetailed, the accuracy of test scores is often negatively impacted; e.g. accuracy is lower, and scores are more uncertain (Monaghan, 2006; Ryan, 2006). The total score is often a more accurate measureofanindividual’sknowledgeorskillsinasubdomainof interestthanasubscorederivedonlyfromthoseitemsthatpurport tomeasurethesubdomaindirectly(Monaghan,2006).Assessment organizations have a duty to provide teachers with sufﬁcient informationabouttheseaccuraciestoallowthemtomakevalid inferencesbasedontestresults(e.g.AERAetal.,2014;Newton, 2005). Future research is needed to investigate how to best communicatethisaccuracyinformation.

AppendixA.

TableA.1 Questionnaire1.

Respondents’background

1.Whatisyourpositionwithintheschool? a)Teacher;b)Internalcoach;c)Principal 2.Whatisyoursex?

a)Male;b)Female 1.Whatisyourage?

4.Inwhichdistrictdoyouwork?

a)Friesland;b)Groningen;c)Drenthe;d)Overijssel;e)Gelderland;f)Limburg;g)Noord-Brabant;h)Zeeland;i)Utrecht;j)Noord-Holland;k)Zuid-Holland;l)Flevoland 5.Howmanyyears’experiencedoyouhaveinprimaryeducation?(Forinternalcoaches/principalsonly)

a)Lessthan5years;b)5to10years;c)Morethan10years

6.Howmanyyearshaveyouworkedasateacher/internalcoach/principal? a)Lessthan5years;b)5to10years;c)Morethan10years

7.Whichgradedoyouteachthemost?(Note:Answertheremainingquestionsforthestudentsinthisgrade)(Forteachersonly) a)Lowestgrades(Group1,2);b)Middlegrades(Group3,4,5);c)Uppergrades(Group6,7,8)

8.Inwhichgradedoyoufunctionasinternalcoach?(Forinternalcoachesonly)

a)Lowestgrades(Group1,2);b)Middlegrades(Group3,4,5);c)Highestgrades(Group6,7,8);d)Wholeschool(Group1–8).Other: 9.What,ifany,otherfunctionsdoyoufulﬁllinschool?(Forinternalcoaches/principalsonly)

a)Nootherfunctions;b)Principal;c)Teacher;d)ICTcoordinator;e)Languageormathspecialist;f)Otherwise,namely... 10.Doesyourschooluseanyofthefollowingprinciples?

a)Anthroposophy;b)Dalton;c)Freinet;d)Jenaplan;e)Montessori;f)O4NT;g)noneofthese 11.Whichofthefollowingstudentmonitoringsystemsdoyouuse?

a)Cito-LVS;b)Parnassys;c)Esis;d)Dot.com;e)Other:

(10)

TableA.1(Continued) Actualuseoftestresults

12.Testresultsmaybeusedfordifferentactions.Wepresentedanumberofactionsbelow.Checkthepurposesforwhichyouhaveusedtestresultsinrecentschoolyears (2014–2015,2015–2016).Note:wemeantheuseoftestresultsforactionswiththemajorityofstudents,notforexceptionalcircumstances.Itispossibletogivemorethan oneanswer.Inthelastschoolyears,Ihaveusedtestresultsforthefollowingactions...

Desireduseoftestresults

13.Supposeyouwereallowedtodesigntestscorereportsyourselfsothatitpresentsyouwithalltheinformationyouneed,checkforwhichactionsyouwouldliketouse testresults.Note:wemeantheuseoftestresultsforactionswiththemajorityofstudents,notforexceptionalcircumstances.Itispossibletogivemorethanoneanswer. Inanidealsituation,Iwouldliketousetestresultsforthefollowingactions ...

Mostimportantpurposeofdesireduse

14.Wepresentedyourchosenactionsregardingtestresultsbelow.Whichactiondoyouﬁndmostimportantintheuseoftestresults? Chosethemostimportantone.

Forquestions9,10and11,wepresentedthefollowingactionlist: Nouseoftestresults

Toinformparents/guardiansduringindividualmeetingsorby meansofscorereports

Tocreateindividualactionplansforlowperformingstudents Todetermineindividualstudents’performancecomparedtothe nationalperformance

Toinformparents/guardiansduringgroupmeetings Tocreategroupactionplans

Todeterminethegroupperformancecomparedtothenational performance

Toinformpeopleviatheschoolprospectusorschoolwebsite Tocreateschoolorannualplans

Todeterminetheschool’sperformancecomparedtothenational performance

Toinformtheindividualstudentabouthis/herperformance Tocreateindividualactionplansforhighperformingstudents Todetermineindividualstudents’progressregardinglearninggoals orcontent

Toinformthestudentgroupabouttheirperformance

Toadaptinstructiontoeducationalneeds

Todeterminegroupprogressregardinglearninggoalsorcontent Toinformtheschoolboardorparticipationcouncil

Toformulatepolicyregardingtheselectionofnewteachingmethods Todetermineprogressregardingschoolgoals

Toinformotherschoolsaboutanindividualstudentbymeansofaneducationalreport(school transitionofthestudent)

Togivefeedbacktostudentsinordertoformulatetheirownlearninggoals Tomakedecisionsaboutstudents’transitionyear

Toinformcolleaguesaboutthestudentgroupduringagroupdiscussionortransmission Toplacestudentsintodifferentgroupsfordifferentiation

Tocompareparallelgroupsregardingtheirprogress Toinformtheinspectorate

Tocreateprofessionalplans(performanceappraisals,careerdecisions) Tocomparetheperformanceofastudentgroupwith(thoseof)previousyears.

TableA.2 Questionnaire2.

Respondents’background 1.Whatisyourgender?

a)Male;b)Female 1.Whatisyoursex?

3.Inwhichdistrictdoyourchildrenattendschool?

a)Friesland;b)Groningen;c)Drenthe;d)Overijssel;e)Gelderland;f)Limburg;g)Noord-Brabant;h)Zeeland;i)Utrecht;j)Noord-Holland;k)Zuid-Holland; l)Flevoland

4.Whatisyourhighestlevelofeducation?

a)Noeducation/primaryeducation;b)preparatorysecondaryvocationaleducation;c)generalsecondaryeducation;d)vocationaleducation;e)seniorgeneral secondaryeducation/universitypreparatoryeducationf)univeristyofappliedsciences;g)MasterofArts/Science/PhD

5.Inwhichgradeisyouroldestchild?

a)Lowestgrades(Group1,2);b)Middlegrades(Group3,4,5);c)Uppergrades(Group6,7,8);d)Myoldestchildhasleftprimaryschool 6.Doesyourschooluseanyofthefollowingprinciples?

a)Anthroposophy;b)Dalton;c)Freinet;d)Jenaplan;e)Montessori;f)O4NT;g)noneofthese Centralquestions

7.Whichofthefollowingpurposesdoyouthinkbestsuitsthereasontotestyourchildatschool?

a)Determiningthelevelofyourchild;b)Adaptinginstructiontotheeducationalneedsofyourchild;c)Reportingandcommunicatingtheresultstoparents,theschool boardorinspectorate

8.Doyoureceivethescorereportsofyourchildfromthestudentmonitoringsystem?

a)No;b)Yes,inthescorereportofmychild;c)Yes,duringindividualmeetingswiththeteacher;d)Yes,duringgroupmeetingswithparents;e)Other: 9.Wouldyouliketosupportthelearningprocessofyourchild?

a)No,inmyopinion,thistaskbelongstotheschool;b)Yes,bymeansofthefollowingactions....

10.Thisresearchlooksathowtestresultsarepresented.Whatinformationfromtestresultswouldyouliketoreceiveaboutyourchild?Itispossibletogivemorethanone answer.

a)Thetestscoresofmychild,focusingespeciallyonthedifferentsubjects;b)Theprogressofmychildwithregardtothedifferentsubjects;c)Thelevelatwhichmy childis,comparedtothatofotherchildren,withregardtothedifferentsubects;d)Thelevelatwhichmychildiswithregardtothedifferentpartsofasubject; e)Learningmaterialsuggestionswithregardtothedifferentsubjectsinordertohelpmychildinhis/herlearning;f)Other:

(11)

References

AERA,APA,&NCME(2014).Standardsforeducationalandpsychologicaltesting. Washington,DC:AmericanPsychologicalAssociation.

Aschbacher,P.R.,&Herman,J.L.(1991).Guidelinesforeffectivescorereporting(CSE technicalreport326).LosAngeles,CA:UCLACenterforResearchonEvaluation, StandardsandStudentTesting(CRESST).

Baird,J.A.,Hopfenbeck,T.N.,Newton,P.,Stobart,G.,&Steen-Utheim,A.T.(2014). Stateofthefieldreviewassessmentandlearning(casenumber13/4697).Oslo: KnowledgeCenterforEducation.

Black,P.,&Wiliam,D.(2009).Developingthetheoryofformativeassessment. EducationalAssessment,EvaluationandAccountability,21(1),5–31.http://dx.doi. org/10.1007/s11092-008-9068-5.

Blok,H.,Otter,M.E.,&Roeleveld,J.(2001).Hetgebruikvanleerlingvolgsystemenanno 2000[Theuseofstudentmonitoringsystemsintheyear2000].Amsterdam,the Netherlands:SCO-KohnstamInstituut.

Bradshaw,J.,&Wheater,R.(2009).Nationalfoundationforeducationalresearch: Internationalsurveyofresultsreporting(OFQUAL10/4705).London:Ofﬁceof QualiﬁcationsandExaminations.

Brookhart,S.M.,&Nitko,A.J.(2008).Assessmentandgradinginclassrooms.Upper SaddleRiver.NJ:PearsonEducation.

Davenport,T.H.,&Prusak,L.(1998).Workingknowledge:Howorganizationsmanage whattheyknow.Boston:HarvardBusinessSchoolPress.

Ebbeler,J.,Poortman,C.L.,Schildkamp,K.,&Pieters,J.M.(2016).Effectsofadatause interventiononeducators’useofknowledgeandskills.StudiesinEducational Evaluation,48,19–31.http://dx.doi.org/10.1016/j.stueduc.2015.11.002.

Goodman,D.P.,&Hambleton,R.K.(2004).Studenttestscorereportsand interpretiveguides:Reviewofcurrentpracticesandsuggestionsforfuture research.AppliedMeasurementinEducation,17(2),145–220.http://dx.doi.org/ 10.1207/s15324818ame1702.

Gummer,E.,&Mandinach,E.B.(2015).Buildingaconceptualframeworkfordata literacy.TeachersCollegeRecord,117(4),1–22Retrievedfromhttps://www. tcrecord.org.

Hambleton,R.K.,&Slater,S.C.(1997).AreNAEPexecutivesummaryreports understandabletopolicymakersandeducators?(CSETechnicalReport430).Los Angeles,CA:UCLACenterforResearchonEvaluation,StandardsandStudent Testing(CRESST).

Hambleton,R.K.,&Zenisky,A.L.(2013).Reportingtestscoresinmoremeaningful ways:Aresearch-basedapproachtoscorereportdesign.InK.E.Geisinger(Ed.), Handbookoftestingandassessmentinpsychology(pp.479–494).Washington, DC:APA.

Hattie,J.A.C.,&Brown,G.T.L.(2008).Technologyforschool-basedassessmentand assessmentforlearning:DevelopmentprinciplesfromNewZealand.Journalof EducationalTechnologySystems,36(2),189–201.http://dx.doi.org/10.2190/ et.36.2.g.

Hattie,J.(2009).Visiblylearningfromreports:Thevalidityofscorereports.Paper presentedatthemeetingoftheNationalCouncilonMeasurementinEducation.

Hellrung,K.,&Hartig,J.(2013).Understandingandusingfeedback:Areviewof empiricalstudiesconcerningfeedbackfromexternalevaluationstoteachers. EducationalResearchReview,9,174–190.http://dx.doi.org/10.1016/j. edurev.2012.09.001.

Jaeger,R.M.(1998).NAEPvaliditystudies:Reportingtheresultsofthenational assessmentofeducationalprogress(Workingpaper2003-11).PaloAlto,CA: AmericanInstitutesforResearch.

Ledoux,G.,Blok,H.,Boogaard,M.,&Krüger,M.(2009).Opbrengstgerichtwerken: Overdewaardevanmeetgestuurdonderwijs[Data-drivendecisionmaking:About thevalueofmeasurementorientededucation].Amsterdam,theNetherlands: SCO-KohnstamInstituut.

Lukin,L.E.,Bandalos,D.L.,Eckhout,T.J.,&Mickelson,K.(2004).Facilitatingthe developmentofassessmentliteracy.EducationalMeasurement:Issuesand Practice,23(2),26–32.http://dx.doi.org/10.1111/j.1745-3992.2004.tb00156.x.

Mandinach,E.B.,&Jackson,S.S.(2012).Transformingteachingandlearningthrough data-drivendecisionmaking.ThousandOaks,CA:Corwin.

Mandinach,E.B.(2012).Aperfecttimefordatause:Usingdata-drivendecision makingtoinformpractice.EducationalPsychologist,47(2),71–85.http://dx.doi. org/10.1080/00461520.2012.667064.

TableB.1

Percentageofteachers,internalcoachesandprincipalschoosinganactionasdesireduseandactualuse.

Actions Desireduse Actualuse

Teachers n=119 Internal coaches n=27 Principals n=12 Teachers n=119 Internal coaches n=27 Principals n=12 Communicatinglearning

Toinformparents/guardiansduringindividualmeetingsorbymeansofscorereports 77.3 77.8 66.7 92.4 92.6 75.0

Toinformtheindividualstudentabouthis/herperformance 47.9 74.1 50.0 40.3 48.1 58.3

Toinformotherschoolsaboutanindividualstudentbymeansofaneducationalreport(school transitionofthestudent)

52.1 77.8 50.0 60.5 77.8 75.0

Toinformparents/guardiansduringgroupmeetings 16.0 22.2 33.3 18.5 7.4 25.0

Toinformthestudentgroupabouttheirperformance 20.2 37.0 41.7 14.3 7.4 25.0

Toinformcolleaguesaboutthestudentgroupduringagroupdiscussionortransmission 61.3 77.8 50.0 73.9 70.4 50.0

Toinformpeopleviatheschoolprospectusorschoolwebsite 5.0 14.8 41.7 4.2 7.4 50.0

Toinformtheschoolboardorparticipationcouncil 8.4 44.4 83.3 14.3 48.1 100.0

Toinformtheinspectorate 20.2 55.6 58.3 46.2 77.8 100.0

Supportinglearning

Tocreateindividualactionplansforlowperformingstudents 66.4 85.2 75.0 73.9 77.8 50.0

Tocreateindividualactionplansforhighperformingstudents 63.0 81.5 58.3 53.8 63.0 41.7

Togivefeedbacktostudentsinordertoformulatetheirownlearninggoals 45.4 77.8 50.0 16.0 29.6 33.3

Tocreategroupactionplans 71.4 77.8 66.7 89.1 85.2 66.7

Toadaptinstructiontoeducationalneeds 57.1 85.2 83.3 50.4 77.8 83.3

Toplacestudentsintodifferentgroupsfordifferentiation 68.1 74.1 58.3 75.6 77.8 58.3

Tocreateschoolorannualplans 13.4 44.4 91.7 11.8 25.9 91.7

Toformulatepolicyregardingtheselectionofanewteachingmethod 36.1 51.9 50.0 15.1 33.3 41.7

Tocreateprofessionalplans(performanceappraisals,careerdecisions) 9.2 18.5 33.3 5.9 3.7 50.0

Evaluatinglearning

Todeterminetheindividualstudents’performancecomparedtothenationalperformance 35.3 66.7 58.3 55.5 85.2 66.7 Todetermineindividualstudents’progressregardinglearninggoalsorcontent 66.4 77.8 58.3 73.9 85.2 58.3

Tomakedecisionsaboutstudents’transitionyear 37.8 55.6 33.3 48.7 63.0 58.3

Todeterminethegroupperformancecomparedtothenationalperformance 37.0 77.8 50.0 48.7 77.8 83.3

Todeterminegroupprogressregardinglearninggoalsorcontent 54.6 77.8 50.0 57.1 59.3 50.0

Tocompareparallelgroupsregardingtheirprogress 15.1 44.4 25.0 14.3 29.6 25.0

Todeterminetheschool’sperformancecomparedtothenationalperformance 26.1 66.7 75.0 20.2 74.1 91.7

Todetermineprogressregardingschoolgoals 30.3 74.1 50.0 32.8 66.7 75.0

Tocomparetheperformanceofastudentgroupwith(thoseof)previousyears. 33.6 66.7 58.3 49.6 77.8 75.0

Nouseoftestresults 5.0 8.3 5.1 1.7 0 0

AppendixB.

(12)

Marsh,J.A.(2012).Interventionspromotingeducators’useofdata:Researchinsights andgaps,114(11),1–48Retrievedfromhttp://www.rdc.udel.edu.

Meijer,J.,Ledoux,G.,&Elshof,D.(2011).Gebruikersvriendelijkeleerlingvolgsystemen inhetprimaironderwijs[User-friendlypupilmonitoringsystemsinprimary education](Report849).Amsterdam:KohnstammInstituut.

Monaghan,W.(2006).Thefactsaboutsubscores(ReportNo.RDC-04).Princeton,NJ: EducationalTestingService.

NEGP(1998).Talkingabouttests:Anideabookforstateleaders.Washington,DC:U.S. GovernmentPrintingOfﬁce.

Newby,P.(2010).Researchmethodsforeducation.Harlow,England:Longman.

Newton,P.E.(2005).Thepublicunderstandingmeasurementinaccuracy.British EducationResearchJournal,31(4),419–442.http://dx.doi.org/10.1080/ 01411920500148648.

Popham,W.J.(2009).Assessmentliteracyforteachers:Faddishorfundamental? TheoryIntoPractice,48(1),4–11.http://dx.doi.org/10.1080/

00405840802577536.

Ryan,J.M.(2006).Practices,issues,andtrendsinstudenttestscorereporting.InS. M.Downing,&T.M.Haladyna(Eds.),Handbookoftestdevelopment(pp.677– 710).Mahwah,NJ:LawrenceErlbaum.

Schildkamp,K.,&Kuiper,W.(2010).Data-informedcurriculumreform:Whichdata, whatpurposes,andpromotingandhinderingfactors.TeachingandTeacher Education,26(3),482–496.http://dx.doi.org/10.1016/j.tate.2009.06.007.

Schildkamp,K.,&Lai,M.K.(2013).Data-baseddecisionmaking:Anoverview.InK. Schildkamp,M.K.Lai,&L.Earl(Eds.),Data-baseddecisionmakingineducation: Challengesandopportunities(pp.9–21).Dordrecht:Springer.http://dx.doi.org/ 10.1007/978-94-007-4816-3_1.

Schildkamp,K.,&Teddlie,C.(2008).Schoolperformancefeedbacksystemsinthe USAandinTheNetherlands:Acomparison.EducationalResearchandEvaluation, 14(3),255–282.http://dx.doi.org/10.1080/13803610802048874.

VanGeel,M.,Keuning,T.,Visscher,A.J.,&Fox,J.(2016).Changesineducators’data literacyduringadata-baseddecisionmakingintervention.InT.Keuning,&M.J. M.VanGeel(Eds.),Implementationandeffectsofaschoolwidedata-baseddecision makingintervention:Alargeskillstudy(pp.73–96).UniversiteitTwente: Enschede,TheNetherlands.

VanderKleij,F.M.,&Eggen,H.J.M.(2013).Interpretationofthescorereportsfrom thecomputerprogramLOVSbyteachers,internalsupportteachersand principals.StudiesinEducationalEvaluation,39(3),144–152.http://dx.doi.org/ 10.1016/j.stueduc.2013.04.002.

VanderKleij,F.M.,Eggen,T.J.H.M.,&Engelen,R.J.H.(2014).Towardsvalidscore reportsinthecomputerprogramLOVS:Aredesignstudy.StudiesinEducational Evaluation,43,24–39.http://dx.doi.org/10.1016/j.stueduc.2014.04.004.

Vanhoof,J.,Verhaeghe,G.,Verhaeghe,J.P.,Valcke,M.,&VanPetegem,P.(2011).The inﬂuenceofcompetencesandsupportonschoolperformancefeedbackuse. EducationalStudies,37(2),141–154.http://dx.doi.org/10.1080/

03055698.2010.482771.

Verhaeghe,G.,Vanhoof,J.,Valcke,M.,&VanPetegem,P.(2011).Effectenvan ondersteuningbijschoolfeedbackgebruik[Effectsofsupportinschoolfeedback use].PedagogischeStudien,88(2),90–106.

Wainer,H.,Hambleton,R.K.,&Meara,K.(1999).Alternativedisplaysfor communicatingNAEPresults:Aredesignandvaliditystudy.Journalof EducationalMeasurement,36(4),301–335. http://dx.doi.org/10.1111/j.1745-3984.1999.tb00559.x.

Wiliam,D.,&Thompson,M.(2007).Integratingassessmentwithinstruction:What willittaketomakeitwork?InC.A.Dwyer(Ed.),Thefutureofassessment: Shapingteachingandlearning(pp.53–82).Mahwah,NJ:Erlbaum.

Wiliam,D.(2011).Embeddedformativeassessment.Bloomington,IN:SolutionTree Press.

Zapata-Rivera,J.D.,&Katz,I.R.(2014).Keepingyouraudienceinmind:Applying audienceanalysistothedesignofinteractivescorereportsAssessmentin Education:Principles.Policy&Practice,21(4),442–463.http://dx.doi.org/ 10.1080/0969594X.2014.936357.

Zapata-Rivera,D.,&Zwick,R.(2011).Testscorereporting:PerspectivesfromtheETS scorereportingconference(ETSRR–11-45). Princeton,NJ.

Zapata-Rivera,D.,VanWinkle,W.,&Zwick,R.(2012).Applyingscoredesignprinciples inthedesignofscorereportsforCBALTM

teachers(ETSRM-12-20). Princeton,NJ.

Zenisky,A.L.,&Hambleton,R.K.(2012).Developingtestscorereportsthatwork: Theprocessandbestpracticesforeffectivecommunication.Educational Measurement:IssuesandPractice,31(2),21–26. http://dx.doi.org/10.1111/j.1745-3992.2012.00231.x.

Zwick,R.,Sklar,J.C.,Wakeﬁeld,G.,Hamilton,C.,Norman,A.,&Folsom,D.(2008). Instructionaltoolsineducationalmeasurementandstatistics(ITEMS)for schoolpersonnel:Evaluationofthreeweb-basedtrainingmodules.Educational Measurement:IssuesandPractice,27(2),14–27. http://dx.doi.org/10.1111/j.1745-3992.2008.00119.x.

Dorien Hopster-den Otter is a Ph.D. candidate at the Research Center for Examinations and Certiﬁcation (RCEC), a collaboration between Cito and the University of Twente. Her research interests include formative assessments, psychometricsandscorereportdevelopment.Sheiscurrentlyworkingonprojects relatingtothequalityofeducationaltestsandthedevelopmentofaneducational master’sprogrammefortestdevelopment.

SaskiaWoolsisaneducationalresearcherandmanagerofprototypingatCito.Her researchinterestsincludethevalidityandvalidationofeducationalassessments. Sheiscurrentlyworkingoninnovativeprojectsinvolvingformativeassessments, educationaltechnologyandassessmentquality.

TheoJ.H.M.EggenisaseniorresearchscientistatthePsychometricResearch CenterofCitoandaprofessorofpsychometricsattheUniversityofTwenteinthe Netherlands. He is director of the Research Center for Examinations and Certiﬁcation (RCEC). Hisresearch interestsinclude thequality ofeducational testingandcomputerized(adaptive)testing.HeisalsoworkingonprojectslikePISA (international educational survey)and on a varietyof projects onformative educationalassessment.

Bernard P. Veldkampis head of the Department of Research Methodology, MeasurementandDataAnalysisandthescientiﬁcdirectoroftheResearchCenter forExaminationandCertiﬁcation(RCEC).Hisresearchinterestsinclude measure-mentoptimizationandbehavioraldatascience.Hisinterestsfocusoncomputerized assessment.Hiscurrentprojectsincludeobtrusiveassessmentinonlinelearning andareviewstudyontheconditionsforeffectiveformativeassessment.