Evaluation and testing

(1)

EVALUATION*&*TESTING Wouter'van'den'Hoogen Measurement Measurement*is'the'assignment'of'numbers'to'objects' or'events.' ! Any'measurement'can'be'judged'by'the'following'meta; measurement'criteria'values:'level'of'measurement' (which'includes'magnitude),'dimensions'(units),'and' uncertainty.'' 2 They'enable'comparisons'to' be'done'between'diﬀerent' measurements'and'reduce' confusion.'' ! Even'in'cases'of'clear' qualitaGve'similarity'or' diﬀerence,'increased' precision'through' quanGtaGve'measurement'is' oHen'preferred'in'order'to' aid'in'replicaGon.' Evalua9on*of*user*experience •OHen'lacking'a'direct'relaGon'with'the'concept' •Inexact'' •Noise!' •Problems'with'replicaGon' ! •QuanGfy' •_Reliable' •_Valid

(2)

5 5*important*ques9ons*for*any*research •Why'do'we'perform'the'research?' •_{What'is'the'research'quesGon?'} •_{Who'do'we'want'to'study?'} •_{Where'do'we'perform'the'study?'} •How'do'we'collect'and'analyse'the'data? 6 Measuring*user*experience •QuesGonnaire' •ObservaGon'' ! ! •Interview' •Physiology' •Behaviour' •Eye;tracking' •_Think'Aloud' •_{Eﬃciency'&'Accuracy'(mistakes)'measures}

(3)

9 10 The*SeEng Develop*a*game*in*the*baHle*against*HIV/AIDS The'game'(s)'will'be'a'success'if:' ! •Children'are'empowered'to'make'informed'decisions'about' their'sexual'acGvity'and'other*important*choices'about'their' lives' •The'games'promote*community*involvement'in'HIV'care'and' treatment.'' •Increase*levels*of*knowledge'to'curb'sGgma'and' discriminaGon'in'school'communiGes Outcome*oriented The'details'deﬁne'the'setup:' •_{make'informed'decisions…'(absolute)'} •promote'community'involvement…'(increase)' •_{increase'levels'of'knowledge…'(increase)}

(4)

Outcome*oriented The'details'deﬁne'the'setup:' •make'informed'decisions…'(absolute)' •_{promote'community'involvement…'(increase)'} •increase'levels'of'knowledge…'(increase)' ! Asking'for'a'diﬀerence:' •Pre'and'post'measurement'(baseline)' •_{Control'condiGon'} •SubjecGve'change'paradigm 13

ObservaGon,'QuesGonnaires,''

and'Interviews

14 Comparison

Observation Questionnaire Interview

Direct Indirect Flexibel, explorative

Relatively objective Simple? In depth information

(Unconscious) behaviour Often evaluation Mix

Time intensive Relatively cheap Time intensive

Unobtrusive Intrusive

(5)

17 Levels*of*formality 18 Least*formal Everyday'/'casual Casual'/'SituaGon'specific Formal'/'structural Most*Formal Defining*the*level*of*observa9on Detailed* Small'units' ObjecGve' Precise' DescripGve' ! looks'leH' Takes'of'lid' Touches'hand' ! Macro* Large'units' SubjecGve' Less'accurate' InterpretaGon' ! Walks'home' Eats'candies' Makes'homework Quan9fying*behaviour •_Frequency' •_DuraGon' •_Intensity' •DirecGon' ! •Who'does'what,'where,'when'en'with'whom' •Inter'rater'reliability'should'reflect'this!

(6)

QuesGonnaires

21 About*aEtude*ques9ons "Answering'an'altude'quesGon'entails'several'tasks'(Strack'&'MarGn,' 1987;'Tourangeau,'1984):'Respondents'(a)'need'to'interpret'the'quesGon' to'determine'the'altude'object'and'evaluaGve'dimension'the'researcher' has'in'mind.'Next,'they'(b)'need'to'retrieve'relevant'informaGon'from' memory.'…'Based'on'this'informaGon,'respondents'(c)'need'to'compute'a' judgment.'Having'formed'a'judgment,'they'(d)'can'rarely'report'it'in'their' own'words'but'need'to'map'it'onto'a'set'of'response'alternaGves' provided'by'the'researcher.'Finally,'(e)'respondents'may'want'to'edit'their' private'judgment'before'they'communicate'it'to'the'researcher'for' reasons'of'social'desirability'and'self;presentaGon.”''' ! [Schwartz'&'Bohner] 22 Careful*with*interpreta9on! A'couple'of'interesGng'hypotheses'(see:'Pronin,'Kruger,'Savitsky'&'Ross,'2001:' in'Gilovich,'T.,'Griﬃn'D.,'and'Kahneman,'D.'(2002)'HeurisGcs'and'Biases,'p656)' ! 1. People'perceive'their'own'self;knowledge'and'insights'to'be'more'accurate' and'complete'than'that'of'others' 2. People'perceive'their'knowledge'of'other'people'to'be'more'accurate'and' complete'than'other'people’s'knowledge'of'them' 3. People'perceive'the'discrepancy'between'their'self;knowledge'and'other' people’s'knowledge'of'them'to'be'greater'than'the'corresponding' discrepancy'between'other'people’s'self;knowledge'and'their'knowledge'of' those'people.' 4. People'perceive'their'group’s'knowledge'and'their'knowledge'to'be'more' accurate'and'complete'than'other'group’s'knowledge'of'their'group.

(7)

EvaluaGon'vs'Behaviour

When'possible'consider'behaviour' over'evaluaGon' •Awareness'=/='behaviour'change' •Knowledge'=/='behaviour'change' ! ! MoGvaGon'and'situaGon'can'result' in'persons'to'be' •_{Unconscious'vs'Conscious'} •_{Competent'vs'Incompetent} 25

Designing'a'QuesGonnaire

26 Steps*when*designing*a*ques9onnaire 1. Deﬁne'what'you'want'to'measure' 2. Design'the'quesGons'and'answering'scales' 3. Create'the'quesGonnaire' 4. Test'the'quesGonnaire What*to*measure •Brainstorm,'make'lists,'use'literature' •Reduce' ! •Only'ask'what'you'need'to'know' •Aim'for'no'more'than'15'minutes

(8)

Design*ques9ons QuesGon' ! ! •_{“How'much'fun'did'you'} have'in'the'last'quest?”' •Good'for'gaining' informaGon'(age,'raGngs' of'fun,'etc)  Statement' ! ! •_{“The'last'quest'was'a'lot'} of'fun”'' •Statements'good'for' assessing'agreement' with'an'idea 29 Design*Ques9ons •Use'clear,'everyday'language' •_{Especially'for'children!'} ! ! ' ' ' “I'really'felt'like'I'idenGﬁed'with'the'people'in' ' ' ' the'game”'' ! ' ' ' “The'people'in'the'game'were'just'like'me” 30 Answering*op9ons Closed'(choice,'easy,'reacGve,'guessing,'social'desirable)' vs'Open'(acGve)' ! •Indicate'your'agreement'with'the'following'statement:  I'can'avoid'HIV'by'wearing'a'condom'during'sex.' ! •_{What'are'ways'to'avoid'HIV'infecGon?} ……… ………' Answering*op9ons Both'the'quesGon/statement'and'the'scale'should'be' clear,'and'exhausGve' ! What'educaGon'do'you'have?' Primary'School' Secondary'School' Bachelors' Masters

(9)

Answering*op9ons What'is'your'highest'completed'level'of'educaGon?' ! Primary'School' ' ' Secondary'School' Trade'School' Bachelors' Masters' PhD' Other:______________ 33 Scale*decisions Scale'construcGon' ! •_ConGnuous' ! •Interval' ! •Dichotomous 34 Scale*decisions

Continuous Interval Dichitomous

Place a mark on a continuum

Equal steps, circle one

answer Yes/No - True/False

Rich data Balanced Low variance

Free Guided Forced

Scales

(10)

37

How successful have you been in life, so far?

0 not at all successful 1 2 3 4 5 6 7 8 9 10 extremely successful

How successful have you been in life, so far?

-5 not at all successful -4 -3 -2 -1 0 1 2 3 4 5 extremely successful 38 0 10 20 30 40 0 1 2 3 4 5 6 7 8 9 10 0-10 scale -5 to 5 scale 0 1 2 3 4 -5 -4 -3 -2 -1 0 1 2 3 4 5 0-10 scale -5 to 5 scale Scales*and*the*midpoint

Unipolar Not*at*all _completelyAgree*

0 1 2 3 4

◘ ◘ ◘ ◘ ◘

;2 ;1 0 1 2

(11)

Scale'decisions

•Scale'(numbering)' inﬂuences'answers'and' variance' • bipolar'vs'unipolar' • bipolar'implies'dimension' • ‘Neutral’'point 41 Scale*decisions •Scale'(numbering)'inﬂuences'answers'and'variance' •bipolar'vs'unipolar' •bipolar'implies'dimension'&'‘0’'point' ! •Neutral'point,'' •no'answer?' •do'not'know?' ! •Label'each'answering'point'when'possible 42 Scales*and*the*midpoint

Do*not*agree _SomewhatAgree* Moderately*_agree Agree*a*lot Completely*_Agree

0 1 2 3 4

◘ ◘ ◘ ◘ ◘

;2 ;1 0 1 2

Disagree Moderatly*_disagree agree*/*do*Do*not* not*disagree moderately* agree Agree Neutral Do'not'know Layout Whichever'the'scale:' •Be'consistent'and'don’t'switch'scales' •Problem'with'lazy'responding?' •Use'a'‘trick’'quesGon/statement'e.g.'“Agree'with'this'statement”' ! •Use'even'spacing' ' ' ' 1'''2''''''''3''''''4''5' ! ' ' ' 1''''2''''3''''4''''5

(12)

45 Create*ques9onnaire •Order'eﬀects'/'Context' •_{AssimilaGon'&'Contrast'} ! •_{Careful'placement'} •RandomisaGon 46 Test*ques9onnaire