Amsterdam University of Applied Sciences

Amsterdam University of Applied Sciences

Script concordance test; a tool to assess cognitive integration?

Ramaekers, Stephan

Publication date 2016

Document Version Other version

Link to publication

Citation for published version (APA):

Ramaekers, S. (2016). Script concordance test; a tool to assess cognitive integration?. Paper presented at International Association of Medical Science Educators 2016, Leiden,

Netherlands.

General rights

It is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), other than for strictly personal, individual use, unless the work is under an open content license (like Creative Commons).

Disclaimer/Complaints regulations

If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please contact the library:

https://www.amsterdamuas.com/library/contact/questions, or send a letter to: University Library (Library of the University of Amsterdam and Amsterdam University of Applied Sciences), Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.

Amsterdam School of Health Professions

Script Concordance Test; a tool to assess cognitive integration?

Stephan Ramaekers, PhD

Amsterdam University of applied Sciences IAMSE, Leiden 2016

Cognitive integration – part 3

1

Cognitive integration - part 3

•Characteristics of the SCT

•Example cases and items

•Scoring and concordance

•Research results about the SCT

Amsterdam School of Health Professions

Script Concordance Test (SCT)

1. Method of assessing clinical reasoning

•Case-based

•Involving professional judgements and decision making

•Beyond evidence-based medicine 2. Theory / research grounded

•Authentic problems and circumstances

•Knowledge organization / Scripts

•Decision making under uncertainty

•Variability in reasoning of experts 3. Standardized

•Same stimuli

•Objective automated correction

3

Amsterdam School of Health Professions

SCT: key points

•degree of concordance between examinee performance and a reference panel of experienced physicians on a series of case-based tasks

•Each of these tasks consists of judging the effect a new data has on the status of an option in a clinical situation comporting uncertainty

•Format:

• A briefly described clinical problem

• With new information presented

• reviewing a clinical decision (diagnosis, assessment strategy, preventive / therapeutic intervention)

•Inferences:

• Clinical reasoning quality

• Organisation of knowledge

4

Amsterdam School of Health Professions

Clinical reasoning

(Selective) gathering of information

Organizing information / relations / patterns Interpreting findings / judgements

Drawing conclusions / making decisions / choices Evaluating / reflecting

5

reasoning modes

49 ² =

1) 49 * 40 + 9 * 49 = 1960 + 441 = 2401 2) 50²– 50 – 49 = 2500 – 99 = 2401

3) 2401

Amsterdam School of Health Professions

Clinical reasoning and basic sciences

Problem solving and reasoning:

•require a knowledge base of basic sciences

•reorganize knowledge: theory versus practice, integration (e.g.

Schmidt, 1996; Norman 2004)

•have a limited transfer to new situations (e.g. Hoffmann &

Ritchie, 1997)

•depend on the context in which problems and solutions are embedded (e.g. Wiggins, 1993; Cumming, 1999; Fantano, 2005)

•Situated cognition and learning (e.g. Anderson, 1996;

Newmann, 1996)

7

Amsterdam School of Health Professions

Theory versus practice

• Direction of reasoning (forwards – backwards)

• Scheme- or script development

• Fundamental details

‘embedded’

8

Amsterdam School of Health Professions

Knowledge encapsulation

Explicit basic science knowledge is largely forgotten by the time students

graduate at medical school.

(Ling et al, 2008)

9

SCT and illness scripts

Scripts, knowledge organised for specific tasks:

•Knowledge networks

•Relations / links: clinical features / clinical entities

•Organized for tasks (diagnosis / management . . .)

•Multiple micro-judgements in the process

E. Custers - Thirty years of Illness Scripts Medical Teacher, 2015, 37: 457-462

Amsterdam School of Health Professions

Cognitive integration - part 3

•Characteristics of the SCT

•Example cases and items

•Scoring and concordance

•Research results about the SCT

11

Amsterdam School of Health Professions

Item format SCT: diagnostic

Clinical scenario / stem

If you were thinking of And then you find This hypothesis becomes An hypothesis

new clinical data -2 -1 0 +1 +2

- 2= ruled out or almost ruled out, - 1= less probable

0= neither less or more probable + 1= more probable

+ 2= certain or almost certain

12

Amsterdam School of Health Professions

Example: Pediatric Emergency Medicine

(B. Carrière)

A three-year old girl presents to the ER with important sialorrhea, diminished neck movements, and fever for more than 24 hours. Parents report no recent trauma, and no episode of foreign body obstruction

If you were thinking of And then you find This hypothesis

becomes An epiglottitis An updated vaccination against

Haemophilus influenzae B -2 -1 0 +1 +2

- 2= ruled out or almost ruled out, - 1= less probable

0= neither less or more probable + 1= more probable

+ 2= certain or almost certain

13

Therapeutic decision

If you are considering

to treat with And you find The effect on your therapeutic option is

a therapeutic option

new clinical information

-2 -1 0 +1 +2

-2 Totally contra-indicated -1 Rather contra-indicated

0 Neutral +1 Appropriate +2 Very appropriate

Amsterdam School of Health Professions

Example: Radiation oncology

(Carole Lambert)

A 56-year-old patient is referred to you for the treatment of a T3 N2 M0 non-small cell lung cancer.

If you are considering

to treat with And you find The effect on your therapeutic option is A combined chemoradiation

treatment with curative intent

A 2.5 cm supraclavicular lymph node

-2 -1 0 +1 +2

-2 Totally contra-indicated -1 Rather contra-indicated

0 Neutral +1 Appropriate +2 Very appropriate

15

Amsterdam School of Health Professions

Example: Pre-operative reasoning

(S. Meterissian, McGill)

A 50 year old male sunworshipper presents with a 2 cm. dark pigmented lesion on the R calf. Initial punch biopsy revealed a 2.3 mm. Breslow

thickness superficial spreading melanoma. In your management:

If you were planning… and found… the planned management is:

A wide local excision plus only a sentinel node biopsy of the groin nodes

The hot/blue node to be clearly black with tumor

-2 -1 0 +1 +2

- 2= Absolutely contraindicated - 1= Should be reconsidered 0 = Not affected + 1= Supported + 2= Absolutely correct

16

Amsterdam School of Health Professions

Cognitive integration - part 3

•Characteristics of the SCT

•Example cases and items

•Scoring and concordance

•Research results about the SCT

18

Aggregate scoring method

•A group of experienced practitioners (reference panel)

•All answers are recorded

•Points depend of the number of reference panel answers

Anchors on SCT question -2 -1 0 +1 +2

Times chosen by panel 0 0 2 9 4

Modal answer calculation 0/9 0/9 2/9 9/9 4/9 Points to examinee - modal 0 0 0.22 1.0 0.44 Panel: 15

Amsterdam School of Health Professions

Variability in expert responses

(Ramaekers, 2011)

0 5 10

question 12

0 5 10 15

question 19

0 2 4 6

question 21 0

2 4 6

question 29

20

Amsterdam School of Health Professions

Cognitive integration - part 3

•Characteristics of the SCT

•Example cases and items

•Scoring and concordance

•Research results about the SCT

21

Amsterdam School of Health Professions

Research results SCT

SCT-format developed in 1998. Subsequent studies:

•Validity(scores reflecting extent of CR experience, covering many different (sub)domains, types of cases, SCT-format, etc.)

•Reliability/ generalisability(minimum numbers of

cases/questions, repeated measurements, scoring practices, etc.)

•Reference panel (size and composition, variability/ consensus) and SCT construction

•Participants(perceived value, development tracking, response process in SCT, etc.)

22

e.g. CR experience

•

Clerks

•

Residents

•

Experienced

externesresidentspanel groupe

100 90 80 70 60 50 40 30

op_total

Amsterdam School of Health Professions

Further information

www.cme.umontreal.ca/tcs (PDF files on SCT)

24