Additional file 2 – Critical Appraisal ToolItemExplanation

(1)

Additional file 2 – Critical Appraisal Tool

Item Explanation

Validity and reliability studies

1 If human subjects were used, did the authors give a detailed description of the sample of subjects used to perform the (index) test?

This item can be scored yes if:

1. the sample characteristics (e.g. height, weight, age, diagnosis, symptom status) were described or the manner of recruiting subjects were stated or if selection criteria were applied

If none of the above have been described or if insufficient information was provided, select “no”.

If inhuman or inanimate objects were used, select N/A.

2 Did the authors clarify the

qualification, or competence of the rater(s) who performed the (index) test?

1. the rater(s) characteristics (e.g. qualification,

specialization, amount of experience using the instrument under investigation) have been described

If the above have not been described or insufficient information was provided, select “no”.

Validity

studies 3 Was the reference standard explained?

1. the reference standard is likely to produce correct measurements

2. the reference standard is the best method available 3. details (name of the instrument, references to the

accuracy of the instrument) of the reference standard is reported

If none of the above is applicable to the reference standard’s description then select “no”.

Reliability

studies 4 If interrater reliability were tested, were raters blinded to the findings of other raters?

1. it is stated that the raters were blinded to each other’s findings or if a description, that implies that the raters were blinded, were reported

If no information is provided then select “no”.

If intrarater reliability were examined then select “N/A”.

Reliability

studies 5 If intrarater reliability were tested, were raters blinded to their own prior findings of the test under evaluation?

1. rater(s) have examined the same subjects on more than one occasion, it should be stated whether the rater(s) were blinded to the subjects they have examined previously

If insufficient information is provided then select “no”.

If interrater reliability were examined then select “N/A”.

Reliability

studies 6 Was the order of examination varied?

1. the order in which subjects were tested varied between raters if interrater reliability were tested.

2. the order of subjects were varied when intrarater reliability were tested

If varied order of examination is unnecessary or impractical (e.g. rater(s) digitizing or reading X-rays) then select “N/A”.

Validity

studies 7 If human subjects were used, was the time period between the reference standard and the index test short enough to be reasonably sure that the target condition did not change between the two tests?

1. results from the index test and the reference standard were collected on the same subjects at the same time 2. a delay between measurements occurs, it is important that

the target condition should not change between measurements.

If the time period between performing the index test and the reference standard was sufficiently long that the target condition may have changed between the two tests or if insufficient information is provided then select "no".

If inhuman or inanimate objects were used then select N/A.

Reliability

studies 8 Was the stability (or theoretical stability) of the variable being measured taken into account when determining the suitability of the time-interval between repeated measures?

1. the stability of the variable is known or reported and reviewers then decide on an appropriate time interval between repeated measures (stability of a test variable can only be determined if there is a reference standard) 2. there is no reference standard then the reviewers should

agree upon the theoretical stability of the variable and decide on an appropriate time interval between repeated measures

Validity

studies 9 Was the reference standard independent of the index test?

1. it is clear from the study that the index test did not form part of the reference standard

If it appears that the index test formed part of the reference standard then select "no".

10 Was the execution of the (index) test described in sufficient detail to permit replication of the test?

1. the study reported a clear description of the measurement procedure (e.g. the positioning of the instrument or rater, execution sequence of events)

2. if citations of methodology were supplied

(2)

The extent to which details is expected to be reported depends on the ability of different procedures to influence the results and on the type of instrument or test under evaluation If insufficient information is provided then select “no”.

Validity studies

11 Was the execution of the reference standard described in sufficient detail to permit its replication?

1. the study reported a clear description of the measurement procedure (e.g. the positioning of the instrument or rater, execution sequence of events)

2. if citations were supplied

12 Were withdrawals from the study explained?

1. it is clear what happened to all subjects who entered the study

2. if subjects who entered but did not complete the study is taken into account

If it appears that subjects who entered but did not complete the study were not accounted for or if insufficient information is provided then select "no "

If inhuman or inanimate objects were used then select N/A.

13 Were the statistical methods appropriate for the purpose of the study?

1. the analysis is appropriate in terms of the research question

2. the analysis is appropriate in terms of the type of data 3. a justification for the choice of statistical tests is provided 4. statistical analysis were reported using appropriate

statistical measures

5. where possible, statistical analysis for validity and reliability studies incorporates measures of variability e.g.

95% CI

If the analysis is not appropriate then an explanation should be provided

If insufficient information was provided then select “no”.

(index test / test = measurements from the tested instrument)

Definitions

Validity: The degree to which an instrument measures what it is intended to measure and the extent to which the values obtained are similar to the true values (Portney and Watkins 2009).

Criterion-related validity: The ability of one test (index test) to predict results obtained on an external criterion (gold standard/reference standard) that is already established or assumed to be valid. When both tests are performed on the same subjects, the scores from the index test are correlated with those achieved by the criterion measure (Portney and Watkins 2009).

Concurrent validity: When the index test and the criterion measure are taken at the same time so that it reflects the same incident of behaviour (Portney and Watkins 2009).

Predictive validity: When the index test is performed and followed by a period of time after which the criterion measure is obtained, the relationship between the index test and the criterion scores determines whether the index test is a valid predictor of the outcome of the criterion measure (Portney and Watkins 2009).

Construct validity: The ability of an instrument to measure an abstract concept, which cannot be observed directly and which has been constructed to represent an abstract trait (Portney and Watkins 2009).

Convergent validity: Indicates that two measures, which are believed to reflect the same construct, will have similar results or will correlate highly (Portney and Watkins 2009).

Divergent validity: Indicates that two measures, which are believed to measure different constructs, will correlate poorly (Bannigan and Watson 2009).

Discriminative validity: The extent to which measures from a measurement instrument distinguishes between individuals or populations that would be expected to differ (Bannigan and Watson 2009).

Reliability: The degree of consistency with which an instrument or observer(s) measure a variable and to which extent the measurement is free from error (Portney and Watkins 2009).

Test-retest reliability: Describes the stability of the assessment instrument in obtaining the same results with repeated measurements using the identical test on two or more separate occasions, keeping all testing conditions as constant as possible (Portney and Watkins 2009)

Intrarater reliability: The stability of data recorded by one observer across two or more trials of which the variables being rated are fixed and time is the only factor that varies between administrations (Karanicolas et al. 2009).

Interrater reliability: The extent to which two or more observers obtain similar scores when rating the same individuals (Karanicolas et al. 2009).