A Comparison of Multiple Informants on the Behavioral and Emotional Problems in Children and Adolescents: The Agreement Between the Child Behavior Checklist (CBCL), the Youth Self Report (YSR) and the Health of the Nation Outcome Scale for Children and
Adolescents (HoNOSCA).
Masterscriptie Forensische Orthopedagogiek Graduate School of Child Development and Education Universiteit van Amsterdam C.C. Doll 11346876 Begeleiding: Dr. M.J. Noom Tweede beoordelaars: Dr. Y. Nijssen & Dr. A. Boon
2
Abstract
Clinicians often use various screening tools to assess the behavioral and emotional problems in children and adolescents. It has been suggested that the use of multiple informants for the
assessment of children and adolescents may be better than the use of one informant. Therefor the present study was aimed at examining the correspondence between the parent/caregivers,
patients and clinicians using the Child Behavior Checklist (CBCL), Youth Self Report (YSR) and Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA). Our total sample existed of 1393 patients. Total scores of the CBCL, YSR and HoNOSCA were compared using correlations. Correlations were also calculated to compare the CBCL domains and scales with the YSR domains and scales and with the HoNOSCA items. We also examined whether a disorder diagnosis was reflected in higher or lower scores on relevant scales and items of the CBCL and YSR. Lastly we explored which combination of CBCL and YSR scales best predicted the HoNOSCA items. We found that there was substantial agreement between the multiple informants, but not complete correspondence. Scores on CBCL, YSR scales and HoNOSCA items were able to reflect a diagnosed disorder, although not very good. Not all HoNOSCA items could be predicted with the CBCL and YSR, indicating that these measurements are not
completely comparable. We conclude that the use of multiple informants in the assessment is preferable. Also we recommend that measures should be more alike to better compare the unique contributions of the different informants.
3
A Comparison of Multiple Informants on the Behavioral and Emotional Problems in Children and Adolescents: The Agreement Between the Child Behavior Checklist (CBCL), the Youth Self Report (YSR) and the Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA).
Clinicians often use various screening questionnaires to gain information about the behavioral and emotional problems of children and adolescents. Mash & Hunsley (2005) suggest the use of multiple measurements with different informants in the assessment of children and adolescents, since the different views on the patient (caregiver, clinician, patient) complement each other to give a more complete picture of the patient. The Achenbach System of Empirically Based Assessment (ASEBA) produced the well-known and internationally recognized Child Behavior Checklist (CBCL), the Youth Self report (YSR) and the Teacher’s Report Form (TRF) (ASEBA, 2014). The CBCL, YSR and TRF are identical measures, only the CBCL is completed by the parents/caregivers, the YSR by the children and adolescents and the TRF by the teacher.
However, ASEBA did not produce a measurement with the clinicians point of view. The Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA) could be an instrument that fulfills that role.
In the Netherlands the CBCL, YSR and HoNOSCA have already been recommended to be used next to each other for Routine Outcome Monitoring (Bonnet, 2015). But, contrary to the Dutch versions of the CBCL and YSR, the Dutch version of the HoNOSCA has not yet had an extensive research on its psychometric properties. One way to test if an instrument is valid, is by its convergent validity. A good convergent validity means that an instrument correlates high with another instrument that is supposed to measure the same. The CBCL, YSR and HoNOSCA generally try to measure the same: the presence/severity of psychopathological symptoms in children and adolescents. Therefor we expect them to generate similar results. Yet prior research has shown that there may only be moderate agreement between different informants and one measure as a ‘golden standard’ may not be found. For example, even between the CBCL and YSR, which are identical measurements, correlations vary between .20 and .44 (Achenbach, McConaughy & Howell, 1987; Salbach-Andrae, Lenz & Lehmkuhl, 2009). To further explore the agreement between multiple informants the present study was aimed at examining the correspondence between the CBCL, YSR and HoNOSCA.
4 The CBCL is a questionnaire that is developed for assessing emotional and behavioral problems, competencies, and adaptive functioning for children and adolescents aged 6-18 years (ASEBA, 2014). Parents or caregivers of the children and adolescents assess many specific behaviors about the adolescents, which add up to scores on three domains: Internalizing, Externalizing and Other. These domains are made by adding up a combination of 8 underlying scales, which are made by adding up a combination of the items. The items can also be added up to form six DSM-oriented scales. Validity and reliability research has been done in multiple countries and showed good results (Achenbach et al., 2008). For instance, a systematic review on the comparison between the CBCL and the Strengths and Difficulties Questionnaires (SDQ) showed that the CBCL was valid and reliable, although it was less specific than the SDQ (Warnick, Bracken & Kasl, 2008). The YSR is the self-report version of the CBCL, designed for adolescents aged 11-18 years. The YSR contains 112 similar items to the CBCL’s 120 items. Just as the CBCL validity and
reliability research has been done and showed good results (Achenbach, 2008).
HoNOSCA
The HoNOSCA is an instrument that was developed as a routine global outcome measure
(Gowers et al., 1999). The HoNOSCA exists of 2 sections with 13 items in the first section and 2 items in the second section. With the first section clinicians can quickly rate the severity on emotional and behavioral problems of children and adolescents. The 13 items together generate a Total problems score. The second section is used to determine whether there may have been lacking information or accessibility to the treatment. Multiple research has been conducted which supports acceptable validity and reliability of the HoNOSCA (Bilenberg, 2003; Bran & Coleman, 2010; Garralda, Yates & Higginson, 2000; Gowers et a.l, 1999; Holzer et al., 2006; Lesinskiene, Senina, Ranceva, 2007). Studies that have examined the convergent validity of the HoNOSCA found that it was satisfactory (Harnett, Loxton, Sadler, Hides, Baldwin, 2005;
Garralda, Yates, Higginson, 2000; Urben et al., 2014). For instance, Garralda et al., (2000) found that HoNOSCA change in problems was correlated with the Children’s Global Assessment Scale (CGAS), a measure clinicians use to measure the functioning of children and adolescents.
For the assessment of the children and adolescents Gowers et al. (1999) suggested the use of 5 scales made by adding a combination of items of the HoNOSCA rather than the use of just the Total problems score. However, Tiffin & Rolling (2012) note that the internal reliability consistency of these scales were poor. They suggest that using two symptom scales, based on
5 emotional problems items and behavioral problems items may be better. One study has shown that the use of these two subscales is promising compared to the use of just the total score, as they may provide a more differentiated representation of a patient, since it could also
discriminate between emotional problems and externalizing problems (Urben et al., 2015).
A comparison of the CBCL, YSR and HoNOSCA
Although the CBCL, the YSR and the HoNOSCA have differences in their format and content, they also share common ground in assessing mental-health problems in children and adolescents. Hanssen-Bauer, Langsrud, Kyernmo, and Heyerdahl (2010) examined the agreement between the HoNOSCA, the CBCL and the YSR. Their results have shown that the HoNOSCA Total score had a medium correlation (r = .49) with CBCL Total problems and a medium correlation (r = .41) with YSR Total problems. The HoNOSCA item Aggressive behavior had a large
correlation with the externalizing domain of the CBCL (r = .62) and a medium correlation with the externalizing domain of the YSR (r = .46). The HoNOSCA item Emotional symptoms had a medium correlation with the internalizing domain on the CBCL (r = .43) and the internalizing domain of the YSR (r = .52). Hanssen-Bauer and colleagues also found that when predicting HoNOSCA items with both a CBCL scale or the YSR equivalent, the CBCL was better at predicting the HoNOSCA items Aggressive behavior, Overactivity or Attention problems and Peer problems and the YSR was better at predicting the HoNOSCA item Emotional symptoms. Yet they do conclude that despite the correspondence, the results on the HoNOSCA items do not entirely match with the ASEBA scales. The psychosomatic scale on the HoNOSCA revealed a small correlation (r = .25) when compared with somatic problems of the CBCL and showed no correlation with somatic problems of the YSR. Furthermore the HoNOSCA item Self-injury had a large correlation with the YSR domain Internalizing (r = .63) and the HoNOSCA item
Abnormal thoughts or perceptions had a relatively small correlation with the YSR domain Internalizing (r = .34), but the same CBCL domains did not correlate with these HoNOSCA items. One other study has also examined the agreement between the clinician and the child or adolescents using the CBCL, which was filled in by the clinician and the YSR (Deighton, Croudace, Fonagy, Brown, Patalay, & Wolpert, 2014). They found medium correlations at the beginning of a treatment trajectory and more agreement as the treatment progressed.
The agreement between the CBCL and the YSR has been examined a little more. A recent study across 25 countries found a general correlation of .45 between CBCL Total
6 problems and YSR Total problems, but correlations varied between countries as low as .16 until as high as .66 (Rescorla et al., 2012). The general finding for the Internalizing domain was r = .45 ranging from .21 to .61 and for the Externalizing domain was r = .46 ranging from .13 to .66. The general correlation findings on the 8 scales and the 6 DSM-oriented scales ranged from r = .34 to .44. It varied more between countries than between the kind of problem. The correlations on the items were a lot better ranging from r = .72 to .94.
An advantage of a clinician based measure like the HoNOSCA compared to a parent or self-report is that clinicians can observe problems more objectively than the parents or the adolescents themselves. Clinicians had a lower agreement with out-patients than they had with in-patients, where they could observe the patients themselves, rather than relying on information given to them (Gowers, Levine, Bailey-Rogers, Shore & Burhouse, 2002).
As far as we know little research about the agreement between the CBCL, YSR, and HoNOSCA has been done, besides Hanssen-Bauer, Langsrud, Kvernmo, & Heyerdahl (2010). The aim of the present study was to further explore the agreement between the CBCL, YSR, and HoNOSCA. To explore the agreement between the CBCL, YSR and HoNOSCA this research had three research questions. First, what is the strength of the relationships between the HoNOSCA, CBCL and YSR scales and items at the beginning of a treatment trajectory? We hypothesize that similar items and scales of the HoNOSCA, CBCL and YSR will show high correlations. Secondly, can the presence or absence of a diagnosis be reflected in scores on the relevant scales and items of the CBCL, YSR and the HoNOSCA? We hypothesize that children and adolescents will have higher scores on items and scales of the CBCL, YSR and HoNOSCA when they are diagnosed with a disorder relevant to those items and scales. Thirdly, how well do the CBCL and YSR scales predict the HoNOSCA items? For the last explorative research
question we set the clinician based HoNOSCA as the golden standard, since the clinician decides what treatment will be used for the behavioral and emotional problems he thinks are necessary.
Method Sample
The sample consisted of 1393 participants of which 54.8% were male. The sample had an average age of 13.31 (SD = 3.40, range 6-18) and an average treatment duration of 364 days (SD = 226.93, range 7-1352). Table 1 presents the sample description. The participants were included by a consortium of multiple Dutch mental health services (ROMCKAP) who collected and
7 combined their data. This resulted in a database with anonymized patient information about the HoNOSCA, the CBCL and, the YSR. For the present study participants were included if they had a measurement at the admission of treatment (with a 90 day margin) and if they had at least a combination of two measurements (HoNOSCA-CBCL, HoNOSCA-YSR, CBCL-YSR). Of the 1393 participants 751 participants had a completed HoNOSCA, 1180 participants had a
completed CBCL and 1007 had a completed YSR. 152 participants of the 1393 had all three measurements completed.
Table 1: Sample description (N = 1393)
Total Sample
Mean (SD) or Frequency (%) Sex
Male 764 (54.8)
Female 629 (45.2)
Age, mean years 13.31 (2.76)
Duration of treatment, mean days 364.07 (226.93) Diagnosis
Mood Disorders 146 (10.5)
Anxiety Disorders 200 (14.4)
Attention Disorders 344 (24.7)
Autism Spectrum Disorders 274 (19.7)
Disorders of Childhood, or Adolescence NAO 127 (9.1) Conduct Disorders 87 (6.2) Personality Disorders 45 (3.2) V-Codes 19 (1.4) Other Disorders 131 (9.4) No Classification 20 (1.4) Measures
Health of the Nation Outcome Scale for Children and Adolescents (HoNOSCA) (Gowers et al., 1999)
8 The Dutch translation, HoNOS Jeugd, of the HoNOSCA was used (Staring, Hofman & Mulder, 2003). Clinicians answer 15 items about the adolescents on a five-point scale ranging from 0 (No problems) until 4 (severe problems) based on occurrence in the last two weeks. Clinicians score an item with 8 when they have no knowledge at all about the item. The first 13 items measure problems on: ‘Disruptive/antisocial/aggressive behavior’,
‘hyperactivity/attention/concentration problems’, ‘self-injury’, ‘substance abuse’,
‘scholastic/language skills’, physical disease/disability’, ‘hallucinations/delusions/abnormal perceptions’, ‘non-organic somatic symptoms’, ‘emotional symptoms’, peer relationships’, selfcare/autonomy’, ‘family relationships’, and ‘school attendance’. The last two items measure knowledge and understanding about the nature of the problems, and problems with information about the treatment. These two items were excluded in this research, because they do not give information about the mental health of children.
Child Behavior Checklist (CBCL) (Achenbach, 1991)
The Dutch translation of the CBCL for children and adolescents between 6 and 18 years old was used (Verhulst & Van der Ende, 1996). Parents or caregivers answer 120 items about their child on a three-point scale ranging from 0 (not true, as far as you know) until 2 (very true or often true) based on behavior in the past 2 months. The items are divided into different domains and scales. The domain ‘internalizing’ consists of the scales ‘anxious/depressed’,
withdrawn/depressed’, and ‘somatic complaints syndromes’. The domain ‘externalizing’ consists of the scales ‘rule breaking syndromes’ and ‘aggressive behavior’. The domain ‘other’ consists of the scales ‘social problems’, ‘thought problems’, and ‘attention problems’. The ‘other problems’ scale is not added in any domain, however it is included with all scales to form the score on the ‘total problems’ domain.
Youth Self Report (YSR) (Achenbach, 1991)
The Dutch translation of the YSR for adolescents between the age of 11 till 18 years was used (Verhulst & Van der Ende, 1997). This is a questionnaire with a similar 112 items and similar division into domains and subdomains as the CBCL. The adolescents answer the questions themselves on a three-point scale ranging from 0 (not true, as far as you know) until 2 (very true or often true) based on their behavior during the past six months.
9 All statistical analyses were conducted using SPSS 22. Pearson correlation analyses were
calculated to investigate the linear relationship between the HoNOSCA, CBCL, and YSR. To minimize the probability of a type I error, Bonferroni’s correction was applied.
Three-way and two-way analyses of variance (ANOVA) were conducted to examine if the presence or absence of a diagnosed disorder can be reflected in scores on relevant scales and items of the CBCL, the YSR and the HoNOSCA. Four diagnosed disorders were examined: Mood disorders, anxiety disorders, attention disorders and conduct disorders, since these disorders had CBCL and YSR scales that were directly related to the diagnoses that were
examined in this sample. For this analysis the CBCL and YSR items were rearranged into scales that correspond with the Diagnostic and Statistical Manual of Mental Disorders (DSM)
(Verhulst, Ende van der & Koot, 1996). The combined scales were, Affective problems, Anxiety problems, Somatic problems, Attention Deficit/Hyperactivity and Oppositional Defiant
problems. The HoNOSCA does not have DSM scales, but it does have items that should theoretically correspond with the disorders. Table 2 shows the explored disorders with their corresponding HoNOSCA, CBCL and YSR scales. Participants with one of these four diagnoses were expected to show a higher mean on the corresponding CBCL, YSR scales and HoNOSCA items, as compared to participants without this diagnose. To control for age, two groups were made, the first ‘younger’ group ranged from 3 to 10 and the second ‘older’ group ranged from 11 to 18. For the YSR only two-way ANOVA’s were conducted since the minimum age for a completed YSR was 11.
Table 2: Diagnosed disorders and their corresponding HoNOSCA, CBCL, YSR scales.
Diagnose HoNOSCA Scales CBCL/YSR DSM Scales
Mood disorders Emotional symptoms Affective problems Anxiety disorders Emotional symptoms Anxiety problems
Attention disorders Overactivity Attention Deficit/Hyperactivity problems Conduct disorders Disruptive Oppositional Defiant problems
Multiple regression analyses were conducted to investigate how much of the variability in the HoNOSCA scales could be explained by the CBCL and YSR scales. With multiple
regressions we could also investigate which combination of CBCL and YSR scales was best at predicting a HoNOSCA scale and investigate if the CBCL and YSR had the same scale as the best predictor. The ASEBA scales Total score, Internalizing, Externalizing and Other were not included in these models, because they are made up of the other scales and would therefore be
10 counted twice. To minimize the probability of a type I error, Bonferroni’s correction was
applied.
Results Measurements
All measurements were at the start of a treatment trajectory. Participants had an average total score of 12.1 (SD = 6.22) for the HoNOSCA which is an average to high score based on a Dutch patient sample (Boon, de Boer, de Haan, Nijssen & Klasen, n.d.) . Participants had an average total score of 61.11 (SD = 27.64) for the CBCL and 76.50 (SD = 26.25) for the YSR. For the domain Internalizing participants had a mean score of 18.91 (SD = 10.58) for the CBCL and 20.13 (SD = 11.84) for the YSR. For the domain Externalizing participants had a mean score of 15.88 (SD = 11.15) for the CBCL and 17.30 (SD = 8.21) for the YSR. For the domain Other participants had a mean score of 20.71 (SD = 9.81) for the CBCL and 25.07 (SD = 9.16) for the YSR. Both the CBCL and the YSR mean scores seem higher than scores found by Hanssen-Bauer, Langsrud, Kyernmo, and Heyerdahl (2010) and also a lot higher than the mean scores in 21 countries (Rescorla et al., 2012). The YSR scores were higher than the CBCL scores, which has been reported before (Achenbach et al., 2008). Table 3 presents the scores for the HoNOSCA and table 4 presents the scores for the CBCL and, YSR.
Table 3: Score distribution HoNOSCA (N = 751)
Mean SD Total score 12.1 6.22 Disruptive problems .93 1.09 Overactivity 1.61 1.27 Self-injury .31 .75 Substance Misuse .18 .64 Scholastic 1.23 1.25 Physical Illness .38 .85 Hallicinations Delusions .24 .65 Somatic .61 1.03 Emotional symptoms 1.70 1.29 Peers 1.62 1.23 Self-Care .79 .98 Family Problems 1.76 1.24 School Attend .75 1.33
11
Table 4: Score distribution CBCL (N = 1180) and YSR (N = 1007)
CBCL YSR Mean SD Mean SD Total score 61.11 27.64 76.50 26.25 Internalizing 18.91 10.58 20.13 11.84 Externalizing 15.88 11.15 17.30 8.21 Other 20.71 9.81 25.07 9.16 Anxious Depressed 8.28 5.40 8.46 6.20 Withdrawn Depressed 5.87 3.79 5.55 3.62 Somatic Complaints Syndromes 4.76 3.80 6.13 3.82 Social Problems 5.81 3.83 5.31 3.55 Thought Problems 5.66 4.02 10.32 4.21 Attention Problems 9.24 4.36 9.44 3.66 Rule Breaking Syndromes 5.01 4.54 7.80 3.77 Aggressive Behavior 10.87 7.58 9.50 5.35 Other Problems 5.60 3.35 14.00 3.44 Correlations
The level of significance was set at α = .0035 after the Bonferroni’s correction. In further tables the HoNOSCA item names are shortened to the first part of the official item names.
* Pearson correlation is significant at the 0.05 level after Bonferroni correction (2-tailed).
** according to Cohen (1988) correlation effect sizes are small = < .3, medium = .3 to .49 , large = > .5. Bold numbers are correlations expected to be high
Table 5: Correlations HoNOSCA-CBCL (N = 538)
HoNOSCA***/CBCL Internalizing Externalizing Other
Disruptive .056 .549* .161* Overactivity -.119 .137* .211* Self-injury .348* .143* .135* Substance Misuse .159* .253* .063 Scholastic -.045 .071 .130* Physical Illness .060 -.103 .020 Hallucinations Delusions .188* .058 .111 Somatic .186* -.096 -.057 Emotional symptoms .449* .064 .083 Peers .251* .152* .162* Self-Care .066 .084 .187* Family Problems .265* .321* .118 School Attend .229* .223* .009
12
HoNOSCA-CBCL
HoNOSCA Total score had a medium correlation (r = .360, p < .001) with CBCL Total score. The CBCL Internalizing domain had a medium correlation with the HoNOSCA item Emotional symptoms (r =.449, p < .001) and the CBCL Externalizing domain had a high correlation with the HoNOSCA item Disruptive problems (r = .549, p < .001). The CBCL domain Other did have a few significant correlations with HoNOSCA items, but they were all considered small. The correlations between the HoNOSCA items and the CBCL domains are shown in table 5.
High correlations were found between the HoNOSCA item Disruptive problems and the CBCL scales Rule breaking syndromes and Aggressive behavior. Medium correlations were found between the HoNOSCA item Emotional symptoms and the CBCL scales Anxious-depressed and Withdrawn-Anxious-depressed. There were also medium correlations between the HoNOSCA item Overactivity and the CBCL scale Attention problems and between the HoNOSCA item Substance misuse and the CBCL scale Rule breaking syndromes. We only
Table 6: Correlations HoNOSCA-CBCL (N = 538)
HoNOSCA/CBCL Anxious Depressed Withdrawn Depressed Somatic Complaints Syndromes Social Problems Thought Problems Attention Problems Rule Breaking Syndromes Aggressive Behavior Other problems Disruptive .081 .131* -.087 .157* .068 .164* .506* .520* .250* Overactivity -.052 -.133* -.120 .143* .002 .354* .113 .137* .150* Self-injury .335* .247* .229* .087 .254* -.012 .189* .104 .058 Substance Misuse .119 .123 .143* -.047 .109 .082 .430* .134* .090 Scholastic -.019 .033 -.124 .130* .016 .167* .059 .071 .030 Physical Illness -.024 .034 .160* .028 .042 -.019 -.085 -.104 -.019 Hallucinations Delusions .193* .135* .106 .074 .192* .007 .056 .054 -.011 Somatic .125 .033 .291* -.066 .077 -.146* -.078 -.096 -.041 Emotional symptoms .415* .365* .276* .075 .203* -.071 .070 .055 .011 Peers .203* .310* .096 .219* .182* .003 .153* .137* .088 Self-Care .057 .139* -.033 .166* .108 .176* .069 .085 .095 Family Problems .235* .247* .146* .128* .154* .010 .355* .272* .138* School Attend .169* .202* .183* -.011 .050 -.016 .303* .159* .044 * Pearson correlation is significant at the 0.05 level after Bonferroni correction (2-tailed).
** according to Cohen (1988) correlation effect sizes are small = < .3, medium = .3 to .49 , large = > .5 Bold numbers are correlations expected to be high.
13 found a small correlation between HoNOSCA item Somatic and CBCL scale Somatic complaints syndromes and between HoNOSCA item Peers and the CBCL scale Social problems.
Unexpected medium correlations were found between the HoNOSCA items Family problems and School attendance with the CBCL scale Rule Breaking Syndromes. All correlations between the HoNOSCA items and the CBCL scales are shown in table 6.
Correlations HoNOSCA-YSR
HoNOSCA Total score had a medium correlation (r = .406, p < .001) with YSR Total score. The YSR Internalizing domain had a high correlation with the HoNOSCA item Emotional symptoms (r =.547, p < .001) and the YSR Externalizing domain had a medium correlation with the
HoNOSCA item Disruptive (r = .363, p < .001). The YSR domain Other did have a few significant correlations with HoNOSCA items, but they were all considered small. The correlations between the HoNOSCA items and the YSR domains are shown in table 7.
High correlations were found between the HoNOSCA item Emotional symptoms and the YSR scales Anxious-depressed and Withdrawn-depressed. Medium correlations were found between the HoNOSCA item Disruptive problems and the YSR scales Rule breaking syndromes
Table 7: Correlations HoNOSCA-YSR (n = 365)
Internalizing Externalizing Other
Disruptive -.001 .363* .053 Overactivity -.118 .181* .143 Self-injury .468* .186* .241* Substance Misuse .158* .336* .133 Scholastic .054 .119 .102 Physical Illness .143* -.049 .045 Hallicinations Delusions .232* .197* .195* Somatic .225* -.080 .045 Emotional symptoms .547* .061 .273* Peers .404* .094 .235* Self-Care -.036 .062 .067 Family Problems .328* .213* .213* School Attend .271* .191* .142
* Pearson correlation is significant at the 0.05 level after Bonferroni correction (2-tailed).
** according to Cohen (1988) correlation effect sizes are small = < .3, medium = .3 to .49 , large = > .5 Bold numbers are correlations expected to be high.
and Aggressive behavior. There were also medium correlations between the HoNOSCA items Overactivity, Substance abuse, and Peers with the CBCL scales Attention problems, Rule
14 breaking syndromes, and Social problems respectively. We only found a small correlation
between the HoNOSCA item Somatic problems and the CBCL scale Somatic complaints
syndromes. Multiple unexpected medium correlations were found between the HoNOSCA items and YSR scales and are presented with all the correlations in table 8
Table 8: Correlations HoNOSCA-YSR (n = 365)
* Pearson correlation is significant at the 0.05 level after Bonferroni correction (2-tailed).
** according to Cohen (1988) correlation effect sizes are small = < .3, medium = .3 to .49 , large = > .5 Bold numbers are correlations expected to be high.
Correlations CBCL-YSR
CBCL Total score had a medium correlation (r = .421, p < .001) with YSR Total score. The CBCL domain Internalizing had a high correlation with YSR domain Internalizing (r = .559, p < .001). The CBCL domain Externalizing also had a high correlation with the YSR domain
Externalizing (r =.586, p < .001). The Other domain correlated medium with each other (r = .403). The CBCL Externalizing did not have a correlation with the YSR domain Internalizing, however the YSR domain Externalizing did have a significant small correlation with the CBCL domain Internalizing. The correlations between the CBCL domains and the YSR domains are shown in table 9. Anxious Depressed Withdrawn Depressed Somatic Complaints Syndromes Social Problems Thought Problems Attention Problems Rule Breaking Syndromes Aggressive Behavior Other problems Disruptive .039 .021 -.090 .028 .041 .060 .345* .321* -.006 Overactivity -.120 -.114 -.060 .044 .008 .316* .187* .149 -.013 Self-injury .473* .438* .262* .208* .297* .067 .228* .127 .084 Substance Misuse .158* .167* .072 .057 .111 .155* .466* .192* -.007 Scholastic .064 .008 .057 .096 .028 .137 .151 .078 .070 Physical Illness .128 .078 .165* .123 .064 -.079 -.033 -.054 .116 Hallicinations Delusions .234* .212* .134 .103 .254* .099 .182* .177* .035 Somatic .180* .167* .249* .079 .129 -.113 -.094 -.057 .122 Emotional symptoms .552* .510* .307* .300* .328* .024 .100 .024 .149 Peers .382* .455* .194* .329* .223* .022 .104 .072 .125 Self-Care -.049 .015 -.049 .076 .015 .081 .060 .054 .046 Family Problems .323* .300* .204* .216* .217* .081 .265* .144 .184* School Attend .251* .290* .155* .136 .157* .045 .237* .129 .054
15 Medium and high correlations were found for all CBCL and YSR corresponding scales except for Other problems which only had a small correlation. The scales Anxious Depressed and Withdrawn depressed also had medium correlations with each other, just as the scales Rule breaking syndromes and Aggressive behavior both from the CBCL and YSR point of view. The correlations between the CBCL scales and the YSR scales are shown in table 10.
Table 9: Correlations CBCL-YSR (n = 794)
CBCL/YSR Internalizing Externalizing Other
Internalizing .559* .139* .299*
Externalizing .036 .586* .195*
Other .171* .318* .403*
* Pearson correlation is significant at the 0.05 level after Bonferroni correction (2-tailed).
** according to Cohen (1988) correlation effect sizes are small = < .3, medium = .3 to .49 , large = > .5 Bold numbers are correlations expected to be high.
Table 10: Correlations CBCL-YSR (n = 794)
* Pearson correlation is significant at the 0.05 level after Bonferroni correction (2-tailed).
** according to Cohen (1988) correlation effect sizes are small = < .3, medium = .3 to .49 , large = > .5 Bold numbers are correlations expected to be high.
CBCL/YSR Anxious Depressed Withdrawn Depressed Somatic Complaints Syndromes Social Problems Thought Problems Attention Problems Rule Breaking Syndromes Aggressive Behavior Other problems Anxious Depressed .553* .401* .317* .341* .316* .082 .100* .145* .111* Withdrawn Depressed .311* .525* .194* .255* .128* .033 .105* .031 -.057 Somatic Complaints Syndromes .351* .313* .517* .219* .228* .080 .147* .078 .155* Social Problems .180* .148* .107* .438* .153* .089 .080 .274* .074 Thought Problems .328* .228* .195* .291* .456* .180* .189* .227* .119* Attention Problems -.043 -.042 -.036 .176* .114* .432* .240* .292* .057 Rule Breaking Syndromes .017 .017 .084 .081 .090 .251* .636* .452* .098* Aggressive Behavior .034 -.011 .040 .155* .085 .202* .321* .541* .099* Other problems .024 -.020 .109* .184* .104* .211* .263* .396* .282*
16 Secondly, can the presence or absence of a diagnosis be reflected in scores on the relevant scales and items of the CBCL, YSR and the HoNOSCA
Reflecting the presence or absence of a disorder in HoNOSCA, CBCL, and YSR scores
Multiple Three-way ANOVA’s were conducted to explore if the presence or absence of a
diagnosed disorder can be reflected in higher or lower scores on relevant scales of the CBCL and the YSR and items of the HoNOSCA controlled for Age and Sex. Because most analyses
violated the assumption of homogeneity a more stringent alpha level was necessary. The new significant value was set at α = .01 for all analyses. Mostly main effects were found throughout the analyses. The main effects for diagnosis are shown in table 11. Two interactions were found, but no three-way interactions.
Main effects for the diagnosis Mood disorder were found for the HoNOSCA and YSR. Children with a mood disorder scored higher on the relevant scale than children without a mood disorder. There was a significant interaction effect between diagnosis and age for the CBCL F(1, 1426) = 8.747, p = .003. In the older group, children with a mood disorder diagnosis scored higher than children without a mood disorder diagnosis, but in the younger group, children with a mood disorder diagnosis scored lower than the children without a mood disorder diagnosis.
Main effects for diagnosis Anxiety disorder were found for the HoNOSCA, CBCL and, YSR. Children with an anxiety disorder scored higher on emotional problems than children without an anxiety disorder and Children with an anxiety disorder scored higher on the Anxiety scale than children without an Anxiety diagnosis.
Main effects for the diagnosis Attention disorders were found for the HoNOSCA, CBCL and, YSR. Children with an Attention disorder scored higher on HoNOSCA Overactivity and CBCL/YSR Attention problems than children without an Attention disorder. A significant interaction effect was found between diagnosis and sex for the CBCL F(1, 1426) = 12.713, p < .001. Both Males and females with an attention disorder scored higher on attention problems than males and females without an attention disorder. However females without an attention disorder scored lower on attention problems than males without an attention disorder, but
females with an attention disorder scored as high as males with an attention disorder on attention problems.
Main effects for the diagnosis Conduct disorders were found for the HoNOSCA, CBCL, and YSR. Children with Conduct disorder scored higher on Disruptive behavior than children
17 without a Conduct disorder and Children with a Conduct disorder scored higher on the ODD scale than children without a Conduct disorder diagnosis.
Table 11: Three-way ANOVA significant main effects for diagnosis
Diagnosis Instrument Scale F df p ETA
Mood disorder HoNOSCA Emotional symptoms 43.20 1, 3195 <.001 .013 YSR Mood disorders 119.96 1, 1125 <.001 .096 Anxiety
disorder
HoNOSCA Emotional symptoms 200.66 1, 3195 <.001 .059 CBCL Anxiety disorder 31.81 1, 1426 <.001 .022 YSR Anxiety disorder 22.48 1, 1125 <.001 .020 Attention
disorder
HoNOSCA Overactivity 719.46 1, 3195 <.001 .184 CBCL Attention problems 116.32 1, 1426 <.001 .075 YSR Attention problems 157.04 1, 1125 <.001 .122 Conduct
disorder
HoNOSCA Disruptive 129.05 1, 3195 <.001 .039
CBCL ODD 25.27 1, 1426 <.001 .017
YSR ODD 29.65 1, 1125 <.001 .026
* according to Cohen (1988) Partial Eta squared are small = <.01, medium = .06 to .137 and large = >.138 Predicting HoNOSCA scales with ASEBA scales.
Table 12: Multiple regressions: Predicting HoNOSCA scales with CBCL and YSR
HoNOSCA scales Model with CBCL scales Model with YSR scales
ΔR2 F(9, 528) p ΔR2 F(9, 355) p Disruptive behavior .397 38.62 <.001 .228 11.62 <.001 Overactivity .187 13.45 <.001 .184 8.87 <.001 Self-injury .176 12.51 <.001 .303 17.14 <.001 Substance Abuse .267 21.39 <.001 .271 14.66 <.001 Scholastic .058 3.64 <.001 .048 2.01 .038 Physical Illness .069 4.34 <.001 .077 3.28 .001 Hallucinations Delusions .061 3.78 <.001 .117 5.23 <.001 Somatic .128 8.63 <.001 .137 6.29 <.001 Emotional symptoms .247 19.20 <.001 .382 24.41 <.001 Peers .147 10.10 <.001 .252 13.28 <.001 Self-Care .061 3.84 <.001 .034 1.41 .183 Family Problems .209 15.49 <.001 .174 8.29 <.001 School Attendance .184 13.24 <.001 .139 6.35 <.001
18 Multiple regressions were used to assess the ability of ASEBA scales to predict HoNOSCA items. Every individual HoNOSCA item was separately predicted with all the CBCL scales in
Table 13: Significant (p = < .005) predictors for the HoNOSCA scales
HoNOSCA Scale CBCL scales B YSR Scales B
Disruptive Behavior Somatic Complaints Syndromes
-.173 Somatic Complaints Syndromes -.256
Thought Problems -.144
Rule Breaking Syndromes .331 Rule Breaking Syndromes .314 Aggressive Behavior .489 Aggressive Behavior .315 Overactivity Withdrawn Depressed -.185
Attention Problems .413 Attention Problems .387 Self-injury Anxious Depressed .271 Anxious Depressed .428 Withdrawn Depressed .255 Attention Problems -.165
Rule Breaking Syndromes .233 Rule Breaking Syndromes .190 Substance Abuse Social Problems -.200
Rule Breaking Syndromes .613 Rule Breaking Syndromes .565 Aggressive Behavior -.190
Scholastic No scale No scale
Physical Illness Anxious Depressed -.207 No scale Somatic Complaints
Syndromes
.217
Hallucinations Delusions No Scale No Scale Somatic Somatic Complaints
Syndromes
.296 Somatic Complaints Syndromes .251
Attention Problems -.210 Emotional symptoms Anxious Depressed .348 Anxious Depressed .450 Withdrawn Depressed .201 Withdrawn Depressed .266 Attention Problems -.177 Attention Problems -.188 Peers Withdrawn Depressed .270 Withdrawn Depressed .371
Social Problems .205
Attention problems -.227 Attention problems -.197
Self-Care No Scale No Scale
Family Problems Attention Problems -.240
Rule Breaking Syndromes .354 Rule Breaking Syndromes .263
School Attendance Withdrawn Depressed .246
Rule Breaking Syndromes .374 Rule Breaking Syndromes .233 one model and with all the YSR scales in one model. The significant cut-off point was set at α = .004 after the Bonferroni’s correction. Every HoNOSCA item was significantly explained by the CBCL and by the YSR except for the Self-care item and Scholastic item. The YSR did not
19 significantly predict these items.The variances explained by each model are presented in table 12. In table 13 only the significant predictors of these models are presented with their
corresponding Beta.
Discussion
Because research has shown that the use multiple informants in the assessment of behavioral and emotional problems of children and adolescents is more informative than the use of just one informant (Mash & Hunsley, 2005), the aim of the present study was to explore the agreement between multiple informants (clinician, parent/caregiver and patient) on behavioral and
emotional problems in children and adolescents using the CBCL, YSR and HoNOSCA. Firstly, it was hypothesized that similar items and scales of the CBCL, YSR and
HoNOSCA would show high correlations. We found that the HoNOSCA total score correlated medium with both the CBCL and the YSR total score which was comparable to what Hanssen-Bauer, et al. (2010) found. Also the CBCL total score and the YSR total score correlated medium with each other. Our findings were slightly higher than what Rescorla, et al. (2012) found in a Dutch sample. The medium correlations on the total scores implicate that there is a good
correspondence between the clinician’s, parent’s, and children’s and adolescent’s perspective on the total problems the children and adolescents have, however it is not a complete
correspondence.
Our results also showed that some HoNOSCA items have strong relations with CBCL and YSR domains and scales. As we expected the HoNOSCA item
Disruptive/antisocial/aggressive behavior had a strong relation with the CBCL and a medium relation with the YSR domain Externalizing and the Rule breaking syndromes and Aggressive behavior scales. Also the HoNOSCA item Emotional symptoms had a strong relation with the YSR and a medium relation with the CBCL domain Internalizing and the Anxious-depressed and Withdrawn-depressed scales. Our findings together with the similar findings of Hanssen-Bauer and colleagues seem to support the findings of Youngstrom, Loeber & Stouthamer-Loeber (2000), who found that adolescents underreport externalizing problems, while parents are less aware of the internalizing problems in adolescents. We also expected to find good relations between the HoNOSCA item Hyperactivity/attention/concentration problems with the CBCL and YSR scale Attention problems and the HoNOSCA item Substance Misuse with the CBCL and YSR scale Rule breaking problems, which they were, although they were also of medium
20 strength. Furthermore, we expected the HoNOSCA item Peer relationships to have a good
relation with the CBCL and YSR scale Social problems, but it only had a medium relation with the YSR scale and a small relation with the CBCL scale. This could mean that parents and caregivers may not have good insight in the social relations of children and adolescents. Also all the CBCL and YSR Scales and domains had medium to large relations with their corresponding scales and domains. Our findings were slightly higher than the findings in a Dutch sample in a study across 21 countries (Rescorla, 2012). It can also be seen that the scales that make up the Internalizing domain correlate well with each other and that the scales that make up the Externalizing domain also correlate well with each other.
We did not expect to find relations between the HoNOSCA items Scholastic/language skills, Physical disease/disability, Hallucinations/delusions/abnormal perceptions,
Self-care/autonomy and School attendance with any of the CBCL and YSR scales domains, since they did not seem to have theoretical correspondence with the CBCL and YSR scales and domains. Our findings support that these HoNOSCA items are not comparable with the CBCL and YSR. We also did not expect a relation between the HoNOSCA item Family problems with any of the CBCL and YSR scales and domains. However it did have a medium relation with the CBCL domain Externalizing and a medium relation with the YSR domain Internalizing.
Additional to the findings of Youngstrom, Loeber & Stouthamer-Loeber (2000), Adolescents may feel that family problems are more related to their internalizing problems, while parents may feel that the externalizing problems are due to family problems. Further research into this could be useful.
In contrary to what we expected the HoNOSCA item Non-organic somatic symptoms did not have a relation with the YSR and CBCL scale Somatic complaints syndromes. This could probably be, because the HoNOSCA item Non-organic somatic symptoms seems to measure handicaps due to a lacking physical health (Staring, Hofman, & Mulder, 2003), while the CBCL item Somatic complaints syndromes measures physical problems due to internalizing problems (Drotar, D., Stein, & Perrin, 1995). Our results also showed that the CBCL and YSR scale Other problems did not have a relation, this suggests that this scale may not be a consistent scale.
Secondly, it was hypothesized that a diagnosis of a specific disorder should be reflected in relatively high scores on relevant scales and items of the CBCL, YSR and HoNOSCA. We examined 4 specific disorders: Mood disorder, anxiety disorder, attention disorder and conduct
21 disorder. Our results revealed that a diagnosed mood disorder was reflected in higher scores on the HoNOSCA item Emotional symptoms and the YSR and CBCL scale Mood disorder. Although the actual effect on the HoNOSCA item was small and the CBCL scale Mood
disorders seems to have a major issue, since young children with a mood disorder had relatively lower scores on the Mood disorder scale than children without a mood disorder diagnosis.
As expected we also found that children and adolescents with an anxiety disorder,
attention disorder and/or conduct disorder was reflected in higher scores on the HoNOSCA items Emotional symptoms, Hyperactivity/attention/concentration problems and
Disruptive/antisocial/aggressive behavior respectively and on the CBCL and YSR scales Anxiety disorder, Attention problems and ODD respectively. But the actual effects on the scores were higher for the HoNOSCA items than for the CBCL and YSR scales. This means that the
HoNOSCA items seem to be better at discerning these disorders than the CBCL and YSR scales. Thirdly, we explored the possibility of predicting the HoNOSCA items with a
combination of the scales of the CBCL or the YSR. We assumed that most HoNOSCA items could be predicted with the CBCL or YSR scales and we examined which combination of scales might be best to predict the HoNOSCA items. Our results revealed that all HoNOSCA items had a significant relation with the CBCL scales, in contrary to the YSR, which did not have a
significant relation with the HoNOSCA items Scholastic/language skills and Self-care/autonomy. This means that the YSR can’t be used to predict these two HoNOSCA items.
Our results revealed that the CBCL explained more variance in the HoNOSCA item Disruptive/antisocial/aggressive behavior than the YSR did. Rule breaking syndromes and Aggressive behavior were the best predictors. Interesting here is that Somatic Complaints syndromes had a negative effect on Disruptive Behavior as seen by their parents. Adolescents with Somatic Complaints Syndromes are probably more reliant on other people. Disruptive behavior may make people less likely to help you and children with somatic complaints syndromes may therefore engage in less disruptive behavior.
We found that the CBCL and the YSR seem to be equally good in predicting the HoNOSCA item hyperactivity/attention/concentration problems. Both the CBCL and YSR had the scale Attention problems as best predictor. In contrary to the findings that depression and Attention Deficit Hyperactivity Disorder (ADHD) have a high comorbidity (Biederman,
22 a negative relation with the HoNOSCA item hyperactivity/attention/concentration problems. It may be that children and adolescents who are more withdrawn are less hyperactive than children who are not withdrawn.
In our results it seems to be that the YSR may be better at predicting the HoNOSCA item Self-injury than the CBCL is. Also where the CBCL only had Anxious/Depressed as a
significant predictor, the YSR also had Withdrawn/Depressed as a significant predictor.
Deliberate Self-Harm has been linked to Depression (Andersson, Tannå, & Nordin, 2013). The YSR being better at predicting Self-injury can therefor also be linked with the YSR being better at assessing internalizing problems than the CBCL.
The HoNOSCA item Substance abuse may be predicted equally good with the CBCL as with the YSR. For both the CBCL and YSR the scale Rule breaking syndromes was the best significant predictor. This makes sense, since under aged drinking and smoking drugs, for instance, could be counted as breaking the rules. The CBCL scale Social problems had a small negative relation with the HoNOSCA item Substance abuse. This means that caretakers may think that children and adolescents use substances more often when they don’t have social problems. Or that children and adolescents use less substances when they have social problems. For instance, children and adolescents may not go out as much to drink when they don’t have friends, but peer use of substances is a strong predictor for substance use by children and adolescents (Hawkins, Catalano, & Miller, 1992).
We found that only a low or no significant variance in the HoNOSCA items Scholastic/language skills, Physical disease/disability, Hallucinations/delusions/abnormal perceptions, Non-organic somatic symptoms and, Selfcare/autonomy could be explained by either the CBCL or the YSR scales. Only the HoNOSCA item Non-organic somatic symptoms could be predicted a little bit by the CBCL and YSR scale Somatic complaints syndromes. This means that these HoNOSCA items do not seem to be comparable with the CBCL and YSR scales.
Our results present that the HoNOSCA scale Emotional symptoms may be better predicted with the YSR than with the CBCL. This is in line with the YSR being better at detecting internalizing problems. Both CBCL and YSR had Anxious/depressed and
Withdrawn/depressed as the best predictors for emotional problems. For both the CBCL and YSR it was also seen that the scale Attention problems had a negative relation with the
23 HoNOSCA scale Emotional problems. This is the opposite of what could be expected, since ADHD has been linked with emotional lability (Sobanski, et al., 2010).
We found that the YSR scales may be better at predicting the HoNOSCA item Peer relationships than the CBCL scales. Interestingly, it was not the YSR scale Social problems that predicted the HoNOSCA item Peer relationships, but it was mostly predicted with the scale Withdrawn/depressed. The best explanation we have is that children and adolescents who are more withdrawn will have a harder time making friends or keeping relations. We do recommend more research into why the HoNOSCA item Peer relationships is not predicted with the YSR and CBCL scale Social problems.
Lastly we found that the HoNOSCA item Family relationships and School attendance could be predicted with both the CBCL and the YSR, although they did not predict them very well. The CBCL and YSR scale Rule breaking syndromes was the best predictor for both
HoNOSCA items. For the HoNOSCA item school attendance this probably means that not going to school is also breaking the rules. But for the HoNOSCA item Family relationships we would expect that the CBCL and YSR scale Social problems to be better at predicting relationships. The CBCL and YSR scale Social problems, seems to not be able to predict peer or family relations.
General discussion
In general, The HoNOSCA, CBCL and YSR are corresponding best with each other when scales are theoretically comparable, but there is not a complete agreement between the clinicians, parents/caregivers and children and adolescents. Found discrepancies may be due to the different perspectives on the behavioral and emotional problems children and adolescents have. Our findings seem to support that the clinicians and the children and adolescents are more in agreement about internalizing problems than the clinicians and the parents/caregivers. Our findings also seem to support that the clinicians and the parents/caregivers are more in agreement about the externalizing problems than the clinicians with the children and adolescents. Overall the parents/caregivers and the children and adolescents seem to be more in agreement with the each other than they are with the clinicians, however this may also be because the CBCL and YSR are identical questionnaires, while the HoNOSCA is a different measurement. The difference in the measurements is also an important factor. Multiple HoNOSCA items did not have a relation with the CBCL and YSR. The CBCL and YSR could also not predict these
24 HoNOSCA items well. Besides, a variety of unexpected relations were found that could be useful for further research.
Methodological issues
One advantage of this study was that it could analyze data that was gathered by multiple Dutch clinics throughout the country. This means that the implications have a greater generalization across the Dutch population. Another advantage in our study was the larger sample size, therefor our results are more reliable. However this came with the disadvantage that not all participants had a combination of the three measurements. This means that based on our results alone, we may not make strong statements in the comparison of the HoNSOCA, CBCL and YSR. The used dataset also had a few disadvantages. The large dataset had a lot of missing or erroneous data. It was not possible to retrieve information about missing data or the erroneous data since the dataset was anonymized. Lastly, since it was unknown with what information clinicians filled in the HoNOSCA at the start of a treatment trajectory, correlations between the HoNOSCA and the CBCL and YSR could be inflated. If for instance the clinicians had the results of the CBCL or YSR when they filled in the HoNOSCA correlations would obviously be better than they would have been in reality.
Clinical implications
Just as Mash & Hunsley (2005) we recommend the use of multiple informants in the assessment of behavioral and emotional problems in children and adolescents. Our results support that clinicians, parents/caregivers and children and adolescents are not completely in agreement about the problems. Therefor we also believe that the different informants contribute to the complete assessment of the children and adolescents. Even though the CBCL, YSR and HoNOSCA have scales and items that can reflect a disorder, we advise against the use of these scales and items as an assessment tool for the diagnosis, since they were only able to make small distinguishes and would therefor probably generate a lot of false positives and false negatives. We also underline the importance of treating data more carefully. A lot of information was lost due to the many mistakes we found in the dataset. We recommend further research that aims at comparing the HoNOSCA with the CBCL and YSR in their ability to measure treatment effects. Since we were not able to do that due to missing and erroneous data. Lastly, we would recommend in the future that measures with multiple informants are more alike to better compare the unique contributions
25 of the different informants on the behavioral and emotional problems of children and
26 References:
Achenbach, T. M., McConaughy, S. H., Howell, C. T. (1987) Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational
specificity Psychological Bulletin, 101(2) 213-232
Achenbach, T. M. (1991) Manual for the Child Behavior Checklist/ 4-18 and 1991 profiles. Burlington, United states of America: Department of Psychiatry, University of Vermont. Achenbach, T. M. (1991) Manual for the Youth Self Report form and 1991 profiles. Burlington,
United states of America: Department of Psychiatry, University of Vermont.
Achenbach, T. M., Becker, A., Döpfner, M., Heiervang, E., Roessner, V., Steinhausen, H., & Rothenberger, A. (2008). Multicultural assessment of child and adolescent
psychopathology with ASEBA and SDQ instruments: research findings, applications, and future directions. Journal of Child Psychology and Psychiatry,49(3), 251-275.
Achenbach System of Empirically Based Assessment (2014) Instrumenten voor emotionele en gedragsproblemen. Retrieved on 1 may 2017, from http://www.aseba.nl/home
Andersson, M. J., Tannå, H., & Nordin, S. (2013). Self-image in adolescents with deliberate self-harm behavior. PsyCh Journal,2(3), 209-216.
Biederman, J., Faraone, S., Mick, E., Moore, P., & Lelon, E. (1996). Child Behavior Checklist Findings Further Support Comorbidity between ADHD and Major Depression in a Referred Sample. Journal of the American Academy of Child & Adolescent
Psychiatry, 35(6), 734-742.
Bilenberg, N. (2003). Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA). European Child & Adolescent Psychiatry,12(6), 298-302.
Bonnet, J. (2015, October 21). ROM binnen jeugd-GGZ en gemeenten. Inventarisatie en voorstel aan gemeenten. Retrieved from https://vng.nl/files/vng/151021_rom_binnen_jeugd- ggz_en_gemeenten.pdf
Boon, A., Boer de, S., Haan de, A., Nijssen, Y., & Klasen, H. (n.d.). Normtabellen voor de HoNOSCA naar leeftijd en geslacht. Retrieved from
http://www.kenniscentrum-kjp.nl/app/webroot/files/tmpwebsite/downloadables_publicaties/Normtabellen_HoNOSC A_Lucertis_Jutters.pdf
27 Brann, P., & Coleman, G. (2010). On the Meaning of Change in a Clinician's Routine Measure
of Outcome: HoNOSCA. Australian & New Zealand Journal of Psychiatry,44(12), 1097-1104.
Cohen, J.W. (1988). Statistical power analysis for the behavioral sciences (2nd edn). Hillsdale,
NJ: Lawrence Erlbaum Associates.
Drotar, D., Stein, R. E., & Perrin, E. C. (1995). Methodological issues in using the child behavior checklist and its related instruments in clinical child psychology research. Journal of Clinical Child Psychology, 24(2), 184-192.
Garralda, M. E., Yates, P., & Higginson, I. (2000) Child and adolescent mental health service use – HoNOSCA as an outcome measure. British Journal of Psychiatry, 177, 52-8
Gearing, R. E., Schwalbe, C. S., Mackenzie, M. J., Brewer, K. B., & Ibrahim, R. W. (2014). Assessment of Adolescent Mental Health and Behavioral Problems in Institutional Care: Discrepancies Between Staff-Reported CBCL Scores and Adolescent-Reported YSR Scores. Administration and Policy in Mental Health and Mental Health Services Research,42(3), 279-287.
Gowers, S. G., Harrington, R. C., Whitton, A., Lelliott, P., Beevor, A., Wing, J., & Jezzard, R. (1999). Brief scale for measuring the outcomes of emotional and behavioural disorders in children. Health of the Nation Outcome Scales for children and Adolescents
(HoNOSCA). The British Journal of Psychiatry,174(5), 413-416.
Gowers, S Levine, W., Bailey-Rogers, S., Shore, A., Burhouse, E. (2002). Use of a routine, self-report outcome measure (HoNOSCA--SR) in two adolescent mental health services. The British Journal of Psychiatry,180(3), 266-269.
Harnett, P. H., Loxton, N. J., Sadler, T., Hides, L., & Baldwin, A. (2005). The Health of the Nation Outcome Scales for Children and Adolescents in an adolescent in-patient sample. Australian and New Zealand Journal of Psychiatry,39(3), 129-135.
Hanssen-Bauer, K., Langsrud, Ø, Kvernmo, S., & Heyerdahl, S. (2010). Clinician-rated mental health in outpatient child and adolescent mental health services: associations with parent, teacher and adolescent ratings. Child and Adolescent Psychiatry and Mental Health,4(1), 29.
28 Hawkins, J. D., Catalano, R. F., & Miller, J. Y. (1992). Risk and protective factors for alcohol
and other drug problems in adolescence and early adulthood: Implications for substance abuse prevention. Psychological Bulletin,112(1), 64-105.
Holzer, L., Tchemadjeu, I. K., Plancherel, B., Bolognini, M., Rossier, V., Chinet, L., & Halfon, O. (2006). Adolescent Drug Abuse Diagnosis (ADAD) vs. Health of Nation Outcome Scale for Children and Adolescents (HoNOSCA) in clinical outcome
measurement. Journal of Evaluation in Clinical Practice,12(5), 482-490.
Lesinskiene, S., Senina, J., & Ranceva, N. (2007). Use of the HoNOSCA scale in the teamwork of inpatient child psychiatry unit. Journal of Psychiatric and Mental Health
Nursing,14(8), 727-733.
Mash, E. J., & Hunsley, J. (2005). Evidence-Based Assessment of Child and Adolescent Disorders: Issues and Challenges. Journal of Clinical Child & Adolescent Psychology, 34(3), 362-379
Rescorla, L. A., Ginzburg, S., Achenbach, T. M., Ivanova, M. Y., Almqvist, F., Begacov, I., Bilenberg, N., Bird, H., Chahed, M., Dobrean, A., Dopfner, M., Erol, N., Hannesdottir, H., Kanbayashi, Y., Lambert, M. C., Leung, P. W. L., Minaei, A., Novik, T .S., Oh, K., Petot, D., Petot, J.M., Pomalima, R., Rudan, V., Sawyer, M., Simsek, Z., Steinhausen, H. C., Valverde, J., van der Ende, J., Weintraub, S., Metske, C. W., Wolanczyk, T., Zhang, E. Y., Zukauskiene, R., & Verhulst, F. C. (2012) Cross-informant Agreement Between Parent-Reported and Adolescent Self-Reported Problems in 25 Societies. Journal of Clinical Child & Adolescent Psychology 42(2), 262-273
Salbach-Andrae, H., Lenz, K., & Lehmkuhl, U. (2009). Patterns of agreement among parent, teacher and youth ratings in a referred sample. European Psychiatry, 24(5), 345-351. Sobanski, E., Banaschewski, T., Asherson, P., Buitelaar, J., Chen, W., Franke, B., Holtmann, M.,
Krumm, B., Sergeant, J., Sonuga-Barke, E., Stringaris, A., Taylor, E., Anney, R., Ebstein, R.P., Gill, M., Miranda, A., Mulas, F., Oades, R. D., Roeyers, H., Rothenberger, A., Steinhausen, H., & Faraone, S. V., (2010). S15-01 - Emotional lability in children and adolescents with ADHD. Clinical correlates and familial prevalence. The Journal of Child Psychology and Psychiatry, 51(8), 915-923
Staring, T., Hofman, E., & Mulder, N. (2003). Health of the Nation Outcome Scales Jeugd (HoNOS Jeugd): Instructies bij het invullen van de HoNOS Jeugd. Retrieved from
29
http://www.kenniscentrum- kjp.nl/app/webroot/files/tmpwebsite/Downloadable_PDFs_Instrumenten/HoNOSCA-lijst.pdf
Tiffin, P. A., & Rolling, K. (2012). Structure of the Health of the Nation Outcome Scales for Children and Adolescents: An ordinal factor analysis of clinician ratings of a sample of young people referred to community mental health services. Psychiatry Research,197(1-2), 154-162.
Urben, S., Baier, V., Mantzouranis, G., Schwery, J., Mahi, C., Courosse, S., . . . Holzer, L. (2014). The French adaptation of the Health of the Nation Outcome Scale for Children and Adolescents Self-Rated Form (F-HoNOSCA-SR): Validation and clinical routine use. Psychiatry Research,218(1-2), 229-235.
Urben, S., Pihet, S., Graap, C., Baier, V., Dyson, C., Courosse, S., & Holzer, L. (2015). Clinical Utility of the 2 New Scales of the Health of the Nation Outcome Scales for Children and Adolescents (HoNOSCA): A Naturalistic, Prospective Study in a Psychiatric Unit for Adolescents. Journal of Psychiatric Practice, 21(3), 232-240
Verhulst, F.C., Ende, J. van der & Koot, H. M. (1996). Handleiding voor de CBCBl/4-18. Rotterdam, Netherlands: Sophia Kinderziekenhuis, Erasmus MC.
Verhulst, F.C., Ende, J. van der & Koot, H. M. (1997). Handleiding voor de Youth Self-Report (YSR). Rotterdam, Netherlands: Sophia Kinderziekenhuis, Erasmus MC.
Warnick, E. M., Bracken, M.B., & Kasl, S. (2008). Screening Efficiency of the Child Behavior Checklist and Strenghts and Difficulties Questionnaire: A Systematic Review. Child and Adolescent Mental Health, 13(3), 140-147
Youngstrom, E., Loeber, R., & Stouthamer-Loeber, M. (2000). Patterns and correlates of agreement between parent, teacher, and male adolescent ratings of externalizing and internalizing problems. Journal of Consulting and Clinical Psychology, 68(6), 1038-1050.
