Multi-informant Discrepancies in Reports of Parenting and Child Behavior Martina A. Moens
University of Amsterdam
Faculty of Social and Behavioral Sciences Graduate School of Child Development and Education
Student number: 5934869
Name supervisor: Prof. Dr. G. Overbeek Date: 19 December 2014
Number of words: 5501
Author Note
Martina A. Moens, Department of Child Development and Education, University of Amsterdam.
Correspondence concerning this paper should be addressed to Martina A. Moens, Department of Child Development and Education, University of Amsterdam, Room D7.12, Nieuwe Achtergracht 127, 1018 WS, Amsterdam, The Netherlands. E-mail:
Abstract
Multi-informant discrepancies have mainly been seen as measurement error, because of their failure to provide definitive answers in the measurement of child psychopathology. However, discrepancies can actually reveal meaningful information about child psychopathology. The current study aims to examine multi-informant discrepancies by investigating associations between parent and observer reports of parenting and child behavior. We expected that these associations would be weaker when parents reported higher levels of dysfunctional parenting or child behavior. Measures for child behavior and parenting were the DPICS coding system (observer reports), and ECBI, MESSY and PPI questionnaires (parent reports), respectively. Data were used from 175 families with children aged 4-8 years, who participated in the first cohort of the ORCHIDS study (Chhangur, Weeland, Overbeek, Matthys, & Orobio de Castro, 2012), a RCT on the effectiveness of the Incredible Years family training program. Results from multigroup CFA show small to medium agreement between parent and observer reports of parenting and child behavior. Moreover, several trends suggest that discrepancies may be explained by underlying problems within families, although with the current sample not enough power was present to yield significant findings. Overall, results show the importance of interpreting informant discrepancies based on underlying family problems.
Keywords: parenting, childhood, problem behavior, prosocial behavior,
Multi-informant Discrepancies in Reports of Parenting and Child Behavior The mainstay of evidence-based assessment in clinical research and practice involves the use of reports taken from multiple informants. Multi-informant reports are needed to obtain a comprehensive view on psychopathology in children, because there is no single measure or method that provides a definitive answer or ‘gold standard’ (Hunsley & Mash, 2007). Each informant contributes unique information about a child’s problems. Therefore, the use of ratings from multiple informants has been strongly advocated (Dirks, De Los Reyes, Briggs‐Gowan, Cella, & Wakschlag, 2012; Van der Ende, Verhulst, & Tiemeier, 2012). However, a significant challenge is that multiple informants report differently, even when observing a child’s behavior in similar contexts or situations. That is, discrepancies often are manifest between reports of multiple informants. For example, mother-father, parent-child, teacher-child, or observer-parent discrepancies. Low agreement between these different informants is the rule, rather than the exception (Ferdinand, Van der Ende, & Verhulst, 2004). Achenbach (2006) also mentions that levels of agreement about child psychopathology are mostly small to moderate in strength. Why these informants are only small to moderate in their agreements still remains unclear.
Despite the growing body of research on explaining multi-informant discrepancies, several important gaps remain. Studies of predictors of discrepancies have mostly focused on discrepancies between mothers, fathers, teachers, and children (e.g., Harvey, Fischer,
Weieneth, Hurwitz, & Sayer, 2013). Also, studies have focused on explanatory mechanisms, such as socioeconomic factors (Stone, Speltz, Collett, & Werler, 2013). However, parent-observer discrepancies are currently still underexposed. Therefore, the current study aims to examine such multi-informant discrepancies by investigating the associations between parent and observer reports, and by looking at whether the strength of these associations depends on
the extent to which parents report on their own parenting as more dysfunctional, and report on their child’s behavior as more problematic.
Multi-informants discrepancies are the most consistent and yet most poorly
understood phenomena in mental health research (Achenbach, 2006). They have been found in a wide range of parenting and child behavior ratings, both with respect to ratings of psychopathology and prosocial behavior (Gresham, Elliot, Cook, Vance, & Kettler, 2010; Waaktaar, Borge, Christie, & Torgersen, 2005). Discrepancies have mainly been seen as measurement error or statistical noise. Therefore, random error is a common explanation for discrepancies between informants (De Los Reyes, 2011). However, discrepancies can actually reveal meaningful information about child psychopathology or family functioning and parent-child interactions (De Los Reyes & Kazdin, 2005). Viewing multi-informant discrepancies as meaningful information sheds light on the underlying mechanisms that might result in differences between reports. When comparing reports of parents and
observers, these different reports can be due to parents’ subjective experiences in contrast to observational data (Sessa, Avenevoli, Steinberg, & Morris, 2001). Specifically, self-reports of parents reflect subjective experiences about events, while observational data assess events more objectively.
Importantly, parent report biases may be informative because of the pathological processes that they might reflect. On the one hand, informant discrepancies have been related to negative parenting practices (i.e., harshness), parenting stress, and hostile parent-child interactions (De Los Reyes & Kazdin, 2005; Ferdinand et al., 2004; Grills & Ollendick, 2003; Treutler & Epkins, 2003). Such family risk factors may influence parent’s perceptions of family functioning in a negative way, leading parents to more readily and intensely view specific child behaviors as “problematic” and perceive their own behavior in terms of strict discipline and harsh control. On the other hand, informant discrepancies may be due to
parents’ unawareness, disinterest or inability to recognize their children’s problem behavior. These parental perception biases have been related to adverse development such as
delinquent behavior (De Los Reyes, 2011; Ferdinand et al., 2004), again indicating that informant discrepancies may be informative for identifying underlying pathological
processes. In other words, disinterested or highly stressed parents might under- or overreport their child’s behavior, respectively, resulting in discrepancies with other reporters (i.e., observers).
Overall, multi-informant discrepancies are particularly interesting when they are viewed as a meaningful phenomenon; reflecting underlying problems that are correlated with child psychopathology (De Los Reyes, Thomas, Goodman, & Kundey, 2013). When
discrepant reports are viewed in this way, they can provide useful information and may actually lead to more accurate assessments of family functioning, instead of introducing noise and confusion. For example, if informant discrepancies stem from a high level of family conflict, this is information that clinicians can incorporate into their diagnostic and treatment process (Guion et al., 2009).
The Present Study
In line with this reasoning, the current study aims to examine multi-informant discrepancies by investigating associations between parent and observer reports of child
behavior (see Figures 1a and 1b). Acknowledging that discrepancies between informants
might reflect underlying problems such as parenting stress or negative parenting (Ferdinand et al., 2004, 2006; Grills & Ollendick, 2002; Guion et al., 2009), we expect that the
association between parent and observer reports of child behavior will be weaker when parents report relatively high levels of dysfunctional parenting (i.e., negative parenting), and/ or relatively low levels of positive parenting. In other words, when parents report underlying
problems with respect to their parenting, then the association between parent and observer reports of child problem behavior are expected to be weaker.
In a similar fashion, the current study aims to examine multi-informant discrepancies by investigating associations between parent and observer reports of parenting (see Figures 2a and 2b). Acknowledging that greater discrepancies in informant reports have been associated with worse child behavior outcomes (Gaylord, Kitzmann, & Coleman, 2003; Guion et al., 2009), we expect that the association between parent and observer reports of
parenting is weaker when parents report higher levels of problem behavior of their child, and
relatively low levels of prosocial behavior. That is, the more child externalizing behavior and less prosocial behavior is reported, the weaker the association will be between parent and observer reports of parenting.
Method Participants
A total of 175 parents (Mage = 38.87, SDage = 5.07) and their biological children aged 4–8 (Mage = 6.11, SDage = 1.38), who show mild to (sub)clinical externalizing behavior problems, participated in the present study. See Table 1 for demographic characteristics of the sample. The sample was drawn from the first cohort of the ongoing ORCHIDS study (Chhangur, Weeland, Overbeek, Matthys, & Orobio de Castro, 2012), which is a randomized controlled trial on the effectiveness of the Incredible Years family training program in the Netherlands. In order to acquire an ‘at-risk’ sample of families, without excluding children and parents with subclinical or even normal-range functioning, families were selected on the basis of a child behavior cut-off score at or above the 75th percentile. We used the Eyberg Child Behavior Inventory (ECBI; Eyberg & Pincus, 1999) to obtain the cutoff score for children’s problem behavior.
Families were included when both parent and observer reports of child behavior and parenting were available. Exclusion criteria were mental retardation of the parent and/ or child (IQ≤70) and not mastering the Dutch language. Approval for the data collection
procedures was obtained from the central committee on research involving human subjects in The Netherlands (METC UMC Utrecht).
Procedure
Families were approached for participation via two Dutch regional health care organizations. These families received a personalized information letter, including the ECBI to screen for children’s problem behavior. Parents received a monetary reward of 7,50 euros for their participation. Once included, parents were asked to fill out questionnaires of their children’s behavior and questionnaires of parenting. Parents spent a maximum of one hour filling out these online questionnaires. Children and one of their parents were also video-taped in a structured-play-situation at home. These videos were coded by trained coders with the Dyadic Parent-Child Interaction Coding System (DPICS; Eyberg, Nelson, Duke, & Boggs, 2005).
Structured-play-situations were filmed during house visits. The video-taped situations were divided in four different standard episodes. Each episode had a duration of five minutes. The degree of parental control varied in each situation, for which parents received clear instructions. Toys with which dyads could play were Lego toys, blocks, Dominos, and a pay desk. Within the first structured-play-situation all parent and child dyads played with Lego toys to get used to the situation. The second situation was the child directed interaction (CDI), in which the child was allowed to play his or her game according to his or her rules. During the third situation, the parent-directed interaction (PDI), the parent was instructed to lead the play by the parent’s rules. The fourth situation was the clean-up (CU), where the parent was instructed to direct the child to put all the toys away without assistance.
Four trained observers watched the video-taped structured-play-situations and systematically coded all verbal and non-verbal categories of DPICS for parenting and child behavior. For example, acknowledgements, (critical) statements, parental affect, child smart talk, and child positive affect. The DPICS manual offers a standard coding procedure and coding sheet (see Appendix A).In addition to the DPICS categories, global ratings for parenting and child behavior were coded (see Appendix B). Before coding individually, observers received 45 hours of training with videotapes. The observers were required to meet a minimum of 75 percent inter-observer reliability. On approximately 15 percent of the coded data, inter-observer agreement checks were conducted.
Measures
Parent-reported Child Problem and Prosocial Behavior. For measuring parent-reported Child Problem Behavior, we used a Dutch version of the Eyberg Child Behavior Inventory (ECBI), translated by the University of Utrecht (Raaijmakers, Posthumus, & Matthys). The ECBI is a widely used 36-item parent rating scale to measure disruptive behavior problems in children. This inventory assesses the frequency (i.e., intensity) of the behavior and its identification as a problem. Frequency ratings range from 1 (never) to 7 (always), and are summed to yield an overall problem behavior Intensity score ranging from 36 to 252. In this study only the Intensity scale was considered, following Webster-Stratton et al. (2004). Norms and cutoff scores were based on the original ECBI manual (Eyberg & Pincus, 1999). Cronbach’s α in the present study was .85.
To measure parent reported Child Prosocial Behavior, we used the Dutch version of the Matson Evaluation of Social Skills with Youngsters (MESSY-II; Matson, 2010; Matson, Neal, Worley, Kozlowski, & Fodstad, 2012). The MESSY-II is a measure for social skills for children aged 2-16, and is designed to provide information on a child’s communication and general social skills behavior. MESSY-II questions are about observations of both
appropriate and inappropriate social behaviors. For this study we only used the Appropriate scale, which consists of 20 items. Answers range from 1 (not at all) to 5 (very much). Cronbach’s α was .90.
Observer-reported Child Problem and Prosocial Behavior. For measuring observer reported Child Problem and Prosocial Behavior, we used the Dyadic Parent-Child Interaction Coding System (DPICS; Eyberg et al., 2005). The DPICS was originally
developed by Robinson and Eyberg (1981). It is an extensively researched observational measure for recording behaviors of children and their parents at home. It includes categories to record behaviors of young children and their parents during parent-child interactions. Examples of items indicative of Child Problem Behavior are non-compliance, deviance (e.g. cry, whine, yell, smart talk), physical negative, and destructive (Webster-Stratton, Reid, & Hammond, 2004). Inter-observer reliability coefficients (ICC) for DPICS items of Child
Problem Behavior was .92. Examples of items indicative of Child Prosocial Behavior are
compliance, (non)verbal positive affect, and physical warmth (Webster-Stratton et al., 2004). ICC for DPICS items of Child Prosocial Behavior was .79.
Observer-reported global rating for Child Problem and Prosocial Behavior. Extra global ratings for Child Problem and Prosocial Behavior were measured in addition to DPICS measures. Two Likert scales (range 1-5) were created, one for child negative affect and one for child positive affect. At the end of every five minute block (child directed play, parent directed play, and clean-up) a score was given for negative/ positive affect. The
System for Coding Interactions and Family Functioning (SCIFF; Lindahl & Malik, 2001) and the System for Coding Interactions in Parent-Child Dyads (SCIPD; Lindahl & Malik, 1996) were obtained for these extra variables. These coding systems have been used successfully to code parent-child play interactions in earlier studies. ICC for the global rating of Child
Parent-reported Negative/ Positive Parenting. These reports were measured by the Parenting Practices Interview (PPI; Webster-Stratton, 1985, Webster-Stratton, Reid, & Hammond, 2001), which is an 72-item questionnaire adapted from the Oregon Social
Learning Center’s Discipline Questionnaire and revised for young children. We used a Dutch translation of this questionnaire. The two summary scores are Negative Parenting (i.e., harsh–inappropriate discipline), consisting of seven items including spank–swat–whip, slap– hit, yell, raise voice, and Positive Parenting (i.e., supportive parenting), consisting of five items including discussing problems, teaching another behavior, praising or rewarding good behavior, verbal encouragement, and physical affection for good behavior. PPI scores are rated on a 7-point scale ranging from 1 (never) to 7 (always). Cronbach’s α for Negative
Parenting was .79. Cronbach’s α for Positive Parenting was .72.
Observer-reported Negative/ Positive Parenting. For measuring observer reported
Negative/ Positive Parenting, we used the DPICS. Following Webster-Stratton et al. (2004),
critical statements, negative commands, physical intrusion, and physical negative were seen as indicators for measuring Negative Parenting, and acknowledgments, unlabeled and labeled praise, positive affect, and encouragement were taken in order to measure Positive Parenting. ICC for DPICS items of Positive Parenting was .86. ICC for DPICS items of Negative
Parenting was .80.
Observer-reported global rating for Negative/ Positive Parenting. Similar to measures of Child Behavior, extra global ratings for Negative/ Positive Parenting were measured in addition to DPICS measures. Two Likert scales (range 1-5) were created, one for parental negative affect and one for parental positive affect. As with the child affect variables, a score was given for the amount of negative/positive affect at the end of each five minute block. Again, these created affect scales were based on the SCIFF (Lindahl & Malik, 2001)
and the SCIPD (Lindahl & Malik, 1996).ICC for the global rating of Negative Parenting was .79. ICC for the global rating of Positive Parenting was .46.
Data Analyses
To test the associations between parent and observer reports of parenting and child
behavior, MG CFA were conducted using version 20 of the AMOS program (Byrne, 2013).
Specifically, we investigated whether these associations were different in strength across a ‘less problems’ and ‘more problems’ groups, in four models: (1) Positive Parenting, (2) Negative Parenting, (3) Child Prosocial Behavior, and (4) Child Problem Behavior. Group cutoff scores for the Parenting models were based on the ECBI, where scores above 131 on the Intensity scale were considered problematic (Eyberg & Pincus, 1999), resulting in a ‘more child behavior problems’ and ‘less child behavior problems’ group. Cutoff scores for the Child Prosocial Behavior and Child Problem Behavior models were based on the median split values of PPI positivity and harshness, respectively. Resulting in less and more
positivity and harshness groups.
The models were tested by using the estimation method maximum likelihood (ML). Model fit was evaluated by means of the chi-square (χ2) statistic, Root Mean Square Error of Approximation (RMSEA), and the Comparative Fit Index (CFI). The chi-square test is a measure of exact fit. A significant chi-square value (p<.05) indicates that the model does not fit the data (Kline, 2011). The RMSEA is a measure of approximate fit. RMSEA values should be ≤.08 (Hu & Bentler, 1999; Kline, 2011, p. 208), and CFI > .97 is indicative for good fit (Kline, 2011, p.208). Further, we used chi-square difference tests (∆χ2) in order to test for measurement invariance. A significant value (p<.05) means that a more restricted model fits worse than a less restricted model (Kline, 2011).
In order to fit the models correctly, parcels were created for both parent and observer reports (see Figures 3 to 6). In addition, if needed, negative error variances (Heywood case;
Kline, 2011, p. 158) were constrained to be equal to the error variance of the group were there was no negative error variance (McDonald, 1985). For the observer constructs in each model, a global rating was included besides the DPCIS parcels.
Within each model, factor covariances were compared to indicate the associations between parent and observer reports. Because measurements should be invariant across groups –Measurement invariance concerns whether scores from the operationalization of a construct have the same meaning under different conditions (Meade & Lautenschlager, 2004)– in order to be able to compare the factor covariances, in each of the four models we followed the same steps. First, we performed MG CFA to test for configural invariance or equal form invariance (Hform). In such a model, the same measurement model is specified across the groups (Kline, 2011, p. 252). Next, we tested for construct-level metric invariance or equal factor loadings (HΛ; Kline, 2011, p. 253). If the equal factor loadings model (HΛ) showed equally good fit as the configural invariance model (Hform), then the assumption of construct-level metric invariance was accepted. The latter indicates that the items measure similar constructs in different groups –in our case, whether parent and observer reports of parenting or child behavior were measured similarly across groups.
If construct-level metric invariance (HΛ) did hold indeed, then factor covariances between the groups could be compared adequately. In order to test this, we performed a test for equality of factor covariances across groups (HΦ). With the latter analysis, our hypotheses were tested. As hypothesized, the covariances were expected to be unequal.
Results Positive Parenting
In order to test the associations between parent and observer reports of positive parenting, we performed a positive parenting MG CFA. Two groups were obtained on the basis of a child problem behavior cutoff score of 131 (ECBI; Eyberg & Pincus, 1999),
resulting in a ‘less problems’ (n = 83; M = 118.11, SD = 11.89) and ‘more problems’ (n = 82;
M = 150.07, SD = 13.17) group. More problems indicate clinical scores of child problem
behavior.
Figure 3 shows the Positive Parenting model with its standardized estimates in both groups. Table 2 shows the measurement invariance results of the Positive Parenting model. The chi-square from a Positive Parenting model with all parameters allowed to be unequal across groups was compared to the chi-square from a Positive Parenting model with the factor loadings constrained to be equal across groups. The model with all parameters allowed to be unequal across groups (Model 1a) showed good fit, according to the chi-square test of exact fit, the CFI and the RMSEA. The model with factor loadings constrained to be equal across groups (Model 1b) performed equally well as the previous model, indicating that measurement invariance of factor loadings holds. Therefore, we continued with the more stringent equal factor loadings model, where measurement invariance of factor loadings holds.
Having established construct-level metric invariance across groups, covariances were then compared between the ‘less problem’ and the ‘more problem’ groups. The correlations (i.e., standardized covariances) provide insight in the agreement between parent and observer. Although the correlation between parent and observer reported negative parenting is smaller in the ‘more problems’ group (r = .49; p < .001) than in the ‘less problems’ group (r = .58; p < .001), results of chi-square difference tests (∆χ2) for Model 1c show that the difference in correlation between the groups is not significant (see Table 2).
Negative Parenting
In the negative parenting model, we tested the associations between parent and observer reports of negative parenting with MG CFA (see Figure 4). Again, ‘less problems’ (n = 83; M = 118.11, SD = 11.89) and ‘more problems’ (n = 81; M = 150.15, SD = 13.24)
groups were obtained on the basis of a child problem behavior cutoff score of 131 (ECBI; Eyberg & Pincus, 1999). More problems indicate clinical scores of child problem behavior.
Table 2 shows the measurement invariance results of the Negative Parenting model. The configural invariance model with all parameters allowed to be unequal across groups (Model 2a) showed good fit, according to all fit measures. The model with factor loadings constrained to be equal across groups (Model 2b) performed equally well as the previous model, indicating that measurement invariance of factor loadings holds. Therefore, we continued with the more stringent equal factor loadings model, where measurement invariance of factor loadings holds.
Since construct-level metric invariance across groups was established, correlations were compared between the ‘less problem’ and the ‘more problem’ groups. In case of the Negative Parenting model, the correlation between parent and observer in the ‘more problems’ group (r = -.15; p = .278) appeared to differ from the correlation in the ‘less problems’ (r = .04; p = .748). However, as the results of a chi-square difference test (∆χ2
) for Model 2c show, the difference in correlation between the groups is not significant (see Table 2). Thus, there is not enough evidence to support our hypothesis, although the results show that agreement seems different.
Child Prosocial Behavior
In order to test the associations between reports of child prosocial behavior, we performed a child prosocial behavior MG CFA. In this model, two near equal groups were obtained on the basis of a PPI positivity scale using a median split cutoff. This split yielded a ‘more positivity’ (n = 76; M = 76.34; SD = 3.76) and ‘less positivity’ (n = 99; M = 62.72; SD = 6.03) group. More positivity indicates higher scores on positive parenting.
Figure 5 shows the Child Prosocial Behavior model with its standardized estimates in both groups. Table 2 shows the measurement invariance results of this model. The model
with all parameters allowed to be unequal across groups (Model 3a) showed good fit, according to the chi-square test of exact fit, the CFI and the RMSEA. The model with factor loadings constrained to be equal across groups (Model 3b) performed equally well as the previous model, indicating that measurement invariance of factor loadings holds. Therefore, we continued with the more stringent equal factor loadings model, where measurement invariance of factor loadings holds.
Having established construct-level metric invariance across groups, covariances were then compared between the ‘less positivity’ and the ‘more positivity’ groups. As expected, the correlation between parent and observer reports is significantly smaller in the ‘less
positivity’ group (r = .00; p = .983) than in the ‘more positivity’ group (r = .19; p = .245), as results of chi-square difference tests (∆χ2) for Model 3c show (see Table 2). In other words, results show that the agreement between parent and observer reports on prosocial behavior is significantly weaker when parents report less positivity in parenting.
Child Problem Behavior
In the child problem behavior model, we tested the associations between reports of child problem behavior with MG CFA (see Figure 6). In order to obtain two near equal groups, we performed a median split cutoff on the basis of a PPI harshness scale. This split yielded a ‘less harshness’ (n = 94; M = 2.78; SD = 0.47) and ‘more harshness’ (n = 81; M = 4.01; SD = 0.43) group. More harshness indicates higher scores on harsh parenting.
Table 2 shows the measurement invariance results of the Child Problem Behavior model. The model with all parameters allowed to be unequal across groups (Model 4a) showed good fit, according to all fit measures. The model with factor loadings constrained to be equal across groups (Model 4b) performed equally well as the previous model, indicating that measurement invariance of factor loadings holds. Therefore, we continued with the more stringent equal factor loadings model, where measurement invariance of factor loadings
holds. Because construct-level metric invariance across groups did hold, correlations were compared between the ‘less harshness’ and the ‘more harshness’ groups. Although the
correlation between parent and observer was smaller in the ‘more harshness’ group (r = .03; p = .810) than in the ‘less harshness’ group (r = .20; p = .104), results of chi-square difference tests (∆χ2) for Model 4c show, that the difference in correlation between the groups is not significant (see Table 2).
Discussion
This study examined multi-informant discrepancies between parent and observer reports of child behavior and parenting. Results demonstrated that, in general, associations between multi-informant reports about child behavior and parenting are low to moderate. Moreover, in the current sample we did not find enough evidence to explain parent-observer discrepancies by underlying problems within families (i.e., dysfunctional child behavior and parenting). However, we did find trends that may indicate that parent-observer discrepancies may be explained by patterns of dysfunctional parenting or child behavior. Specifically, we found a significant weaker association between parent and observer reports of child prosocial behavior if parents reported less positivity in parenting, in comparison to parents who
reported more positivity. In addition, we found two trends that pointed in the same direction. There were weaker associations between parent and observer reports of child problem behavior if parents reported more harshness in parenting, compared to parents who reported less harshness. Also, there were weaker associations between reports of positive parenting if parents reported more child behavior problems, in comparison to parents who reported less child behavior problems.
Findings of this study are in line with earlier research that demonstrated low to moderate agreement between different informants (Achenbach, 2006; Ferdinand et al., 2004; Guion et al., 2009; Sessa et al., 2001). More specifically, the current study demonstrated low
agreements in reports of child problem and prosocial behavior, and moderate agreements in positive parenting reports. This can perhaps be explained by the fact that informants reporting in different contexts may only provide information about the context in which they are
involved themselves, or are biased (Kraemer et al., 2003). Additionally, our study suggests that agreement between informants are more consistent if reports are about positive parenting than if reports are about child behavior or negative parenting. Thus, only if parents report on their own positive parenting practices, they might be less biased. This can be explained by the willingness to report, which has been related to discrepancies (Karver, 2006; Stokes, Pogge, Wecksell, & Zaccario, 2011). Probably parents are more willing to report on their positive parenting practices, rather than on their negative parenting practices.
Our present findings do not show enough proof for underlying family problems as an explanation for multi-informant discrepancies. Of the four multigroup analyses performed, only one finding (on discrepancies in reports of child prosocial behavior) came out
significant. Although previous research defined underlying factors otherwise, our findings are not in line with literature that actually did successfully explain discrepancies by underlying family problems (De Los Reyes, 2011; De Los Reyes & Kazdin, 2005; Guion et al., 2009; Harvey et al., 2013). These contrasting findings could be due to sample size, which probably has been rather small for finding significant discrepancies between the parent and observer reports. It is also possible that the group dichotomy obtained with the used cut-offs did not differ enough in child problem levels or dysfunctional parenting to be able to find different discrepancies per group. Another possibility is the issue of self-reports (i.e., subjective experiences) versus objective observational data (Sessa et al., 2001). It could be that discrepancies strictly stem from the difference in subjectivity versus objectivity, barely reflecting any underlying mechanisms.
Other contrasts to our hypotheses are findings concerning agreements between reporters on negative parenting. Negative associations between parent and observer reports were found in families that deal with more child problem behavior. Possible explanations for this unexpected finding, are lack of insight of parents (Ferdinand et al., 2004; Grills &
Ollendick, 2002). Other explanations might be that negative parenting is more susceptible for social desirable answers (Bornstein et al., 2014), or that harsher parents view their child’s behavior as problematic, rather than their own parenting behavior. In this way, in contrast to the observer, parents could choose not to report their negative parenting while there are in fact negative parenting practices.
Nevertheless, although with the current sample not enough evidence was found to explain discrepancies by underlying problems within families, we did find several trends and one significant finding in line with our hypotheses. Specifically, for reports on child prosocial behavior we did find that the agreement between reporters is weaker when parents report less positivity in parenting. Here, less positive parenting might reflect underlying mechanisms indicative of family problems, which in turn determine more discrepancy between
informants. This suggests that discrepancies might be larger when families deal with more dysfunctional parenting or child behavior issues. But, these findings were not consistent across all models that we tested.
Future research could focus on longitudinal discrepancy patterns in order to shed light on explanatory mechanisms for child developmental processes in the long term. For instance, it could be that larger discrepancies due to parents’ under- or overreporting of child problem behavior are predictors of child problem behavior in the long term (Ferdinand et al., 2004). Furthermore, subsamples of different ethnicity groups would be interesting to consider in further research. Specifically, cultural differences might affect family dynamics or parenting, possibly resulting in different patterns of discrepancies (Vendlinkski, Silk, Shaw, & Lane,
2006; Guion et al., 2009; Waters et al., 2006). Therefore, we would gain even more insight in parent-observer discrepancies by taking into account several cultural backgrounds. For example, in African American families the use of strict discipline is more common and acceptable compared to other cultural backgrounds (Deater-Deckard, Dodge, Bates, & Pettit, 1996; De Los Reyes & Kazdin, 2005). Furthermore, research in this matter could take into account measures of parental psychopathology, yet another interesting explanatory
mechanism for discrepancies (De Los Reyes, 2011).
Although this study sheds light on multi-informant discrepancies, several limitations warrant mentioning. In the first place, the sample of this study might have been too small to be able to find systematic patterns of explanatory factors for parent-observer discrepancies. When replicating this study, therefore, a larger sample should be considered in order to claim stronger statements in this matter. Secondly, some measures we used are not optimal in terms of their psychometric quality. Inter-observer reliability of global rating measures for
observer-reported positive parenting and child prosocial behavior were rather low. In addition, parent-reported parenting was measured by the PPI (Webster-Stratton, 1985; Webster-Stratton et al., 2004), from which the psychometric qualities are not thoroughly examined yet. These drawbacks could have altered the outcomes.
Despite these limitations, the study strengths should also be acknowledged. First, the current study focused not only on multiple informants, but investigated also multiple
methods, by looking at the agreement on child behavior and parenting between the parent (who filled out questionnaires) and the observer (who systematically coded by using a coding system). Secondly, the sample of this study is carefully selected on the basis of a 75th
percentile, resulting in a relevant ‘at-risk families’ sample, without excluding children and parents with subclinical or normal-range functioning. Another strength of this study is the important knowledge gap that it attempted to explain. Up till now, the growing body of
research on explaining multi-informant discrepancies mostly focused on discrepancies between mothers, fathers, teachers, and children (e.g., Harvey et al., 2013), or focused on predictors such as socioeconomic factors (Duhig, Renk, Epstein, & Phares, 2000; Stone et al., 2013). Before our study, parenting and child behavior predictors of parent-observer
discrepancies were underexposed.
In conclusion, the most important message of this study is the existence of low to moderate agreements between parents and observers in their reports about parenting and child behavior. Also, several trends point in the direction that discrepancies may be explained by underlying problems within families, although with the current sample not enough evidence was found to conclude this. Therefore, we should first shed more light on the interpretation of discrepancies by means of underlying family problems, before we are able to acknowledge the contribution of such patterns as meaningful information.
References
Achenbach, T. M. (2006). As others see us: Clinical and research implications of cross-informant correlations for psychopathology. Current Directions in Psychological
Science, 15, 94–98. doi:10.1111/j.0963-7214.2006.00414.x
Byrne, B. M. (2013). Structural equation modeling with AMOS: Basic concepts, applications,
and programming. Routledge.
Chhangur, R. R., Weeland, J., Overbeek, G.J., Matthys, W. C. H. J., & Orobio de Castro, B. (2012). ORCHIDS: An observational Randomized Controlled Trial on childhood differential susceptibility. BMC Public Health, 12, 917–924. doi:10.1186/1471-2458-12-917
Deater-Deckard, K., Dodge, K. A., Bates, J. E., & Pettit, G. S. (1996). Physical discipline among African American and European American mothers: Links to children’s
externalizing behaviors. Developmental Psychology, 32, 1065–1072. doi:10.1037/0012-1649.32.6.1065.
De Los Reyes, A. (2011). Introduction to the special section: More than measurement error: Discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. Journal of Clinical Child & Adolescent Psychology, 40(1), 1–9. doi:10.1080/15374416.2011.533405
De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin,131(4), 483–509. doi:10.1037/0033-2909.131.4.483
De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. (2013). Principles underlying the use of multiple informants' reports. Annual Review of Clinical
Psychology, 9, 123–149. doi:10.1146/annurev-clinpsy-050212-185617
Dirks, M. A., De Los Reyes, A., Briggs‐Gowan, M., Cella, D., & Wakschlag, L. S. (2012). Annual research review: Embracing not erasing contextual variability in children’s behavior–theory and utility in the selection and use of methods and informants in developmental psychopathology. Journal of Child Psychology and Psychiatry, 53(5), 558–574. doi:10.1111/j.1469-7610.2012.02537.x
Duhig, A. M., Renk, K., Epstein, M. K., & Phares, V. (2000). Interparental agreement on internalizing, externalizing, and total behavior problems: A meta‐analysis. Clinical
Psychology: Science and Practice, 7(4), 435–453. doi:10.1093/clipsy.7.4.435
Eyberg, S. M., Nelson, M. M., Duke, M., & Boggs, S. R. (2005). Manual for the dyadic parent-child interaction coding system. 3rd edition. Unpublished manuscript. http://www.PCIT.org
Eyberg, S. M., & Pincus, D. B. (1999). ECBI: Eyberg Child Behavior Inventory and
SESBI-R: Sutter-Eyberg Student Behavior Inventory-Revised. Professional manual. Odessa,
FL: Psychological Assessment Resources.
Ferdinand, R. F., Van der Ende, J., & Verhulst, F. C. (2004). Parent-adolescent disagreement regarding psychopathology in adolescents from the general population as a risk factor for adverse outcome. Journal of Abnormal Psychology, 113(2), 198–206.
doi:10.1037/0021-843X.113.2.198
Ferdinand, R. F., Van der Ende, J., & Verhulst, F. C. (2006). Prognostic value of parent– adolescent disagreement in a referred sample. European Child & Adolescent
Psychiatry, 15(3), 156-162. doi:10.1007/s00787-005-0518-z
Gaylord, N. K., Kitzmann, K. M., & Coleman, J. K. (2003). Parents' and children's perceptions of parental behavior: Associations with children's psychosocial adjustment in the classroom. Parenting: Science and Practice, 3(1), 23–47. doi: 10.1207/S15327922PAR0301_02
Gresham, F. M., Elliot, S. N., Cook, C. R., Vance, M. J., & Kettler, R. (2010). Cross-informant agreement for ratings for social skills and problem behavior ratings: An investigation of the social skills improvement system rating scales. Psychological
Assessment, 22, 157–166. doi:10.1037/a0018124
Grills, A. E., & Ollendick, T. H. (2002). Issues in parent–child agreement: The case of structured diagnostic interviews. Clinical Child and Family Psychology Review, 5, 57–83. doi:10.1023/ A:1014573708569.
Grills, A. E., & Ollendick, T. H. (2003). Multiple informant agreement and the anxiety disorders interview schedule for parents and children. Journal of the American
Academy of Child & Adolescent Psychiatry, 42(1), 30–40.
Guion, K., Mrug, S., & Windle, M. (2009). Predictive value of informant discrepancies in reports of parenting: Relations to early adolescents’ adjustment. Journal of Abnormal
Child Psychology, 37(1), 17–30. doi:10.1007/s10802-008-9253-5
Harvey, E. A., Fischer, C., Weieneth, J. L., Hurwitz, S. D., & Sayer, A. G. (2013). Predictors of discrepancies between informants’ ratings of preschool-aged children's behavior: An examination of ethnicity, child characteristics, and family functioning. Early
Childhood Research Quarterly, 28(4), 668–682. doi:10.1016/j.ecresq.2013.05.002
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure
analysis: Conventional criteria versus new alternatives. Structural Equation Modeling,
6, 1–55. doi:10.1080/10705519909540118
Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical
Psychology, 3, 29–51.
Karver, M. (2006). Determinants of multiple informant agreement on child and adolescent behavior. Journal of Abnormal Child Psychology, 34, 251–262. doi:10.1007/s10802-005-9015-6
Kline, R. B. (2011). Principles and practice of structural equation modeling: Third edition. New York: The Guilford Press.
Kraemer, H. C., Measelle, J. R., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessment and research: Mixing and matching contexts and perspectives. American
Journal of Psychiatry, 160(9), 1566–1577. doi:10.1176/appi.ajp.160.9.1566
Lindahl, K. M., & Malik, N. M. (1996). System for coding interactions in parent-child dyads (SCIPD): A coding system for structured and unstructured parent-child task.
Lindahl, K. M., & Malik, N. M. (2001). System for coding interactions and family functioning (SCIFF). Unpublished manuscript, University of Miami.
Matson, J. L. (2010). The Matson evaluation of social skills with youngsters-II (MESSY-II). Baton Rouge, LA: Disability Consultants, LLC.
Matson, J. L., Neal, D., Worley, J. A., Kozlowski, A. M., & Fodstad, J. C. (2012). Factor structure of the Matson Evaluation of Social Skills with Youngsters-II (MESSY-II).
Research in Developmental Disabilities, 33(6), 2067–2071.
doi:10.1016/j.ridd.2010.09.026
McDonald, R. P. (1985). Factor analysis and related methods. Hillsdale NJ: Erlbaum. Meade, A. W., & Lautenschlager, G. J. (2004). A comparison of item response theory and
confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7(4), 361–388. doi:10.1177/1094428104268027
Robinson, E. A., & Eyberg, S. M. (1981). The dyadic parent-child interaction coding system: Standardization and validation. Journal of Consulting and Clinical Psychology, 49, 245–250. doi:10.1037/0022-006X.49.2.245
Sessa, F. M., Avenevoli, S., Steinberg, L., & Morris, A. S. (2001). Correspondence among informants on parenting: Preschool children, mothers, and observers. Journal of
Family Psychology, 15(1), 53–68. doi:10.1037/0893-3200.15.1.53
Stokes, J., Pogge, D., Wecksell, B., & Zaccario, M. (2011). Parent–child discrepancies in report of psychopathology: The contributions of response bias and parenting stress.
Journal of Personality Assessment, 93(5), 527-536.
doi:10.1080/00223891.2011.594131
Stone, S. L., Speltz, M. L., Collett, B., & Werler, M. M. (2013). Socioeconomic factors in relation to discrepancy in parent versus teacher ratings of child behavior. Journal of
Psychopathology and Behavioral Assessment, 35(3), 314–320.
doi:10.1007/s10862-013-9348-3
Tabachnick, B. G., & Fidell, L. S. (2012). Using Multivariate Statistics. Upper Saddle River: Pearson Education.
Treutler, C. M., & Epkins, C. C. (2003). Are discrepancies among child, mother, and father reports on children's behavior related to parents' psychological symptoms and aspects of parent–child relationships? Journal of Abnormal Child Psychology, 31(1), 13–27. doi:10.1023/A:1021765114434
Van der Ende, J., Verhulst, F. C., & Tiemeier, H. (2012). Agreement of informants on emotional and behavioral problems from childhood to adulthood. Psychological
Assessment, 24(2), 293–300. doi:10.1037/a0025500
Vendlinkski, M., Silk, J. S., Shaw, D. S., & Lane, T. J. (2006). Ethnic differences in relations between family process and child internalizing problems. Journal of Child
Psychology and Psychiatry, and Allied Disciplines, 47, 960–969.
doi:10.1111/j.1469-7610.2006.01649.x.
Waaktaar, T., Borge, A. I. H., Christie, H. J., & Torgersen, S. (2005). Youth–parent consistencies on ratings of difficulties and prosocial behavior: Exploration of an at-risk sample. Scandinavian Journal of Psychology, 46, 179–188. doi:10.1111/j.1467-9450.2005.00447.x
Waters, E., Doyle, J., Wolfe, R., Wright, M., Wake, M., & Salmon, L. (2000). Influence of parental gender and self-reported health and illness on parent-reported child health.
Pediatrics, 106(6), 1422–1428. doi:10.1542/peds.106.6.1422
Webster-Stratton, C. (1985). Parenting Practices Interview. Unpublished assessment instrument.
Webster-Stratton, C., Reid, M. J., & Hammond, M. (2001). Preventing conduct problems, promoting social competence: A parent and teacher training partnership in Head Start.
Journal of Clinical Child Psychology, 30, 283–302.
Appendix A DPICS Coding Sheet
Appendix B
Table 1
Demographic Characteristics of Sample (N=175)
Demographic measures n %
Child's gender (% female) 86 49 Parent's gender (% female) 161 92 No. of children in family (% 2 children) 101 58 No. of adults in family (% 2 adults) 143 82 Marital status parents (% married) 117 67 Job occupation parents (% employed) 127 73 Father's education (% IVE) 44 26 Mother's education (% IVE) 59 34 Religion father (% Christian) 54 31 Religion mother (% Christian) 63 36
Note. No. = number.
Table 2
Values of selected fit statistics for hypotheses about measurement invariance for two-factor CFA models of Positive Parenting, Negative Parenting, Child Prosocial Behavior, and Child Problem Behavior.
Hypothesis χ2(df) p ∆χ2(df) p RMSEA [90% CI] CFI
Pos. Parenting Model 1a Hforma 27.42 (24) .285 0.030 [0;0.073] 0.980 Model 1b HΛb 39.05 (31) .152 11.63 (7)d .113 0.040 [0;0.075] 0.952 Model 1c HΦc 39.65 (32) .166 0.60 (1)d .439 0.038 [0;0.073] 0.955 Neg. Parenting Model 2a Hforma 79.15 (63) .082 0.040 [0;0.065] 0.940 Model 2b HΛb 95.35 (72) .034 16.21 (9)d .063 0.045 [0.013;0.067] 0.914 Model 2c HΦc 95.97 (73) .037 0.62 (1)d .43 0.044 [0.012;0.067] 0.915
Child Pr.soc. Beh.
Model 3a Hform a
49.51 (40) .144 0.037 [0;0.068] 0.966
Model 3b HΛb 54.63 (46) .179 5.12 (6)d .529 0.033 [0;0.063] 0.969
Model 3c HΦc 64.76 (49) .065 10.13 (3)d .017* 0.043 [0;0.069] 0.944
Child Prob. Beh.
Model 4a Hforma 22.02 (17) .184 0.041 [0;0.085] 0.982
Model 4b HΛb 23.87 (22) .353 1.87 (5)d .867 0.022 [0;0.069] 0.993
Model 4c HΦc 24.70 (23) .366 0.82 (1)d .367 0.021 [0;0.067] 0.994
Note. Model 1c, 2c, 3c, and 4c are the final models, used for interpretation of the covariances between the factors (i.e., parent and observer reports). CI =
Confidence Interval; Pos. Parenting = Positive Parenting model; Neg. Parenting = Negative Parenting model; Child Pr.soc. Beh. = Child Prosocial Behavior model; Child Prob. Beh. = Child Problem Behavior model.
Figure 1a. Association between parent and observer reports of Child Problem Behavior
explained by parent reports of Negative Parenting.
Figure 1b. Association between parent and observer reports of Child Prosocial Behavior
explained by parent reports of Positive Parenting.
Parent reported Negative Parenting Association between reports Observer reported Child Problem Behavior Parent reported Child Problem Behavior Parent reported Positive Parenting Observer reported Child Prosocial Behavior Parent reported Child Prosocial Behavior Association between reports
Figure 2a. Association between parent and observer reports of Negative Parenting explained
by Parent reports of Child Problem Behavior.
Figure 2b. Association between parent and observer reports of Positive Parenting explained
by parent reports of Child Problem Behavior.
Parent reported Child Problem Behavior Observer reported Negative Parenting Parent reported Negative Parenting Association between reports Parent reported Child Problem Behavior Observer reported Positive Parenting Parent reported Positive Parenting Association between reports
Figure 3. Positive Parenting MG CFA model, with standardized estimates.
Note. PP valence = global rating positive parenting; pos = positivity. Parcel 1 en 2 are parcels for positivity, parcel 3 and 4 are parcels for praise.
Figure 4. Negative Parenting MG CFA model, with standardized estimates.
Note. PI = parental intrusion; PN = physical negative; CS = critical statements; NP valence = global rating negative parenting.
Figure 5. Child Prosocial Behavior MG CFA model, with standardized estimates.
Note. CPAN/V = child positive affect non-verbal/ verbal; CP valence = child positive affect; pos = positivity.
All parcels are parcels for prosocial behavior.
Figure 6. Child Problem Behavior MG CFA model, with standardized estimates.
Note. noncom = noncompliance; oppo = oppositional; CN valence = global rating child negative affect;
inattdes = inattention/destructive, a combined parcel.
