Faculty of Social and Behavioural Sciences
Graduate School of Child Development and Education
How Effective are Online Parent Support Programs
for Children’s Behavioral Problems?
A Multilevel Meta-analysis and Qualitative
Comparative Analysis
Research Master Child Development and Education Research Master Thesis
Therdpong Thongseiratch Supervisor: Patty Leijten, Ph.D. Date: 2 May 2018
How Effective are Online Parent Support Programs for Children’s Behavioral Problems? A Multilevel Meta-analysis and Qualitative Comparative Analysis
Therdpong Thongseiratch University of Amsterdam
IMPORTANCE There is a rapid increase in the development and use of online parent
support programs. This calls for the need to understand how effective these strategies are for improving children’s mental health.
OBJECTIVES First, to meta-analyze the effects of online parenting support programs on
children’s behavioral problems, and on children’s emotional problems and parental mental health. Second, to explore the combinations of program components to yield higher program effectiveness.
DATA SOURCES We searched for randomized trials evaluating online parent support
programs for children’s mental health that were published until June 30, 2017 in Medline, PsycINFO, Web of Science, and the Cochrane Library. Search terms included parent support program, parent training, online, internet, computer, website, and application.
STUDY SELECTION We included randomized controlled studies and quasi-experimental
studies evaluating the effect of a parent support program on children’s behavioral problems, and/or on children’s emotional problems and parental mental health. We included studies where children’s mean age was up to 12 years. We excluded studies if the article had not undergone peer review, or if it did not include sufficient statistics to compute a standardized effect size.
DATA EXTRACTION AND SYNTHESIS Of 2,941 articles, 12 articles with a total of
2,025 participants met the inclusion criteria. All studies were randomized controlled studies. Data were independently extracted by 2 reviewers. The quality of the included studies was examined using the Cochrane risk of bias tool. Effect sizes (Hedges’ g) were calculated from post intervention means and standard deviations. We used multilevel meta-analysis to
summarize the program effects, and Qualitative Comparative Analysis (QCA) to identify pathways to effectiveness, and individual content and delivery components that seem sufficient or necessary for yielding high effectiveness.
MAIN OUTCOMES AND MEASURES The primary outcome was children’s behavioral
problems after the program. Secondary outcomes were children’s emotional problems and parental mental health after the program.
RESULTS Parent support programs had small but significant effects on children’s
behavioral problems (Hedges’ g = −0.32; 95% CI, −0.47 to −0.17), emotional problems (Hedges’ g = −0.22; 95% CI, −0.31 to −0.13), and parental mental health (Hedges’ g = −0.30; 95% CI, −0.42 to −0.17). In the QCA, we identified 4 combinations of components as
pathways to high effectiveness on children’s behavioral problems. Sending parents reminders to work on the program was the only 1 sufficient component; all programs that sent parents reminders yielded high effectiveness.
CONCLUSIONS AND RELEVANCE There is evidence that online parent support
programs reduce children’s behavioral and emotional problems, and improve parental mental health. Effect sizes are similar to effects of preventive face-to-face parent support programs. Sending parents reminders to work on the program seems sufficient to yield high
effectiveness, none of the other program characteristics were associated with high
effectiveness. Evaluating the effects of online parenting support programs is an emerging field, with a small but rapidly increasing number of randomized studies. The field currently relies on parent-reported effects only, and should invest in including less subjective outcome measures.
Introduction
Current prevalence estimates for children’s behavioral and emotional problems typically range from 10% to 20%.1,2 These problems are one of the most common reasons for
referrals to child mental health clinics.3 Not only are these problems common, they are
typically stable and predict negative outcomes in later life, such as academic
underachievement and various mental health disorders.3,4 Moreover, children’s behavioral
and emotional problems place immense burdens on parents.5 It is therefore important to
develop and disseminate effective programs to reduce these problems.
Parent support programs can successfully and sustainably reduce behavioral problems in children.6,7 These programs typically aim to improve the parent-child relationship and to
break coercive interaction cycles.8 The strong evidence-base for their effectiveness led them
to be widely recommended as the primary intervention strategy for reducing children behavioral problems (e.g., by the United Kingdom's National Institute for Health and Care Excellence).9
A serious problem with most programs, however, is that they are not easily accessible. Fewer than 25% of the families of preschool children with behavioral problems receive an evidence-based parent support program.10,11 Access may depend on referral to child’s mental
health care services, long waiting lists11, the often high costs, and sometimes inconvenient
locations of the services.12 Moreover, even if families are offered a parent support program,
parental adherence to these programs is often problematic. The average attendance rates do not exceed 60%.13
To increase their use, and thus the public health impact, programs need to become more accessible. Online programs may serve this goal. More than 60% of all people worldwide, including socioeconomically disadvantaged populations, have access to the internet, and this number is rising quickly.14 For example, 78% of disadvantaged low income
parents in Los Angeles use the internet.15 This makes an online parenting support program a
promising approach to reach a large number of families,16 including high-risk parents. The
question remains, however, how effective these programs actually are at reducing children’s behavioral problems.
While it is well-known that face-to-face programs can reduce disruptive child behavior, it is unknown whether online parent support programs can also do this. First, an online parent support program does not offer therapist contact, while families with severe problems may need this contact in order to improve children’s behavior problems. When there is no direct, face-to face contact, parents cannot get feedback on their practices. This might make it harder for them to develop an effective skill to deal with children’s behavioral problems. Indeed, evidence suggests that providing effective feedback is critical for skill development and utilization.17,18,19 Second, since most online programs are self-administered,
it might be harder for parents to complete the program, without the aid of a dedicated therapist to encourage them to regularly work on the program.20-22 Understanding the
aggregated effects of online parent support programs in a well-designed meta-analysis can improve our understanding of how effective these programs, and can guide decision making process about implementing online parent support program in daily clinical practice.
This understanding is not only important to inform clinical practice, but also to increase our insights into the critical ingredients of psychosocial programs. If online parenting programs can effectively reduce children’s behavior problems, this questions the assumption that face-to-face therapist contact is essential for effective intervention. If, instead, online parenting programs are relatively ineffective, this questions whether online teaching methods can replace face to face teaching methods. These insights are important not only for understanding how parenting support programs work, but for understanding how psychosocial intervention works more generally.
Related to this, it is important to not only estimate the overall effectiveness of online parenting programs, but to also examine what makes these programs less or more effective. The internet allows for diverse teaching methods, including interactive instructions, online exercises and sending reminders to parents. Knowledge of the content and delivery components of online programs that drive effectiveness can help optimize online parent support programs by making them briefer, more effective and cost-effective, and improving implementation and dissemination. For example, time and costs can be saved on phone calls or therapist involvement if we find that these components are not necessary for high
effectiveness of online parenting support programs on child behavior.
In addition to the effects of online parent support programs on child’s behavioral problems, we studied the effects of online parenting programs on child’s emotional problems and parental mental health. The effects of parent support programs for children with
emotional problems have received far less attention. Although almost all parent support programs have not actively developed to improve child’s emotional problems, these programs could improve this problem by changing the same parenting practices that improve child’s behavioral problems.23 Many face-to-face parent support programs have shown that
improving the parent-child relationships is an important strategy to reduce children’s
emotional problems.24 Programs that focus on improving the parent-child relationship might
therefore be effective for reducing both behavioral and emotional problems. However, the extent to which emotional problems are affected by parenting programs for behavioral problems is unclear. Some studies suggest that these parent support programs reduce
children’s emotional problems24, but others did not report on emotional problems25 or failed
to replicate these findings.26 Systematic reviews of the effects of parent support programs for
reducing children’s emotional problems suggest that the evidence-base is relatively weak.27
programs on children’s emotional problems. We expected that online parent support
programs could potentially improve children’s emotional problems, because these programs focus on multiple risk factors for child’s emotional problems (e.g., unpredictable environment and disrupted parenting)24 rather than on risk factors for child’s behavioral problems only.
Similarly, many parent support programs also aim to improve the mental health of families more generally.28 Previous meta-analyses showed that traditional group parent
support programs also reduced parental depressive symptoms, stress, and improved self-efficacy.28,29 It is , however, not clear whether these improvements reflect the impact of
parent support programs directly targeting parental mental health or whether they occur as an indirect consequence of the improvement of their abilities to with children’s behavior. A growing number of meta-analyses have supported the application of online psychosocial support program for adult psychological disorder treatment and prevention across different population groups, but relatively few meta-analyses have concentrated on improving parental mental health.30 In this review, we therefore also investigated the impact of online parent
support programs on parental mental health such as stress, anxiety, and depression. We expected that these online programs may also benefit parental mental health even they were primarily aimed to improve children’s behavioral or emotional problems.31 These
improvements may be a result of the parents’ application of online program strategies (e.g., self-problem management, self-emotional regulation) to themselves in addition to the use of strategies focused on improving child’s behavior problems.
To the best of our knowledge, this study is the first to meta-analyze the effects of online parent support programs on children’s behavioral problems. Secondary outcomes are children’s emotional problems and parental mental health. Moreover, to explore pathways to high effectiveness (or less effectiveness) of online parent support programs, we used
components and combinations of components that are associated with stronger program effects.
Methods
We independently reviewed the evidence from randomized controlled and quasi-experimental design studies on the effects of online parenting support programs on children’s behavioral and emotional problems. This meta-analysis was developed in line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (www.prisma-statement.org). This study was registered with PROSPERO, number
CRD42017080051.
Search Strategy
We searched for studies evaluating online parent support program studies that were published until June, 30 2017 in the databases of Medline, PsycINFO, Web of Science, and Cochrane Library. To identify as many studies as possible, we used lists of synonyms separated by the Boolean ‘‘or’’ operator (with word stems to identify all word variants), including: (a) parent, mother, father, family, and caregiver for the concept of parenting; (b) support, coach*, advice, and train* for the concept of parent support; and (c) internet, computer, online, mail, chat*, and website for the concept of online context. Moreover, we also examined bibliographical references of existing systematic reviews and of the primary studies identified by the systematic search.
Study Selection Criteria
Studies were selected for inclusion if they (a) evaluated the effect of a parenting support program on children’s behavioral or emotional problems (child mean age up to 12 years maximum), (b) were written in English, (c) compared families receiving the program with families in an untrained control, wait-list, or care as the usual condition, (d) were randomized, including parallel group or cluster randomization, or quasi-randomization (e.g.,
a matched-controlled group) (e) delivered more than 50% of the parenting support program online, (f) reported post-test scores of children’s behavioral and emotional problems with sufficient detail to allow the calculation of an effect size, or the data could be requested from the authors, and (g) were published in a peer-reviewed journal.
We excluded studies if they (a) were directed at parents of special child populations that were not defined by their behavioral and emotional problems, including children in foster care, children with autism spectrum disorders, intellectual disabilities, physical disabilities, or mental illness, (b) did not isolate the effects of parenting program but involved other aspects of services to the children and families, and (c) had sample sizes smaller than N = 10.
One author assessed the abstracts and full texts of studies that were likely to meet inclusion criteria. Discrepancies and the final list of studies included in the review were assessed by both authors.
Data Extraction
We extracted the following data: (1) means and standard deviations of post-test scores on measures of children’s behavioral problems (e.g., conduct problems, disruptive behavior), emotional problems (e.g., anxiety, depression), and parent’s mental health (e.g., stress, depression, anxiety); (2) methodological characteristics (e.g., the sample size, study design); (3) program characteristics (e.g., online approach, therapist involvement, duration) and (4) sample characteristics (e.g., participant’s age and sex).
In addition, to identify pathways to effectiveness, we coded the presence versus absence of each of the following components for each study: (1) psychoeducation; (2) social learning theory; (3) proactive parenting; (4) relationship building; (5) engage in child-led play; (6) client-centered therapy; (7) parent problem management; (8) parent self-emotional regulation; (9) peaching parents how to teach children problem solving, emotion regulation, or social skills for peer relations; (10) written text ; (11) video; (12) online
exercise or homework; (13) therapist analysis with feedback; (14) phone call; (15)
supplemental information; (16) parent reminder. The first 9 components reflect the content of the program; the latter 7 reflect the delivery process. eTable 1 explains the components in more detail.
Effect Size Calculation
We calculated Hedges’ g to decrease small sample bias for some of the included studies.32 First, the difference between the mean post test scores of families in the
intervention and control condition was divided by the pooled standard deviation. Further, the estimate was corrected for small sample bias using Hedges correction.32
Assessment of Risk of Bias
We used the Cochrane Collaboration risk of bias tool to categorize risk of bias in (1) random sequence generation, (2) blinding of participants and personnel, (3) incomplete outcome data, and (4) selective reporting. Ratings of high, unclear or low were assigned for each domain within the 12 individual studies. A high risk of bias was assigned if the report made it clear that the method potentially introduced the outcomes that could be biased. An unclear bias was assigned if the authors’ report was unclear as to whether the study findings were likely to be biased. A low risk of bias was assigned if it was clear from the report that the assessed issues could not have biased the study./33
Data Analyses Meta-analysis
Three-level multilevel meta-analysis was used to account for the clustering of effects (e.g., from multiple measures and/or on multiple follow-up occasions) within the studies. Level 1 represents sampling variance for each effect size. Level 2 is implied for variance between effect sizes within a study. Level 3 represents variance between effect sizes across studies. We estimated the size of the intervention effect by fitting meta-analysis models
without an intercept. The statistical analysis was designed to evaluate both within-study and between study variables.
Heterogeneity was evaluated using Cochran's Q test of heterogeneity and the I2
statistic, which measured the proportion of inconsistency among studies that could not be explained by chance. To assess the possibility of publication bias, we inspected asymmetry of funnel plots and the Egger regression test when the number of studies in an analysis exceeded 20.33
Qualitative Comparative Analysis (QCA)
We used QCA to identify the single individual component and the combination of components that were associated with either high or less effectiveness. We also determined sufficient pathway (a set of components representing one of possibly several pathways to the outcome) and necessary pathway (a set of components within which every instance of the outcome occurs) to high and less effectiveness. We chose to use QCA for this because QCA was specifically designed for small samples of studies.34 We used the Fussy software
application in STATA version 15 to build the truth table.35 The truth table gives an overview
of all possible combinations of components (i.e., configurations) that have a similar outcome (i.e., highly effective versus less effective).
We classified studies as “highly effective” if the intervention group demonstrated moderate to high effect on children’s behavioral problems (effect size ≥ 0.3). We chose to use the threshold of 0.3 because effect sizes around 0.3 are of policy interest.36 We classified the
other studies as “less effective”.34 Although included studies used a variety of behavioral
outcomes measured, almost all studies (9 studies) used the Eyberg Child Behavior Inventory (ECBI) as the primary outcome. We therefore used the effect size based on ECBI for
outcome set calibration. We calibrated the 2 studies that did not use the ECBI based on the effect sizes of the Strengths and Difficulties Questionnaire and Child Behavior Checklist.
eTable 1 maps the empirical components of the parent support programs covered in the QCA on 2 types of components. We sent emails to all authors to ask them to code which components were included in each program. In the eTable 2, a score of 1 indicates presence of the component and 0 indicates absence.
Results
Included Studies
Online database searches yielded 2941 results. An additional 15 hits resulted from the searching of study registries and clearinghouse websites, and hand searching of prior reviews. In total, after de-duplication, 1942 records were captured (Figure 1).
Figure 1. PRISMA Flow Diagram
The first author and a research assistant independently reviewed records for eligibility. Twelve studies remained eligible and were included in this review. All 12 included studies were randomized controlled trials (RCTs) with 2-arm trial (comparing an intervention to a control condition). The sample sizes varied from 37 to 464.37-48 Study
Table 1. Characteristics of Included Studies Source Sa mpl e Size (N) Child Age Range (years) Mean Child Age (years) %
Boys Country Study Design Type of Control Type of Progra m
Name of Program
Morgan et al, 37
2017 433 3-6 4.8 47.34 Australia RCT Waitlist Website Cool Little Kids
Sourander et al,
38 2016 464 4 4 61.9 Finland RCT Education Website Strongest Families Smart Website
Sanders et al, 39
2014 193 3-8 5.63 67 New Zealand RCT Workbook Website Triple-P
Baker et al, 40
2017 200 2-9 4.4 55 Australia RCT Waitlist Website Triple-P
Porzig-Drummond et al,
41 2015
84 2-10 5.27 50 Australia RCT Waitlist Video 1-2-3 Magic
Parenting Program Sanders et al, 42
2012 116 2-9 4.7 67 Australia RCT Internet-use-as usual Website Triple-P Morawska et al,
43 2014 165 2-10 6.06 61.9 Australia RCT Waitlist Podcast Triple-P
Enebrink et al, 44
2012 104 3-12 6.83 57.7 Sweden RCT Waitlist Website Parent Management Training Breitenstein et al,
45 2016 79 2-5 NR 43 US RCT Attention App Chicago Parent Program
Jones et al, 46
2017 97 3-10 NR NR UK RCT Waitlist Website Integrated Bipolar Parenting
Intervention Antonini et al, 47
2014 37 3-9 5.6 62.5 US RCT IRC Website I-Interact
Franke et al, 48
2016 53 3-4 4 71.7 New Zealand RCT Waitlist Website Triple-P
The 12 studies evaluated 8 different online parent support programs. The contents of all programs but 1 were reasonably similar, with a shared focus on teaching and practicing parenting skills and child management strategies to break cycles of coerciveness in parent– child interaction. One program included modules on child emotional health37 and 1 study
included modules on parental mental health.46 While most programs were delivered on a
website,37-41,45-48, 1 program was delivered through downloadable Video,42 1 through
podcast,43 and 1 through an app.44
Seven studies used wait-list control groups.37,40,41,43,44,46,47 The wait period varied,
from 6 weeks to 4 months, across studies. All studies offered usual services during the waiting period. Five studies used alternative treatments to the control condition, including education,38 workbook,39 and Internet-use-as usual. 41, 47
Risk of Bias
All studies had low risk of bias for selective reporting. Most studies were rated as having a high risk of bias regarding the blinding of participants and personnel (91.7%), because blinding is virtually impossible to achieve in a psychological program. Study attrition was between 3% and 15% and only 5 studies reported intention-to-treat analysis. Risk of selection bias was low for all studies (Table 2).
Table 2. Risk of Bias at the Study Level
Source Reporting Bias (Selective Reporting) Attrition Bias (Incomplete Outcome Data) Performance Bias (Blinding of Participants and Personnel) Selection Bias (Random Sequence Generation) Morgan et
al,372017 low low high low
Sourander et
al,38 2016 low low high low
Sanders et
al,39 2014 low high high low
Baker et al,40
2017 low low high low
Drummond et al,41 2015 Sanders et
al,42 2012 low high high low
Morawska et
al,43 2014 low low high low
Enebrink et
al,44 2012 low high high low
Breitenstein
et al,45 2016 low high high low
Jones et al,46
2017 low high high low
Antonini et
al,47 2014 low high high low
Franke et
al,48 2016 low low unclear low
A high risk of bias was assigned if the report made it clear that the method potentially introduced findings that could be biased. An unclear rating was assigned if the report made it unclear whether the study findings were likely to be biased. A low risk of bias was assigned if it was clear from the method and reporting that the issues assessed could not have biased the study.
Parent Support Program Effects
Parent support programs (k = 11; n = 28) significantly reduced children’s behavioral problems (Hedges’ g = −0.32; 95% CI, −0.47 − −0.17, heterogeneity, Q = 72.66, P = <.001;
I2= 62.84%; eFigure 1). Programs (k = 6; n = 8) had smaller, but significant effects on
children’s emotional problems (Hedges’ g = −0.22; 95% CI, −0.30 − −0.13, heterogeneity, Q = 6.77, P = .45; I2= 0%; eFigure 2). Parent support programs (k = 9; n = 39) also significantly
improved parental mental health (Hedges’ g = −0.30 (95% CI, −0.42 − −0.17, heterogeneity,
Q = 78.69, P = <.001; I2= 53.65%; eFigure 3). We were unable to evaluate publication bias because of the small numbers of the included studies (n < 20).33
QCA Results
Six studies were classified as highly effective (Hedges’ g > −.30). Because studies lacked variability on 9 components (i.e., components were either present or absent in almost all studies), these components were excluded from the QCA model. The final QCA therefore included 6 components: 3 reflecting content (engage in child-led play, parental self-emotional regulation, and teaching parents how to teach children problem solving, emotional regulation,
or social skills for peer relations) and 3 reflecting delivery methods (additional phone calls, therapist feedback, and sending parents reminders).
Single Components as Paths to Effectiveness
No single component was both necessary (i.e., all highly effective programs would have this component) and sufficient (i.e., all programs with this component would be highly effective) for high effectiveness. Sending parents reminders to work on the program was the only sufficient individual component for more improvement in children’s behavioral
problems: all programs that included this component were highly effective (consistency 100 %). No other individual component was near the threshold for sufficiency. Similarly, no individual component was identified as individually necessary.
No single component was both necessary and sufficient for less effectiveness. Adding phone calls to the online program was the only sufficient individual component for less improvement in children’s behavioral problems: the outcome was present in 2 of the 2 studies that have this component (consistency 100 %; Table 3).
Combinations of Components as Paths to Effectiveness
With 6 components, there was a total of 36 (62) possible different combinations (i.e.,
“configurations”) of components. Our included programs included 8 of these possible 36 configurations, with good spread across the included programs (Table 3). None of these 8 configurations were contradictory. This means that none of the configurations were present in both highly effective and in less effective programs. Instead, there were 4 configurations for highly effective programs and 4 configurations for less effective programs (solution
consistency and coverage 100%; Table 3).
Figure 2 illustrates these 4 pathways for high effectiveness and 4 pathways for less effectiveness. A pathway to high effectiveness shared by 3 programs was teaching parents how to teach children problem solving, emotional regulation, or social skills for peer relations
and sending parents reminders to work on the programs. A pathway to less effectiveness shared by 2 programs was included only components on teaching parents how to teach children problem solving, emotional regulation, or social skills for peer relations, and not providing other components. The other 6 pathways were unique for individual programs (Figure 2).
Table 3. Coding Framework as Applied to Included Programs
Type of
components Intervention features
Highly effective
programs Less effective programs
To tal hi ghl y ef fec tiv e (n= 6) To tal les s ef fec tiv e ( n= 5) P or zi g-D ru mmo nd et al , 20 15 S an der s et al , 20 12 M or aw sk a et al , 2 014 E ne bl ink et al , 2 012 Jon es et al , 20 17 A nt oni ni et al , 201 4 S our and er et al , 201 6 S an der s, e t al 20 14 B ak er , et al 20 17 B rei te ns tei n, e t al 201 6 Fr ank e, e t al 20 16 1. Content components Engage in
child-led play (A) 0 0 0 1 1 0 1 0 0 1 0 2 2 Parental self-emotional regulation (B) 1 0 0 0 1 1 1 0 0 1 0 3 2 Teaching parents how to teach children (C) 1 1 1 0 1 0 1 1 1 0 1 4 4 2. Delivery components Additional phone call (D) 0 0 0 0 0 0 1 0 0 0 1 0 2 Therapist analysis with feedback (E) 0 0 0 1 0 1 1 0 0 0 1 2 2 Parent reminder (F) 1 1 1 0 0 1 0 0 0 0 0 4 0
Figure 2. Configurations of Parent Support Program Components
A, Engage in child-led play; B, Parental self-emotion regulation; C, Teaching parents how to teach the child; D, Additional phone call; E, Therapist analysis with feedback; F, Parent reminder
Discussion
There has been a rapid increase in the development of online parent support programs. In this meta-analysis, we tested the effects of online parent support on children’s behavioral problems, and on children’s emotional problems and parental mental health. We found that online parent support programs have a small but significant impact on child’s behavioral and emotional problems, and on parental mental health.
The effect size of 0.32 of online parent support programs for child’s behavioral problems compares very favorably with those of 0.30 from meta-analyses of the conventional face-to-face parent programs, such as Incredible Years.49 Although most online programs
were self-directed and lacked therapist involvement, their effects are of similar magnitude as those of conventional face-to-face programs.We did, however, not directly test the relative effects of online and conventional face-to-face programs. To our knowledge, such studies are rarely conducted. Thus, future studies that compare outcome measures for online parent support programs vs. conventional face-to-face programs are needed to confirm the comparative effect of this innovative approach.
The effects of online parent support programs on children’s emotional problems and parental mental health were also similar to the effects of face-to-face parent support programs on these outcomes.50,51 Program effects on emotional problems were, however, somewhat
smaller than program effect on behavioral problems. Most online programs did not explicitly include strategies to prevent or manage child’s emotional problems. In this review, there was only 1 online parent support program that primarily aimed to prevent emotional problems.37
However, although almost all online parent support programs were focused on reducing child’s behavioral problems, these online programs also targeted many of the proposed mechanisms and risk factors for child’s emotional problems such as non-nurturing,
unpredictable, and unstructured, parenting behaviors.24 Moreover, these online programs also
included strategies that have been proved to prevent child’s emotional problems such as relationship building, and positive parenting based on social leaning theory.37-40 This finding
is in line with previous studies on face-to-face parent support programs suggesting that programs originally targeting child’s behavior problems also reduce child’s emotional problems.24,53 This evidence suggests that online parent support program may offer a viable
strategy for reducing child’s emotional problems.
Parents who have a child with behavioral or emotional problems are more likely to have mental health problems.53 Our meta-analysis revealed small but significant positive
effects of online parent support programs on parental mental health. The effect on parental mental health found in this meta-analysis is in line with the effect of group-based and others self-directed parent support programs.24, 54-57. However, the positive effects on parental
mental health should be interpreted with caution. The extent to which parent support programs affect parental mental health might depend on program or sample characteristics, such as baseline stress, anxiety or depression severity. These severities varied substantially between studies in our sample ( I2= 53.65%), including 1 study targeting on parents with
bipolar disorder.46 Moderator analyses and future research targeting different parental
characteristics would be needed to identify whether online programs effects vary between subgroups of parents. Although there was only 1 program that primarily aimed to improve parental mental health,37 the effects of online parent support programs on parental mental
health was significant and close in magnitude to the effect on child’s behavioral problems. The overall effect of online parent support programs on parental mental health confirmed that online programs also benefit parental mental health, even if they are primarily aimed to improve children’s behavioral or emotional problems.This meta-analysis however could not show whether the improvement was a result of the parents’ application of online program strategies to themselves, of the use of strategies focused on improving child’s behavior problems, or both.
In the QCA, we showed that there was no single component that was necessary to ensure the highly effective outcome. Instead, different combinations of components (presence or absence of them) led to high effectiveness. This supported evidences from previous meta-analyses that individual components rarely contributed to less or more effective
psychotherapy.54,55 We identified 4 combinations of components as pathways to high
effectiveness for improving children’s behavioral problems. In other words, when one of these combinations was present within a program, the intervention demonstrates highly improved children’s behavior. The results revealed that the combinations of both components (content and delivery) were important to ensure the highly effective outcome. In fact, almost all programs classifying as highly effectiveness included both types of the components, while almost all programs classifying as less effectiveness provided only 1 type of the components. The combination of teaching parents how to teach children problem solving, emotional regulation, or social skills for peer relations (content component) and sending parents
highly effective studies. However there were also 3 other pathways to a highly successful program. These results generated some hypotheses about what works to improve online program outcome on child’s behavioral problems. This statement however should be confirmed by a future research.
Sending parents reminders to work on the program was the only single component that leaded to high effectiveness. Parents’ retention is a major challenge in any parent support program, but particularly in online programs. This finding pointed to the importance of adjunctive contact by program generated-parent reminder or online messaging. These delivery methods can be used as supportive strategies for maximizing engagement over time in online programs.
Surprisingly, none of the highly effective combinations of components included the added phone calls. In other words, therapist contact was not associated with program
effectiveness. The previous meta-analysis on self-directed parent support program found that the effect sizes reduced when interventions that involved regular program contact via phone or internet were removed.56 However the moderator analysis in this previous meta-analysis
combined all types of program contact (e.g. added phone calls and online messaging) as 1 moderator. Our meta-analysis showed that what was associated with program effectiveness was sending parents reminders via email, or program-generated messages. This suggests that messages, even when program-generated, might be sufficient to encourage parents to stay engaged in the program. We suggest that that intensive therapist involvement might not be necessary for higher effectiveness.
Limitations
Some limitations of this meta-analysis should be taken into consideration when interpreting our results. The literature on online parenting programs is only just emerging, and the quality of these studies is not yet as high as that of studies on traditional parenting
programs. First, although we included any type of outcome measure, all available outcome measures were parent-reported. Thus, outcomes were not blinded to the families’ condition. A previous meta-analysis showed that the effects of parenting programs to reduce attention deficit hyperactivity symptoms disappeared when analyzing data from informants who were blind to treatment allocation.58 Other meta-analysis, however, show similar effects of
parenting programs for parent-reported and observed measures of child behavior.29 Second, 8
out of 12 included studies did not report intention to threat analysis. These studies might overestimate program outcomes, if parents who have negative program experiences are more likely to drop out and are not included in the analyses. Third, most studies only tested the immediate effects of parent support programs. We therefore cannot be sure to what extent treatment effects of online parent support programs are maintained. Lastly, in the QCA, we had to exclude many components because the included studies did not vary in whether they included or excluded these components. Most online parent support programs include the same program components, including psychoeducation on positive parenting, proactive parenting, and relationship building.37-48 We thus could not explore which of these
components lead to high effectiveness, because they were presented in almost all programs.24 Implication for Future Research and Clinical Practice
To extend the promise of online parent support programs, future researches are needed to determine the optimal program components of online parent support programs. This is because most online parent support programs include the similar set of components. Knowing which of these components drive effectiveness can help optimize programs by making them cost-effective. In addition, because online programs may not be appropriate for every family (e.g., families without access to the internet, parents with literacy problems), future research should test how individual family characteristics influence program outcomes. Finally, to draw stronger conclusions about whether online parenting programs can actually
replace traditional parent support programs in some cases, future studies should directly test the effects of online and traditional parent support programs against each other.
For clinical practice, given the ubiquity of online programs, they appear poised to be a viable parent’s behavior change modality in clinical practice. Our findings are the supportive evidence that automatic program-generated parent reminders, without actual therapist
contact, can be used as a delivery method to encourage parents to regularly work on the parent support programs. We recommend to include sending parents reminders to work on a program in a future online parent support program to enhance program effectiveness.
Conclusions
This meta-analysis and QCA suggest that online parent support programs reduce parent-reported children’s behavioral and emotional problems, and improve parental mental health. The effect sizes are similar to the effects of face-to-face programs. While 4 pathways for high effectiveness were identified in the QCA, sending parents reminders to work on the program seemed to be the most important way to yield high effectiveness. Our findings support the use of online parent support programs, especially if they include reminders for parents to work on the program. However, research on online parent support programs currently relies on parent-reported effects only, and should invest in including less subjective outcome measures.
References
1. Bronsard G, Alessandrini M, Fond G, et al. The prevalence of mental disorders among children and adolescents in the child welfare system: a systematic review and meta-analysis. Medicine (Baltimore). 2016;95(7): e2622.
doi:10.1097/MD.0000000000002622.
2. Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry. 2015;56(3):345–65.
3. Loeber R, Capaldi DM, Costello E. Gender and the development of aggression, disruptive behavior, and early delinquency from childhood to early adulthood. In: Tolan PH, Leventhal BL, ed. Advances in development and psychopathology: Brain
Research Foundation symposium series. Disruptive behavior disorders. New York,
NY, US: Springer Science + Business Media; 2013:137–160.
4. Lavigne JV, Arend R, Rosenbaum D, Binns HJ, Christoffel KK, Gibbons RD.
Psychiatric disorders with onset in the preschool years, I: stability of diagnoses. J Am
Acad Child Adolesc Psychiatry. 1998;37(12):1246–1254.
5. Vaughan EL, Feinn R, Bernard S, Brereton M, Kaufman JS. Relationships between child emotional and behavioral symptoms and caregiver strain and parenting stress. J
Fam Issues. 2013;34(4):534–556.
6. Kaminski JW, Valle LA, Filene JH, Boyle CL. A meta-analytic review of components associated with parent training program effectiveness. J Abnorm Child Psychol. 2008;36(4):567–589.
7. Van Aar J, Leijten P, Orobio de Castro B, Overbeek G. Sustained, fade-out or sleeper effects? a systematic review and meta-analysis of parenting interventions for
disruptive child behavior. Clin Psychol Rev. 2017;51:153–163.
8. Patterson GR. A social learning approach: coercive family process. Eugene, OR: Castalia; 1982.
9. Pilling S, Gould N, Whittington C, Taylor C, Scott S. Guideline Development Group. Recognition, intervention, and management of antisocial behaviour and conduct disorders in children and young people: summary of NICE-SCIE guidance. BMJ. 2013;346:f1298.doi: 10.1136/bmj.f1298.
10. Kataoka SH, Zhang L,Wells KB. Unmet need for mental health care among US children: variation by ethnicity and insurance status. Am J Psychiatry.
2002;159(9):1548–1555.
11. Prinz RJ, Sanders MR. Adopting a population-level approach to parenting and family support interventions. Clin Psychol Rev. 2007;27:739–749.
12. Owens PL, Hoagwood K, Horwitz SM, et al. Barriers to children’s mental health services. J Am Acad Child Adolesc Psychiatry. 2002;41(6):731–738.
13. Baker CN, Arnold DH, Meagher S. Enrollment and attendance in a parent training prevention program for conduct problems. Prev Sci. 2011;12(2): 126–138.
14. Lebo H, Dunahee M. The world internet project international report 7th edition. Los
Angeles, CA: Center for the Digital Future; 2017.
15. Love SM, Sanders MR, Metzler CW, Prinz RJ, Kast EZ. Enhancing accessibility and engagement in evidence-based parenting programs to reduce maltreatment:
conversations with vulnerable parents. J Public Child Welf. 2013;7(1):20–38. 16. MacDonell KW, Prinz RJ. A review of technology-based youth and family–focused
17. Becci AA, Yueqi Y, Thomas M, Jungrim M. Changes in parenting practices during Parent Management Training Oregon model with parents of children in foster care.
Child Youth Serv Rev. 2017;76:181–191.
18. Billings D. Efficacy of adaptive feedback strategies in simulation-based training. Mil
Psychol. 2012;24(2):114–133.
19. Davis WD. The interactive effects of goal orientation and feedback specificity on task performance. Hum Perform. 2005;18(4):409–426.
20. MacDonell KW, Prinz RJ. A review of technology-based youth and family–focused interventions. Clin Child Fam Psychol Rev. 2017;20:185–200.
21. Taylor TK, Webster-Stratton C, Feil EG, Broadbent B, Widdop CS, Severson HH. Computer-based intervention with coaching: an example using the incredible years program. Cogn Behav Ther. 2008;37:233–246.
22. McGoron L, Ondersma SJ. Reviewing the need for technological and other
expansions of evidence-based parent training for young children. Child Youth Serv
Rev. 2015;59:71–83.
23. Murray L, Pella JE, De Pascalis L, et al. Socially anxious mothers’ narratives to their children and their relation to child representations and adjustment. Dev Psychopathol. 2014;26(4):1531–1546.
24. Webster-Stratton C, Herman KC. The impact of parent behavior-management training on child depressive symptoms. J Couns Psychol. 2008;55:473–484.
25. Gross D, Garvey C, Julion W, Fogg L, Tucker S, Mokros H. Efficacy of the Chicago parent program with low-income African American and Latino parents of young children. Prev Sci. 2009;10:54–65.
26. Axberg U, Broberg AG. Evaluation of “The Incredible Years” in Sweden: the transferability of an American parent-training program to Sweden. Scand J Psychol. 2012;53:224–232.
27. Baumel A, Pawar A, Kane JM, Correll CU. Digital parent training for children with disruptive behaviors: systematic review and meta-analysis of randomized trials. J
Child Adolesc Psychopharmacol. 2016;26(8):740–749.
28. Barlow J, Smailagic N, Huband N, Roloff V, Bennett C. Group-based parent training programmes for improving parental psychosocial health. Cochrane Database Syst
Rev. 2014;17(5):CD002020.
29. Furlong M, McGilloway S, Bywater T, Hutchings J, Smith SM, Donnelly M. Behavioral and cognitive-behavioural group-based parenting interventions for early-onset conduct problems in children age 3-12 years. Cochrane Database Syst Rev. 2012;15(2):CD008225.
30. Lau Y, Htun TP, Wong SN, Tam WSW, Klainin-Yobas P. Therapist-supported internet-based cognitive behavior therapy for stress, anxiety, and depressive
symptoms among postpartum women: a systematic review and meta-analysis. J Med
Internet Res. 2017;19(4):e138.doi:10.2196/jmir.6712.
31. Taylor-Richardson KD, Heflinger CA, Brown TN. Experience of strain among types of caregivers responsible for children with serious emotional and behavioral
disorders. J Emot Behav Disord. 2006;14:157–168.
32. Hedges LV. Distribution theory for Glass’s estimator of effect size and related estimators. J Educ Stat. 1981;6:107–128.
33. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of Interventions. Chichester, England: Wiley-Blackwel; 2011.
34. Thomas J, O'Mara-Eves A, Brunton G. Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst Rev. 2014;3(1):1–14.
35. Longest KC, Vaisey S. Fuzzy: a program for performing qualitative comparative analyses (QCA) in Stata. Stata J. 2008;8:79–84.
36. Hedges LV, Hedberg EC. Intraclass correlation values for planning group-randomized trials in education. Educ Eval Policy Anal. 2007;29:60–87.
37. Morgan AJ, Rapee RM, Salim A, et al. Internet-delivered parenting program for prevention and early intervention of anxiety problems in young children: randomized controlled trial. J Am Acad Child Adolesc Psychiatry. 2017;56(5):417–425.e1. 38. Sourander A, McGrath PJ, Ristkari T, et al. Internet-assisted parent training
intervention for disruptive behavior in 4-year-old children a randomized clinical trial.
JAMA Psychiatry. 2016;73(4):378–387.
39. Sanders MR, Dittman CK, Farruggia SP, Keown LJ. A comparison of online versus workbook delivery of a self-help positive parenting program. J Prim Prev.
2014;35:125–133.
40. Baker S, Sanders MR, Turner KMT, Morawska A. A randomized controlled trial evaluating a low-intensity interactive online parenting intervention, Triple P online brief, with parents of children with early onset conduct problems. Behav Res Ther. 2017;91:78–90.
41. Porzig-Drummond R, Stevenson RJ, Stevenson C. Preliminary evaluation of a self-directed video-based 1-2-3 magic parenting program: A randomized controlled trial.
Behav Res Ther. 2015;66:32–42.
42. Sanders MR, Baker S, Turner KMT. A randomized controlled trial evaluating the efficacy of Triple P online with parents of children with early-onset conduct problems. Behav Res Ther. 2012;50:675–684.
43. Morawska A, Tometzki H, Sanders MR. An Evaluation of the efficacy of a Triple P-positive parenting program Podcast series. J Dev Behav Pediatr. 2014;35(2):128–137. 44. Enebrink P, Högström J, Forster M, Ghaderi A. Internet-based parent management
training: a randomized controlled study. Behav Res Ther. 2012;50:240–249.
45. Breitenstein SM, Fogg L, Ocampo EV, Acosta DI, Gross D. Parent use and efficacy of a self-administered, Tablet-based parent training intervention: a randomized controlled trial. JMIR Mhealth Uhealth. 2016;4(2):e36. doi: 10.2196/mhealth.5202. 46. Jones SH, Jovanoska J, Calam R, et al. Web-based integrated bipolar parenting
intervention for parents with bipolar disorder: a randomized controlled pilot trial. J
Child Psychol Psychiatry. 2017;58(9):1033–1041.
47. Antonini TN, Raj SP, Oberjohn KS, et al. A pilot randomized trial of an online parenting skills program for pediatric traumatic brain injury: improvements in parenting and child behavior. Behavior Therapy. 2014;45:455–468.
48. Franke N, Keown LJ, Sanders MR. An RCT of an online parenting program for parents of preschool-aged children with ADHD symptoms. J Atten Disord. 2016; pii: 1087054716667598. [Epub ahead of print].
49. Menting ATA, Orobio DC, Matthys W. Effectiveness of the incredible years parent training to modify disruptive and prosocial child behavior: a meta-analytic review.
Clin Psychol Rev. 2013;33:901–913.
50. Sanders MR, Kirby JN, Tellegen CL, Day JJ. The triple P-positive parenting program: a systematic review and meta-analysis of a multi-level system of parenting support.
Clin Psychol Rev. 2014;34(4):337–357.
51. Teubert D, Pinquart M. A meta-analytic review on the prevention of symptoms of anxiety in children and adolescents. J Anxiety Disord. 2011;25:1046–1059
52. Ginsburg GS, Schlossberg MC. Family-based treatment of childhood anxiety disorders. Int Rev Psychiatry. 2002;14:143–154.
53. Kellam SG, Rebok GW, Mayer LS, Ialongo N , Kalodner CR. Depressive symptoms over first grade and their response to a developmental epidemiological based
preventive trial aimed at improving achievement. Dev Psychopathol. 1994;6:463–481. 54. Ahn H, Wampold BE. Where oh where are the specific ingredients? A meta-analysis
of component studies in counseling and psychotherapy. J Couns Psychol. 2001;48:251–7.
55. Bell EC, Marcus DK, Goodlad JK. Are the parts as good as the whole? A meta-analysis of component treatment studies. J Consult Clin Psychol. 2013;81:722-36. 56. Tarver J, Daley D, Lockwood J, Sayal K. Are self-directed parenting interventions
sufficient for externalising behaviour problems in childhood? a systematic review and meta-analysis. Eur Child Adolesc Psychiatry. 2014;23(12):1123-37.
57. Baumel A, Pawar A, Kane JM, Correll CU. Digital parent training for children with disruptive behaviors: systematic review and meta-analysis of randomized trials. J
Child Adolesc Psychopharmacol. 2016;26(8):740-749.
58. Sonuga-Barke E, Brandeis D, Cortese S, Daley D, Ferrin M, Holtmann M, et al. Nonpharmacological interventions for ADHD: systematic review and meta-analyses of randomized controlled trials of dietary and psychological treatments. Am J
Supplementary Online Content
eTable 1. Lists of the QCA Components
Components Function Items
1.Psychoeducation Change parental behavior by informing parents about child development and child’s needs
1A.Explain child developmental stages
1B.Explain appropriate parent's response to child's emotion and behaviors
2.Social learning theory Change child behavior by parental reactions: (reward and punishment)
2A.Positive Reinforcement of positive behavior 2B.Management of negative behavior
3.Proactive parenting Change child behavior by parental actions: prositivity and structure
3A.Clear limit setting
3B.Give positive & direct commands 4.Relationship building Improve quality of parent-child
relationship 4A.Spend quality time with the child 4B.Sensitivity and responsivity to the child’s needs 5.Engage in child-led
play Provide child with child-led play-based activity 5.Engage in child-led play 6.Client-centered therapy Provide child with an opportunity
to develop sense of self 6A.Active listening 6B.Negotiation 7.Parent self-problem
management Inform and monitor parents about problem solving in their family 7A.Parent monitoring 7B.Problem solving between adults 8.Parent self-emotional
regulation Inform parents about self-emotional regulation 8.Parent self-emotional regulation 9.Teaching parents how
to teach children Inform parents about how to teach the child in different topics 9A.Problem solving 9B.Emotion regulation
9C.social skills for peer relations 10.Written text Inform parents via written text 10.Written text
11.Video Inform or model parents via video 11A.Instruction Video 11B.Video vignettes 12. Online exercise or
homework Encourage parents to work with home activity 12A.Online exercise 12B.Homework or practice assignment 13.Therapist analysis Contact parents by therapist to 13.Therapist analysis with feedback
with feedback provide feedback on their practices
14.Phone call Parent receive phone call in
between session 14.Phone call
15.Supplemental
information Provide additional source of information to parents 15A.Downloadable information or book 15B.Downloadable worksheet 15C.Downloadable Podcast
15D.Automated summary email 16.Parent reminder Send parents reminders to work
eTable 2. Study’s Components and Coding Components 1. Morg an et al,20 17 2. Sour ande r et al, 2016 3. Sand ers et al, 2014 4. Bake r et al, 2017 5. Porzi g-Drum mon d et al, 2015 6. Sand ers et al, 2012 7. Mora wska et al, 2014 8. Eneb rink et al, 2012 9. Breit enste in et al, 2016 10. Jone s et al, 2017 11. Anto nini et al, 2014 12. Fran ke et al, 2016 1. Psychoeducation 1 1 1 1 1 1 1 1 1 1 1 1
1A. Explain child developmental
stages 0 1 0 0 0 0 0 0 0 0 0 0
1B. Explain appropriate parent's response to child's emotion and behaviors
1 1 1 1 1 1 1 1 1 1 1 1
2. Psychoeducation 1 1 1 1 1 1 1 1 1 1 1 1
2A. Positive Reinforcement of
positive behavior 1 1 1 1 1 1 1 1 1 1 1 1
2B. Management of negative
behavior 1 1 1 1 1 1 1 1 1 1 1 1
3. Proactive parenting 0 1 1 1 1 1 1 1 1 1 1 1
3A. Clear limit setting 0 1 1 1 1 1 1 1 1 1 1 1
3B. Give positive & direct commands 0 1 1 1 0 1 1 1 1 1 1 1
4. Relationship building 1 1 1 1 0 1 1 1 1 1 1 1
4A. Spend quality time with the child 1 1 1 1 0 1 1 1 1 1 1 1
4B. Sensitivity and responsivity to the
child’s needs 1 1 1 1 0 1 1 0 1 1 1 1
5. Engage in child-led play 0 1 0 0 0 0 0 1 1 1 0 0
6. Client-centered therapy 0 1 1 0 0 0 0 0 1 0 1 1
6A. Active listening 0 1 1 0 0 0 0 0 1 0 1 1
7. Parent self-problem management 1 1 1 1 1 1 1 1 1 1 1 1
7A. Parent monitoring 0 1 1 1 1 1 1 0 0 1 1 1
7B. Problem solving between adults 1 1 1 0 0 0 0 1 1 0 1 1
8. Parental emotion regulation /
dealing with negative thoughts 1 1 0 0 1 1 0 0 1 1 0 0
9. Teaching parents how to teach
children 1 1 1 1 1 1 1 1 0 1 1 1
9A. problem solving 0 0 1 1 0 0 1 0 0 1 1 1
9B. emotion regulation 1 1 1 1 1 1 1 1 0 1 1 1
9C. social skills for peer relations 1 1 0 0 0 0 0 0 0 0 1 0
10. Written text 1 0 0 0 0 0 0 1 1 0 1 0
11. Video 1 1 1 1 1 1 1 1 1 1 1 1
11A. Instruction Video 1 1 1 1 1 1 1 1 1 1 1 1
11B. Video vignettes 1 1 1 1 1 1 0 1 1 1 1 1
12. Online exercise or homework 1 1 0 0 0 1 0 1 1 1 1 1
12A. Online exercise 1 1 0 0 0 1 0 1 1 1 1 1
12B. Homework or practice
assignment 1 0 0 0 0 1 0 1 1 1 1 1
13. Therapist analysis with feedback 1 1 0 0 0 0 0 1 0 0 1 1
14. Phones call 1 1 0 0 0 0 0 0 0 0 0 1
15. Supplemental information 1 0 0 0 0 1 0 1 1 1 0 1
15A. Downloadable information 0 0 0 0 0 1 0 1 1 1 0 1
15B. Downloadable worksheet 0 0 0 0 0 1 0 1 0 1 0 1
15C. Downloadable Podcast 0 0 0 0 0 1 0 1 0 1 0 1
15D. Automated summary email 1 0 0 0 0 0 0 0 0 0 0 0
16. Parent reminder 1 0 0 0 0 1 1 0 0 1 0 1
16A. Email 0 0 0 0 0 1 1 0 0 1 0 1
Source
eFigure 1. Forest Plot Displaying 28 Effect Sizes of Online Parent Support Programs on Children’s Behavioral Problems
Each study is followed by 2 or 3 letters: the first represents the measurement tools (C, Child Behavior Checklist; E, Eyberg Child Behavior Inventory; and S, Strengths and Difficulties Questionnaire), the second represents the behavioral subtest (B, Behavior; C, Conduct; E, Externalizing; I,
Intensity; H, Hyperactivity; and P, Problem including peer problem for the Strengths and Difficulties Questionnaire), and the third represents the reporters (M, Mother; and F, Father). The diamond indicates the overall multi-level random effect across all studies.
eFigure 2. Forest Plot Displaying 8 Effect Sizes of Online Parent Support Programs on Children’s Emotional Problems Source
Hedges’ g (95%CI)
Each study is followed by 2 letters: the first represents the measurement tools (A, Child Adjustment and Parent Efficacy Scale; C, Child Behavior Checklist; P, Preschool Anxiety Scale-Revised; and S, Strengths and Difficulties Questionnaire), and the second represents the emotional subtest (A, Anxiety; E, Emotion; I, Internalizing; and L, Anxiety life interference).The diamond indicates the overall multi-level random effect across all studies.
eFigure 3. Forest Plot Displaying 39 Effect Sizes of Online Parent Support Programs on Parental Mental Health
Hedges’ g (95%CI)
Source
Each study is followed by 2 or 3 letters: the first represents the measurement tools (C, Child Adjustment and Parent Efficacy Scale; D, Depression, Anxiety, and Stress Scale; O, Over-Involved/Protective parenting scale; P, Parenting scale; and S, Parenting stress index), the second represents the subtests (A, Anxiety; D, Depression; I, Anxiety life interference; L, Laxness; O, Overreactivity; T, Total; S, Stress; and V, Verbosity), and the third represents the reporters (M, Mother; and F, Father). The diamond indicates the overall multi-level random effect across all studies.
