A Hybrid Application Of Structural Equation Modeling and Network
Analysis
Boris Stapel
University of Amsterdam
Conventionally, relationships between (latent) psychological constructs were studied with the structural model in Structural Equation Modeling (SEM). In this study network analysis serves as an alternative for the structural model to study relations between constructs. Network anal-ysis is applied and evaluated on aggregated data that represents psychological constructs in a simulation study and on empirical data. Data was aggregated according to three methods: 1. Fitting a fully correlated Confirmatory Factor Analysis (CFA), 2. Separate CFA and 3. Sum scores. Simulations showed that overall all measurement methods yielded the same results. On the empirical data different measurement methods produced three different networks. Results are discussed.
Keywords:latent variables, networks, psychological aggregates, structural equation modeling,
confirmatory factor analysis.
Introduction
Structural Equation Modeling(SEM) is a family of statis-tical methods for causal inference based on covariance struc-tures that gained a lot of popularity in psychology. It en-ables users of SEM to use a measurement model to measure psychological constructs and a structural model to measure relations between these constructs. The measurement model stems from the factor analytic approach to continuous vari-ables by Spearman (1904). This is further developed as con-firmatory factor analysis (CFA) to indirectly measure psy-chological constructs (Jöreskog, 1967); commonly referred to as latent variables, whereby the effect that the construct has on the observed variables (Figure 1) (Borsboom, Mellen-bergh, & Van Heerden, 2003). The idea is that CFA isolates the construct from other irrelevant factors that have nothing to do with what the researcher tries to measure. The struc-tural model entails the path model that was also developed in the early 20th century by Wright (1934). The path model enables users to test a system of regression equations to test theories that have multiple relations and variables in them. In the context of testing a system of regression equations be-tween latent variables, the path model is referred to as the structural model. SEM that include both of these models are very appealing because pure constructs can be distilled from distorted data while at the same time it can test how these constructs relate to each other. A model with many variables in which the measurement and the structural model are used in combination requires many participants which are not easily obtainable in practice. Therefore, researchers of-ten use other ways to approach measurement, such as the use of sum scores. Sum scores contain the true
variabil-ity of a construct and random variabilvariabil-ity, leading to biased, but more precise estimates of the model (Ledgerwood & Shrout, 2011). Close to calculating sum scores are indepen-dent factor analysis whereby for every construct a separate factor analysis is carried out to subsequently calculate fac-tor scores for every individual. This paper will focus on the relationships between psychological constructs as measured with different methods. Three measurement methods will be used: the 1. correlated CFA method, the 2. seperate CFAs
methodand the 3. sum score method. The structural model
will not be used to analyze the relationships between these constructs, but instead, a new method that is relatively new in psychology will be used to analyze the relations. This new analysis is called network analysis.
Just like SEM, network analysis can also be applied on co-variance structures that arise from the interactions between variables Costantini et al. (2014). There are three kinds of networks: directed correlation networks, undirected corre-lation networksand undirected partial correlation networks. Undirected partial correlation networks are the most inter-esting as an alternative for the structural model to investi-gate relations between psychological constructs. First, they do not allow the ambiguity of equivalent models. Second, the relationships are symmetrical; the benefit of symmetrical relationships is that problematic causal implications do not affect the interpretation. Third, when two variables show a relation, the effect of other variables is partialled out, thus it is possible to study conditional independence of variables. When two variables are conditionally independent of each other this means that they are independent given all other variables that are included in the analysis.
appealing as a way to study the relations between psycho-logical constructs. Unfortunately, pure statistical indepen-dence is close to impossible to find in psychological data. Both correlation and partial correlation networks will always be fully connected networks because “everything correlates to some extent with everything else” (Meehl, 1990, p.204). The LASSO (least absolute shrinkage and selection opera-tor) (Friedman, Hastie, & Tibshirani, 2008; Tibshirani, 1996) method offers a solution for this problem by estimating par-tial correlations in a sparse manner by shrinking down the arbitrarily small edges to exactly zero creating a network that is more interpretable (Costantini et al., 2014). Figure 2 rep-resents a LASSO network when data is generated according to the model in Figure 1.
SEMs and network analysis can both be applied to co-variance structures as to replace the CFA as a measurement model and as a substitute for the structural model. In the next two subsections I will first elaborate on networks as a substitute for CFA, an application which has already been explored. Second, I will present network analysis as a substi-tute for the structural model as a new application, providing the main goal of this paper: to study how network analysis can be applied on the analysis of the relations between latent variables. This new approach to will go by the name of
hy-brid analysiswhere measurement and network analysis are
combined through a two-step procedure: first the constructs are measured, second the network of the constructs is esti-mated, resulting in a network of latent variables.
Measurement Model Vs. Network Analysis
Conventionally, researchers in psychology use CFA to re-duce complex data patterns of observed variables to latent constructs. A set of positively correlated variables are as-sumed to all be caused by the same construct, and can there-fore be reduced to a single number. In science, the use of this reductionist methodology is widespread but not su ffi-cient because it does not learn us anything about how the object of study works (Barabasi, 2012; Schmittmann et al., 2013). The same is true for psychology, where it is common to assume observed behaviour is influenced by unobservable causes (Bollen, 2002).
Network analysis is gaining interest as an alternative
to the measurement model (Borsboom & Cramer, 2013; Borsboom, Cramer, Schmittmann, Epskamp, & Waldorp, 2011; Cramer, Waldorp, van der Maas, & Borsboom, 2010; Schmittmann et al., 2013). In psychometrics, network anal-ysis is a method used to investigate associative patterns be-tween the indicators of the latent variable, in order to get a better view of the dynamics within the construct rather than postulating an underlying latent variable, thereby dropping the idea that indicators share a single cause (Schmittmann et al., 2013).
Since the network approach to psychological constructs
has been successful and networks are suitable for any co-variance structure, an interesting next step is to extend the use of network analysis. When we accept the use of latent variables, common practice dictates that structural equations have to be estimated between these constructs. In the next section I will set out to elaborate on the structural model and network analysis in order to show that networks have a wider applicability.
Network Analysis Vs. Structural Model
With SEM one can test causal relations between latent variables. A structural relation is represented by an arrow to indicate the direction of the effect. SEM is popular because it enables researchers to test complex causal hypotheses in a normative model to dictate what the data pattern should look like. However, there is a property of SEM that makes it a very inappropriate tool to use in psychology; statistically equiva-lent models exist with different substantial causal hypothe-ses. For the past 30 years, researchers have pointed out that causal SEMs are often ambiguous (Lee & Hershberger, 1990; MacCallum, Wegener, Uchino, & Fabrigar, 1993; Markus, 2010; Stelzl, 1986). That is, when a model shows a good fit, it is still unclear whether this is the correct model and if there are other equivalent models that show the exact same fit but have a different substantial interpretation (Markus, 2010). MacCallum et al. (1993) showed that by using a simple rule derived from Stelzl (1986) it is possible to create multiple equivalent models for models that are found in the literature and even that those alternative models are often consistent with existing literature in the substantial field. Statistically there is no way to choose one model over the other when one is faced with multiple equivalent models. Subsequently this makes it hard to interpret a SEM and especially the structural model because it contains the substantial causal hypotheses. The only arbiter to decide between models is theory, but as MacCallum et al. (1993) showed; multiple equivalent models can be supported by different theories.
This ambivalence is very confusing since it allows for a flexible substantial interpretation, however, the following ex-ample shows that a causally ambivalent model is just what researchers are looking for. The integrated threat theory of Stephan et al. (2002) serves as a good example of why a structural model is not always the best way to test psycholog-ical theories. The theory incorporates different perspectives on threat to provide a mechanism through which intergroup attitudes are explained. A very rough outline of the theory is as follows: First, there are antecedents of threat which influ-ence the different kind of threats. Second, there are the actual threats that are dependent of their antecedents, thus causing negative racial attitudes which are at the end of the chain (antecedents → threats → racial attitudes). This is where the directed theory of integrated threat formally ends. Through antecedents and threats a specific racial attitude is reached
and that is the end of it. It ends formally because this de-scription corresponds to the causal model as depicted in the paper by Stephan et al. (2002). However, in the methodolog-ical limitationssection of their paper the authors warn the reader for such an interpretation of the model:
“Undoubtedly most, if not all, of the causal rela-tions among the variables in the integrated threat theory are reciprocal in nature to a greater or lesser degree.”
Here Stephan et al. (2002) argue that the direction of the arrows should be ignored. A network of dependence and independence relations would be better suited in this situa-tion since there is only a single network instead of multiple equivalent structural models. This means that for example negative contact is associated with realistic threat which is associated with racial attitudes which is associated with neg-ative contact (negneg-ative contact – realistic threat – racial atti-tudes). If Stephan et al. (2002) would have used a network analysis, there would be no need for a section to explain the meaninglessness of the depicted causal relations in the model. It would therefore be more in line with reality, the reality wherein associations are observed instead of causal relations.
Hybrid Analysis
Theoretically, the network approach of latent variables and sum scores that represent psychological constructs ap-pears to be a valuable addition to the SEM framework in the form of hybrid analysis. In this paper the procedure for hy-brid analysis will be explained and will be tested to see how well this procedure works in a simulation study. Data will be simulated according to a SEM with latent variables and a specific causal structure. This structure implies multiple equivalent causal models but just a single network. How often this network is retrieved correctly with the LASSO method under different circumstances will be the measure of the quality of the network estimation. To measure the sim-ulated latent variables the correlated CFA method, separate CFA method and the sum score method will be used to show that different methods of measurement matter. Where the use of sum scores in SEM have been criticized for serious conse-quences due to measurement error (Cole & Preacher, 2014; Ledgerwood & Shrout, 2011), the LASSO method could be the answer to the biased estimates since it is designed to dis-till a sparse network. As the correlated CFA method and the separate CFA method both reduce measurement error, they should suffer less from its consequences. After the hybrid analysis has been tested on simulated data, it will be applied to empirical data provided by Doosje, Loseman, and van den Bos (2013). This dataset was used to calculate 14 sum score scales, representing constructs in a path model. In the cur-rent study, hybrid analysis will be applied to this data set
ξ1 X2 X1 X3 η1 Y2 Y1 Y3 η2 Y5 Y4 Y6 δ1 δ2 δ3 1 2 3 4 5 6 ζ1 ζ2 γ21 γ11 β21 λ7 λ8 λ9 λ1 λ2 λ3 λ4 λ5 λ6 Structural Model Measurement Model Figure 1. Reflective measurement model and the structural model. Latent variables are measured indirectly through the observed variables. Arrows between the latent variables rep-resent causal relations.
by recalculating the sum scores for network analysis and by measuring all constructs with the correlated and the separate CFA method. This yields three different networks that will be evaluated based on their edges.
This paper is structured as follows: first, the 2-step pro-cedure is tested under controllable conditions in a simulation study. In this first step a factor model with correlated fac-tors, independent factors and sum scores will be estimated and calculated. This section researches the ability to recover a network structure of the latent variables and other aggre-gates in a given SEM. Second, I apply the 2-step procedure on empirical data acquired of Doosje et al. (2013). In their ar-ticle Doosje et al. (2013) measure 14 constructs through sum scores to be analyzed in relation to each other. Their analysis will be extended with network analysis of latent variables and sum scores.
Simulation Study
Instead of multiple equivalent models that a SEM implies, there is only a single undirected network that corresponds to any directed SEM. In the present study we aim to simulate data according to a full mediation model (X → Y → Z). The reason to use a full mediation model is because it is very simple but sufficient to simulate dependence and conditional independence relations. In X → Y → Z X is conditionally
ξ1 X2 X1 X3 ξ2 X4 X5 X6 ξ3 X7 X8 X9 δ1 δ2 δ3 δ4 δ5 δ6 δ7 δ8 δ9 LAS S Oweight LAS S Oweight LAS S Oweight λ7 λ8 λ9 λ1 λ2 λ3 λ4 λ5 λ6 Network Measurement Model Figure 2. A hybrid model. In the first step the latent variables are independently being estimated with reflective measure-ment models. In the second step an independent network of latent variables is estimated.
independent of Z:
(X ⊥ Z | Y) (1)
The same conditional independence applies to the common cause model X ← Y → Z and the mediation model with opposite directions X ← Y ← Z making them equivalent (Stelzl, 1986). To rewrite these models to a single network the conditional independence has to be conserved, which can be done by following two steps (Lauritzen & Spiegelhalter, 1988): first, by marrying parents of a node by adding an edge between them, this process is called moralization; second, by dropping the arrows. The mediation model has no such im-moralities, dropping the arrows is sufficient to make it undi-rected resulting in X − Y − Z.
In theory it is now clear which network should be retrieved given a latent mediation model. The following simulations will test if this is also true in practice under different con-ditions. The simulations test the extent to which varying strengths of latent regressions, factor loadings and sample sizes affect the retrieval of a correct network, using the 2-step procedure. The question that will be addressed is to what extent the LASSO method is suitable to retrieve the depen-dence and conditional independepen-dence relationships when data is generated by a SEM with latent variables. The expectation is that 75% of the networks will be retrieved correctly.
Y X5 X4 X6 X X1 X2 X3 Z X8 X7 X9 4 5 6 1 2 3 7 8 9 ζ2 ζ3 β21 β32 λ7 λ8 λ9 λ1 λ2 λ3 λ4 λ5 λ6 (a) Y X Z (b) Figure 3 Method
Data was generated according to the mediation model (X → Y → Z) which implies a specific structure in the co-variances. The covariance matrix with these structures can be determined by multiplying and adding matrices according to matrix algebraic rules that represent different parts of the model. The following formula represents a general SEM:
Σ(θ) = ΛΣηΛ−1+ Θ (2)
Λ represents the factor loadings which are basically weights assigned to the regressions of the observed variables on the latent variable andΘ represents the residuals of the observed variables. Ση represents the covariance matrix of the latent variables which is written as:
Ση= (I − B)−1Ψ(I − B)−1t (3)
I represents an identity matrix, B represents the regression coefficients between the latent variables and Ψ is the
covari-0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor
tion correctly estimated MRFs
0.00 0.25 0.50 0.75 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor tion of tr ue −negativ es (X −Z) 0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor tion of tr ue −positiv es (X −Y) and (Y −Z) Samplesize 050 100 150 200 250
Beta: Correlated Factors
(a) (b) (c)
Figure 4. Density plot for the sparse partial correlations between X and Z
0.00 0.25 0.50 0.75 1.00 0.5 0.6 0.7 0.8 lambda Propor
tion correctly estimated MRFs
0.00 0.25 0.50 0.75 1.00 0.5 0.6 0.7 0.8 lambda Propor tion of tr ue −negativ es (X −Z) 0.00 0.25 0.50 0.75 1.00 0.5 0.6 0.7 0.8 lambda Propor tion of tr ue −positiv es a ver aged o ver (X −Y) and (Y −Z) Samplesize 050 100 150 200 250
Lambda: Correlated Factors
(a) (b) (c)
Figure 5. Density plot for the sparse partial correlations between X and Z
ance matrix of the residuals of the latent variables. Together these functions can be rewritten as:
Σ(θ) = Λ(I − B)−1Ψ(I − B)−1tΛ−1+ Θ (4)
Appendix A covers a detailed explanation of the matrix alge-bra involved in the simulations.
To test the performance of the LASSO method I simulated 1000 data sets for each of the 55 conditions:
• SEM (1 level: 3 indicators, 3 factors)
• Factor loadings (4 levels: λ= 0.5, 0.6, 0.7, 0.8) • Latent regressions (7 levels: β= 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8)
• Sample size (5 levels: N= 50, 100, 150, 200, 250) Factor loadings and latent regressions were each crossed with sample size but not with each other. When lambdas varied betas were fixed .5, when betas varied lambdas were fixed at .7. The values for the factor loadings were chosen to be able to spot possible trends when they increased. These values are not realistic in practice where it is common to not use items with factor loadings lower than .7. This same argu-ment goes for the latent regressions where a large number of values make it easier to spot a trend although values of .2 and .3 are more likely in practice. For the sample sizes with relatively small values were chosen to reflect the common sample sizes in psychology, a field which is plagued by a scarcity in participants.
For every condition the two-step procedure of the hybrid analysis will be applied as follows. After a dataset was
gen-erated given specific values for the sample size, beta and lambda the first step was to measure the latent variables by one of the three measurement methods: 1. correlated CFA, 2. seperate CFA and 3. sum scores. For the correlated CFA the correlation matrix of the latent variables could be extracted from the fit object of lavaan (Rosseel, 2012). For the separate CFA method individual factor scores had to be calculated for each factor separately, yielding a data set with the scores of each individual on each of the three latent variables which than had to be used to calculate the correlation matrix of the latent variables. The sum scores were calculated by summing up the simulated observed variables, creating three columns of data where each column represented the scores of the in-dividuals on each latent variable. These three columns of data were then used to calculate the correlation matrix. Each of these correlation matrices where then used to estimate a sparse network with the LASSO method. The technical de-tails of this process are described in the next subsection. Data Generation and Analysis
Data generation and analysis was carried out with with the R programming language (R Core Team, 2014). For the data generation I created a function that first calculated the implied covariance matrix for a given λ and β accord-ing to the matrix algebra as described in detail in appendix A. Second, the implied covariance matrix was used as input for the mvrnorm function (Venables & Ripley, 2002) to
gen-0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor
tion correctly estimated MRFs
0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor tion of tr ue −negativ es (X −Z) 0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor tion of tr ue −positiv es (X −Y) and (Y −Z) Samplesize 050 100 150 200 250
Beta: Independent Factors
(a) (b) (c)
Figure 6. Density plot for the sparse partial correlations between X and Z
0.00 0.25 0.50 0.75 1.00 0.7 0.8 0.9 lambda Propor
tion correctly estimated MRFs
0.00 0.25 0.50 0.75 1.00 0.7 0.8 0.9 lambda Propor tion of tr ue −negativ es (X −Z) 0.00 0.25 0.50 0.75 1.00 0.7 0.8 0.9 lambda Propor tion of tr ue −positiv es a ver aged o ver (X −Y) and (Y −Z) Samplesize 050 100 150 200 250
Lambda: Independent Factors
(a) (b) (c)
Figure 7. Density plot for the sparse partial correlations between X and Z
erate a dataset according to a multivariate normal distribu-tion. Third, the simulated mediation model was fitted on the data of which the covariance matrix of the latent variables was stored in an object with the Lavaan package (Rosseel, 2012). At last, the covariance matrix, inverse matrix, par-tial correlation matrix (Dethlefsen & Højsgaard, 2005) and the sparse partial correlation matrix (Epskamp, Cramer, Wal-dorp, Schmittmann, & Borsboom, 2012) of the latent vari-ables were all stored for analysis. Data were visualized with the ggplot2 package (Wickham, 2009).
Results
All three methods show the same patterns in the results, therefore the first part of this section goes for all three meth-ods. Results are presented in Figure 4 – 9. The results show very interesting deviations from the expected outcome, which requires more investigation. In only one of the condi-tions of the separate CFA method 75% correctly estimated
networks were retrieved. In general the correct network
shows to be retrieved very poorly. First, the results of the complete networks when beta increases, indicates that when betas increase the accuracy of the estimated networks be-comes better (Figure 4a, 6a and 8a) . This is most likely the effect of the stronger, more reliable betas, but then after β = 0.4 the proportion of correct networks declines drasti-cally. Figure 5a, 7a and 9a shows more positive results in line with the expectations. When the lambdas increase and
thus, the estimates of the latent variables become more reli-able. More reliable estimates also lead to more reliable re-lationships. The results for the complete networks indicate that to investigate the decline in accurate networks we have to take a closer look at the edges in the network and how they individually behave under different circumstances.
Analysis of Figure 4c, 6c and 8c and Figure 5c, 7c and 9c immediately make it evident that the problem with the accu-racy of the estimated network has not much to do with the positive dependence relationships. The stronger β and λ are the more reliable the estimation of these edges become re-sulting in almost 100 percent accuracy when values for both λ and β become .5 and .4 respectively with some variation with respect to the different sample sizes.
The decreasing accuracy in the complete networks (Figure 4a, 6a and 8a) is certainly not due to the inability to estimate dependencies. Therefore the problem must lie in the estima-tion of condiestima-tional independence (X − Z). Contrary to the increasing ability to estimate conditional independence with increasing lambdas in Figure 5b, 7b and 9b the results in Fig-ure 4b, 6b and 8b show that increasing betas decrease to abil-ity to estimate conditional independence. If the relationships between variables become stronger, it is more likely to find a false-positive relationship where there should be no relation-ship at all. The model is designed in a way that X and Z are conditionally independent on without shrinkage. So the rela-tionships that are estimated between X and Z should be spu-rious. This is something that the LASSO method is designed
0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor
tion correctly estimated MRFs
0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor tion of tr ue −negativ es (X −Z) 0.00 0.25 0.50 0.75 1.00 0.2 0.3 0.4 0.5 0.6 0.7 0.8 beta Propor tion of tr ue −positiv es (X −Y) and (Y −Z) Samplesize 050 100 150 200 250 Beta: Sum−scores (a) (b) (c)
Figure 8. Density plot for the sparse partial correlations between X and Z
0.00 0.25 0.50 0.75 1.00 0.5 0.6 0.7 0.8 lambda Propor
tion correctly estimated MRFs
0.00 0.25 0.50 0.75 1.00 0.5 0.6 0.7 0.8 lambda Propor tion of tr ue −negativ es (X −Z) 0.00 0.25 0.50 0.75 1.00 0.5 0.6 0.7 0.8 lambda Propor tion of tr ue −positiv es a ver aged o ver (X −Y) and (Y −Z) Samplesize 050 100 150 200 250 Lambda: Sum−scores (a) (b) (c)
Figure 9. Density plot for the sparse partial correlations between X and Z
to tackle by setting small spurious relationships to zero. Fol-lowing the previous information I can identify two possible explanations for these deviant results: 1) There could be an artefact in the procedure of simulating and model fitting that causes the LASSO method to fail in recognizing conditional independence or 2) there is a problem with estimating con-ditional independence independent from latent variables that has not yet been identified.
For the next part, I designed an extra simulation study to that partially follows the same procedure as for this study. Instead of simulating data according to latent variables, data are simulated according to observed variables. As a result of this I have skipped the model fitting procedure excluding the possibility of biased data.
Replication With Observed Variables
In the first simulations I generated data from a complete SEM. This raised questions about artifacts of the simulations being responsible for the false-positive relations between X and Z. Specifically, the lambdas were suspected to maybe biasing the covariance between X and Z. To test these hy-potheses I replicated the simulations by sampling data from the covariance matrix of the latent variablesΣη(Equation 3), thereby treating them as directly observed variables. By do-ing so, I can determine if the results, as described above, are a characteristic from the SEM or a characteristic of the method with which the network is estimated.
Results from simulations of observed variables do not dif-fer from the results of the latent variables in their qualitative pattern, the exact values are not exactly the same but the de-cline in precision corresponds with the results from the sim-ulation with latent variables. Now I can exclude the option of SEM causing the bias in the LASSO method. Unexpect-edly, this brings us to an interesting problem that is related to LASSO. To dig deeper into this problem it is worth looking at the distributions of the covariances, inverse covariances, par-tial correlations and the sparse LASSO parpar-tial correlations. The expected values for covariances between X and Z are to be distributed around β2. The expected values for the partial correlation, inverse and the sparse partial correlation matrix are all to be distributed around zero.
The distributions of the covariances, inverse of the co-variances and the partial correlations all correspond to the expected distributions. The distributions of the sparse par-tial correlations are way off the expected distributions (Fig-ure 11). Instead of staying centered around zero, the dis-tribution shows a off-zero bump that gets larger when the β values increase. Subsequently, when β becomes larger the LASSO shows a bias towards making partial correlations of zero slightly bigger.
This bias indicates a fundamental problem for the LASSO method which has been further researched by Epskamp
(2015).1 To summarize: the LASSO method is not able to
elabora-(a) (b) (c)
Figure 10. Results of the simulation of three observed variables by varying the values of β. (a) Are the results for the com-pletely correct estimated networks, (b) represents the results for the true-positive relationships and (c) represents the results for the proportion of false-positives for the conditional independence relation.
process partial correlations of zero. Every partial correla-tion receives an adjustment which is almost always shrink-age. However, when a partial correlation is exactly zero the LASSO method still adjusts. This adjustment then results in a partial correlation bigger than zero. An inspection of the population covariance matrix, a matrix in which the par-tial correlations are exactly zero because there is no random variability, shows that it is indeed the case that zero partial correlations become bigger. This effect seems to be stronger when the strength of indirect relations (X − Y & Y − Z) in-crease (Figure 12).
Figure 11. Density plot for the sparse partial correlations between X and Z
Figure 12. Sparse partial correlations for the population re-lationship between X and Z
Correlated Factors, Independent Factors and Sum Scores Until now the interpretation of the results have been col-lapsed over different methods of measurement. Although all methods show the same results, there are interesting di ffer-ences that are explicitly interesting for psychology, where sample sizes often are small. First, if we compare Figure 4a, 6a and 8a correlated factors better overall than independent factors and sum scores. There is less variety between sample sizes and the curve of the graph is less steep (Figure 4a), indicating more stable results over different values for beta. The plots for the complete networks with variable lambda values (Figure 5a, 7a and 9a) also show that, overall, corre-lated factors outperform the other methods of measurement. However, with reasonable sample sizes, sum scores perform almost equally well and sometimes outperform the correlated factors. With lower lambdas, a large sample size is needed to be as reliable as the correlated factors but with higher factor loadings sample sizes of 100 and 150 will be sufficient for a performance equal to the correlated factors measurement. Finally, the use of sum scores or independent factors is bene-ficial for the estimation of conditional independence in rela-tion of the problem, authored by Epskamp is available upon request.
tion to the correlated factors (Figure reffig:betafullb, 6b and 8b). Until values for beta of .5 sum scores and independent factors outperform the correlated factors measurement. Most notably, this goes especially for the small sample size condi-tions of 50.
Conclusion
Concluding this section it has to be mentioned that the conditions under which the model was constructed were ideal. Although random variability was introduced to the data, the values that were taken to sample from were cho-sen in such a way that on a population level there was con-ditional independence resulting in many simulated cases that corresponded to the population in such a way that there was conditional independence. This section shows that in these cases the LASSO method gives a biased estimate of the re-lationship between two variables when other regression co-efficients are very high and the partial correlation between at least two variables is zero. Instead of being independent of each other, the two variables can get a substantial posi-tive relationship. It is rather unlikely that such a situation will present itself to a psychologist, but these results teach us that inspection of the partial correlation matrix or the in-verse covariance matrix is good practice when the LASSO method is used to estimate a network. If there are any zero partial correlations this should influence the choice of tun-ing parameters to avoid a biased estimate. Notably, the bias of the LASSO method happened mostly when the latent re-gressions were exceptionally high. Figure 12 shows that the bias only appears when regression coefficients reach values of .6 or higher. In psychology, SEMs with latent regressions as high as these are very unlikely, because it requires much better measurement and stronger effects, which are absent in psychology. Overall, these results are very positive.
Empirical Data Analysis
The first goal was to test the two-step procedure with sim-ulated data, which yielded interesting results on the use of a correlated factor model, with separate factor models and sum scores as a way of measurement. The second goal of this paper is to test these three methods of measurement on an empirical, substantively meaningful dataset supplied by Doosje et al. (2013). The supplied data were used to measure 14 constructs with sum scores to put in a path model. The constructs had varying amounts of indicators ranging from 2 to 15. The model that Doosje et al. (2013) tested showed an almost perfect fit (Figure 13). However, the perfect fit is mis-leading because of the uncorrected measurement error (Cole & Preacher, 2014). Because of the relative large amount of constructs that are measured, these data are ideal to test the
two-step method with different methods of measurement as
has been demonstrated in the simulation study.
Figure 13. A graph representation of the model by Doosje et al. (2013).
Substantively, every concept is defined as its indicators. The relations between the concepts are modeled with lin-ear regression equations implying a causal model that can be used to do reliable predictions. The concepts in the model by Doosje et al. (2013) are mostly borrowed from other studies, such as Stephan et al. (2002) and together these concepts and the structural equations are to represent a system of causal relationships. The methodological limitations from Stephan et al. (2002) do also apply for Doosje et al. (2013):
“A clear limitation of our work is its correla-tional design. Even though we have tested a structural equation model, it is not possible to draw any firm causal conclusions.”
All relationships, as implied by the model, lose their causal
meaning and might just as well be undirected. Just like
(Stephan et al., 2002) the current data set could just as well benefit from a descriptive network analysis of the data which might reveal more about the dynamics of radicalization than a well-fitting path model with unaccounted-for measurement error.
First, the correlated CFA method will be applied on the data. Second, the separate CFAs method will be applied. Third and last, the sum score method will be applied to calcu-late the constructs as was done in Doosje et al. (2013). A net-work of every measurement method will then be estimated to allow for network comparison according to the same proce-dures that were described in the simulations section. Analysis
To begin with, I applied the two-step procedure on the empirical data. The lavaan package Rosseel (2012) is used
Table 1
Fit measures for SEM and all separate CFAs
Model X2 df p RMSEA CFI TLI
1. Full correlated CFA 7084.765 3749 <.001 0.082 0.572 0.553
2. In-group identification* -1 3. Individual deprivation 23.801 5 <.001 0.169 0.921 0.843 4. Collective deprivation 13.015 5 .023 0.111 0.977 0.953 5. Intergroup Anxiety 225.367 35 <.001 0.204 0.853 0.811 6. Symbolic Threat 147.876 54 <.001 0.115 0.829 0.791 7. Realistic Threat** 0 0 0 0 1 1
8. Personal Emotional Uncertainty 92.990 44 <.001 0.092 0.901 0.876
9. Perceived Injustice 11.604 9 .237 0.047 0.992 0.986
10. Perceived Illegitimacy Authorities 8.701 5 .122 0.075 0.973 0.947
11. Perceived In-group Superiority 4.084 2 .131 0.089 0.980 0.939
12. Distance to Other people** 0 0 0 0 1 1
13. Societal Disconnectedness 68.606 9 <.001 0.225 0.769 0.616
14. Attitude Towards Violence by Others 0.940 2 .625 0.000 1.000 1.015
15. Own Violent Intentions** 0 0 0 0 1 1
Note: For all models N=131. *Underidentified. **Just-Identified
(a) (b) (c)
Figure 14. Three networks with different approaches to data aggregation. (a) Is the result of aggregation with a CFA with correlated factors. (b) Includes data that was aggregated with CFAs without correlated factors. In (c) the variables are mere sums (z-scores) of observed variables
to fit a 14-factor model on the observed data. By using the in-spect()function the correlation matrix of the latent variables was extracted after fitting the model. The correlation matrix was then subsequently made sparse and visualized with the graphical LASSO method by using qgraph (Epskamp et al., 2012). The seperate CFA’s with uncorrelated factors were fitted one by one with Lavaan, so that for every participant an individual factor score could be calculated. The factor scores were then eventually used to calculate the sparse par-tial correlation matrix with qgraph. The sum scores were calculated as z-scores that could also be used to calculate a sparse partial correlation matrix.
Results
The results of the network analyses are presented in Figure 14. The three distinct methods of measurement yield distinct results that reflect the different properties of different
mea-surement methods.
Figure 14a, 14b and 14c represent the correlated CFA, in-dependent CFA and sum scores network respectively. In the correlated CFA network the variable realistic threat was re-moved due to its high correlation with symbolic threat which led to convergence errors in the model fitting procedure. Also, all observed variables that had factor loadings smaller than .7 were removed from the model. The final model did not fit and modification indices mostly suggested cross load-ings to improve the fit. In both the correlated CFA model and the uncorrelated CFA model indicators were removed when having a factor loading smaller than .7 to improve measure-ment. The simulations support this practice because higher factor loadings lead to better edge estimations. The quan-titative fit measures can be seen in Table 1. From Table 1, it can be concluded that the correlated CFA does not have a good fit. In addition, the fit of all independent CFAs are
neither convincing. One model was under-identified and two were just-identified which causes that no fit measures could be calculated for these models. For the independent factor analyses, only perceived injustice and attitude towards vi-olence by others have an acceptable fit when taking all fit measures into account. At first sight we see that the struc-ture of the three networks is quite similar. Most of the posi-tive and negaposi-tive relationships stay stable over the networks, however, they have a tendency to become more sparse. The correlated CFA network shows many and very strong rela-tions in relation to the independent CFA network and the sum score network. Collective deprivation is strongly connected to five other variables in the correlated CFA network. For example, in both the independent CFA network and the sum score network, this is reduced to only two relations with in-dividual deprivation and systematic threat. Then, looking at
the differences between the independent CFA network and
the sum score network, two equally sparse networks with somewhat different relations are observed. These differences reflect the difference between estimating independent factor models and sum scores which handle measurement error or not respectively. However, both contain measurement error but in a different way, the LASSO method is designed to han-dle measurement error so there might be an interaction
be-tween the LASSO method and different measurement errors
which will be discussed in the next subsection. Conclusion and Discussion
The three different methods of measurement produced
three different networks after the LASSO method was ap-plied. These differences can be ascribed to the measurement methods. For the correlated CFA and many of the indepen-dent CFAs model fit can be evaluated as poorly. First, the correlated CFA required a large amount of cross loadings to fit. Missing required cross loadings forces the model to push up the correlations between latent variables in order to com-pensate for the missing cross loadings (Brown, 2006). This explains why the relations between the latent variables are so strong. These are biased relations reflecting misfit of the model, not relations that could reflect reality. An explanation for the misfit of the model could be ascribed to it being the wrong model, which would undermine the rest of the anal-ysis. However, given the fact the the model requires a lot of parameters to be estimated, which in turn requires a large number of participants. Therefore these results do not dis-prove the idea that there are 14 factors underlying this model, the data is just not sufficient to test this. Second, the network of 14 separate factor models (Figure 14b) avoid the problem of the cross-loadings in the network of the full CFA and be-cause the same amount of data was used to estimate fewer pa-rameters for every latent variable the parameter estimates are more reliable. Although relations between all indicators are not accounted for because of the uncorrelated factors. The
independent CFAs are still useful because in theory they are to exclude measurement error. However, it should be noted that factor scores predicted for each individual reduces error, they do not remove it (Bollen, 1989). In this light we can interpret the network as less biased by cross-loadings and with reduced misrepresentation due to measurement error. However, related observed variables under different factors are likely to exert their effect on the factor scores thereby influencing the factor relations. But most importantly Table 1 shows that the fit for each of the separate models is at least questionable to very bad, or there is no fit at all because of just-identification. And one model even is unidentified be-cause of a lack of indicators. Bebe-cause of this we cannot inter-pret the factors as reliable or valid estimates of the variables that we try to measure. Third, the sum score network shows the most sparse network. Very salient is the fact that two relations (illegitimacy of authorities ingroup superiority -distance to others) went from negative to positive. Another salient change is that the relation between individual depri-vation and realistic threat is absent while it was very strong in the network of the independent CFAs. I have not yet got a explanation for these discrepancy in the results. Most likely the explanation must be sought in the way that error and cor-relations between manifest variables are handled. For the correlated CFA network cross loadings clearly explain the structure. However, considering the separate CFA network and the sum score network, there seems not a clear di ffer-ence which will be discussed in the discussion section of this paper.
Discussion
This paper opened up possibilities for a new line of re-search within psychological modeling. The aim of this study was to test a new approach to the analysis of covariance structures of latent variables as alternative for the structural model in SEM. Correlated CFAs, separate CFAs and sum scores were used as methods of measurement and integrated with network analysis in a so-called hybrid analysis. To do so, hybrid analysis of the three methods of measurement were first put to the test in a simulation study before being applied on empirical data by Doosje et al. (2013)). The sim-ulations yielded positive results. In realistic cases the under-lying network was retrieved well. In extreme cases where partial correlations are zero or close to zero and the direct relations were exceptionally high, it was discovered that the LASSO has a positive bias, creating a false-positive edge be-tween variables that were simulated to be conditionally inde-pendent. This is valid for all three kinds of measurement. In addition, increasing factor loadings increase the true-positive rate and decrease the false-positive rate. Increasing latent re-gressions also increased the true-positive rate; but surpris-ingly, when latent regressions increased, the false-positive rate increased. Stronger relations between latent variables
make it increasingly less likely to find conditional indepen-dence. This could subsequently not be demonstrated with empirical data because it was not well suited for the cor-related CFA method and the separate CFA method. The shortcomings of this data did however prove to be useful to demonstrate how different approaches to the data influenced the network. The absence of cross-loadings strengthen the factor correlations to represent correlations between manifest variables of under different factors. When comparing sepa-rate models to sum scores, the results stayed similar to a great extent, but one salient difference was that some edges went from negative to positive. This has not yet been interpreted yet, but is most likely due to measurement error.
The simulations were used to see how well LASSO could be used to estimate relations and conditional independence. An argument against the simulations in this paper could be made as only a very small model was tested under very ideal circumstances, circumstances that were absent in the empirical data set and probably in most data sets that use SEM. It is true that the simulation could have been extended to include varying amounts of indicators, non-normal data, mixed regression weights and more. However, for this study, the simulation served as an illustration of translating a SEM to a network through partial correlations. The simulations were successful in demonstrating a lack of accuracy when the partial correlation was already zero. For the scope of this study this was enough to justify the rest of this paper which is a beginning for the analysis of latent variables in a net-work. However, when this line of research is going to be ex-tended later in the SEM literature more thorough simulation research could serve to investigate the different more realistic conditions in which networks have to be estimated. The in-dependent CFA network and the sum score were both sparse, but the networks differed substantially from each other. The absence of relations in one network were compensated by absent relations in the other network. Presently, we only know that the use of independent CFAs for the calculation of individual factor scores of participants does not remove all measurement error and does not account for relations be-tween observed variables under different factors. The use of sum scores does not account for any measurement error. Notably, little has been written about the relation between the sum score and calculating factor scores in the context of SEM (Grice, 2001). A point that is very interesting is that the LASSO method reduces the likelihood of relations that are due to measurement error, but when we are confronted with two different measurements that both have measurement er-ror we get different results. This is a very interesting result which should be studied in more detail.
Another criticism could be that network analysis as a re-placement for the structural model is not beneficial at all. First, it is not possible to calculate model fit. Second, if a structural model can be written as a network, why then not
represent it as a network but test it with a structural model? This is a valid observation. It is indeed the case that the net-works in this paper are not fit, they are instead the result of a normative procedure. Normally this would be very problem-atic since the data are not to be trusted since there is a lot of measurement error and sampling variability in psychological data. However, with the LASSO method it can be argued that the representation of the network is accurate to the extend that a reliable interpretation is possible.
Lastly, a very positive sign is the network that was es-timated on the empirical data using sum scores. Earlier it was mentioned that Cole and Preacher (2014) showed that path models with sum scores contain too many edges due to measurement error. Also in the path model of Doosje et al. (2013) many arrows are present, which makes it a hard to interpret a complex model where a sparse model might be more suitable. Where path modeling led to a model with many arrows, the LASSO method returned a sparse network that could be much more interpretable.
The present study delivered more questions than answers hower, this study opened up many possibilities for studying network analysis in combination with SEM.
References
Barabasi, A.-L. (2012). The network takeover. Nature Physics, 8, 14–16.
Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.
Bollen, K. A. (2002). Latent variables in psychology and the social sciences. Annual review of psychology, 53, 605–634. Borsboom, D., & Cramer, A. O. (2013). Network analysis: An
inte-grative approach to the structure of psychopathology. Annual review of clinical psychology, 9, 91–121.
Borsboom, D., Cramer, A. O., Schmittmann, V. D., Epskamp, S., & Waldorp, L. J. (2011). The small world of psychopathology. PloS one, 6, e27407.
Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2003). The theoretical status of latent variables. Psychological review, 110, 203–219.
Brown, T. A. (2006). Confirmatory factor analysis for applied
research.New York, USA: Guilford Press.
Cole, D. A., & Preacher, K. J. (2014). Manifest variable path analy-sis: Potentially serious and misleading consequences due to uncorrected measurement error. Psychological Methods, 19, 300–315.
Costantini, G., Epskamp, S., Borsboom, D., Perugini, M., Mõttus, R., Waldorp, L. J., & Cramer, A. O. (2014). State of the art personality research: A tutorial on network analysis of personality data in r. Journal of Research in Personality, 54, 13–29.
Cramer, A. O. J., Waldorp, L. J., van der Maas, H. L. J., & Bors-boom, D. (2010). Comorbidity: A network perspective. Behavioral and Brain Sciences, 33, 137–150.
Dethlefsen, C., & Højsgaard, S. (2005). A common platform for graphical models in R: The gRbase package. Journal of Sta-tistical Software, 14(17), 1–12.
Doosje, B., Loseman, A., & van den Bos, K. (2013). Determinants of radicalization of islamic youth in the netherlands: Per-sonal uncertainty, perceived injustice, and perceived group threat. Journal of Social Issues, 69, 586–604.
Epskamp, S. (2015). personal communication.
Epskamp, S., Cramer, A. O. J., Waldorp, L. J., Schmittmann, V. D., & Borsboom, D. (2012). qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software, 48(4), 1–18.
Friedman, J., Hastie, T., & Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9, 432-441. doi: 10.1093/biostatistics/kxm045
Grice, J. W. (2001). Computing and evaluating factor scores. Psy-chological Methods, 6, 430–450.
Jöreskog, K. G. (1967). A general approach to confirmatory max-imum likelihood factor analysis. Psychometrika, 34, 183– 202. doi: 10.1002/j.2333-8504.1967.tb00991.x
Lauritzen, S. L., & Spiegelhalter, D. J. (1988). Local computa-tions with probabilities on graphical structures and their ap-plication to expert systems. Journal of the Royal Statistical Society. Series B (Methodological), 157–224.
Ledgerwood, A., & Shrout, P. E. (2011). The trade-off between ac-curacy and precision in latent variable models of mediation processes. Journal of Personality and Social Psychology, 101, 1174–1188.
Lee, S., & Hershberger, S. (1990). A simple rule for generating equivalent models in covariance structure modeling. Multi-variate Behavioral Research, 25, 313–334.
MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological bulletin, 114, 185–199.
Markus, K. A. (2010). Structural equations and causal explana-tions: Some challenges for causal sem. Structural Equation Modeling, 17, 654–676.
Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195–244.
R Core Team. (2014). R: A language and environment for statistical computing [Computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. Re-trieved from http://www.jstatsoft.org/v48/i02/ Schmittmann, V. D., Cramer, A. O., Waldorp, L. J., Epskamp, S.,
Kievit, R. A., & Borsboom, D. (2013). Deconstructing the construct: A network perspective on psychological phenom-ena. New ideas in psychology, 31, 43–53.
Spearman, C. (1904). “General intelligence”, objectively deter-mined and measured. The American Journal of Psychology, 15, 201–292.
Stelzl, I. (1986). Changing a causal hypothesis without changing the fit: Some rules for generating equivalent path models. Multivariate Behavioral Research, 21, 309–331.
Stephan, W. G., Boniecki, K. A., Ybarra, O., Bettencourt, A., Ervin, K. S., Jackson, L. A., . . . Renfro, C. L. (2002). The role of threats in the racial attitudes of blacks and whites. Person-ality and Social Psychology Bulletin, 28, 1242-1254. doi:
10.1177/01461672022812009
Tibshirani, R. (1996). Regression shrinkage and selection via
the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 58, 267-288.
Venables, W. N., & Ripley, B. D. (2002). Modern applied statistics
with s(Fourth ed.). New York: Springer.
Wickham, H. (2009). ggplot2: Elegant graphics for data analysis. Springer New York.
Wright, S. (1934). The method of path coefficients. The Annals of
Mathematical Statistics, 5, 161–215.
Appendix A
To simulate a full mediation model I need to obtain the implied covariance matrix that I can derive from the param-eters of the model. All λ’s are 0.7 and all β’s are 0.8. These values are chosen so that the simulated model represents one with substantial relationships. The general model is:
Σ(θ) = ΛΣηΛ−1+ Θ (5)
Where:
Ση= (I − B)−1Ψ(I − B)−1t (6)
Together these functions can be rewritten as:
Σ(θ) = Λ(I − B)−1Ψ(I − B)−1tΛ−1 (7) Λ = λ11 0 0 λ21 0 0 λ31 0 0 0 λ42 0 0 λ52 0 0 λ62 0 0 0 λ73 0 0 λ83 0 0 λ93 (8)
The beta matrix where the beta coefficients are 0.8:
B= 0 0 0 β21 0 0 0 β32 0 (9) B-I−1= −1 0 0 −β −1 0 −β2 −β −1 (10)
The identity matrix:
I= 1 0 0 0 1 0 0 0 1 (11)
The Psi matrix which is a matrix with 1’s on the diagonal and zero’s on the off-diagonal:
Ψ = ψ11 0 0 0 ψ22−β 0 0 0 ψ33−β = 1 − β 0 0 0 1 − β 0 0 0 1 − β (12) Now: Σ(θ) = Λ(I − B)−1Ψ(I − B)−1tΛ−1 (13) = λ11 0 0 λ21 0 0 λ31 0 0 0 λ42 0 0 λ52 0 0 λ62 0 0 0λ73 0 0λ83 0 0λ93 −1 0 0 −β−10 −β2β −1 1 − β 0 0 0 1 − β 0 0 0 1 − β −1−β −β2 0 −1−β 0 0 −1 λ11λ21λ31 0 0 0 0 0 0 0 0 0λ42λ52λ620 0 0 0 0 0 0 0 0 λ73λ83λ93 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 0 0 0 0 0 0 0 0 0 1 − λ2 (14)
