Modeling reality : the problem of distinguishing between Equivalent Models in Covariance Structural Modeling

Modeling reality: The Problem of distinguishing

between Equivalent Models in Covariance Structural

Modeling

10362711

Thesis Psychological Methods University of Amsterdam Supervised by Lisa Wijsen

Abstract

This thesis explored the problem of equivalent models in Covariance Structural Modeling (CSM) in current research. Equivalent models are models that produce the same test statistics, however differ substantially in theoretical interpretation. In this thesis the problem of equivalent models is embedded within a larger theoretical context. Furthermore, the articles in the Journal of Applied Psychology (2015) are investigated on the recognition of equivalent models. None of the analyzed articles mentioned the existence of equivalent models for the original model. The existence of equivalent models is considered to be a declination of the truth-value of the original model and therefore a tremendous downfall to the validity of the original proposed models.

Content

Introduction 3

Modeling reality 4

Modeling by means of covariance analysis 6

Equivalent models 8

Method 11

Results 12

Discussion 12

Introduction

Science consists of the interplay between theory and experiment. The theoretical aspect of science is considered to consist of a conceptual aspect and a predictive aspect. The formation of a theory starts with a conceptual idea. In order to make quantitative predictions for the observations, this conceptual idea is then followed by a quantification of the idea under investigation, usually through mathematical formalism. By means of the quantitative predictions the conceptual idea can then be tested with an experiment. The experimental aspect of science consists of the isolation of the phenomena under study in such a way that the interference of other phenomena is considered small or can be corrected for. Scientists expect the evidence they collect through an experiment is direct and therefore the interpretation of the experiment is at least on some level, unambiguous (Moravcisk, 1988). This means they believe there is a one to one relationship between the phenomena that is studied and the data it derives. In this interaction between theory and experiment scientists expect that only one theory will be agreeable with the data. They believe there is only one correct theory, and at the same time they demand that the scientific method functions in such a way that only one theory will triumph (Newton-Smith, 1978).

Regarding the problem of equivalent models in Covariance Structure Modeling (CSM) there is not only one theory that triumphs. A formal definition of equivalent models will follow later, for now it is sufficient to know that equivalent models are models who produce the same test statistics however differ substantially in theoretical content (Markus, 2004; Raykov & Marcoulides, 2007; Tomarken & Waller, 2003). This means the scientific method functions not in such a way that only one theory will triumph, several theories are able to claim victory. Within the use of Covariance Structure Modeling in psychology it is not clear what the consequences of equivalent models are, what the solutions could be, what this entails for the truth-value of the original model or what it implies for the validity of psychological research. This thesis will provide answers to these questions and where there is no answer available raise awareness and provide ground for discussion. Previous research on equivalent models did not embed the problem of equivalent models in CSM in a larger theoretical framework. With this, the full impact of the problem of equivalent models has never been evaluated before. In this thesis the problem of equivalent models in CSM is

embedded within a larger theoretical framework. Next, modeling by means of CSM will be explained to provide basic knowledge in CSM. Following, there will be elaborated on the problem of equivalent models. First, by giving a formal definition and second by investigate how psychological research that makes use of CSM deals with the problem of equivalent models by means of a literature study. The Journal of Applied Psychology from 2015 will be studied on the mentioning of the problem of equivalent models. At last, the consequences of equivalent models for the validity of psychological research using CSM will be discussed.

Modeling Reality

In psychology models are applied to the world we try to figure out. Generating hypothesis about the fit of the specific models to the particular things we try to measure does this. According to Giere (2004) the function of models is twofold. On the one hand models are a true description of the theory and on the other hand models are used as objects to represent a system in the real world. This means the relationship between models and the world is a representational one. This representational relationship is what makes it possible to explore the word with the use of models. They are designed in such a manner that elements of the model can be identified with features of the world. The world is explored by exploiting the similarities between the models and the world it is being used to represent. It is expected that models are able to help us learn what something is like, help us to discover the structures of reality. The ontological status of models is not a real debate within the science of psychology; rather it is a debate that takes place outside of science, in the philosophy of science. Scientists themselves do not seem to take the practice of referring to models as being either true or false to be problematic (Contessa, 2009). Most scientists believe models are able to help us discover truths about the world.

This idea of how models are used entails scientific realism. Scientific realism states that the evidence that counts in favor of the acceptance of a theory is also evidence for the truth of the theory; the theory being a specification of the causal relations between the observed phenomena under study. This means the experimental evidence for a theory is evidence that the causal relations it describes, and no other causal relations, operate in such a way that it produces the regularities in the observed phenomena under investigation (Boyd, 1973).

The realists’ position generally consists of an ontological and an epistemological aspect (Borsboom, Mellenbergh & Heerden, 2004; Newton-Smith, 1978). Ontology includes the existence of phenomena and their causal influences. In other words ontology states which things inhabit reality and how those things relate to each other. The epistemological aspect of scientific realism consists of the possibility to discover the truth-values of scientific theories. The ontological aspect of scientific realism consists of the believe that theories are either true or false and second, the believe that if a theory is true, the theoretical terms which the theory makes indicate the theoretical entities. These theoretical entities are causally responsible for the observable phenomenon whose occurrence is evidence for the truth of the theory. The epistemological aspect of realism states we need to have justified beliefs about the truth-values of the theories; we need to believe we are able to decide whether or not a theory is true or false. In summary, in order to be a realist, you first of all need to believe true theories exist and second, need to believe you are able to decide whether a theory is true or false.

This requires a correspondence theory of truth, which means that a proposition is true or false in virtue of how the world is. Secondly, the believe in the Law of Bivalence, which amounts to the claim that any proposition is either true or false. We cannot abandon a ‘correspondence’ theory of truth without entirely extinguishing the spirit of realism (Newton-Smith, 1978). This view coincides with the goal of scientific research which is to function in such a manner that the data will confirm only one true theory.

If it is the case that for one phenomenon there are two theories, T and T’, which are logical incompatible, and compatible with all the actual and possible observational data, this imposes a threat on scientific realism. This problem is called the

underdetermination of theory by data, also referred to as the realist’s dilemma

(Newton-Smith, 1973; Hoefer & Rosenberg, 1994). When there is such a situation the question which of the theories is true is empirically undividable. The ontological aspect of the realist position leads the scientific realist to hold that there is something in virtue of either T or T’ which makes it true. However, when it is the case that T and T’ both fit the data equally well, this means the epistemological aspect of scientific realism, the possibility to decide whether or not a theory is true or false, does not hold. In this case there is nothing that could count as evidence for the truth or falsity of T or T’. Weakening either the ontological or the epistemological aspect of scientific realism can solve this dilemma. Mostly the epistemological ingredient is weakened (Newton-Smith,

1973). This means we can still insist that all scientific theories can be either true or false, and maintaining that in most cases, we can in principle have grounded beliefs concerning the truth-values of our theories, however recognize that in some cases this does not hold.

The problem of equivalent models in CSM can be considered such a case where both theories fit to the observed data. There is nothing in virtue of the statistical fit that makes it able to choose between one of the compelling models and therefore between one of the compelling theories. To gain knowledge of the problem of equivalent models in CSM, modeling by means of CSM will be explained in the following chapter.

Modeling by means of a Covariance Structure Model

In Covariance Structure Modeling (CSM) we make use of mathematical models to represent some aspect of the world. These models consist of a mathematical object, a set of equations. These mathematical models are considered to be an expression of the processes that give rise to the observed phenomena. CSM can be used to test whether a hypothesized causal structure is considered consistent or inconsistent with the data (Breckler, 1990). This causal structure is a specification of relations between a set of latent variables where each latent variable is represented by one or more measured variables. Several models that differ in their causal structure are then fit to the sample data. Among other criteria, goodness of fit is the most important criteria. Models cannot be considered as an explanation of the data when they do not fit the data well (MacCallum, Wegerner, Uchino & Fabrigar, 1993). The goal in CSM is to minimize the residual covariance matrix by reproducing the observed covariances (Fornell & Bookstein, 1982). This allows researchers to evaluate models that hypothesize directed relationships between latent and measured variables while using correlational data (MacCallum et al., 1993).

Several models for the analysis of a covariance structure have been proposed. The most common form of representation used is the LISREL (Linear Structural Relations) developed by Jöreskog & Sörbom, (1984) (Luijben, 1991). The model consists of two components: the measurement model and the structural model (Lee & Hershberger, 1990). The measurement model defines hypothetical latent variables in terms of observed measured variables. The structural model defines the relations between the latent variables. Also a distinction is made between exogenous –

independent – and endogenous – dependent – variables. Therefore all variables fall into one of these four categories: q exogenous observed variables which are denoted by (𝑥𝑥1, 𝑥𝑥2, 𝑥𝑥3, … 𝑥𝑥𝑞𝑞) , p endogenous observed variables which are denoted by

(𝑦𝑦1, 𝑦𝑦2, 𝑦𝑦3, … 𝑦𝑦𝑝𝑝), n exogenous latent variables which are denoted by (𝜉𝜉1, 𝜉𝜉2, 𝜉𝜉3, … 𝜉𝜉𝑛𝑛) and

m, endogenous latent variables which are denoted by (𝜂𝜂1, 𝜂𝜂2, 𝜂𝜂3, … 𝜂𝜂𝑚𝑚). See figure 1 for

a visualization of an LISREL model containing a measurement and a structural component.

Figure 1. Example of a LISREL Model

To represent the measurement component of the model two matrix equations are specified:

Χ = Λ𝑥𝑥𝜉𝜉 + 𝛿𝛿, (1)

and

Υ = Λ𝛾𝛾 𝜂𝜂 + 𝜖𝜖 (2)

Where equation 1 specifies the relations for exogenous observed variables and equation 2 specifies the relations for endogenous observed variables. In equation 1 X is a q x 1 vector of exogenous observed variables; 𝚲𝚲𝒙𝒙 is a q x n matrix of factor loadings

exogenous observed variables; 𝝃𝝃 is a n x 1 vector of exogenous latent variables; and 𝜹𝜹 is a q x 1 error terms vector for the exogenous observed variables. In equation 2 Y is a p x

n matrix of endogenous observed variables; 𝜦𝜦𝜸𝜸 is a p x m matrix of factor loadings

(coefficients) that specifies the influence of the endogenous latent variables on the endogenous observed variables; 𝜼𝜼 is a m x 1 vector of endogenous latent variables; and 𝝐𝝐 is een p x 1 error terms vector of the endogenous observed variables. For the structural component of the model the matrix equation is as follows:

𝜂𝜂 = Β𝜂𝜂+ Γ𝜉𝜉 + 𝜁𝜁 (3)

In equation 3, B is a m x m matrix of coefficients that specifies the influence of endogenous latent variables on endogenous latent variables; 𝜞𝜞 is a m x n matrix of coefficients that specifies the influence of exogenous latent variables on endogenous latent variables; and 𝜻𝜻 is a m x 1 vector of errors in prediction for the endogenous latent variables equations. LISREL specifies four more covariance matrices: 𝜽𝜽𝜹𝜹 is a q x q matrix

of covariances between the errors in the measurement for equation 1, 𝜽𝜽∈ is a p x p

matrix of covariances between the errors in the measurement for equation 2, 𝜱𝜱 is a n x n matrix of covariances between the exogenous latent variables and 𝝍𝝍 is a m x m matrix of covariances among the errors in the predictions for the endogenous latent variable equations.

The population matrix 𝜮𝜮(𝑴𝑴) is defined as a function of the parameter matrices 𝜮𝜮(𝑴𝑴) = 𝜦𝜦(𝑰𝑰 − 𝜝𝜝)−𝟏𝟏_{𝜳𝜳(𝑰𝑰 − 𝜝𝜝′)}−𝟏𝟏_𝜦𝜦′_{+ 𝜱𝜱 (4)}

A theoretical model is represented by specifying a pattern of elements in each of the eight-parameter matrices. Goodness of fit can be determined in many ways. Although not the only method, Maximum Likelihood Estimation is most frequently used (Breckler, 1990). The matrix of observed covariances 𝜮𝜮(𝑺𝑺) is then used to estimate the values of the free parameters that reproduce the data the best

Equivalent models

Given a sample covariance matrix 𝜮𝜮(𝑺𝑺), and two models M and M’, the models M and M’ can be fit to 𝜮𝜮(𝑺𝑺). The parameter estimates acquired for model M and M’ produce an model implied covariance matrix, respectively 𝜮𝜮(𝑴𝑴) and 𝜮𝜮(𝑴𝑴′). The goodness of fit of

the models M and M’ is a function of the closeness of 𝜮𝜮(𝑴𝑴) and 𝜮𝜮(𝑴𝑴′) to 𝜮𝜮(𝑺𝑺). The models M and M’ are equivalent when 𝚺𝚺(𝑴𝑴) = 𝚺𝚺(𝑴𝑴′) for any 𝜮𝜮(𝑺𝑺). When this is the case, the goodness of fit of both models will be the same. (MacCallum et al., 1993; Luijben, 1983;).

Especially around 1990 the problem of equivalent models in the applications of Covariance Structure Analysis gained lots of attention (MacCullum et al., 1993; Breckler, 1990; Lee & Hershberger, 1990; Mayekawa 1994; Luijben 1991; Stelzl, 1986; Williams, Bozdogan & Aiman-Smith, 1996) .MacCullum, Wegener, Uchino and Fabrigar, (1993) showed by conducting a literature study how the problem of equivalent models is addressed in psychological research. Their research entailed three prominent journals in psychology on the subjects of educational psychology, industrial-organizational psychology and social and personality psychology, between 1988 and 1991. From the total of 99 applications they found no cases in which the authors would explicitly acknowledge the existence of an equivalent model. MacCullum et al. (1993) offer some explanations for this. First of all they consider the possibility of equivalent models being uncommon in the literature. Second, they suspect that researchers are either unaware of the existence of equivalent models or they choose to ignore the issue of model equivalency. Breckler (1990) reviewed 72 articles form which only one article acknowledged the existence of an equivalent model. In they majority of cases plausible alternative equivalent models could easily formulated.

Lee & Hershberger (1990) came up with an simple rule for generating equivalent models in CSM, the replacing rule. In a relationship like the following: X  Y, is X the source variable and Y the effect variable. 𝑼𝑼𝒙𝒙 and 𝑼𝑼𝒚𝒚 represent the residuals of X and Y.

Lee & Hersherber (1990) define their replacing rule as: “A direct path, that is, X  Y, can

be replaced by a residual correlation between them, the correlation between 𝑼𝑼𝒙𝒙 an𝑑𝑑 𝑼𝑼𝒚𝒚,

as long as the predictor variables of the effect variable Y are the same as or include those of the source variable X”.1 _{This means a covariance path between 𝑼𝑼𝒙𝒙 and 𝑼𝑼𝒚𝒚. can} replace the path between X and Y. The reverse application of this rule will also create an equivalent model. A direct path between X and Y can replace the residual covariance between 𝑼𝑼𝒙𝒙and 𝑼𝑼𝒚𝒚. Figure 2 shows an illustration of the replacing rule.

1 _{Lee, S., & Hershberger, S. (1990). A simple rule for generating equivalent models in covariance structure}

Model 1 𝜉𝜉1 𝜂𝜂1 𝜂𝜂2 Model 2 𝜂𝜂1 𝜉𝜉1 𝜂𝜂2

Figure 2. Illustration of the replacing rule to construct equivalent models, where model 1

and 2 are equivalent models

Theproblem of equivalent models threatens the validity of conclusions drawn in support of the original model. There may be alternative models that fit the data equally well as the original model, which are not being considered in research. These different models will vary in meaning, but they will all produce the same covariance structure. Therefore they cannot be distinguished in terms of their fit to the data, only in terms of their substantive meaning. The phenomenon of equivalent models cannot be overlooked as an occasional problem. For every model that is being studied large numbers of alternative models can be constructed that are equivalent to the original model. Namely, the existence of equivalent models is inherent in the formal expression of the original model, not the design of the model (MacCullem et al., 1993). This entails it is not the case researchers designed a ‘bad’ model. It is however the case that no matter how ‘good’ the

design of the model is, there can always be generated an equivalent model because it is inherent in the formal expression.

Equivalent models can be seen as a case of underdetermination of theory by data. Underdetermination of theory by data entails we will never be able to choose between two theories upon observational data. This is exactly the case regarding the problem of equivalent models. Equivalent models are equivalent in the case of any observational data. For the definition states that two models M and M’ are equivalent when their covariance matrixes, 𝚺𝚺(𝑴𝑴) and 𝚺𝚺(𝑴𝑴′), are equal for any observation covariance matrix 𝜮𝜮(𝑺𝑺). This means the existing of equivalent models in CSM imposes a threat on scientific realism within psychology.

Method

Sampling

To investigating the phenomenon of equivalent models in practice, I choose to catalogue the applications of CSM in a prominent journal on work and organizational phenomena: the Journal of Applied Psychology. This Journal publishes high quality research concerning the application of psychology to better understand work and organizational phenomena. The journal is one of the leading and most influential journals in the fields of organizational psychology (Editorial, Journal of Applied Psychologie, 2015). Among other types of analyses, this area of psychology uses CSM to analyze the relationship between latent and observed variables. To investigate the Journal of Applied Psychology I made use of the search engines APA PsycNet and Ovid. I reviewed all applications of CSM in this journal from 2015. I categorized all the CSM articles according to the type of application (CSM, SEM, PA, CFA). I examined each article to determine, whether in addition to the original model, researchers tested alternative models. I also examined articles to determine whether or not an equivalent model was explicitly acknowledged.

Exclusion criteria

Because CFA is not used to evaluate any particular structural relationship among latent variables, the existence of equivalent models of relationships among latent variables does not threaten the inferences the researchers make on the basis of their analysis (MacCullum at all. 1993). As a result, my analysis of model equivalence did not include applications of CFA.

Results

Literature study

Of the 119 articles reviewed in the Journal of Applied Psychology (2015), 41 articles used applications of CSM to analyze their data. There where 8 articles using SEM, 27 articles using CFA, 1 article using CSM and 5 articles using PA. Zero articles, which used applications of CSM, acknowledged the existence of an equivalent model. However in the majority of instances, plausible alternative equivalent models could easily be generated.

Discussion

Regarding the problem of equivalent models in CSM it should now be clear it is to be considered a serious problem for empirical research, as it imposes a threat on the validity of the original proposed model. The failure to recognize the existence of equivalent models can lead to misleading conclusions. If for the original proposed model many equally well fitting models can be generated, only they are not being considered, the truth-value of the original model declines. Therefore it is important in any application of CSM to acknowledge the problem of equivalent models by generating such models and evaluate their meaningfulness. Rules by Lee & Hershberger (1990) can be used for such purposes. This thesis demonstrated the easiness in their applications. For now there is still no program that can generate equivalent models in CSM. Therefore the generating of equivalent models could be a problem for models for which many equivalent models can be generated. However, it is not scientifically defensible to dismiss the problem by relying on the a priori status of the original model.

The existence of equivalent models should be recognized within the applications of covariance structure. This thesis showed still little attention goes out to the existence of equivalent models in current research. The current state of affairs in psychological research can be brought forward as an explanation for this. The pressure to publish, and the bias from journals towards producing well fitting models, leaves no room for a careful and time-consuming consideration of equivalent models.

This thesis also made a comparison between the problem of equivalent models in CSM and the threat of underdetermination of theory by data for scientific realism. With the existence of equivalent models, if we want to hold on to scientific realism, we cannot do otherwise then weakening either the ontological or the epistemological aspect. Weakening the epistemological aspect is the most logical choice. This means all scientific

theories can still be either true or false, however not in every case we have the means to discover the truth or falsity of a theory.

Equivalent models can be distinguished in terms of their substantive meaning. However, what is considered meaningful can vary per scientists for there are no strict rules for considering meaningfulness. Another way of distinguishing between equivalent models is by examining each of the causal relations within the model. This however takes the purpose of CSM away which is to fit causal relations with correlation data. By no means can I provide solutions to the problem of equivalent models in applications of covariance structure analysis, what I can do however, is raise awareness and start a discussion. Hopefully my thesis did just that.

References

Bentler, P. M., & Satorra, A. (2010). Testing model nesting and equivalence. Psychological

Methods, 15(2), 111.

Boyd, R. N. (1973). Realism, Underdeterination, and a Causal Theory of Eviende. Nous 7(1), 1 -12.

Borsboom, D., Mellenbergh, G. J., & Van Heerden, J. (2003). The theoretical status of latent variables. Psychological review, 110(2), 203.

Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological review, 111(4), 1061.

Breckler, S. J. (1990). Applications of covariance structure modeling in psychology: Cause for concern? Psychological bulletin, 107(2), 260-273.

Contessa, G. (2010). Scientific models and fictional objects. Synthese, 172(2), 215-229. Giere, R. N. (2004). How models are used to represent reality. Philosophy of science,

71(5), 742-752.

Hoefer, C., & Rosenberg, A. (1994). Empirical equivalence, underdetermination, and systems of the world. Philosophy of science, 592-607.

Jöreskog, K. G., & Sörbom, D. (1988). LISREL 7: A guide to the program and its

applications. Chicago:SPSS.

Lee, S., & Hershberger, S. (1990). A simple rule for generating equivalent models in covariance structure modeling. Multivariate Behavioral Research, 25(3), 313-334. Luijben, T. C. (1991). Equivalent models in covariance structure analysis. Psychometrika,

56(4), 653-665.

MacCallum, R. C., Wegener, D. T., Uchino, B. N., & Fabrigar, L. R. (1993). The problem of equivalent models in applications of covariance structure analysis. Psychological bulletin, 114(1), 185.

Markus, K. A. (2002). Statistical equivalence, semantic equivalence, eliminative induction, and the Raykov-Marcoulides proof of infinite equivalence. Structural

Equation Modeling, 9(4), 503-522.

Mayekawa, S. I. (1994). Equivalent path models in linear structural equation models.

Behaviormetrika, 21(1), 79-96.

Moravcsik, M. J. (1988). The limits of science and the scientific method. Research Policy, 17(5), 293-299.

Newton-Smith, W. (1978). The underdetermination of theory by data. In Rationality in science, 91-110.

Raykov, T., & Marcoulides, G. A. (2007). Equivalent structural equation models: A challenge and responsibility. Structural Equation Modeling, 14(4), 695-700.

Raykov, T., & Penev, S. (1999). On structural equation model equivalence. Multivariate

behavioral research, 34(2), 199-244.

Stelzl, I. (1986). Changing a causal hypothesis without changing the fit: Some rules for generating equivalent path models. Multivariate Behavioral Research, 21(3), 309-331.

Tomarken, A. J., & Waller, N. G. (2003). Potential problems with" well fitting" models. Journal of abnormal psychology, 112(4), 578.

Williams, L. J., Bozdogan, H., & Aiman-Smith, L. (1996). Inference problems with equivalent models. Chapter 10. Advanced structural equation modeling: Issues and

techniques, 279-314.

Yves Rosseel (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48(2), 1-36.