The ment as a performance validity test for ASD : detecting feigning in autism spectrum disorders (ASD): fable or fact? : the value of the Morel Emotional Numbing Test in assessing symptom invalidity within ASD

(1)

Detecting feigning in autism spectrum disorders (ASD): fable or fact? The value of the Morel Emotional Numbing Test in assessing symptom invalidity within ASD.

Linda Olde Dubbelink University of Amsterdam

Field of study: Clinical Psychology Student number: 6030726

Supervisors: prof. dr. H. M. Geurts Anne Geeke Lever MSc Second assessor: prof. dr. B. A. Schmand

(2)

THE MENT AS A PERFORMANCE VALIDITY TEST FOR ASD 2

Abstract

Clients seeking treatment for psychological symptoms can report exaggerated or invalid symptoms for beneficial reasons, and thus form a threat for accurate diagnoses and useful treatment allocation. Therefore, symptom invalidity is detected through performance validity tests (PVTs). Although applied for most diagnostic assessments, no PVT exists for autism spectrum disorders (ASD). The current study therefore investigated whether the Morel Emotional Numbing Test (MENT), a PVT which assesses facial emotion recognition, is insensitive to ASD by comparing the test performance of 19 adults with ASD with 19 age-, gender- and IQ-matched controls (27-54 years). Emotion recognition deficiency is however a debatable feature of ASD and could jeopardize the applicability of the MENT in this

population. To overcome the inconsistent findings and shortcomings of earlier tests,

performance on a dynamic emotion recognition test was compared between the two groups. No group differences were found on both tasks and the data support moderate evidence in favor of these null-findings in comparison to the alternative. It can be concluded that an emotion recognition deficiency is not characteristic for adults with ASD and the MENT seems to be insensitive to ASD and therefore could be a useful PVT for future assessment of ASD.

(3)

In clinical psychology, self-reported symptoms and performance on disorder-related measures are used to diagnose a disorder. This practice assumes that the client discloses true impairment. However, this assumption could be violated if external and internal benefits exist when receiving a diagnosis, such as entrance to treatment and assurance of alimony. Clients might then feign or exaggerate their symptoms in order to be diagnosed. Feigning symptoms has been found in several psychological disorders and is often referred to as ‘malingering’ (Diagnostic and Statistical Manual of Mental Disorders – Fourth Edition, Text Revision [DSM-IV-TR], American Psychiatric Association [APA], 2000; DeViva & Bloem, 2003; Lees-Haley & Dunn, 1994).

To prevent invalid diagnoses and unnecessary allocation of clinical resources, feigning clients need to be detected. However, they are hard to identify at face value. For example, psychologists perform below chance in detecting simulating clients (Faust, Hart, & Guilmette, 1988; Faust, Hart, Guilmette, & Arkes, 1988). Therefore, researchers have tried to detect symptom invalidity through objective measures, so-called performance validity tests (PVTs; Pankratz, 1983; Schmand & Ponds, 2004). A PVT is a face-valid measure of the disorder which lures a feigning person to perform beneath his true level of performance. More importantly, it is insensitive to the disorder of interest (Hartman, 2002). Thus, a person diagnosed with the disorder should be detected as honest by the PVT. This test can, therefore, provide evidence of feigned symptoms and is essential to enhance confidence in psychological test performance and self-reported symptoms (Bush et al., 2005; Green, 2007). PVTs are reliable measures to detect symptom invalidity in several psychological disorders including posttraumatic stress disorder (PTSD), psychotic disorders, anxiety disorders, and

(4)

attention-THE MENT AS A PERFORMANCE VALIDITY TEST FOR ASD 4

deficit/hyperactivity disorder (Geraerts et al., 2009; Merten, Thies, Schneider, & Stevens, 2009; Suhr, Sullivan, & Rodriguez, 2011).

Although PVTs have been tested and used in several clinical populations, research with regard to symptom invalidity does not exist within autism spectrum disorders (ASD). This absence seems logical as individuals who suffer from this pervasive developmental disorder are assumed to be unable and unwilling to feign their symptoms (Frith & Happé, 1999; Green, Gilchrist, Burton, & Cox, 2000). However, a person could benefit from a diagnosis of ASD. For example, it could lead to sympathy from people who might otherwise judge the person of being odd. Also, the current mental health organization requires a

diagnosis before starting treatment (Janssen & Soeters, 2010). An ASD diagnosis thus opens doors to special needs such as educational help and therapeutic intervention, which makes exaggeration or feigning of ASD symptoms beneficial. Therefore, symptom validity measures need to be investigated within the autism spectrum.

To detect feigned symptoms within ASD, several PVTs exist. The two most often used are the Test of Memory Malingering (TOMM; Tombaugh, 1996) and the Word Memory Test (WMT; Green, Allen, & Astner, 1996). On these two tests, exaggeration of memory

impairment is assessed through a memory recognition test of either images or word pairs. Both PVTs are validated in neuropsychiatric samples (Green, Rohling, Lees-Haley, & Allen, 2001; Tombaugh, 1997). Although these measures are valid, they are less useful to detect feigned symptoms of ASD because ASD is not characterized by memory impairment. In fact, people with ASD are often excellent in recognizing objects on the short- and the long term (Bennetto, Pennington, & Rogers, 1996). Hence, it is unlikely that a person that feigns ASD will exaggerate memory impairment.

(5)

A more promising PVT within the assessment of feigned ASD symptoms is the Morel Emotional Numbing Test (MENT; Morel, 1998). It assesses suboptimal performance in a measure of facial affect recognition. The MENT is not yet investigated within ASD, but has proven a valid PVT in clients with claimed PTSD (Morel, 2008; Morel & Shepherd, 2008). The MENT consists of 3 trials of 20 items in which facial expressions are matched to emotion words. The expressions reflect happiness, frustration, sadness, anger, fear, calmness, surprise, shyness, confusion, and sleepiness. As a facial affect recognition test, the MENT seems a face valid measure of ASD. A face conveys information about the emotions and intentions of the other person, which people with ASD find hard to respond to and understand. Thus, it seems to relate to the empirically established social deficit within ASD (Diagnostic and Statistical Manual of Mental Disorders – Fifth Edition [DSM-5], APA, 2013; Tager-Flusberg, 2007).

In need to detect symptom invalidity within ASD and the knowledge that a functional PVT needs to be insensitive to the disorder of interest, the current study investigated whether the MENT is indeed insensitive to a diagnosis of ASD. To this end, adults with a diagnosis of ASD completed the MENT and their performance was compared to the performance of age- and gender matched adults without ASD. Based on the belief that people who are already diagnosed with ASD will not feign symptoms because they have no secondary gain to obtain within the current study and are assumed to be poor at predicting and anticipating the

expectations of another person (Frith & Happé, 1999), we assumed no feigning of symptoms within the sample. Furthermore, the MENT assesses emotion recognition in a two-alternative forced choice format with particularly opposing emotions (e.g. picture 1 represents either ‘sad’ or ‘happy’) and is, therefore, a simple task. It was hypothesized that the MENT is

insensitive to a diagnosis of ASD. Therefore, the ASD was expected to perform as well on the MENT as the control group.

(6)

If insensitivity is not confirmed, a possible explanation may be that adults with ASD are poor at recognizing emotions compared to adults without ASD. Based on the social deficit within ASD, studies have investigated the presence of an inability to recognize facial expressions. However, this research, although exhaustive, is conflicting: some studies support a worse

performance in recognizing facial expressions compared to controls (Celani, Battacchi, & Arcidiacono ,1999; Gepner, de Gelder, & de Schonen, 1996), whereas others show intact facial affect recognition (Baron-Cohen, Wheelwright & Jolliffe, 1997; Castelli, 2005; Grossman, 2000; Kätsyri, Saalasti, Tiipana, von Wendt, & Sams, 2008). One reason for the lack of consensus may be that the impairment in recognizing emotional expressions is more subtle than what could be

uncovered with the static measures of emotion recognition that use intense expressions. In addition, the majority of previous studies lacked a comparison group or appropriate matching on age, gender and intelligence. More importantly, the previous used tests are an indirect measure of emotion recognition in people’s everyday life. These tests use static photographs of facial expressions that are matched without time constraint, whereas in everyday communication these expressions are dynamic and require a quick identification. To overcome this shortcoming, the Emotion

Recognition Task was developed (ERT; Kessels, Montagne, Hendriks, Perret, & de Haan, 2013) in which dynamic facial expressions of varying intensities are presented and matched to the six basic emotions (sadness, happiness, anger, fear, surprise, and disgust). The current study investigated emotion recognition in adults with ASD by using this more ecological valid measure. To this end, the two groups completed the ERT and their performance was compared. As the current task varies in intensity and requires more rapid processing of emotions in comparison to previous tests, it was hypothesized that adults with ASD would show a deficit in emotion recognition in comparison to adults without ASD.

(7)

Method

Participants

Twenty-six adults with ASD participated in the study and were individually matched on age and gender to 26 adults without ASD (the control participants). Estimated IQ was also

matched on group level. Matching was executed to keep possible effects of IQ, age and gender constant across groups so differences in performance between groups could be ascribed to differences in emotion recognition. All participants with ASD were diagnosed by a

multidisciplinary team including a psychiatrist or psychologist according to the DSM-IV-TR (APA, 2000) or ICD-10 (World Health Organization [WHO], 1992): 3 were diagnosed with autism, 15 with Asperger syndrome and 8 with Pervasive Developmental Disorder-Not Otherwise specified (PDD-NOS). They were included if they scored above the cut-off of 26 for ASD on the Autism Spectrum Quotient (AQ; Baron-Cohen, Wheelwright, Skinner, Martin, & Clubley, 2001; Hoekstra, Bartels, Cath, & Boomsma, 2008; Woodbury- Smith, Robinson, Wheelwright, & Baron-Cohen, 2005). Six participants did not meet the cut-off and were, therefore, excluded from the analysis, along with their matched controls.

Control participants were recruited via the experimenter’s social surrounding. To correct for ASD within the control group, participants were excluded if they had relatives with ASD and scored above the recommended off for ASD of 32 on the AQ (recommended cut-off for general sample by Woodbury-Smith et al., 2005). All control participants met these inclusion criteria.

Further inclusion criteria for both groups were an age between 25 and 60, an estimated IQ above 80 and no current mood or psychotic disorder. An IQ above 80 was chosen because mental retardation is often comorbid to ASD and will confound interpretation of results on tests that require above average intelligence (Matson & Shoemaker, 2009). The exclusion of

(8)

current mood or psychotic disorder was chosen because a current mood or psychotic disorder affects emotion recognition and, therefore, could influence performance on MENT and ERT. One participant in the ASD group had a current depressive episode and was excluded along with the matched control participant. Nineteen participants with ASD and nineteen

individually matched control participants met the inclusion criteria and were included (10 males and 9 females; for details see Table 1). Eleven of ASD participants and none of the control participants used psychotropic medication on a permanent basis.

Materials

Autism Spectrum Quotient

The Dutch version of the AQ was administered to assess the presence of ASD traits (Baron-Cohen et al., 2001; Hoekstra et al., 2008). On this test, 50 statements about ASD traits are rated with ‘definitely agree’ (1), slightly agree (1), slightly disagree (0) and definitely disagree (0 points). An example item is: “I tend to notice details that others do not”. The total score on the AQ was used as the outcome measure. In a general population sample, a cut-off of 32 is recommended to identify individuals with ‘autistic traits’ (Woodbury-Smith et al., 2005). For a clinically referred sample, this cut-off of is 26 (Woodbury-Smith et al., 2005). The Dutch version of the AQ has good internal consistency (α=.81), good test-retest reliability (α=.78), and good discriminative validity (Hoekstra et al., 2008; Woodbury-Smith et al., 2005).

Intelligence

The two subtest short form of the Dutch version of the Wechsler Adult Intelligence Scale-Third Version (Ringe, Saine, Lacritz, Hynan, & Munro-Cullum, 2002; Uterwyk, 2000; WAIS-III, 1997; Wechsler, 1997) was administered to estimate intelligence. This form contains the subtests Vocabulary and Matrix Reasoning and provides a general IQ with a

(9)

mean of 100 and a standard deviation of 15. This short form is a valid and reliable estimate of general IQ (Ringe et al., 2002).

Current (comorbid) psychological disorders

The Dutch version of the Mini International Neuropsychiatric Interview (M.I.N.I; Van Vliet, Leroy, & van Megen, 2000) was administered to assess current (comorbid) psychological disorders according to the DSM-IV-TR (APA, 2000). First, screening questions were

administered for each disorder. If any of these were answered with a ‘yes’, the accompanying section was assessed to evaluate the presence of a disorder. The M.I.N.I. has excellent inter-rater reliability, good retest reliability, and is a valid measure of psychological disorders (for details, see Sheehan et al., 1998). In the current study, the M.I.N.I was used to exclude individuals with a current mood or psychotic disorder.

Symptom validity

The Morel Emotional Numbing Test (MENT; Morel, 1998) is a pen-and-paper based

performance validity test which assesses facial affect recognition of ten emotions (happiness, frustration, sadness, anger, fear, calmness, surprise, shyness, confusion, and sleepiness). These expressions were derived by Morel (1998) from two university dramatic arts students (one African American male and one White male) and validated within a Veterans Affair Medical Center (VAMC) by military veterans (for details see Morel, 1998). The test booklet contains 3 trials of 20 items in which the participant matches the expressions to emotion words in a two- alternative forced-choice format. Each trial requires a different matching procedure. The duration of the task is self-paced. The outcome measure was the total number of errors, with a maximum total error score of 60. Earlier research within PTSD samples shows that the MENT has high specificity (it can indicate honest participants at a 100% rate),

(10)

Table 1

Means (and standard deviations) of demographic and clinical scores ASD and CP

Measure Group Group comparison

ASD (n=19) Range CP (n=19) Range t value (36) 95% CI p Cohen’s d Gender(males/females) Age (years) TIQ AQ 10/9 49.1 (6.9) 129.2(16.2) 33.8 (4.3) 27-56 101-155 27-42 10/9 49.0 (6.8) 120.8 (12.9) 12.3 (5.5) 27-56 100-145 5-25 0.05 1.77 13.36 -4.41 to 4.63 -1.25 to 18.08 18.21 to 24.73 .96 0.02 .09 0.57 <.01 4.34 Note. ASD = autism spectrum disorder, CP = control participants. The ASD group consisted of adults with a clinical diagnosis of

autism (n=1), Asperger syndrome (n=13) and PDD-NOS (n = 5). TIQ = Total estimated Intelligence Quotient. AQ = Autism Spectrum Quotient. MINI = Mini International Neuropsychiatric Interview. Please note that various participants (ASD group n = 7; control group n = 2) used medication to treat a broad range of medical conditions (e.g. diabetes). None of these drugs affect performance on the tasks in the current study.

(11)

high sensitivity (it can identify suspected stimulants at an 82% rate) and high internal consistency (range from .94 to .87; Morel, 1998).

Emotion recognition

To assess emotion recognition, the Emotion Recognition Task (ERT; Kessels et al., 2013) was administered. On this test, video clips of facial emotional expressions from neutral to four intensities (0-40%, 0-60%, 0-80%, and 0-100%) are presented of the six basic emotions (happiness, sadness, anger, fear, surprise, and disgust; Ekman, 1992). The expressions were developed by the Perrett Lab (for details see Frigerio, Burt, Montagne, Murray, & Perrett, 2002). The duration of the video-clips ranged from approximately 1 (40% intensity) to 3 seconds (100% intensity). Participants were instructed to match the displayed emotion words to facial expressions with no time constraint. All six emotion labels were presented at the left side of the screen. Responses were made by a mouse-click. A practice block was administered to ensure the task was understood. The 4 experimental blocks consisted of 24 trials each and displayed neutral to more intense dynamic facial expressions. The outcome measure was the total number of correctly identified facial expressions. The ERT is a valid and reliable measure of emotion recognition for both clinical and non-clinical samples (for example, see Kessels et al., 2013; Montagne et al., 2008).

Study purposes questionnaire

To evaluate whether the participants believed the study was about emotion recognition and were blind to the assessment of performance validity, a two-item-questionnaire was

administered about what participants thought was the purpose of the study and how they would rate their emotion recognition ability. The experimenter asked the participants the following question: ‘What was, according to you, the purpose of this study?’ The participants’ answers were written down. Participants then rated their ability to recognize emotions on a

(12)

100 millimeter visual analogue scale from ‘0’ (‘I am unable to recognize emotions’) to ‘100’ (‘I am very good at recognizing emotions’). The outcome measure was the distance in millimeters on the scale.

Procedure

Participants with ASD were recruited trough the study ´Autism and aging: A double

jeopardy?´ conducted at the Brain and Cognition Department of the University of Amsterdam. Within that study, a number of people with ASD did not participate but did sign informed consent to contact them for future research. These people were asked to participate in the current study. The control participants were recruited based on matching age, gender and possible intelligence, via the experimenter’s social surrounding. All participants were informed that the study was about emotion recognition. Once informed consent was obtained,

participants were tested. In a two-hour session the examiner administered all above mentioned materials. In addition, the Dutch adult short version of the faux pas test was administered to convince the participant that the sole purpose of the study was to evaluate emotion recognition (for details, see Spek, Scholte, & Van Berckelaer-Onnes, 2010; Stone, Baron-Cohen, &

Knight, 1998). Performance on this test was not analyzed because it is irrelevant to the current research questions. After four tests, a short break was inserted. Other questionnaires were administered which are not of interest within the current study. The task order was

counterbalanced across participants.

Upon study completion, all participants received full debriefing about the study. This debriefing explained the two-folded purpose of the current study and elaborated on the performance validity test. It was stressed that we did not question the participants’ honesty throughout the test battery and thus assumed that they would not feign their performance in the absence of secondary gain. The experimenter explained that the current study was the very first

(13)

investigation towards a possible performance validity test. It was stressed that such a test is important to enhance credibility and validity of a diagnosis of ASD in future diagnostic evaluations. After debriefing, participants were asked if they understood our purposes. Any residual concerns or questions were discussed. Finally, participants with ASD received a reimbursement of travel expenses (maximum of €20) and control participants were compensated with €15 for their participation.

Statistical analysis

To determine whether there were group differences in MENT performance and emotion

recognition, two independent sample t-tests were performed. These t-tests evaluate whether the data supports a rejection of the null-hypothesis or not (there is no difference between the two groups based on the data). However, these classic hypothesis tests cannot quantify evidence in favor of the proposed (null-) hypotheses. The procedure of Bayesian hypothesis testing does allow this quantification (Jeffreys, 1961, as cited in Rouder, Speckman, Sun, Morev, & Iverson, 2009). This statistical procedure calculates the extent to which the data support hypothesis A (e.g. null-hypothesis) over hypothesis B (e.g. alternative hypothesis), expressed by a Bayes factor (Rouder et al., 2009). For example, when the Bayes factor is 3 the data are 3 times more likely to have occurred under the null-hypothesis than under the alternative

hypothesis. In terms of evidential strength, a Bayes factor of 1 indicates no evidence, between 1 and 3 anecdotal evidence, between 3 and 10 moderate evidence, between 10 and 30 strong evidence, between 30 and 100 very strong evidence and above 100 extreme evidence for the null-hypothesis (recommended categories by Jeffreys, 1961 as cited in Rouder et al., 2009). Several procedures exist to obtain the Bayes factor. In the current two-sample case, the scaled Jeffrey-Zelner-Siow Prior Bayes factor (JZS Bayes factor) is recommended (Rouder et al.,

(14)

2009) and therefore was used for interpretation. For the Bayesian hypothesis tests, the web-based Bayesian procedure of Rouder (2009) was run.

Explorative, answers on the study purposes questionnaire were evaluated qualitatively to show that participants were blind to performance validity testing. To assess whether people with ASD rated themselves less able to recognize emotions in comparison to people without ASD, an independent t-test with rated emotion recognition ability as the dependent variable and group (ASD versus control) as the independent variable was performed. Assumptions were met for this analysis.

Assumptions and outliers

The MENT performance was not normally distributed across groups. Within the ASD group specifically, the MENT performance was positively skewed. Therefore, performance on the MENT was transformed using log transformation (as recommended by Field, 2009, p. 153-161). After transformation, all assumptions were met to execute the independent t-test on MENT performance (Field, 2009, p. 326). The ERT performance was also positively skewed within the ASD group. However, this variable was normally distributed across groups. All assumptions were met to execute the independent t-test on ERT performance. No significant outliers were detected within the transformed MENT data and ERT data using the Outlier Labeling Rule (Hoaglin & Iglewicz, 1987). With this method, the bounds of the lower and upper quartile of the data are estimated and data points that exceed these bounds are identified as outliers.

(15)

Results

Matching

Individual matching was successful. No group differences existed in age, gender and estimated IQ. In accordance with our expectations, the ASD group had significantly higher AQ scores than the control group (see Table 1).

Confirmative analyses

MENT

As expected, participants with ASD (M = 1.84, SE = .29) did not perform significantly worse on the MENT than their matched controls (M = 2.22, SE = .40), t (36) = -0.58, p = .57, Cohen’s d = 0.25 (Cohen, 1992). The Bayesian independent sample t-test yielded a scaled JZS Bayes factor of 3.63. The observed data are therefore 3.6 times more likely to have occurred under the null-hypothesis than under the alternative hypothesis. Given the data, there is moderate evidence in favor of the null hypothesis in comparison to the alternative

hypothesis.

ERT

Contrary to our expectations, participants with ASD (M = 59.84, SE = 1.95) did not perform significantly worse on the ERT in comparison to their matched controls (M = 60.26, SE = 2.01), t (36) = -0.15, p = .88, Cohen’s d = 0.05. The Bayesian independent t-test indicated a scaled JZS Bayes factor of 4.17. The observed data are therefore 4.2 times more likely to have occurred under the null-hypothesis than under the alternative hypothesis. Given the data, there is moderate evidence in favor of the null hypothesis in comparison to the alternative

(16)

Explorative analyses

Purpose of the study questionnaire

None of the participants indicated that validity of their performance was evaluated during assessment. All described that the purpose of the study was to investigate emotion recognition in individuals with ASD and controls. Eight participants (five ASD and three control

participants) correctly indicated that intelligence was taken into account in this investigation. Two ASD participants stated that the results of our study would be implemented in clinical interventions for ASD and one ASD participant thought that the purpose of the study was to exclude that poor emotion recognition is a key characteristic of ASD.

The ASD group (M = 58.68, SE = 5.04) rated themselves significantly lower in ability to recognize emotions in comparison to the control group (M = 78.16, SE = 2.30), t (25.19) = -3.51, p < .01, Cohen’s d = 1.14.

Discussion

In the current study we tested whether the MENT, a facial affect recognition test, could be a performance validity test (PVT) within the assessment of ASD. To this end, performance of adults with ASD and adults without ASD was compared on the MENT and an emotion recognition task (ERT). According to Hartman’s criteria for a PVT (Hartman, 2002), such a test is insensitive to the disorder of interest, is a face-valid measure of this disorder, and lures a feigning person to perform beneath his true level of ability. Based on our findings, there are three lines of evidence that suggest that the MENT is insensitive to a diagnosis of ASD and is a face-valid measure of this disorder.

First, performance on the MENT did not differ between the two groups and all participants performed well on this emotion recognition test. In line with Hartman’s criteria

(17)

and our hypothesis, we can conclude that the MENT is insensitive to a clinical diagnosis of ASD.

Second, in contrast to our expectations, performance on the ERT did not differ between the two groups and all participants were able to recognize the six basic emotions (sadness, happiness, anger, fear, surprise, and disgust). This is in line with previous studies that show intact emotion recognition in ASD (Baron-Cohen et al., 1997, Kätsyri et al., 2008) but does not replicate findings that support an emotion recognition deficit (Celani et al., 1999; Lindner & Rose, 2006; Pelphrey et al., 2002). If we compare the studies that indicate a deficit with the current study, we see that tasks that do find a deficit use a time-bound opportunity to respond whereas tasks that are self-paced, such as the ERT, do not. It seems that individuals with ASD have a problem in processing emotional stimuli quickly rather than in recognizing emotions. This is in line with established difficulties in processing emotional stimuli within earlier research (e.g. Gepner, Deruelle, & Grynfeltt, 2001). The applicability of the MENT, a self-paced emotion recognition test, as a PVT within the assessment of ASD is, therefore, not jeopardized by an emotion recognition deficit within the autism spectrum.

Third, explorative analyses demonstrated that adults with ASD rate their emotion recognition ability lower in comparison to adults without ASD. This finding indicates that adults with ASD have an overall belief that they are poor in recognizing emotions. This is in line with previous findings that individuals with ASD experience deficits in social-emotional reciprocity and may find it hard to understand and respond to emotions of another person (APA, 2013; Dawson et al., 2002; Sigman, Kasari, Kwon, & Yirmaya, 1992). However, this perceived inability is not quantified on the measure of emotion recognition (ERT). Thus, they belief they are impaired in emotion recognition whereas they perform well in the actual recognition of emotions. This discrepancy may, again, be explained by impairment in quickly

(18)

processing emotional information. As this is required in everyday emotion recognition, this could lead to qualitative emotion recognition problems but intact identification of emotional expressions on a self-paced measure. The observed discrepancy increases the applicability of the MENT as a PVT even further as it strikes to be a face-valid measure of a disability that is, according to the individuals within the autism spectrum, a characteristic of ASD.

Alternative explanations of our findings must however be considered. A first

alternative explanation may be that participants merely guessed on both emotion recognition tasks which led to good performance on the MENT and the ERT. Explorative, we therefore evaluated whether participants performed above chance level on both tasks (see supplemental materials for detailed information). On the MENT, one item was performed below chance level by 9 participants with ASD and 12 controls. We can conclude that the expressions on this item were too vague to match the emotional label. Therefore, in future assessment of the MENT, it is recommended to adjust these expressions. Overall performance was, however, above chance level on the MENT, indicating that guessing cannot explain our finding that the MENT is insensitive to a diagnosis of ASD. On the ERT, the emotion specific trials of fear and sadness were performed below chance level. Although this seems to be in line with previous research indicating that these emotions are difficult to recognize by adults with ASD (Boraston, Blakemore, Chilvers, & Skuse, 2007; Humphreys, Minshew, Leonard, &

Behrmann, 2007), the current finding demonstrates that an equal amount of adults without ASD were unable to identify them. Thus, the expressions of fear and sadness were

insufficiently distinguishable from the other basic emotions by both groups. Therefore, it is recommended to cautiously interpret the performance on these emotions in future assessment of the ERT. Overall emotion recognition performance was however above chance level and therefore guessing cannot explain the intact basic emotion recognition we observed in adults with ASD.

(19)

A second issue related to our null-finding on the ERT may be that the sex-ratio in our sample did not match the sex ratio in the clinical population of ASD and this could have overshadowed a difference between ASD and controls in emotion recognition. In a clinical ASD population, there are four to five times more males than females (Fombone, 2003), whereas in our sample an even amount of both genders was included. As females perform better in recognizing emotions in comparison to males (Montagne, Kessels, Frigerio, de Haan, & Perrett, 2005), their overrepresentation in our sample made it possible that their

performance evened out the poor performance of the males with ASD in the ASD group. This could explain our null-finding and may have prevented us to uncover a possible worse

performance by individuals with ASD (especially males) in comparison to individuals without ASD. Explorative, we therefore evaluated whether males and females differed in their

performance on the ERT and subsequently whether the effect of gender differed across conditions (gender x group interaction, see supplemental materials). Finally, we evaluated whether our null-finding was still supported if we excluded females from our analyses (see supplemental materials). We found that there was indeed an overall better performance of females than males on the ERT. However, the effect of gender on performance was equal across groups. A rerun of our analyses without females supported the observed null-finding. In sum, these findings indicate that females could not have overshadowed a worse

performance of the ASD sample in comparison to the control group and, thus, our null-finding is not biased by the sex ratio in our sample.

Our study was the first to examine a performance validity test for ASD. To this end, we tried to establish a strong study design. Although matching on age, gender and intelligence was successful and our participants were indeed blind to the investigation of performance validity, there are still various unanswered questions with regard to performance validity testing within ASD. First of all, while we could establish with the current design whether the

(20)

MENT was insensitive to the diagnosis of ASD, we could not test how well the MENT will perform as a PVT as we cannot determine the specificity (proportion of honest clients with ASD that is not identified as dishonest) and sensitivity (proportion of feigning clients that is identified as dishonest) of this test. To this end, in a future study, the MENT should be administered to a group of individuals who are instructed to feign ASD and matched to a group of individuals with an actual ASD on gender, age and intelligence. If high sensitivity and specificity will be obtained in future studies, it would be of interest to implement the MENT in ASD assessment procedures. Implementation of PVT will increase credibility of a diagnosis and useful allocation of treatment.

Another unanswered question that needs to be addresses in future research, is to what extent the MENT is insensitive to the entire autism spectrum. The current sample included only adults with a normal intelligence that could function independently, and therefore we are unsure to what extent our findings generalize to individuals with ASD that deviate from our sample on these characteristics. To prevent that these individuals are misclassified as feigners on the MENT during diagnostic assessment, it is important to investigate in future research whether there is a threshold at which the MENT is no longer insensitive to a diagnosis of ASD and thus no longer a useful index for detecting feigning of ASD. Finding such a threshold is important because previous research indicates that these lower limits exist. For example, research on the TOMM demonstrated that individuals with severe dementia are unable to pass this PVT (Teichner & Wagner, 2003). Thus, future research should investigate (a) to what extent our results can be replicated in a greater ASD sample with a broader age range and symptom severity and (b) if, at some point within the autism spectrum, there may be a threshold of the applicability of the MENT as a PVT within the assessment of ASD.

(21)

Third, the explanation we opposed earlier of our discrepant findings with regard to qualitative and quantitative emotion recognition in ASD should be validated in future research. As our observed findings seem to indicate that individuals with ASD might be slower in recognizing emotions, a future study could evaluate whether response time on a quantitative measure is indeed greater for individuals with ASD in comparison to individuals without ASD.

Finally, including participants solely on a clinical diagnosis in combination with a self-report instrument which assesses ASD traits may have not been a sufficient and reliable screening procedure for the ASD research sample. Twenty-three percent of the original ASD sample was namely excluded based on this screening procedure and therefore, in future research, a more exhaustive screening of ASD is recommended. For example, a

semi-structured interview such as the Autism Diagnostic Observation Schedule (Generic) (ADOS-G; Gotham, Risi, Pickles, & Lord, 2007; Lord et al., 1989, 2000) could be included to assess ASD.

In sum, we have evaluated whether the MENT could be a useful performance validity test within ASD assessment. Revisiting the criteria for a performance validity test (Hartman, 2002), the current study was the first to establish that the MENT is well performed by individuals with ASD and at the same time strikes as a test of ASD (Hartman, 2002). Thus, the MENT has possible merit as a performance validity test for ASD and, therefore, future research on the applicability of the MENT as a performance validity test within the

(22)

References

American Psychiatric Association (2000). Diagnostic and Statistical Manual of Mental

Disorders (4th ed., Text Revision). Washington, DC: American Psychiatric Association.

American Psychiatric Association (2013). Diagnostic and Statistical Manual of Mental

Disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The Autism- Spectrum Quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders, 13(1), 5-17.

Baron-Cohen, S., Wheelwright, S., & Jolliffe, T. (1997). Is there a ‘Language of the Eyes’? Evidence from normal adults, and adults with autism or Asperger syndrome. Visual Cognition, 4(3), 311-331.

Bennetto, L., Pennington, B. F., & Rogers, S. T. (1996). Intact and impaired memory functions in autism. Child Development, 67(4), 1816-1835.

Boraston, Z., Blakemore, S-J., Chilvers, R., Skuse, D. (2007). Impaired sadness recognition is linked to social interaction deficit in autism. Neuropsychologia, 45, 1501-1510.

Bush, S. S., Ruff, R. M., Tröster, A., Barth, J., Koffler, S. P., Pliskin, N. H., et al. (2005). NAN position paper: Symptom validity assessment: Practice issues and medical necessity. Archives of Clinical Neuropsychology, 20, 419-426.

Castelli, F. (2005). Understanding emotions from standardized facial expressions in autism and normal development. Autism, 9(4), 428-449.

Celani, G., Battacchi, M. W., & Arcidiacono, L. (1999). The understanding of the emotional meaning of facial expressions in people with autism. Journal of Autism and Developmental Disorders, 29(1), 57-66.

(23)

Cohen, J. (1992). A power primer. Psychological Bulletin, 122, 155–159.

Dawson, G., Webb, S., Schellenberg, G. D., Dager, S., Friedman, S., Aylward, E., et al. (2002). Defining the broader phenotype of autism: Genetic, brain, and behavioral perspectives. Development and Psychopathology, 14, 581-611.

DeViva, J. C., & Bloem, W. D. (2003). Symptom exaggeration and compensation seeking among combat veterans with posttraumatic stress disorder. Journal of Traumatic Stress, 16(5), 503-507.

Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3), 169-200. Faust, D., Hart, K., & Guilmette, T. J. (1988). Pediatric malingering: the capacity of children

to fake believable deficits on neuropsychological testing. Journal of Consulting and Clinical Psychology, 56, 578-582.

Faust, D., Hart, K., Guilmette, T. J., & Arkes, H. R. (1988). Neuropsychologists’ capacity to detect adolescent malingerers. Professional Psychology: Research and Practice, 19(5), 508-515.

Field, A. P., 2009. Discovering statistics using SPSS (3rd edition). London: SAGE Publications.

Fombonne, E. (2003). Epidemiological surveys of autism and other pervasive developmental disorders: An update. Journal of Autism and Developmental Disorders, 33(4), 365-382.

Frigerio, E., Burt, D. M., Montagne, B., Murray, L. K., &Perrett, D. I. (2002). Facial affect perception in alcoholics. Psychiatry Research, 113, 161–711.

Frith, U., & Happé, F. (1999). Theory of mind and self-consciousness: What is it like to be autistic? Mind & Language, 14(1), 1-22.

(24)

Gepner, B., Deruelle, C., & Grynfeltt (2001). Motion and Emotion: A novel approach to the study of face processing by young autistic children. Journal of Autism and Developmental Disorders, 31(1), 37-45.

Gepner, B., de Gelder, B., de Schonen, S. (1996). Face processing in autistics: Evidence for a generalised deficit? Child Neuropsychology, 2(2), 123-139.

Geraerts, E., Kozaric-Kovacic, D., Merchelbach, H., Peraica, T., Jelicic, M., & Candel, I. (2009). Detecting deception of war-related posttraumatic stress disorder. The Journal of Forensic Psychiatry & Psychology, 20(2), 278-285.

Gotham, K., Risi, S., Pickles, A., & Lord, C. (2007). The Autism Diagnostic Observation Schedule: Revised algorithms for improved diagnostic validity. Journal of Autism and Developmental Disorders, 37, 613-627.

Green, P. (2007). The pervasive influence of effort on neuropsychological tests. Physical Medicine and Rehabilitation Clinics of North America, 18(1), 43-68.

Green, P., Allen, L., & Astner, K. (1996). The Word Memory Test: A user’s guide to the oral and computer-administered forms, US version 1.1. Durham, NC: CogniSyst.

Green, J., Gilchrist, A., Burton, D., & Cox, A. (2000). Social and psychiatric functioning in adolescents with Asperger syndrome compared with conduct disorder. Journal of Autism and Developmental Disorders, 30(4), 279-293.

Green, P., Rohling, M. L., Lees-Haley, P. R., & Allen, L. M. (2001). Effort has a greater effect on test scores than severe brain injury in compensation claimants. Brain Injury, 15(12), 1045-1060.

Grossman, J. B., Klin, A., Carter, A. S., & Volkmar, F. R. (2000). Verbal bias in recognition of facial emotions in children with Asperger syndrome. Journal of Child Psychology and Psychiatry, 41(3), 369-379.

(25)

Hartman, D. E. (2002). The unexamined lie is a lie worth fibbing. Neuropsychological

malingering and the Word Memory Test. Archives of Clinical Neuropsychology, 17, 709-771.

Hoekstra, R. A., Bartels, M., Cath, D. C., & Boomsma, D. I. (2008). Factor structure, reliability and criterion validity of the Autism-Spectrum Quotient (AQ): A study in Dutch population and patient groups. Journal of Autism and Developmental Disorders, 38, 1555-1566.

Hoaglin, M. F. D. C., & Iglewicz, B. (1987). Fine tuning some resistant rules for outlier labeling. Journal of the American Statistical Association, 82(400), 1147-1149.

Humphreys, K., Minshew, N., Leonard, G. L., & Behrmann, M. (2007). A fine-grained analysis of facial expression processing in high-functioning adults with autism. Neuropsychologia, 45, 685-695.

Janssen, R., & Soeters, P. (2010). DBC’s in de GGZ, ontwrichtende of herstellende werking? GZ- Psychologie, 7, 36-45.

Kätsyri, J., Saalasti, S., Tiippana, K., von Wendt, L., & Sams, M. (2008). Impaired recognition from low- spatial frequencies in Asperger syndrome. Neuropsychologia, 46, 1888-1897.

Kessels, R. P. C., Montagne, B., Hendriks, A. W., Perrett, D., & de Haan, E. H. F. (2013). Assessment of perception of morphed facial expressions using the Emotion Recognition Task: Normative data from healthy participants aged 8-75. Journal of Neuropsychology, 1-19.

Lees-Haley, P. R., & Dunn, J. T. (1994). The ability of naive subjects to report symptoms of mild brain injury, post-traumatic stress disorder, major depression, and generalized anxiety disorder. Journal of Clinical Psychology, 50(2), 252-256.

(26)

Lindner, J. E., & Rosén, L. A. (2006). Decoding of emotion through facial expression, prosody and verbal content in children and adolescents with Asperger’s Syndrome. Journal of Autism and Developmental Disorder, 36, 769-777.

Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., et al. (2000). The Autism Diagnostic Observation Schedule-Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205-223.

Lord, C., Rutter, M., Goode, S., Heemsbergen, J., Jordan, H., Mawhood, L., et al. (1989). The Autism Diagnostic Observation Schedule: A standardized observation of communicative and social behavior. Journal of Autism and Developmental Disorders, 19(2), 185-212.

Matson, J. L., & Shoemaker, M. (2009). Intellectual disability and its relationship to autism spectrum disorders. Research in Developmental Disabilities, 30, 1107-1114.

Merten, T., Thies, E., Schneider, K., & Stevens, A. (2009). Symptom validity testing in claimants with alleged posttraumatic stress disorder: Comparing the Morel Emotional Numbing Test, the Structured Inventory of Malingered Symptomatology, and the Word Memory Test. Psychological Injury and Law, 2, 284-293.

Montagne, B., De Geus, F., Kessels, R. P. C., Denys, D., De Haan, E. H. F., & Westenberg, H. G. M. (2008). Recognition of facial expressions in obsessive-compulsive disorder: A dimensional approach. European Psychiatry, 23, 26–28.

Montagne, B., Kessels, R. P. C., Frigerio, E., de Haan, E. H. F., & Perrett, D. I. (2005). Sex differences in the perception of affective facial expressions: Do men really lack emotional sensitivity? Cognitive Processes, 6, 136-141.

(27)

Morel, K.R. (1998). Development and preliminary validation of a forced-choice test of response bias for posttraumatic stress disorder. Journal of Personality Assessment, 70, 299–314.

Morel, K.R. (2008). Comparison of the Morel Emotional Numbing Test for posttraumatic stress disorder to the word memory test in neuropsychological evaluations. The Clinical Neuropsychologist, 22(2), 350–362.

Morel, K. R., & Shepherd, B. E. (2008). Meta-analysis of the Morel Emotional Numbing Test for PTSD: Comment on Singh, Avasthi, & Grover. The German Journal of Psychiatry, 11(1), 128-131.

Pankratz, L. (1983). A new technique for the assessment and modification of feigned memory deficit. Perceptual and Motor Skills, 57, 367-372.

Pelphrey, K. A., Sasson, N. J., Reznick, S., Paul, G., Goldman, B. D., & Piven, J. (2002). Visual scanning of faces in autism. Journal of Autism and Developmental Disorders, 32(4), 249-261.

Ringe, W. K., Saine, K. C., Lacritz, L. H., Hynan, L. S., & Munro-Cullum, C. M. (2002). Dyadic short forms of the Wechsler Adult Intelligence Scale-III. Assessment, 9(3), 254-260.

Rouder, J. N., Speckman, L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 225-237.

Schmand, B., & Ponds, R. (2004). Suboptimaal presteren: Simuleren, aggraveren en somatiseren. In: B. Deelman, P. Eling, E. de Haan, A. H. van Zomeren (Eds.), Klinische Neuropsychologie (pp. 523-539). Amsterdam: Boom.

Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E. et al. (1998). The Mini-International Neuropsychiatric Interview (M.I.N.I.): The

(28)

development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry, 59(20), 34-57.

Sigman, M. D., Kasari, C., Kwon, J-H., & Yirmiya, N. (1992). Responses to the negative emotions of others by autistic, mentally retarded, and normal children. Child Development, 63(4), 796-807.

Spek, A. A., Scholte, E. M., & Van Berckelaer-Onnes, I. A. (2010). Theory of mind in adults with HFA and Asperger syndrome. Journal of Autism and Developmental Disorders, 40, 280-289.

Stone, V. E., Baron-Cohen, S., & Knight, R. T. (1998). Frontal lobe contributions to theory of mind. Journal of Cognitive Neuroscience, 10(5), 640-656.

Suhr, J. A., Sullivan, B. K., & Rodriguez, J. L. (2011). The relationship of noncredible performance to continuous performance test scores in adults referred for Attention-Deficit/Hyperactivity Disorder evaluation. Archives of Clinical Neuropsychology, 26, 1-7.

Tager-Flusberg, H. (2007). Evaluating the theory-of-mind hypothesis of autism. Current Directions in Psychological Science, 16(6), 311-315.

Teichner, G., & Wagner, M. T. (2003). The Test of Memory Malingering (TOMM): normative data from cognitively intact, cognitively impaired and elderly patients with dementia. Archives of Clinical Neuropsychology, 19, 455-464. Tombaugh, T. N. (1996). Test of Memory Malingering (TOMM). New York: Multi

Health Systems.

Tombaugh, T. N. (1997). The Test of Memory Malingering (TOMM): Normative data from cognitively intact and cognitively impaired individuals. Psychological Assessment, 9, 260-268.

(29)

Uterwijk J. WAIS-III. Nederlandstalige Bewerking. Technische Handleiding. Lisse: Swets & Zeitlinger, 2000.

Van Vliet, I. M., Leroy, H., & Van Megen, H. J. G M. (2000). M.I.N.I. Plus: M.I.N.I. Internationaal Neuropsychiatrisch Interview: Nederlandse versie 5.0.0. Retrieved February 1 , 2014, from http://www.tijdschriftvoorpsychiatrie.nl/assets/measuring instruments/meetinstrumenten_90pdf.pdf

Wechsler, D. (1997). Wechsler Adult Intelligence Scale-Third edition administration and scoring manual. San Antonio, TX: The Psychological Corporation.

Woodbury-Smith, M. R., Robinson, J., Wheelwright, S., & Baron-Cohen, S. (2005). Screening adults for Asperger syndrome using the AQ: A preliminary study of its diagnostic validity in clinical practice. Journal of Autism and Developmental Disorders, 35(3), 331-335.

World Health Organization.(1992). The ICD-10 classification of mental and behavioural disorders: Clinical discriptions and diagnostic guidelines. Geneva: Author.

(30)

Supplemental material for

Explorative analyses

Emotion recognition

A univariate analysis of variance, with the total score on the ERT as the dependent variable and gender and group as the independent variables, indicated that females (M = 63.50, SE = 1.99) performed significantly better on this task in comparison to males (M = 56.95 , SD = 1.66), F (1, 37) = 6.32 , p = 0.02, Cohen’s d = 0.14.. However, the interaction effect of gender and group was not significant, indicating that the effect of gender did not differ between adults with ASD and adults without ASD, F (1, 37) = 1.401, p = 0.25. An independent t-test was rerun on the ERT data without women. Individuals with ASD (M = 58.20, SE = 2.32) did not perform worse on the ERT in comparison to individuals without ASD (M = 55.70, SE = 2.45), t (18) = 0.741, p = .468, Cohen’s d = 0.33.

Above chance performance

First, we evaluated whether participants performed above chance on the MENT. The chance of choosing a correct response while guessing is 50% on each item of the MENT. Therefore, above chance was indicated as a total score higher than 30/60 and an individual item score of 1 in at least 19/38 participants. Overall, all participants performed above chance on the MENT. However, 55% of the participants answered item 18 on set 2 incorrectly (9

participants with ASD and 12 controls) and therefore participants did not score above chance on this specific item.

(31)

Second, we evaluated whether participants performed above chance on the ERT. Above chance was indicated as a total score higher than 16/96 and an emotion total score (per emotion) higher than 2.67/16.Overall, all participants performed above chance on the ERT. However, on the fear trials, 23.7% of the participants (5 participants with ASD and 4 controls) performed below chance level. Also, on the sadness trials, 10.5% of participants performed below chance level (2 participants with ASD and 2 controls).