Implicit artificial grammar learning: effects of complexity and usefulness of the structure

Implicit artificial grammar learning: effects of complexity and usefulness

of the structure

Bos, E.J. van den

Citation

Bos, E. J. van den. (2007, June 6). Implicit artificial grammar learning: effects of complexity

Institute for Psychological Research, Faculty of Social Sciences, Leiden University. Retrieved

from https://hdl.handle.net/1887/12037

Version: Corrected Publisher’s Version

License:

Downloaded from:

Note: To cite this publication please use the final published version (if applicable).

Chapter 7

Can grammar complexity be counterbalanced by a semantic

reference field?

Abstract

By comparing performance on a simple and a complex finite state grammar, the present study replicated the finding from Chapter 2 that implicit learning is

negatively affected by complexity. For both grammars, learning could be enhanced by providing a semantic reference field and instructing participants to decode the

meaning of each exemplar. Unlike participants who memorized exemplars, however, participants in the decoding condition acquired explicit knowledge of salient letter chunks. Moreover, learning by decoding exemplars was as much affected by

complexity as implicit learning. These findings stress the influence of complexity on artificial grammar learning.

Introduction

In Chapter 2 we found that implicit learning of artificial grammars was negatively affected by the complexity of the grammar. The higher a grammar’s topological entropy (Bollt & Jones, 2000), a measure reflecting the number of unique exemplars of any given length that the grammar can generate, the lower was

participants’ performance on a grammaticality judgment test. This finding seems at odds with suggestions that implicit learning plays a role in the acquisition of natural languages (Ellis, 2005; Perruchet & Pacton, 2006; Reber, 1967; Reber & Allen, 2000;

Saffran, Newport, Aslin, Tunick & Barrueco, 1997), which, in Chomsky’s (1957) hierarchy, are far more complex than the finite state grammars used in Chapter 2.

However, it was suggested that, in the case of natural languages, the effects of complexity might be counterbalanced by situational characteristics that were absent from our experiment.

One obvious deviation from the natural situation was the absence of semantics. Experiments on miniature artificial language learning have generally shown beneficial effects of making exemplars meaningful by adding a reference field (Meier & Bower, 1986; Mori & Moeser, 1983; Nagata, 1976; Valian & Coulson,

1988). Moreover, Moeser and Bregman (1972) and Nagata (1977) found that the presence of a reference field mediated effects of complexity. In the absence of a reference field, increasing complexity hampered grammar learning. In contrast, when a reference field was presented that made the constraints of the grammar meaningful (e.g. a type of border cannot appear without a figure), complexity did not affect learning.

Although these findings suggest that the negative effect of complexity on implicit learning observed in Chapter 2 may be removed by adding a reference field, straightforward generalization is prevented by methodological differences. For one thing, the miniature artificial language learning experiments used phrase structure grammars, which specify the relations between classes of words, instead of the simpler finite state grammars, which typically consist of rules pertaining to specific terminal elements. The results of Moeser and Bregman (1972) suggest that one of the beneficial effects of a reference field is facilitation of word class learning. Therefore, the influence of a reference field on learning a complex finite state grammar remains to be established.

In addition, miniature artificial language learning experiments have not investigated whether grammar learning in the presence of a reference field was implicit or explicit. Implicit learning can be defined as acquiring implicit knowledge of a structure without intending to do so. Establishing whether or not knowledge is implicit is often problematic (see Shanks & St.John, 1994). Nevertheless two criteria have been proposed that seem to make a meaningful distinction between implicit and explicit knowledge. Firstly, Jacoby (1991) proposed that knowledge is implicit when it is applied in absence of any intention to do so: i.e. when people lack control over their knowledge. Secondly, Dienes and Berry (1997) proposed that implicit

knowledge is characterized by a lack of meta-knowledge. According to the guessing criterion, people have implicit knowledge when they perform above chance while they claim to be guessing. For both criteria, it has been demonstrated that knowledge classified as explicit is affected by divided attention, while knowledge classified as implicit is not (Dienes, Altmann, Kwan & Goode, 1995; Jacoby, Toth & Yonelinas, 1993).

In Chapter 5, some participants acquired knowledge that was implicit according to the ‘lack of control’-criterion. After trying to decode which ice-cream exemplars of a finite state grammar referred to, they could not refrain from

completing exemplars in a grammatical way when they were instructed to create ungrammatical exemplars. Other studies, however, suggest that the presence of a reference field induces explicit learning. Reference fields have been shown to enhance learning by making the constituent phrases of each exemplar salient (Meier & Bower, 1986; Morgan & Newport, 1981) and salience has been shown to enhance explicit rather than implicit learning (Reber, Kassin, Lewis & Cantor, 1980; though see Turner

& Fischler, 1993).

The aims of the present study were threefold. A preliminary aim was to replicate the negative effect of complexity on implicit learning by comparing a simple and a complex grammar used in Chapter 2. The second and main goal was to

investigate whether providing a reference field for these two finite state grammars could counterbalance the effect of complexity. Finally, the guessing criterion was used to examine whether decoding exemplars to identify the correct referent led to implicit or explicit knowledge.

Method Participants

There were 102 participants in this study (18 male, 84 female; 17 – 37 years of age, M = 21.13, SD = 3.55). All participants were undergraduate students of Leiden University, who received either course credits or money for their participation. The reward depended on the duration of the experiment.

Design

The experiment consisted of an induction phase and a test phase. There were three independent variables. Firstly, the task in the induction phase was varied between participants. Thirty-six participants were instructed to memorize the

exemplars of an artificial grammar (memorize). Thirty-six participants were instructed to decode the exemplars and decide whether or not they described one of two trains (decode). To prepare for their task, these participants first learned the meaning associated with each letter. For the remaining 30 participants there was no induction phase; they formed the control group. Secondly, the complexity of the grammar was varied between participants. One half of the participants with each task worked with a simple grammar; the other half worked with a complex grammar. Finally, different types of exemplars were presented on the grammaticality judgment test (varied within

subjects). The dependent variable was the proportion of correct grammaticality judgments in the test phase.

Materials

The stimuli were exemplars of two of the finite-state grammars used to vary complexity in Chapter 2. Both grammars consisted of the same 11 states and used the letters J, M, N, P, Q, R, S, T, W, X and Z. Complexity was manipulated by creating different connections between the states (see Figure 1). Topological entropy (Bollt &

Jones, 2000) was .71 for the simple and 2.05 for the complex grammar.

A computer program generated a set of 120 unique exemplars of 5 to 11 letters for each grammar. Of each set, 60 exemplars were assigned to the induction phase and 48 to the test phase, so that the paths of the grammar were represented according to the same ratio in both phases of the experiment (see Appendix B). Five of the remaining exemplars were used on practice trials in the test phase; the rest was discarded. In the decoding condition, the same exemplars were used on the practice trials of the induction phase, while number strings unrelated to the grammars were used in the memorize condition. On the experimental trials of the induction phase, the exemplars were the same for both tasks. The exemplars presented in the test phase were the same for all conditions using each grammar.

In the decoding condition, each letter referred to either a shape (M = circle, Q

= diamond, R = triangle, T = rectangle) or a color (J = light green, N = yellow, P = dark green, S = purple, W = red, X = blue, Z = orange). To learn these associations, the letters were first presented together with pictures of shapes and color patches. In the subsequent induction phase, each exemplar was accompanied by two pictures of trains pulling wagons with various shapes of various colors. One third of the

exemplars referred to the train in the upper picture, one third referred to the train in the lower picture and one third referred to a train that was not presented.

Two rules of reference were applied to create the correct train for each

exemplar. First, if a color was followed by a shape, it applied to that shape. Second, if a color was followed by another color, it produced a stripe on the preceding shape.

For example, NRXTZ referred to a train pulling a wagon with a yellow triangle and a wagon with a blue rectangle with an orange stripe. Incorrect alternatives were created by substituting a random color or shape for one specified by the exemplar.

Substitutions were balanced over the position and frequency of the substituted letter.

Four types of exemplars were used on the grammaticality judgment test: 1 grammatical and 3 ungrammatical. The ungrammatical exemplars were created by switching two adjacent inner letters (excluding the first and last). For each grammar, this resulted in 8 exemplars containing 3 violations of first-order dependencies (illegal sequences of 2 letters), 8 exemplars containing 2 first-order and 1 second-order violation (a sequence of two letters that is illegal given the letter preceding it) and 8 exemplars containing 1first-order and 2 second-order violations. The remaining 24 exemplars were unaltered grammatical strings.

0 0

1

2

3

4

5

6

7

8

9

10 Z

N

T

Q

M

R

P

S

W

X Z

J

N P S

J R

Q

M

T

J

N

Z

0 0

1

2

3

4

5

6

7

8

9

10 Z

N

N T

Q

M

R

W P

S

W

X Z

J

N R

Q

M

T

J

N

Z

Figure 1. The simple (top) and complex (bottom) finite state grammars used in the present study.

Procedure

Participants were tested individually in a dimly lit test booth. They were seated in front of a computer monitor, on which the stimuli were displayed, at a distance of about 50 cm. They reacted by pressing keys on a keyboard. The procedure for the induction phase depended on the participants’ task.

Participants in the memorize condition were instructed to study each exemplar for a memory test. They were presented with 5 practice trials, after which they were notified that the real task began. The 60 experimental trials were presented in random order. Each trial began with a fixation cross in the middle of the screen. After 1 second the cross was replaced by a grammatical exemplar centered at the fixation point. The exemplar was displayed for 5 seconds. Then participants were prompted to reproduce it. After pressing the enter-key, participants were again presented with the original exemplar for 2 seconds so that they could check their answer. Finally, the screen turned blank for 1 second before the next trial began.

The decoding condition started with a vocabulary acquisition phase.

Participants were informed that this was a necessary preparation for their next task. A screen pairing each letter with its corresponding color or shape was presented and participants were instructed to study it until they knew the referent of each letter.

After pressing the enter-key, they received a vocabulary test. The letters were

presented in random order together with three numbered pictures of colors or shapes.

Participants had to indicate which picture each letter referred to by pressing 1, 2 or 3.

This study-test procedure was repeated three times.

In the subsequent induction phase, participants were informed that they would see two pictures of trains and one letter string. They were instructed to press 1 if the string referred to the upper train, 2 if it referred to the lower train or 3 if it referred to neither. Each trial (5 practice and 60 experimental) started with the presentation of the trial number. After one second, an exemplar of the grammar was presented in the lower half of the screen together with two pictures of trains presented above each other in the upper half. When the participant pressed 1, 2 or 3, the two pictures were replaced by one picture of the correct referent, accompanied by the word “Correct!” if the answer was right. After 5 seconds, the feedback screen was followed by the next trial.

The induction phase was followed by the grammaticality judgment test.

Experimental participants were informed that the previously presented exemplars had been generated according to a complex set of rules. They were instructed to judge whether or not new exemplars followed the same rules. Control participants received a similar instruction, which did not refer to the induction phase. Participants were instructed to press the ‘j’-key (for ‘ja’: ‘yes’) if they thought that an exemplar

followed the rules or the ‘n’-key (for ‘nee’: ‘no’) if they thought that it did not follow the rules. In addition, they were required to indicate their confidence in each judgment on a scale from 1 (very little) to 5 (very much) by pressing one of the number keys on the keyboard.

Participants received 5 practice trials followed by 48 experimental trials, presented in random order. Each trial began with a fixation cross appearing in the middle of the screen. After 1 second the cross was replaced by an exemplar centered at the fixation point. When the participant pressed the ‘j’ or ‘n’-key, the screen turned blank for 1 second. Subsequently, the confidence scale was presented until the participant pressed a number from 1 to 5. A final blank screen separated two consecutive trials by 1 second.

After the test, participants were thanked for their participation. The duration of the experiment varied from 10 minutes for the control condition to 30 minutes for the memorize condition and 45 minutes for the decoding condition.

Analyses

The data from the grammaticality judgment test were analyzed by means of a 2 x 3 x 4 mixed-model analysis of variance (ANOVA) with grammar (simple vs.

complex) and task in the induction phase (memorize vs. decode vs. control) as between-subjects variables and type of exemplar (grammatical vs. 3 first-order violations vs. 2 first, 1 second-order violation vs. 1 first, 2 second-order violations) as within-subjects factor. To investigate whether knowledge was implicit according to the guessing criterion (Dienes et al., 1995) a 2 x 3 ANOVA with grammar and task in the induction phase as between-subjects variables was performed on the proportion of correct grammaticality judgments for trials on which participants had (very) little confidence. Simple contrasts were used to investigate whether experimental participants were more often correct than control participants on such trials.

Results

The ANOVA on the proportion of correct grammaticality judgments showed main effects of grammar (F(1,96) = 12.317, MSE = .045, p = .001), task in the induction phase (F(2,96) = 30.864, MSE = .045, p < .001) and type of exemplar (F(3,94) = 13.558, MSE = .028, p < .001). However, these effects were modified by significant interactions between grammar and task in the induction phase (F(2,96) = 3.372, p = .038) and between task in the induction phase and type of exemplar (F(6,190) = 6.749, p < .001), which will be examined below.

Figure 2. Mean proportion correct with 95% confidence interval for each task in the induction phase and grammar.

The interaction between grammar and task in the induction phase is illustrated by Figure 2. For both the simple and the complex grammar, separate ANOVA’s showed that the proportion of correct grammaticality judgments depended on the task in the induction phase (simple: F(2,48) = 32.961, p < .001; complex: F(2,48) = 16.931, p < .001). Orthogonal contrasts showed that the proportion correct was higher

for the experimental conditions than for the control condition (simple: t(35.0) = 8.604, p < .001; complex: t(43.4) = 6.979, p < .001) and higher for the decoding condition

than for the memorize condition (simple: t(28.4) = 2.881, p = .007; complex: t(29.4) = 2.131, p = .042). However, independent samples t-tests showed that performance on the grammaticality judgment test after the three tasks was differentially affected by grammar complexity. In the memorize condition (t(34) = 2.547, p = .016) and the decoding condition (t(34) = 2.308, p = .027), the proportion correct was higher for the simple grammar than for the complex grammar; in the control condition, there was no difference (t(28) < 1).

The interaction between task in the induction phase and type of test exemplar was examined by performing separate ANOVA’s for each type of exemplar.

Proportion correct (see Table 1) varied with the task in the induction phase for grammatical exemplars (F(2,99) = 46.832, p < .001), exemplars with 3 first-order violations (F(2,99) = 8.648, p < .001) and exemplars with 2 first and 1 second-order violation (F(2,99) = 24.453, p < .001). For each of these types, orthogonal contrasts showed that the experimental conditions did better than the control condition (t(67.9)

= 9.397, p < .001; t(99) = 4.158, p < .001; t(99) = 5.995, p < .001, respectively). For grammatical exemplars and exemplars with 2 first and 1 second-order violation, the proportion correct was also higher after decoding than after memorizing (t(62.2) = 4.389, p < .001; t(99) = 3.601, p < .001, respectively). In contrast, there was no difference between the tasks in the induction phase for exemplars with 1 first and 2 second-order violations (F(2,99) < 1). Neither memorizing nor decoding enabled participants to perform better than the control group on this type of exemplar.

Table 1

Mean proportion of correct grammaticality judgments with standard deviations for each type of exemplar and task in the induction phase.

Task

Type of exemplar Memorize Decode Control

Grammatical .667 (.119) .821 (.173) .487 (.115) 3 first-order violations .680 (.178) .676 (.220) .505 (.166) 2 first, 1 second-order violations .590 (.184) .759 (.221) .415 (.189) 1 first, 2 second-order violations .534 (.195) .467 (.201) .503 (.228) Note. Standard deviations are in parentheses.

Low-confidence grammaticality judgments

The ANOVA on the proportion of correct low-confidence responses only showed a main effect of task in the induction phase (F(2,81) = 3.417, MSE = .044, p = .038). Simple contrasts showed that the proportion correct in the memorize condition (M = .624, SD = .156) was higher than in the control condition (M = .499, SD = .125;

p = .024), whereas the proportion correct in the decoding condition (M = .507, SD =

.316) did not differ from the control condition (p = .886). This suggests that

participants in the decoding condition had meta-knowledge about their knowledge of the grammar. Participants in the memorize condition, in contrast, showed knowledge of the grammar although they thought they were guessing. This suggests that part of their knowledge was implicit.

Discussion

The first aim of the present study was to replicate the negative influence of grammar complexity on implicit learning that was observed in Chapter 2. Indeed, the present experiment showed that participants who memorized exemplars from a simple grammar were better at judging the grammaticality of new exemplars than participants who memorized exemplars from a complex grammar. The main difference between the studies was a failure to replicate the finding that increasing complexity

particularly hampered implicit learning of second-order dependencies. An explanation for this discrepancy may be that the difference in complexity between the grammars was smaller in the present study. The regression equation for items containing multiple violations of second-order dependencies in Chapter 2 predicted a difference in proportion correct of .22 for the grammars used here, which is at the limit of the 95% confidence interval for the observed difference in the present study (M = .09, 95% CI = -.04 – .22). This suggests that the regression equation in Chapter 2 was

influenced by the more extreme levels of complexity. A difference in the acquisition of second-order dependencies may therefore be observed with larger differences in complexity between grammars. Further research will have to show whether or not implicit learning of second-order dependencies is particularly vulnerable to increasing complexity and, if so, what increases are critical.

The main question in the present study was whether the negative effect of complexity could be counterbalanced by adding a reference field, which made the

exemplars meaningful. For both the simple and the complex grammar, participants who had to decode this meaning performed better on the grammaticality judgment test than participants who memorized the exemplars. However, the presence of the

reference field did not change the negative effect of complexity. Both tasks in the induction phase enabled participants working with the simple grammar to perform better on the grammaticality judgment test than participants working with the complex grammar. Although further studies may identify other ways to enhance artificial grammar learning, this finding suggests that complexity is an important determinant of success.

As noted in the introduction, reference fields may enhance the acquisition of phrase structure (and even more complex, natural) grammars even more than the acquisition of finite state grammars. Some artificial language learning studies have shown that the reference field produced its beneficial effect by enhancing the salience of phrases constituting the exemplars. Adding a reference field had the same effect as introducing spatial grouping (Morgan & Newport, 1981) and suffix markers (Meier &

Bower, 1986). Finite state grammars (by definition) do not involve this phrase structure. Nevertheless, the reference field may have enhanced learning in the present study by making certain aspects of the structure salient. Identifying a wagon in the reference field always required combining the meanings of two or three individual letters (in contrast to Chapter 5, in which the size of the chunks specifying one ball was more variable and ranged from 1 to 5 non-words). Consistently processing the exemplars as concatenations of two and three letter chunks is likely to enhance the salience of those chunks and may have led to explicit knowledge of bigrams and trigrams.

This suggestion was supported by the analysis of trials on which participants provided a low confidence rating. Participants in the decoding condition seemed to know whether or not they had relevant knowledge to make a grammaticality

judgment. If they had little confidence in their judgment, they were not more likely to be correct than control participants who had not been previously exposed to

grammatical exemplars. Consequently, decoding the meaning of grammatical exemplars produced explicit knowledge according to the guessing criterion.

Participants in the memorize condition, however, acquired implicit knowledge. They did better than control participants, even when they had little confidence in their

judgments, suggesting that their knowledge was not accompanied by meta- knowledge.

In conclusion, the present study produced three findings. Firstly, it provided further evidence that implicit artificial grammar learning is hampered by increasing complexity of the grammar. Secondly, it showed that learning can be enhanced by providing a semantic reference field. Participants who decoded the meaning of

exemplars from an artificial grammar acquired more knowledge than participants who memorized them. However, decoding the exemplars led to explicit knowledge of letter chunks that were made salient by the reference field. This suggests that the advantage of memorizing over explicit learning, observed for complex materials in Chapter 2, can be reversed if participants are cued what to learn. Thirdly, both implicit structure learning and explicit learning of salient letter chunks were reduced for the complex grammar. This suggests that the complexity of finite state grammars is an important determinant of how well (or with how much effort) these structures can be induced.