Visualising Cross-Recurrence Patterns in Natural Language Interaction

Visualising Cross-Recurrence

Patterns in Natural Language

Interaction

Nick J.G. de Wolf

10165401

Bachelor thesis

Credits: 18 EC

Bachelor Opleiding Kunstmatige Intelligentie

University of Amsterdam

Faculty of Science

Science Park 904

1098 XH Amsterdam

Supervisor

Raquel F

ERNANDEZ

´

Institute for Logic, Language & Computation

Faculty of Science

University of Amsterdam

Science Park 904

1098 XH Amsterdam

Abstract

Using recurrence quantification analysis, this thesis develops and dis-cusses a tool which can assist dialogue analysts with extracting both qual-itative and quantqual-itative information from the cross-recurrence plots of dia-logues. First, the techniques used in this thesis are explained in more detail. Second, the implementation of the tool is discussed, evaluating all the fea-tures which can be used for both quantitative and qualitative analysis. Finally, the power of the tool is then explored by looking into two case studies. The first case study provides a good overview of the general usefulness of the visualization tool, while the second case study evaluates how the tool can be used to detect certain features of the dialogues. The three features of child-adult dialogues that were discovered using the visualization tool are: “repe-tition games”, acknowledgements and backchannels, and language adaption. The results of these case studies suggest that the tool is a useful extension to the capabilities of a dialogue analyst. The visualization tool successfully as-sists in both quantitative and qualitative analysis for both general and specific tasks. Additionally, it provides a quick and easy to use interface for interact-ing with the data and the different measures that can be applied, while it can also successfully assist the dialogue analysts with tasks such as feature detection.

Contents

1 Introduction 4

2 Recurrence Quantification Analysis 5

3 RQA of Dialogue Interaction 5

4 Turn-Based Cross-Recurrence 8

4.1 Measures of Convergence . . . 8

4.1.1 Lexical Convergence . . . 9

4.1.2 Syntactical Convergence . . . 9

4.2 Recurrence Measures . . . 9

5 The Visualization Tool 10 5.1 Analysis of Requirements . . . 11 5.1.1 Data Analysis . . . 11 5.1.2 Tool Analysis . . . 12 5.2 Implementation of Visualization . . . 12 5.2.1 Turn Information . . . 13 5.2.2 Dialogue Information . . . 14 5.2.3 Text Display . . . 14 5.2.4 Display Controls . . . 14 5.2.5 Dialogue Display . . . 15 6 Analysis 15 6.1 Case Study 1: General Overview . . . 15

6.1.1 Aims . . . 15

6.1.2 Method and Results . . . 15

6.2 Case Study 2: Feature Detection . . . 18

6.2.1 Aims . . . 18

6.2.2 Method and Results . . . 19

1

Introduction

In recent years, researchers have attempted to analyze conversational data from dialogue corpora to learn more about language acquisition. For exam-ple, Clark [2009] studied language acquisition in a broad way, looking at nearly all of the different aspects involved (e.g. phonetics and syntactical structure), while Veneziano and Parisse [2010] studied language acquisition by studying more specific aspects. In their study they monitored the acquisi-tion of early verbs by children, by analyzing conversaacquisi-tions between children and their caregivers. This type of research is crucial to improving the knowl-edge of language acquisition, which in turn can further enhance the linguis-tic capabilities of autonomous systems. In contrast to other species, humans can only communicate with humans who have acquired the same language. Hence, the system will have to acquire this language as well. Additionally, Tomasello [2009] has argued that by studying language acquisition, more can be discovered about the use of language as well. Thus, new insights on lan-guage acquisition can enhance the capabilities of these autonomous systems to communicate with humans in at least two ways. First, the system would be able to apply language acquisition itself, improving its linguistic capabilities during a conversation with humans. Second, using the results of previous research, the system could already start off with a general understanding of the use of a specific language. In order to improve the understanding of language acquisition this thesis will focus on characterising and visualizing coordination between interlocutors in dialogue data.

Interlocutors tend to converge on different aspects during a dialogue (for instance, the use of certain words [Brennan and Clark, 1996], or syntactical constructions [Pickering and Garrod, 2004]). It still remains an open question as to how these convergences affect language acquisition. Recurrence quan-tification analysis (RQA) is a technique that can be used to investigate these convergences within a dialogue by analyzing the cross-recurrence plot (CRP) of a dialogue. An important feature of RQA is that this technique allows both qualitative and quantitative investigation. Both of these investigation types will play a significant role in this thesis.

The aim of this thesis is to develop a visualization tool that can assist dialogue analysts, with both the qualitative and the quantitative analysis of conversational data using RQA. By using both qualitative and quantitative analysis, the tool supports decision making and learning by providing a level of confidence in perceived relationships within the data [Amar and Stasko, 2004]. This thesis presents the basics of RQA and CRP, and describes the developed visualization tool which applies the techniques of RQA in combi-nation with CRPs.

The general concepts of RQA and CRPs will be introduced in the fol-lowing section. Section 3 will present an overview of recent work related to applying RQA on conversational data. Section 4 will explain the basic units of analysis, and the RQA measures that are implemented in the visualiza-tion tool. Subsequently, the most important implementavisualiza-tion choices will be described in Section 5. The results of the analysis of some conversational data will be displayed in Section 6. Finally, Section 7 will present with a discussion of the results.

2

Recurrence Quantification Analysis

RQAis a technique introduced by Eckmann et al. [1987], which can be ap-plied to recurrence plots (RPs) to retrieve quantified information. An RP is a method for visualizing the recurrence between multiple multi-dimensional states within the same dynamical system. As states are usually not one, two or three dimensional, which would allow it to be visualized, the plot is pro-jected into the two or three dimensional sub-spaces instead. Each cell within this recurrence plot represents the recurrence between two states. Eckmann et al. used a binary matrix, where the value at the cell equals 1 when the two states match, and in all other cases 0. Therefore an RP is a matrix that visu-alizes those times at which a state recurs within a dynamical system. Closely related, a cross-recurrence plot (CRP) displays those times at which a state in one dynamical system recurs simultaneously in the second dynamical sys-tem.

Eckmann et al. [1987] have shown that a recurrence plot is a useful visu-alization technique for identifying features in complex time series data. Be-cause the plots themselves provide no quantification, these features can only be used for qualitative analysis. In order to apply RQA, Zbilut and Webber Jr [1992] developed several measures to quantify features within an RP (For instance: Recurrence Rate (RR) and Percentual diagonal line length). Two years later, Webber Jr and Zbilut [1994] developed several more measures that, combined with the previous measures, formed the basis for RQA. They added measures for features such as: Entropy, Ratio, Trend, and Determin-ism. In short, RQA is a method of nonlinear data analysis, which is mainly used for investigating dynamical systems (e.g. dialogues). By applying RQA on a cross-recurrence plot (CRP) one can compute the similarity between two dynamic systems. As RQA can be applied to nearly all types of data, it has been used in many fields of research (e.g. physiology, engineering, chemistry, etc.). As the visualization tool developed in this thesis applies both the qualitative (RPs) and the quantitative (RQA) technique to retrieve information from a CRP, both of these techniques will be further explained. The next section will elaborate further on how RQA has been applied to the field of linguistics.

3

RQA of Dialogue Interaction

In this section, the various ways in which RQA has been applied in the past for the purpose of dialogue analysis, will be discussed. Firstly, the reasons which allow RQA to be applied to conversational data will be briefly explained. Secondly, the recent applications of RQA in the field of linguistics will be explored. Finally, the most important applications of RQA in the field of linguistics, which will also be presented in this thesis, will be summarized at the end of this section.

Although RQA has been used in a variety of disciplines, the technique has only just recently been applied in the field of linguistics. Some recent re-search has suggested that the language use of children and caregivers display syntactical coordinationduring the process of language acquisition. Lewis and Elman [2001] showed that children can correctly generalize rules of a

language from the statistical structure of the utterances they receive from their caregivers. In addition, Chouinard and Clark [2003] noticed that by correcting and reformulating incorrect utterances of the child, language ac-quisition can occur. These error corrections work on all kinds of different types of mistakes that could be made by the child (e.g phonological, mor-phological, syntactical, lexical, etc.). It became clear that children learn to align their own language use with the correct utterances of their caregivers. As the research has shown, children and caregivers can come to align their language used during conversation, hence this can also be interpreted as two dynamic systems of language aligning themselves to each other during a con-versation. This idea, of displaying interlocutors as dynamic systems, will be further explained.

The dynamic systems used in a CRP of a dialogue represent the two in-terlocutors of that dialogue. Each dynamic system contains a list of states of that specific dynamic system, which represents the sentences uttered by the corresponding interlocutor. Assuming there are only two interlocutors, the x-axis and the y-axis each represent one of the dynamic systems. The value of each cell in the CRP represents the convergence between an utterance of each of the interlocutors, corresponding to the row and column of the cell. Hence, it is possible to apply RQA to conversational data, and thus the tech-nique can provide insights in how the utterances of interlocutors converge during a dialogue.

The first researchers to apply RQA in linguistics were Dale and Spivey [2005, 2006]. By applying this technique to conversational data between child and caregivers from the corpora contain in the CHILDES Database [MacWhinney, 1992], they tried to identify similarities between utterances of child and caregivers. The RQA was split into two different categories of measures. The first category of measures detect lexical coordination (word sequences), and the second category of measures detect syntactical coordi-nation(word class sequences). The conclusions of the analysis of the lexical coordinationwere that: The lexical recurrence is higher at younger ages, and the recurrence is higher within the same dialogue compared to different dia-logues. Likewise, the analysis of the syntactical coordination showed similar results, although the values were consistently higher compared to the lexical coordination. Finally, the results indicated that bigrams of syntactic classes are temporally organized in local coordination (there is more convergence between utterances that are closer to each other in time). As the results found were retrieved using both RQA and the visualization of the recurrence plot, it can be concluded that it is worthwhile to look at both the qualitative in-formation stored in the CRP, as well as the quantitative inin-formation from RQA.

In contrast to Dale and Spivey [2005, 2006], Angus et al. [2012a] did not display recurrence at a purely syntactical or term-based level. Instead, their recurrence measure worked on a conceptual level. The words in a sentence are classed under terms and stored in a semantic model, where each term represents the general meaning of the word (e.g. house and apartment are closely related in meaning, thus these would be stored under a single term in the semantic model). Every unique word in a sentence is then assigned to one of these terms, which in turn gets stored in an occurrence vector to keep track

of whether a term occurs in an utterance. These occurrence vectors of utter-ances are then compared to calculate the similarity between two utterutter-ances. Angus et al. [2012a] claim that the conceptual level is more accurate than the syntactical- or term-based level, when analysing whether two utterances that share the same subject. This conceptual level is then visualized in a concep-tual recurrence plot. It should be noted that the data set used in their research is not related to language acquisition, but instead originates from the long running Australian television series Enough Rope1_{. In this show the host}

interviews prominent celebrities. Another corpus they used was extracted from the Insight program2_{, which is a show where an audience of}

approxi-mately 40 people discuss a controversial topic. Their analysis showed that this visualization technique can help dialogue analysts with their analysis, as it provides a better representation of the convergence between utterances. Although their results might not be of direct use to analyzing language acqui-sition, their results can still be used for the general purpose of analyzing the convergence between utterances. Angus et al. conclude with a list of various topics that they believe to be crucial for building a visualization tool. Their conclusion is that the conceptual recurrence plots are a useful visualization technique that provides assistance in qualitative assessment of conversations. In the same year, Angus et al. [2012b] publish another paper, describing several multi-participant recurrence (MPR) metrics, which can all be applied to conversation data. To create these metrics they split recurrence into its un-derlying dimensions (time, direction and type). Angus et al. [2012b] then use these to create eight metrics that can be used to analyze recurrence. The use-fulness of these metrics is then demonstrated with three case studies. Their conclusions are: the MPR metrics can help identify critical points in the con-versation and they can help track the topic of a concon-versation.

Essentially combining the methods of the four previous papers, Fern´andez and Grimm [2014] propose a new method showing some resemblance to the previously discussed papers. Similar to Dale and Spivey [2005], they used three English corpora from the CHILDES Database: Sarah from the Brown corpus [Brown, 1973], Abe from the Kuczaj corpus [Kuczaj, 1977], and Naomi from the Sachs corpus Sachs [1983]. In addition, Fern´andez and Grimm seem to use the same division between lexical and syntactical coor-dination. Similar to Angus et al. [2012a], Fern´andez and Grimm [2014] also propose to apply RQA with a measure of conceptual convergence using a se-mantic model. In addition, their method provides a quantitative turn-based recurrence model, and the goal in their paper is to offer tools that could im-prove computational approaches to language acquisition. The analysis of their method shows that turn-by-turn temporal development of dialogue is an important factor in explaining the recurrence patterns that indicate con-versational joint action. The method of turn-based cross-recurrence, will be introduced in the next section.

Using the results of the previously discussed papers, this thesis develops and discusses a visualization tool for CRPs of conversational data. The vi-sualization tool is mainly influenced by the methods described in the paper

1_{Produced by Zapruder’s Other Films Pty. Ltd. and broadcast by Australian Broadcasting}

Cor-poration(ABC).

2

of Fern´andez and Grimm [2014], although the various visualization aspects discussed by Angus et al. [2012a] have been considered as well.

4

Turn-Based Cross-Recurrence

The approach suggested by Fern´andez and Grimm [2014], which is the start-ing point of the work presented in this thesis, will be explained in this section by dividing the approach into three parts. Firstly, the general method is dis-cussed. Secondly, the measure of convergence proposed by Fern´andez and Grimm [2014] will be explained in more detail in Subsection 4.1. Finally, I elaborate on the recurrence measures proposed by Fern´andez and Grimm [2014] in Subsection 4.2. Both the measures of convergence and the recur-rence measures will be implemented in the visualization tool.

In contrast to Dale and Spivey [2005, 2006], which take individual words as the basic units of analysis, Fern´andez and Grimm [2014] suggest that the dialogue turnsshould be the basic units of analysis. A dialogue turn consists of an uninterrupted stretch of speech by one of the speakers. The visualiza-tion tool will work on the assumpvisualiza-tion that there will be a two-party dialogue, with speakers A and B. These turns are stored, in the same sequence as they occur over time, per speaker individually, resulting in A = (a1, a2, ..., an)

and B = (b1, b2, ..., bm). Thus, the CRP will be a grid of n × m where the x

and y axes correspond to each of the turn sequences of A and B. As a result, each cell represents a pair of turns (i,j) where i is the ith element of B (x-axis) and j is the jth element of A (y-axis). Because similarity is a matter of degree, the binary values as suggested by Eckmann et al. [1987] will not work on ut-terances in a dialogue, as humans rarely copy each others utut-terances exactly in a dialogue. Hence, the visualization tool will use values between 0 and 1, to represent the similarity of a pair of turns, where 1 means the two turns con-verge perfectly and 0 means that there is no concon-vergence. The measures of convergence will be discussed in Section 4.1. The methods that are applied in the process of extracting quantitative information from a dialogue using RQAare called recurrence measures. The recurrence measures proposed by Fern´andez and Grimm [2014] have been stored as standard measures in the visualization tool, these measures will be discussed in Section 4.2.

4.1

Measures of Convergence

A CRP of dialogue data has many different features that can be used to calculate the convergence between two utterances. Fern´andez and Grimm [2014] proposed three types of measures of convergence, however only two types of these measures will be used as standards in the visualization tool. First, the lexical convergence which measures converges on word sequences. Second, the syntactic convergence which measures converges on Part-Of-Speech-tag(POS)-tag sequences.

4.1.1

Lexical Convergence

Two types of measures for lexical convergence have been implemented. The first measure uses the number of shared lexical stem n-grams3_{, and the}

sec-ond measure uses the number of shared lexeme n-grams. A lexeme is a pair consisting of a POS-tag and a lexical stem, e.g. <noun, child>. For each turn there is a list of the n-grams it contains, and these lists are then matched. The total number of n-grams that appear in both lists is then normalised by the total number of n-grams in the longest list. Hence, only when two turns con-tain exactly the same n-grams, the measure will return the maximum value of 1.

4.1.2

Syntactical Convergence

For the syntactical convergence there are two types of measures as well. The first measure uses the number of shared POS-tag n-grams, and the second measure uses the number of shared POS-tag n-grams that do not share the same lexical stem. The first measure matches can work with both lexemes and just plain POS-tag matching, whenever lexeme n-grams are used, the measure will simply ignore the lexical part. The last measure effectively disentangles the syntactical and lexical coordination by not considering the n-grams that share the same POS-tag and lexical stem. Because only the POS-tags are considered that are not the result of the usage of the same word in a sentence, the syntactical coordination is no longer influenced by the pos-sible lexical coordination. Thus, whenever two turns are identical on a lexical level, the measure will return a 0, as all the lexemes will be equal in the n-grams.

4.2

Recurrence Measures

Different types of measures can be used within RQA. The simplest measure that has been implemented is the global recurrence rate (RR), which is essen-tially the average value of the CRP. This rate is computed by counting all the recurrence values in the recurrence plot and dividing them by the total num-ber of cells. As this measure does not capture any local coordination, there is a measure that computes the average of the points in the line of incidence (LOI). This is a set in the CRP that contains all of the cells, that correspond to pairs of turns, that are directly adjacent to each other in the dialogue.4 Thus, this line captures the recurrence between an utterance and the utterance that directly follows in the dialogue. This measure can provide information on how a speaker adapts his own speech depending on the utterances directly preceding his own. As the most exact matches are usually found around the line of incidence, the local recurrence measure exists. This measure takes a set of points that are within a range of distance d from the line of incidence, and computes the average of this set. This results in a local recurrence rate,

3_{By default unigrams or bigrams.} 4

Note that this is not only the case for (i, i), if the child has uttered the first turn (i, i − 1) also belongs to the set, while (i, i + 1) belongs to the set if the caregiver has uttered the first sentence.

which differs from the global recurrence rate as this rate can capture tempo-ral structure as long as d remains a small number. The local recurrence rate can be split into three different parts, the LOI rate, the upper local recurrence rateand the lower local recurrence rate. An example of the regions used in the local recurrence rate, with d = 4 is shown in Figure 1. The LOI rate is computed by taking the average of the values contained in the LOI. The up-perand lower rates are respectively the average value of the areas above and below the (LOI). The upper rate represents the convergence of the utterances of the speaker on the y-axis in response to an utterance of the speaker on the x-axis, while the lower rate represents the convergence of the utterances of the speaker on the x-axis in response to an utterance of the speaker on the y-axis.

Figure 1:

Example of cross-recurrence plot with highlighted regions.

5

The Visualization Tool

The approach in building the visualization tool was divided into three steps. First the requirements for the visualization tool were analyzed. This analysis showed the minimum requirements for the tool and input data. Second, the tool was implemented. Third, the tool was tested and evaluated. Steps 1 and 2 of this process will be discussed in Subsections 5.1 and 5.2, respectively. Step 3 will be discussed in Section 6.

5.1

Analysis of Requirements

In order to apply both quantitative and qualitative analysis using a single tool, a certain amount of information is required from the corpus. Next, the visual-ization tool requires the functionality to process this information. Using the CHILDES corpus5 _{some minimum requirements for the visualization tool}

were discovered. These findings will be discussed in the following subsec-tions.

5.1.1

Data Analysis

As the data within the CHILDES corpus consists of both meta-data and dialogue-data, these will first have to be separated. This meta-data should at least contain the names of all participants in the dialogue, while the dialogue-data should contain information about the lexical stems and POS-tags used in the conversation6_{. Secondly the punctuation and symbols should be removed}

from the dialogue-data, as these are not required in the matching process. Thirdly, every lexical part of the utterance that has no matching POS-tag should be removed from the data. Fourthly, all the utterances in the speech data should be split into turns. Next, the plain-text version of the text can be saved into the database for later display within the tool, while the lexeme version can be used to calculate the convergence value for the CRP. Finally, startand end tags will be added to each sentence. The amount of tags added to each side of the sentence, depends on the n-gram measure being used. Every sentence will have n − 1 start tags added to the front of the sentence, and n − 1 end tags added to the back of the sentence. This way, even when the sentence has less than n words, the measure can still be applied while the Unigram models remain unchanged. What follows is an example of the entire data processing. Consider the input to be:

FAT: do you want more soup ?

%mor: mod|do pro|you v|want qn|more n|soup ?

The first line shows the uttered text, while the second line displays the POS-tag—stemcombination per uttered word. The first step of the process is to removes all punctuation, in this case only the question mark is removed:

FAT: do you want more soup

%mor: mod|do pro|you v|want qn|more n|soup

Secondly, the lexemes are created, using both the POS-tags and the stems of the sentence, resulting in:

FAT: <mod,do> <pro,you> <v,want> <qn,more> <n,soup> The final step means adding n−1 start and end tags to the sentence, assuming n = 2. The output of the data processing of this single sentence looks like:

FAT: <start,start> <mod,do> <pro,you> <v,want> <qn,more> <n,soup> <end,end>

5_{The raw text files were used, although the requirements should apply to the xml files too.} 6

5.1.2

Tool Analysis

Next, the minimum requirements of the tool were analyzed. This analysis was performed in the different levels of a CRP. First there is the information that has to be stored in the plot as meta-data. Next, there is the turn-data that should be contained within each cell of the plot. Last, the requirements for the controls of the tool were analyzed.

Meta-data

In order for the CRP to be more useful in the visualization tool, it will have to contain some meta-data. First, the plot should contain the file name. Otherwise, the user would not know which dialogue he is currently looking at. Second it should contain the list of turns of both users, this makes it easier to divide the turns over the axis, and the size of both the x-axis and the y-axis can also be retrieved from the size of the lists. Next, the plot should contain the first speaker, as this is important for determining the line of incidence. Finally, the meta-data contained within the plot can also be used to create a display for the entire conversation.

Turn-data

As all the cells in a plot represent the convergence value of two turns, it should be clear that the turns should contain this convergence value. Additionally, it should contain information on which axis speaks first in that specific turn pair. By saving that data into a turn, the dialogue analyst can easily spot which of the interlocutors is the first speaker of that turn.

Controls

The first of the basic controls that are needed within the tool, are the controls for changing between the corpora, dialogues and measures. The visualization would not be easy to use, if it has to be restarted every time the user wishes to change one of these three parameters. Using these 3 basic controls, the user already has the ability to easily compare the different dialogues within the corpus, while being able to apply different measures. As some dialogues can become quite large, and the size of a monitor is limited a zoom-functionality is necessary. As a last control, there should be a way of changing the distance value d for the local recurrence rate.

5.2

Implementation of Visualization

The visualization tool has been written in Java, and the processed data from the corpus has been stored in SQLLite databases. Although there are numer-ous formats for storing SQL data (e.g. MySQL, Oracle, Microsoft Access, etc.), SQLLite was used because it stores its database in a single file. For each corpus there is a single database, which contains the original text of each dialogue, and for each measure separately a score per dialogue.

The visualization tool contains different modules that are each created for doing a specific task, that should help in analyzing dialogue data. Figure 2 displays a snapshot of the information frame of the tool, and Figure 6 displays the dialogue display module. The most important modules that are visible in the figure will be discussed below.

Figure 2: Snapshot of the visualization tool with highlighted modules. Following

the numbers displayed in the highlighted areas: 1 = Turn information, 2 = Dialogue

information, 3 = Text display, 4 = Display controls

5.2.1

Turn Information

The turn information module displayed in Figure 2.1, shows all the tion that is contained within the selected cell. The module contains informa-tion about the two turns that are being matched. First, the number of the turn in the list is displayed per interlocutor, next to the number of the turn in the entire dialog. Second, the recurrence rate between the two turns is displayed. Third, the text of the utterances in the turn of each interlocutor is displayed, which allows the user to quickly see which utterances have been matched. If one of the text areas is clicked in this module, the text display jumps to the turn in the dialog. Last, if the selected utterance is changed in the dialogue display, the information in this module is updated.

5.2.2

Dialogue Information

The dialogue information module displayed in Figure 2.2, shows all the quantitative information about the currently selected dialogue in the dialogue controls. First, the global recurrence rate is displayed, which is calculated by calculating the average of all the cells in the dialogue display. Next, the local recurrence rateis displayed, this value is calculated by taking the weighted average of the line of incidence, the upper part of the diagonal and the lower part of the diagonal. These values are also displayed in this module.

5.2.3

Text Display

Figure 2.3 displays a part of the text display module, which contains a table that presents the content of the turns in chronological order. Two turns in the table are highlighted with red, representing the turns that are contained within the currently selected cell in the dialogue display. This module can be useful to put the matched turns into context, as these turns do not neces-sarily have to be adjacent. The numbers in the first column represent the the global turn number of the specific sentence in the row, which means that this number represents the turn in perspective to the entire dialogue, and not to a interlocutor.

5.2.4

Display Controls

Figure 2.4 contains the display controls module, which holds the controls for the dialogue selection and the dialogue display module. The dialogue displaymodule is shown in Figure 1 and will be described in the next sub-section. From top to bottom, the display controls module contains three text fields from which the corpus, measure and dialogue can be selected, a check-box for inverting the display colors, the controls for the local recurrence measure and controls for the zoom in the dialogue display. The first field provides the available corpus, while the second field displays the available measures for the selected corpus. The final field lets the user choose one of the dialogs from the corpus, which is then displayed in the dialogue display. The last item that affects the entire displayed dialogue, is the inverse option. By checking this box the colors of the cells in the dialogue display will be inversed. The usefulness of this function will be described in Section 6.2.

Next, there are the controls for the local recurrence measure in the dia-logue display. The first element allows the user to alter the distance d of the measure. The next three checkboxes allow the user to hide the upper local recurrence rate, the line of incidence and the lower local recurrence rate re-spectively. Thus, when the user wishes to only analyze a single direction in the temporal domain, he can choose to disable either the lower or the upper rate. Last, there are the zoom options. The “Reset view” button, resets the view to its default size. The “Zoom in” button allows the user to zoom in on the CRP, while the “Zoom out” button allows the user to zoom out, up to the default starting size.

5.2.5

Dialogue Display

The last module of the visualization tool is displayed in Figure 1, and con-tains the dialogue display. This module is separated from the other modules and has its own frame that can be moved around. It contains the CRP of the currently selected dialog. The values of the cells are represented by a gray-scale color. When the inverse option is not selected, white represents the value 0, and black represents the value 1. By clicking on one of the cells, the corresponding pair of turns in the dialogue is selected and the dialogue informationmodule gets updated.

6

Analysis

As the aim of this thesis was to create a visualization tool that could aid di-alogue analysts, this section will explore to what extent the tool can actually support analysts. Two case studies have been designed to explore the useful-ness of the visualization tool. The first case study presents an overview of the particular usefulness of each module in the tool. The second case study highlights specific features that can be found with the help of the visualiza-tion tool. The conversavisualiza-tional data used in following the case studies comes from the Abe corpus from the CHILDES Database. The two case studies will be discussed in the next subsections.

6.1

Case Study 1: General Overview

6.1.1

Aims

The fundamental task of the visualization tool is to decrease the workload of a dialogue analyst, and to support exploration of the conversational data. Hence, the aim of this case study is to show how the basic design and func-tionality of the visualization tool can support investigation of patterns in con-versational data, and to provide the general ideas behind each module of the tool.

6.1.2

Method and Results

Although the specific implementations of the modules where already pro-vided in Section 5, the specific design choices are still to be discussed. Hence, the design choices and the specific uses of each module will be discussed and illustrated in the following paragraphs.

Display controls

The main controls of the visualization tool can be found in the display controls, although no information of the dialogue is represented here, it is still a very essential module. For all the different features in this module, there was a range of options regarding the specific component to use for each of them. The selection of the corpus, measures, and dialogues is processed by drop-down boxes, because it enables the user to quickly change between the available options. This also allows the user to instantly select

Figure 3: An example of a dialogue display with a hidden upper local recurrence

region.

the dialogue he wants to analyse, without first have to scroll through all of them.

The check boxes for the areas of the diagonal are useful when, for exam-ple, the user only wishes to look for the convergence of the speaker on the x-axis, instead of both, this would be possible by disabling the upper local recurrence region. An example of this functionality is displayed in Figure 3. Another example would be to disable the LOI, as this would result in only measuring the convergence of turns that are not directly linked. The diago-nal width component works by using a slider. This slider allows the value to be incremented and decremented with a set value, while the component also allows the user to just insert a specific value. The ability to increment and decrement the value with a mouse click, made the slider a more preferable option in comparison to a plain text field.

Qualitative analysis

The qualitative information is mostly displayed in the dialogue display module, although it can be argued that the text display module contains some qualitative information as well. The dialogue dis-playis essential to the users in order to apply the qualitative analysis. The qualitative information is produced by the visualization of the cells in the cross-recurrence plot, which allow the user to spot numerous things about the dialogue (e.g. little/much convergence between pair of turns, distribu-tion of the convergence, length of the dialog, etc.). The diagonal is split into three different colors7_{, to clearly display the parts of the diagonal to the user.}

7_{For this case the upper diagonal region is cyan, the LOI is blue, and the lower diagonal region is}

As mentioned in Section 5 the dialogue display is separated from the other modules, because the frame would become too large, if the dialogue display was integrated into the main frame as well. Another useful feature for qual-itative analysis, is the ability to inverse the colors of the dialogue display by checking the inverse option in the display controls module. By applying the inverse option, cells that have a significant difference in value, in compari-son to their surrounding cells, can be more easily spotted. Hence this feature can be used to detect single isolated points. These points are caused by con-vergences between pairs of utterances that occur by pure chance [Webber Jr and Zbilut, 1994], as a result these point in the cross-recurrence plots are usually surrounded by low convergence values. Figure 4 displays a dialogue containing a single isolated point.

Figure 4: Visualization of the Abe012 dialogue combined with the POS Unigram

measure. The left image displays the default display, and the right image displays

the inversed display. Both images have the isolated point highlighted.

As stated above, the text display also shows some qualitative information. Because this module provide a general overview of the entire text within the selected dialogue, this display can provide a quick look into the nature of the utterances (e.g. short/long sentence, many different words, child copying caregivers, gibberish, etc.), and can thus help in deciding if the hypothesis that an user might have made should be further investigated or not. Fur-thermore, this module can improve the efficiency involving the process of checking entire dialogues, because the user has all the relevant data readily available in the tool.

Quantitative analysis

The quantitative analysis can be performed by us-ing the remainus-ing modules: the general information module, and the turn informationmodule. The main difference between the two modules is that the general information module displays only general quantified information (no information of specific turns) about the dialogue, while the turn infor-mationmodule only displays information on a turn-based level. Next, the usefulness of the general information module will be discussed, and the turn

informationmodule will be further explained.

The general information module presents numerous values as discussed in 5. One of the displayed features is the first speaker in the dialogue, this feature is important to an user, since it changes the starting direction of the diagonal in the dialogue display, as is shown in Figure 6. Here the care-giver starts the conversation and the LOI starts horizontally. On the other hand, when the child starts the conversation the LOI will start vertically, as is shown in Figure 7. The second field in this module is the general recur-rence value, which can be used to quickly examine the average convergence in the dialogue. The next components of the module are the values from the diagonal region. These values can assist in determining which side of the dialogue converges more to the other, by comparing the values of the upper diagonal recurrence and the lower diagonal recurrence. Combining both the global and the local recurrence values with the dialogue display can provide a good impression of the average spread of the convergence in the dialogue. In conclusion, it should be noted that all values in this module are reduced to five decimal places in the display, the differences between the values after this point become insignificant to the overall results, and by allowing just five decimals a good readability is maintained.

The turn information module displays more details about the currently selected turn. This information consists of features such as: the turn num-bers of each speakers, the utterances of the speakers, and the convergence value of the pairs of turns. By also showing the number of the turn in rela-tion to the entire conversarela-tion, the module allows the user to easily look up the corresponding utterances in the text display. To speed up this process, one of the text areas containing the utterances of one of the speakers can be clicked to set the view of the text display to the clicked turn. By providing this functionality, the module provides a quick way of retrieving the context around the turn. The convergence value displayed in this module is retrieved from the dialogue using the current measure, and it allows the user to see the exact convergence value instead of just the gray-scale color in the dialogue display.

6.2

Case Study 2: Feature Detection

6.2.1

Aims

One of the possible activities of a dialogue analyst is to discover certain fea-tures or patterns in a conversation. Because these feafea-tures might describe certain conversational behaviours, identifying such features in a conversation between a child and its caregiver can possibly lead to a better understanding of language acquisition. Thus, the aim of this case study is to determine what features can be detected in the cross-recurrence plot by using the devel-oped tool, and to what extend the meaning of these features can be explained. Three features have been detected using the visualization tool, and each of these features will be discussed in a separate case, in an attempt to explain how to identify the features in dialogues, and to explain their significance for the analysis of conversational data. Firstly, the method of analysis will be described in the next subsection. Subsequently, the three features will each be discussed individually.

6.2.2

Method and Results

This case study used the visualization tool to discover features in the dia-logues of the corpus. All of the measures that were used in this case study are displayed in Table 1. This table presents the measures sorted by the type of coordination each of them can measure. For each feature there will be a discussion about which measure, or combination of measures, is best used to detect them. Additionally, each feature will be evaluated on its specific use to a dialogue analyst. Once these topics have been discussed, a possible explanation will be provided for the presence of a each feature.

Lexical coordination

Syntactical coordination

Lexeme Unigrams

Postag Bigrams

Lexeme Bigrams

Postag Bigrams w/o Lexeme Bigrams

Table 1: Display of the convergence measures used in Case Study 2, sorted by their

type of coordination.

High recurrence rate (Hypothesis: Repetition)

One very distinct feature of a dialogue is a very high recurrence rate on the LOI. Provided that the average of all dialogues combined is around 0.05 per measure8, some dialogues have a value of 0.326099. In the Abe dialogue corpus there are two dialogues (Abe009 and Abe010) clearly showing this features, both dialogues share an occurrence called the ”repetition game”. This is a game between caregiver and child, where the child has to repeat what the parent says. Hence the hypothesis is, that this high recurrence rate is caused by a high amount of exact repetition in the dialogue.

This hypothesis is further supported by the fact that the upper recurrence rateis consistently higher than the lower recurrence rate in all methods. Because the child’s utterances are always on the y-axis, this shows that the child’s utterances converge stronger to those of the the caregiver, than the caregiver’s utterances converge to those of the child. In Table 2 the recurrence values of one of these two dialogues is displayed for all measures.

The next support to the hypothesis comes from the high value of the LOI in the Lexeme Bigrams measure. Taking into account that the repetition in the turns not adjacent in the dialogue is significantly lower, as is displayed in Table 2, it can be concluded the the high recurrence value on the LOI was caused by the child repeating the directly preceding utterances of his caregiver. The results of this measures can be combined with the clearly visible differences between the values of POS-tag-measures and the values of the POS-tag-without-Lexeme-measures in this dialogue. When the latter type of measures show a lower value than the POS-tag measures, it is because only a few POS-tags match when the lexical stems are not allowed to be identical. Given that the values in the POS-tag measure are high, it can be concluded that many lexical stems did in fact match in the turns on the LOI.

8_{The Unigram models usually have a value around 0.10, while higher n-grams usually have}

values between 0.02 and 0.05

Convergence measure

Global

Local

Upper diagonal

LOI

Lower diagonal

Lexeme Unigram

0.07304

0.07304

0.07313

0.31286

0.05683

Lexeme Bigrams

0.02398

0.02398

0.02261

0.18085

0.01485

POS-tag Bigrams

0.14946

0.14946

0.15710

0.32609

0.12969

POS-tag Bigrams w/o

Lexeme Bigrams

0.12336

0.12336

0.13265

0.13905

0.11334

Table 2: Table displaying the recurrence values of the Abe010 dialogue for several

convergence measures. Distance d for the diagonal is maxed, in order to contain

the entire dialog. Hence, the local recurrence rate is equal to the global recurrence

rate.

Hence, it can be concluded the the recurrence on the LOI is mostly caused by the lexical coordination between the child and caregiver.

Although the high recurrence rate on the LOI can be explained by lexi-cal coordination, the recurrence in the upper and lower diagonal region can not. Clearly visible in Figure 5, is that the utterances, that are several turns apart in the dialogue, show convergences with the POS-tag measures, but not with the Lexeme measures. Thus, it can be concluded that these turns share a syntactical coordination, instead of a lexical coordination. The addi-tional hypothesis about the repetition game that follow from these findings, is that the utterances of the caregiver share a common syntactical structure. The evidence for this hypothesis is not directly visible using just quantita-tive analysis, although the additional evidence from the qualitaquantita-tive analysis makes this hypothesis quite convincing.

To sum up, the high recurrence rates found within these dialogues are indeed caused by the repetition game. Although it should be noted that the high recurrence rates have different causes depending on the distance from the LOI. On the LOI, the high recurrence is mostly caused by lexical coordi-nation, while the high recurrence in turns that are not adjacent in this dialogue is mostly caused by syntactical coordination.

Differences in upper- and lower recurrence rate

Another clear fea-ture that is visible in many dialogues, is the upper local recurrence rate being higher than the lower recurrence rate, or vice versa. This type of information is hard to retrieve from the visualization, but easily retrieved from the quan-tification. The difference in upper local recurrence and lower local recurrence is usually larger if the width d, of the diagonal, is kept low. Although it must be noted that choosing a too low value for d can give a wrong interpretation of the results, as too few points would be included into the diagonal to give a good representation of the local recurrence rate. For the next example,

Figure 6 shows an example of a dialogue where the upper local recurrence rate is higher than the lower local recurrence rate, the diagonal has been set to the maximum width to ensure that no important information is left out of the computation. The resulting values are displayed in Table 3. The first thing that can be noticed from the table is that the lower local recurrence rate of the Lexical Bigram measure equals 0. Although this values is very low,

Figure 5: Visualization of the Abe010 dialogue. All visualization have d = 5. Top

left is the POS Bigram measure, top right is the POS Bigram w/o Lexeme Bigrams

measure, and bottom left is the Lexeme Bigrams measure.

the results of the other measures can provide an explanation. Comparing the POS-tagmeasure with the POS-tag-without-Lexeme measure shows that both have equal results for the lower diagonal, which means that all the recurrence in the lower diagonal is from a syntactical nature (instead of a lexical nature which would be needed for a high Lexical Bigram value). Hence, it can be concluded that because the convergence in the lower diagonal is purely syntactical on a bigram level. Another thing that can be noticed is that, apart from the Lexical Unigram measure, the upper local recurrence rate is always higher than the lower recurrence rate. A possible explanation for the results of the Lexical Unigram measure being different compared to the remaining measures, follows from the fact that words such as “the”, “oh”, and “it” are

Figure 6: The visualization of the Abe006 dialogue using the POS-bigram w/o

Lexeme Bigrams

measure, with d = 17 (maximum width of the diagonal).

also considered in the matching process. Because these types of words can occur in combination with many other words, these words are less significant when using higher n-gram measures. As has been noted, the chance of these words occurring in the same n-gram with two short utterances is quite low. In conclusion, the hypothesis for the occurrence of this feature is that the difference between the upper- and lower local recurrence rate is caused by one of the interlocutors more strongly adapting its own speech depending on the utterances of the other interlocutor. In this specific example, the child and caregiver mostly show syntactical coordination, while the child is more strongly adapting his linguistic patterns to the preceding utterances of the caregiver.

Convergence measure

Upper diagonal rec.

Lower diagonal rec.

Lexeme Unigram

0.02535

0.04289

Lexeme Bigrams

0.00321

0.00000

POS-tag Bigrams

0.04951

0.03547

POS-tag Bigrams w/o Lexeme Bigrams

0.04620

0.03547

Table 3: Upper and lower recurrence values of the Abe006 dialogue, with d = 17

(maximum width of the diagonal).

Multiple black dots in a row

Using the dialogue display in combi-nation with a POS-n-gram measure, black dots (cells with a value of 1) on a straight line appear in most dialogues. Further investigation showed, that these black dots are short turns consisting only of acknowledgements or backchannels uttered by the caregiver or child (e.g. “uhhuh”, “right”, and “okay” are the most common). The only criteria for these dots to ap-pear on the screen, is that both participants in the conversation needs to have uttered at least one of these type of phrases. Once these dots form a contin-uous streak, as displayed in Figure 7, you can trace these utterances back to

the LOI very easily (moving either horizontally or vertically, depending on the direction of the line in comparison to the LOI). If the line of black dots is vertical, and actually consists of acknowledgements or backchannels, the caregiver is telling something while the child is giving the caregiver notifica-tions that he/she is still listening. If the line is horizontal, it is the opposite, and the caregiver is notifying the child that he/she is still listening.

Figure 7: The visualization of the Abe002 dialogue using the POS-bigram measure,

with d = 5. The left image displays a row of black dots in the first column, while

the right image displays its inversed version.

In case of doubt, the text display or the turn information module can al-ways be used for verification. A screenshot of the text module of the tool, displaying the text belonging to the dialogue displayed in Figure 7 is shown in Figure 8. By using the qualitative information contained within the vi-sualization tool, this type of information is easy to retrieve. On the other hand, this feature would be hard to notice if only the quantitative informa-tion was used, as the convergence between the two utterances that are linked in a turn where one of the participants is talking while the other is listen-ing, is very low. The lines of black dots can be found with every method, although the POS-tag measures yield the best result, because the acknowl-edgements or backchannels share the same POS-tag (Communicator (CO)). The line of black dots is easier to detect when enabling the inverse function-ality (although the black dots will now be white dots), as it greatly improves the contrast in comparison to the default view. The right image in Figure 7 displays the inversed view.

7

Conclusions and Future Work

In this thesis a tool for visualizing cross-recurrence plots was introduced. The purpose of this visualization tool is to assist dialogue analysts, with both the qualitative and the quantitative analysis of conversational data using RQA. The techniques behind this tool were then further explained, and in the

pre-Figure 8: Screenshot of the visualization tool, displaying the selected test from the

dialogue displayed in Figure 7

vious section several applications of the visualization tool were dicussed. From these case studies it can be concluded that the tool can assist dialogue analysts with several tasks. Firstly, the visualization tool can provide more insight into the conversational data by providing a comprehensible display of both qualitative and quantitative analysis. Secondly, it can assist an analyst with the detection of various features within a dialogue.

As a result of the modularity of the visualization tool, it should be easy to add new corpus information and new measures by minor adjustments to the reader and matcher. It would also be interesting to see whether the same features can be found within other corpora, and to see whether other types of conversational data between children and caregivers can provide more fea-tures. For instance, one could look whether the linguistic coordination be-haves differently at an older/younger age, or whether the same type of co-ordination occurs with human-computer conversation corpora. Another in-teresting feature would be to apply new kinds of convergence measures that focus on different aspects of an utterance. For instance, Danescu-Niculescu-Mizil and Lee [2011] has recently investigated linguistic coordination within fictional conversation data (movie dialogues), and concluded that these dia-logues display linguistic convergence as well.

Finally, the tool itself could be further improved. The dialogue display could be updated to display its recurrence with a color display, and the func-tionality of applying two measures as once for easy comparison could also be implemented in the future. Furthermore, the tool could be equipped with an XML-file reader, as the tool is currently only capable of reading plain-text file. As several corpora are stored in XML format, this could potentially increase the amount of usable databases.

References

Robert Amar and John Stasko. A knowledge task-based framework for de-sign and evaluation of information visualizations. In Information Visual-ization, 2004. INFOVIS 2004. IEEE Symposium on, pages 143–150. IEEE, 2004.

Daniel Angus, Andrew Smith, and Janet Wiles. Conceptual recurrence plots: Revealing patterns in human discourse. Visualization and Com-puter Graphics, IEEE Transactions on, 18(6):988–997, 2012a.

Daniel Angus, Andrew E Smith, and Janet Wiles. Human communication as coupled time series: Quantifying multi-participant recurrence. Audio, Speech, and Language Processing, IEEE Transactions on, 20(6):1795– 1807, 2012b.

Susan E Brennan and Herbert H Clark. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22(6):1482, 1996.

Roger Brown. A first language: The early stages. Harvard U. Press, 1973.

Michelle M Chouinard and Eve V Clark. Adult reformulations of child errors as negative evidence. Journal of child language, 30(3):637–670, 2003.

Eve V Clark. First language acquisition. Cambridge University Press, 2009.

Rick Dale and Michael J Spivey. Categorical recurrence analysis of child language. In Proceedings of the 27th annual meeting of the cognitive science society, pages 530–535, 2005.

Rick Dale and Michael J Spivey. Unraveling the dyad: Using recurrence analysis to explore patterns of syntactic coordination between children and caregivers in conversation. Language Learning, 56(3):391–430, 2006.

Cristian Danescu-Niculescu-Mizil and Lillian Lee. Chameleons in imagined conversations: A new approach to understanding coordination of linguis-tic style in dialogs. In Proceedings of the 2nd Workshop on Cognitive Modeling and Computational Linguistics, pages 76–87. Association for Computational Linguistics, 2011.

J-P Eckmann, S Oliffson Kamphorst, and David Ruelle. Recurrence plots of dynamical systems. EPL (Europhysics Letters), 4(9):973, 1987.

Raquel Fern´andez and Robert M. Grimm. Quantifying categorical and con-ceptual convergence in child-adult dialogue. To appear in Proceedings of CogSci 2014, the 36th Annual Conference of the Cognitive Science Soci-ety, 2014.

Stanley A Kuczaj. -Ing,-s, and-ed: A study of the acquisition of certain verb inflections. PhD thesis, ProQuest Information & Learning, 1977.

John D Lewis and Jeffrey Elman. Learnability and the statistical structure of language: Poverty of stimulus arguments revisited. In Proceedings of the 26th annual Boston University conference on language development, volume 1, pages 359–370. Citeseer, 2001.

Brian MacWhinney. The CHILDES project: Tools for analyzing talk. Child Language Teaching and Therapy, 8(2):217–218, 1992.

Martin J Pickering and Simon Garrod. Toward a mechanistic psychology of dialogue. Behavioral and brain sciences, 27(02):169–190, 2004.

Jacqueline Sachs. Talking about the there and then: The emergence of dis-placed reference in parent-child discourse. Childrens language, 4:1–28, 1983.

Michael Tomasello. Constructing a language: A usage-based theory of lan-guage acquisition. Harvard University Press, 2009.

Edy Veneziano and Christophe Parisse. The acquisition of early verbs in french: Assessing the role of conversation and of child-directed speech. First Language, 30(3-4):287–311, 2010.

Charles L Webber Jr and Joseph P Zbilut. Dynamical assessment of phys-iological systems and states using recurrence plot strategies. Journal of Applied Physiology, 76(2):965–973, 1994.

Joseph P Zbilut and Charles L Webber Jr. Embeddings and delays as derived from quantification of recurrence plots. Physics letters A, 171(3):199–203, 1992.