Sentence meaning in Conceptual Spaces
Suzan Zuurmond 10791167
Bachelor thesis Credits: 18 EC
Bachelor Opleiding Kunstmatige Intelligentie
University of Amsterdam Faculty of Science Science Park 904 1098 XH Amsterdam Supervisor dr. M.A.F. Lewis
Institute for Logic, Language and Computation Faculty of Science
University of Amsterdam Science Park 107 1098 XG Amsterdam
Abstract
The contribution of this research is to clarify the ideas given in (G¨ardenfors, 2014) in such a way that they can be implemented computationally. In his book, G¨ardenfors gives a systematic way of describing nouns and verbs, among other word classes. However, the book does not provide detailed examples of word representations and the description of how these words interact to form sentences is not so clearly formulated.
This research analyses G¨ardenfors’s theory of conceptual spaces and provides detailed exam-ples of noun and verb representations. These representations are then combined to represent sentences in conceptual spaces. Using a set of simple sentences, the implementation is evalu-ated by creating a two-dimensional visualisation of a sentence space using Principal Component Analysis (PCA) for sentence comparison.
In conceptual spaces, nouns refer to object categories represented as convex regions and names refer to objects represented by points in these regions. Verbs refer to the force or result vector (not both) of an event, and have a magnitude, a point of origin and a direction. Sentences express a construal of an event, and at least contain one vector (force or result) and one object. Analysing these descriptions leads to representing the words in the sentence set as follows: the aspects of objects, coloured dots, can be represented as dots in a colour domain (existing of Hue, Saturation and Lightness dimensions) and a spatial domain (existing of a x-, y- and z-co¨ordinate). Verbs, pushing and pulling motions, can be represented as a vector in a force domain (existing of a x-, y- and z-co¨ordinate).
To represent a sentence, all domains (including the prominence weights) of the agent and patient space should be multiplied with the Cartesian product, to form a point in the sentence space. To visualise this, and to mimic the relevancy weights of the domains, PCA can be used. That the sentence representation and visualisation are implemented rightly is derived from seeing similar sentences visualised near each other.
Contents
1 Introduction 5 2 Theoretical Foundation 7 2.1 Conceptual Spaces . . . 7 2.2 Word Classes . . . 7 2.2.1 Nouns . . . 7 2.2.2 Verbs . . . 9 2.3 Sentences . . . 9 3 Analysis 10 3.1 Data set and Domains . . . 103.2 Implementation details . . . 14 4 Evaluation 18 5 Conclusion 23 5.1 Discussion . . . 23 5.2 Conclusion . . . 24 6 Future Research 25 References 27
Acknowledgement
I want to thank Martha Lewis for her supervision, her help and ideas during the whole process of this project. Her broad knowledge about Natural Language Processing in general and Conceptual Spaces in specific made her able to push me in the right direction when needed and provided me with a good theoretical background.
1
Introduction
How can we represent the meaning of words? And how can these words be combined to form sentences? These are important questions in Computational Linguistics (CL), and also forms the basis to create human-level artificial intelligence that understands and forms language (Schubert, 2019; Jurafsky & Martin, 2014). Creating a meaning representation is important, since it can greatly improve Natural Language Processing (NLP), which can better automatic translation, question answering and text summarisation (among other things) (Clark & Pulman, 2007; Derrac & Schockaert, 2015). An advantage of basing NLP on meaning is that it is more similar to how humans process language (G¨ardenfors, 2004) and it will, potentially, give a more understandable insight in the working of these NLP implementations, than for instance neural networks that are hard to explain (Koh & Liang, 2017).
The way of representing meaning in CL can be split in two: a distributional and a symbolic approach (Clark & Pulman, 2007; Coecke, Sadrzadeh, & Clark, 2010). Clark clarifies that the distributional approach uses large corpora and statistics to represent the meaning of words with vectors based on context, while the symbolic approach uses logic to express the meaning of sen-tences by combining the meaning of its constituent parts. The difference between the two is the focus on the meaning of individual words, versus focus on the combination of words (Clark & Pulman, 2007). This distinction is thus related to the two questions asked above: How can we represent the meaning of words? And how can these words be combined to form sentences?
Early CL research mainly focused on answering the first question (Gasparri & Marconi, 2016). The most common way of representing words is with Vector Space Models (VSM) (Sahlgren, 2006; Turney & Pantel, 2010). These models are firstly introduced by Hinrich Sch¨utze, who says the following: ”Vector similarity is the only information present in Word Space: semantically related words are close, unrelated words are distant” (Sch¨utze, 1993). To clarify; these VSM’s use context, being the text around a word, to represent the meaning of words with vectors (Turney & Pantel, 2010). Putting these vectors together in a space, makes it possible to compare meanings of different words (Sch¨utze, 1993).
Since SVM’s are limited to single words (mainly nouns, adjectives and verbs) and do not capture the meaning of sentences, more recent research focused on compositionality (Socher et al., 2013; Baroni & Zamparelli, 2010) to find an answer on the second question. Smolensky, for instance, uses tensor products to multiply vectors representing fillers (the individual words in a sentence), with vectors representing associated roles (the sequence positions) (Smolensky, 1990), on which other researchers build on (Clark & Pulman, 2007; Coecke et al., 2010). Next to Smolensky’s approach, there are also Neural Network approaches to represent compositionality, such as Skip-Thought Vectors (Kiros et al., 2015) and the Recursive Neural Tensor Network (Socher et al., 2013).
Another way of representing words and sentence meaning is with conceptual spaces. G¨ arden-fors uses his theory of Conceptual Spaces to represent the meaning of concepts and events (G¨ardenfors, 2004; G¨ardenfors, 2014). His theory is based on human reasoning, instead of a set of linguistic rules, which starts from the idea that humans represent concepts geometrically. A concept can be described in a conceptual space that is formed by different fundamental domains such as space, colour and motion. Concepts can then be seen as a set of convex spaces in the relevant domains. That these spaces are convex is important, since this makes it possible to say something about similarities and other objects. To clarify, if two points in the colour space represent green, we know that all points in between these points also represent green. Thereby, concepts that are similar to each other will also be represented closer to each other in space: the convex space representing green will be closer to yellow, than to red.
This research will elaborate on G¨ardenfors’s meaning representations using conceptual spaces. In his book The geometry of meaning: Semantics based on conceptual spaces (G¨ardenfors, 2014) G¨ardenfors gives a systematic way of describing nouns and verbs, among other word classes. However, the book does not provide detailed examples of word representations and the descrip-tion of how these words interact to form sentences is not so clearly formulated. This research will analyse G¨ardenfors’s theory and tries to provide detailed examples of noun and verb rep-resentations. Thereby, this research tries to combine these examples to represent sentences in conceptual spaces. At last, the implementation of G¨ardenfors’s sentence representation is evalu-ated using a small dataset and Principal Component Analysis (PCA), which is used to make a two-dimensional visualisation. This leads to the research question:
Based on G¨ardenfors’s description of words and sentence meanings, how can word representations be combined to represent sentence meanings in Conceptual Spaces?
To get an answer to this question, multiple steps are taken corresponding to the chapters in this thesis. Firstly, G¨ardenfors’s theory about nouns, verbs and sentences is given in the Theoretical Foundation. Secondly, his theory is analysed and narrowed down to concrete examples of word and sentence representations in the Analysis. Thereafter, the representation of sentences is evaluated by visualising and comparing multiple sentences in a two-dimensional space, which is created with Principal Component Analysis. This step is explained in the Evaluation. From that, an answer to the research question is formed, given in the Conclusion. This chapter also includes points of discussion. At last, this thesis will suggest Future Research.
2
Theoretical Foundation
The next section will explain the theory given by G¨ardenfors in his books Conceptual spaces: The geometry of thought (G¨ardenfors, 2004) and The geometry of meaning: Semantics based on conceptual spaces (G¨ardenfors, 2014). In these books Gardenf¨ors introduces his theory of Conceptual Spaces, in which the meaning of words and sentences can be given. The next section will explain the relevant parts of his theory for this research.
2.1
Conceptual Spaces
To represent words, G¨ardenfors introduces conceptual spaces. These spaces consist of dimensions that have geometric or topological features, like shape, weight or location. These dimensions are then sorted into domains. For instance, the domain ’taste’ could exist of the dimensions sweetness, sourness, saltiness, bitterness, and umami. He uses these spaces to describe different word classes, which will be explained in the next section.
2.2
Word Classes
In linguistics, word classes are used to put words into distinct groups with different semantic and syntactic functions. In English there are traditionally eight classes: nouns, pronouns, adjec-tives, verbs, adverbs, prepositions, conjunctions and interjections. The aim of G¨ardenfors is to introduce a cognitively grounded theory of events that constrains the semantics of word classes. The second part of his book The Geometry of Meaning Gardenfors (G¨ardenfors, 2014) de-scribes a semantic analysis of these word classes. Important is to note that this analysis is syntax-free, which means that the semantic notion should not depend on any grammatical cate-gories.
The next section will elaborate on this syntax-free analysis concerning the two word classes that are relevant for this research: nouns and verbs. For both these word classes a summary of Gardenfors’ theory is given, which will form the theoretical foundation of this research.
2.2.1 Nouns
Before describing G¨ardenfors’ view on noun representation it is important to know what makes a word a noun. The Cambridge dictionary describes a noun as a word that refers to a person, place, thing, event, substance, or quality. It includes a great variety; woman, tree, happiness, sunset, danger, laughter, doctor all fall into the noun class. It thereby is the largest word class of all eight classes (https://dictionary.cambridge.org/dictionary/english/noun, n.d.).
G¨ardenfors starts his theory from the idea that nouns contain information about several domains. For instance, the word ‘apple’ contains information about its size, its colors, its taste, etcetera. All these different parts of information are combined in what G¨ardenfors calls object categories. This is in contrast with adjectives, another word class, that only refer to a single domain. For example, ‘red’ only refers to the color domain and ‘sweet’ to the taste domain.
Object categories
Gardenfors describes object categories as concepts that are used to classify objects, that is, nouns. Although object categories do exist of the different domains that describe an object, they can not be seen as merely bundles of properties. He describes that there are two kinds of aspects of object categories that should be taken into account to form a complete description of an object: domain and meronomic aspects.
At first, G¨ardenfors notes the domain aspects. He states that while we learn new things about objects the set of domains can be expanded. Also, dependent on the context, particular domains become more or less important. For instance, eating an apple makes its taste domain more important, while letting an apple fall will make its weight more important. This is why Gar-denfors suggests to add prominence weights of the domains. Next to prominence, the suggested representation for an object category also includes information about the correlations between the regions. For instance, the colour domain of an apple correlates with its taste domain, since the redness of an apple says something about its sweetness.
Secondly, G¨ardenfors discusses the meronomic aspects. Meronomy can be described as the relations between an objects and its parts. For instance, the relationship between an apple and its seeds, skin, flesh, core and stem.
In short, G¨ardenfors describes nouns as object categories that contain information in different domains, including domain and meronomic aspects.
Objects
Gardenfors describes objects as a special kind of category. When we talk about a particular object, for instance a certain apple that lies on the table and is about to fall, we refer to something different than when we talk about liking an apple with breakfast. In the latter case we talk about the concept of an apple. We do not eat the same apple each morning, but a new similar one.
Where nouns refer to object categories, names refer to objects. Object categories represent a concept that extend to regions in different domains. For instance, the color of an apple will be represented by a region of green, red and yellow in the color domain; the taste of an apple will be a region that covers both sweet and sour. These regions are always convex, that is, every point in these regions can be connected with a line that lies completely in the region.
Objects can be seen as very narrow object categories. A particular apple may have a particular color of red that lies in the region of apple colors and a particular sweetness that lies in the region of apple tastes. Thereby, an object can be reduced to a point in the convex spaces that describe the corresponding concept.
In short, we can see nouns as object categories represented as convex spaces and names as objects represented by points in these spaces.
2.2.2 Verbs
To understand the semantics and representations of verbs, it is necessary to understand Gar-denf¨ors’s model of events based on conceptual spaces. This section will explain Gardenf¨ors’s two-vector model of events and notes how he bases the semantics of verbs on this model.
Describing an event is as describing what is happening. A prototypical event has an agent, that generates a force vector (or force pattern), which causes a change in the state of a patient (or a path of changes). For instance, ’Jan kicks a ball’ describes an event where Jan is the agent that generates a force vector (via kicking the ball). This force vector changes the state (the location) of the patient, in this case the ball. Gardenf¨ors proposes that it is necessary for an event to contain at least two vectors and one object; a force vector causing the change and a result vector representing the change of an object.
Important in an event are both the roles of the agent and the patient. To model an event, two spaces are needed: the agent and the patient space. Both these spaces exists of the domains that are relevant to the event that is modelled. For instance, eating an apple will make the taste domain more relevant, than painting an apple. Thereby, an agent space needs to at least contain a force domain, in which the force vector can be represented.
The paragraphs above describe the most basic events. As Gardenf¨ors notes, for describing more complex events, other domains, roles and forces could be added. For instance, most events are intentional, which could be modelled by adding a goal domain to the agent space. Thereby, the patient could be very heavy and create a counterforce, or the agents needs to use an instrument to perform an action. These are a few, of many, reasons why a two-vector model could be extented. Similar to how object categories are narrowed down to specific objects, the description of one result vector could be extended to a space that represents general event categories. Gardenf¨ors notes that these spaces, in contrast to concepts, are not necessarily convex.
Using this two-vector model, Gardenf¨ors gives his thesis about verbs: they refer to either the force or result vector (not both) of an event. These vectors have a magnitude, a point of origin and a direction.
In short, we know verbs are represented as either the force or result vector of an event.
2.3
Sentences
The last section explains the analysis of nouns and the semantics of verbs. This research aims to look at how these are put together. G¨ardenfors notes that when he is talking about sentences, he is actually addressing utterances. Utterances include all spoken words or statements and are dependent on communicative context, while in philosophy and much of linguistics sentences are seen as independent of context.
G¨ardenfors starts from the basic idea that sentences express events and a description of an event is based on a construal. Thus, for describing an event, interpretation plays an important role. G¨ardenfors narrows important aspects of interpretation down to attention, perspective, categorization and common ground. G¨ardenfors’ thesis about construals of events is that they at least contain one vector (force or result) and one object. From this notion he then forms a thesis about sentences: ‘A (declarative) sentence typically expresses a construal of an event’ (G¨ardenfors, 2014).
Where it is complex to make a mental model of an event, a sentence (or utterance) only captures certain aspects of an event, dependent on a particular focus. The same event can be described by different sentences and thereby different construals: ‘Eva hits Jan’ and ‘Jan is hit by Eva’ describe the same event, but Eva and Jan are, respectively, put in focus.
In short, the central components of a sentence are explained by the model of events and the thesis about construals.
3
Analysis
This section will analyse Gardenfors’s theory, described in the Theoretical Foundation, and narrow it down to individual word and sentence representations. At first, it is discussed which domains are relevant for the used data set. Then, G¨ardenfors’s view on nouns, verbs and sentences is reduced to clear requirements.
3.1
Data set and Domains
The data set will exist of different combinations of two objects and one verb formed into sen-tences. The objects will represent dots of colour in an Euclidean space, while the verbs represent a pushing or pulling motion. Examples of sentences are ‘Blue pushes Red rightwards’ and ‘Green pulls Orange from the left’. The next section will discuss the different domains and their dimen-sions that are used to represent the sentences.
Colour domain
G¨ardenfors describes that our cognitive representation of colours can be divided in three impor-tant dimensions: hue, intensity and brightness. The hue dimension refers to the colour circle, intensity refers to a range from grey to a greater colour-intensity and the brightness refers to a range from white to black(G¨ardenfors, 2014). The HSL colour space also exists of three di-mensions resembling the same things: hue, saturation and lightness. Figure 1 illustrates the similarities between both G¨ardenfors’s colour spindle and the HSL colour cone.
The last paragraph explains why the three dimensions hue, saturation and lightness are used to represent the colour of the objects. The hue dimension is given by a number between 0 and 360, corresponding to the colour circle. In this circle 0 corresponds to red, 120 to green and 240 to Blue. Saturation is given by a number between 0 and 100, where 0 corresponds to a grey colour and 100 to a very bright colour. Lightness is also given by a number between 0 and 100, where 0 corresponds to black and 100 to white. This means that the all colours for this research can be represented in the colour cube of figure 2.
Concluding, the colour domain described by G¨ardenfors can be represented as the three dimensional space consisting of hue, saturation and lightness.
Figure 1: In this figure, the similarities between both G¨ardenfors’s colour spindle and the HSL colour cone are shown. Left: G¨ardenfors’s colour spindle including Hue, Intensity and Bright-ness (G¨ardenfors, 2014). Right: the HSL colour cone including Hue, Saturation and Lightness (Commons, 2005)
Figure 2: The three dimensional colour space existing of Hue, Saturation and Lightness dimen-sions. The crosses represent individual colours in the colour space.
Spatial domain
Next to a colour each dot will have a place in an Euclidean space. This space will be relative towards the agent, the object that is pushing or pulling. This means that the agent will always be in the centre of the space at coordinates (0, 0, 0), while the patient can be at different locations. For simplicity, a space is created that expands 10 steps in each direction. These steps do not represent any particular distance like meters, feet or inches. The used spatial domain is represented in figure 3.
Force domain
The force domain is used to represent verbs. As mentioned in the Theoretical Foundation, verbs can be seen as vectors with a magnitude, a point of origin and a direction. The verbs used in this research are pushing and pulling motions.
Regarding the sentences in the data set, we state that the sentence ’Blue pushes Red right-wards’ represents a motion with the location of the Agent (0, 0, 0) as the point of origin and a rightwards direction away from this point. The magnitude, for now, is not clearly defined in the sentence and thereby will differ from 0 to 10.
The sentence ‘Green pulls Orange from the left’ represents a motion with the location of the Patient as the point of origin, which can be any point the in the left halve of the spatial domain, towards the location of the Agent (0, 0, 0). The magnitude, for now, is not clearly defined in the sentence and thereby will differ from 0 to 10.
In this research a distinction is made in different pushing directions: left-, right-, for-, back-, up- and downwards. These directions are used to evaluate similarities in the sentences. A similar distinction is made for the pulling motion, where the direction determines the position of the
Figure 3: In this figure, the three dimensional spatial space is showed existing of the x-, y-and z-coordinates. Note that these dimensions do not correspond to any particular distance like meters, feet or inches. The cross represents the Agent standing in the middle of the spatial space.
Patient relative to the Agent. Examples of these motions are shown in figure 4.
In short, the pulling and pushing motions can be represented by vectors with different points of origins, magnitudes and directions in the force domain.
Figure 4: In this figure, different pushing motions are showed.
3.2
Implementation details
In the Theoretical Foundation G¨ardenfors’s theory is discussed. Before implementing his theory, his view on nouns, verbs and sentences needs to be reduced to clear requirements. This section will elaborate on which steps are taken to reduce G¨ardenfors’s theory and how his theory is interpreted regarding the sample sentences.
Narrowing down a domain of an object category to a domain of an object
Where a big part of G¨ardenfors’s theory is focused on the description of concepts, this research will look at particular objects. As mentioned in the Theoretical Foundation objects are repre-sented by points that lie in the convex space of an object category, that is a concept. Since G¨ardenfors’s description of an object category is clear, we will use this description to reduce it to an object later. G¨ardenfors summarises his theory about object categories in the following definition (G¨ardenfors, 2014).
A category is determined by
1 a set of relevant domains (may be expanded over time)
2 a set of convex regions in these domains (in some cases, the region may be the entire domain)
3 prominence weights of the domains (dependent on context)
4 information about how the regions in different domains are correlated 5 information about meronomic (part-whole) relations.
Representing the first two requirements seem straightforward. As Gardenf¨ors notes, domains exist of multiple dimensions and object categories are described by convex spaces in these domains (G¨ardenfors, 2014). Reducing the object categories to specific objects will mean that objects can be described by a singular point in the corresponding convex space. The last three requirements are harder to determine, since it requires more knowledge about the aspects of concepts and their relation to each other. The prominence weights of the domains can be seen as an extra dimension added to a domain (G¨ardenfors, 2014; G¨ardenfors, 2004). G¨ardenfors gives no clear description of representing requirement 4, but this can be interpreted as something implicit. He does mention that requirement 5 can be seen as a different domain (G¨ardenfors, 2014).
In this research, very simple objects with few aspects are used, which keeps the set of relevant domains small (meeting the first requirement). Also, by using simple concepts as colour and location, the convex regions can be determined precisely (meeting requirement two). These two requirements can thus be narrowed down to points represented in the domains discussed in section 3.1, as shown in figure 6. When describing an object or sentence (which have multiple domains), the third requirement will be added implicitly using Principal Component Analysis (PCA). PCA is a multivariate technique that looks at patterns of similarities to extract the most important features of multivariate data. It then represents the data as a new set of orthogonal variables called principal components (Jolliffe, 2011). The fourth and fifth requirements will not be taken into account, since the fourth requirement is seen as implicit and the simplicity of the sentences makes that there are no meronomic relations.
In short, requirements 1, 2 and 3 will be taken into account for representing objects, while requirement 4 and 5 are left out for this research.
Figure 6: In this figure, the difference between an object category and an object is represented by showing the differences in their colour domain. Left: the colour space of the concept Blue dot. Right: the colour point of a specific Blue dot. The colour of a specific Blue dot should always lie in the color space represented on the left.
Combining multiple domains to form a representation of an object
Different domains can be put together by applying the Cartesian product (G¨ardenfors, 2004). The Cartesian Product of two sets is given by the set of all ordered pairs (Bagaria, 2019). Putting different domains of an object together will lead to the vector showed in figure 7. This vector represents a point in a Conceptual Space consisting of all the dimensions of the domains corresponding to an object. As mentioned in the previous section, adding the weights and combining the different domains will be done with PCA. Note that PCA will produce a two-dimensional vector, that is only a reflection of the higher-two-dimensional vector.
a1 a2 ... an wa × b1 b2 ... bm wb × ... × z1 z2 ... zo wz = a1 a2 ... an wa ... z1 z2 ... zo wz
Figure 7: In this figure, the mathematical representation of an object is given. The different vectors represent the different domains that describe aspects of an object. Each domain has a weight that is added at the end of the vector. If relevant, two of the vectors can describe the correlation and meronomic relations between the domains. All domains are multiplied with the Cartesian product to form one long vector that describes an object.
Combining different domains to form a sentence
From the Theoretical Foundation we know a sentence expresses an event and a description of an event is based on a construal. G¨ardenfors’s thesis about construals of events is that they at least contain one vector (force or result), in this case the force vector.
Events consist of both an agent and a patient space. Following the steps mentioned in the sections above, we can thus represent a sentence by all domains in figure 8. In this figure, the sentence ’Yellow pushes Blue rightwards’ is represented, where the dimensions of the agent are (60, 50, 100), corresponding to the hsl colour dimensions, (0, 0, 0) to the xyz location dimensions and (5, 0, 0) to the xyz force dimensions. The patient space exists of (260, 50, 100) representing the colour dimensions and (0, 0, 0) representing the xyz location dimensions.
In total, we can view this representation as three domains (colour, space and force), where fifteen digits are used to describe both the agent and patient. To form a complete representation, five digits should be added to determine the prominence weights. A sentence representation can thus be seen as similar to figure 8. These weights will be added implicitly using PCA, which forms a visualisation of the multidimensional sentence space. These visualisations are given in the Evaluation.
Figure 8: In this figure, the representation of all domains in a sentence is given. The sentence that is represented is ’Yellow pushes Blue rightwards’. Note that these domains all have an extra dimension: the weight of the domain.
4
Evaluation
By comparing the results of either similar or different sentences, the implementation can be evaluated. This is done with ‘simple’ sentences that have straightforward meaning with colours as nouns and movements as verbs, as described in the Analysis. Principal Component Analysis (PCA) is used to reduce all the dimensions corresponding to the sentences to two component dimensions, making the visualisation and comparison of sentences less complicated.
First of, a set of sentences is generated. Every sentence exists of 15 dimensions: three colour dimensions and three location dimensions for both the agent and patient, and three dimensions for the force vector. See 8 and the section about the Data set and Domains for more information about the domains and their values.
The set of sentences exists of 2000 randomly generated sentences, with 1000 sentences that represent push motions and 1000 sentences that represent pull motions. For the both the pull and push-sentences the colours of the agent and patient is randomly picked from the colour domain shown in figure 2, and the location of the agent is always (0, 0, 0). The difference between the pull and push-sentences is the location of the patient: for push-sentences the location of the patient is always (0, 0, 0), while for pull-sentences it is randomly picked from the spatial domain showed in 3. What is also different between the push and pull-sentences is the force vector: for push-sentences this vector points from the location of the patient towards a randomly picked point in the spatial domain, while the pull-sentences work the other way around, since the force vector will point from the random location of the patient towards the location of the agent (0, 0, 0).
Next to random sentences, there are also 108 specific sentences added to use for comparison. These specific sentences contain strong push and pull motions from and towards all six directions (-10, 0, 0), (0, -10, 0), (0, 0, -10), (10, 0, 0), (0, 10, 0) and (0, 0, 10), which we can think of as representing very typical values for the sentences. Besides the different directions and motions, the specific sentences also contain all colour combinations for both the patient and agent limited to the colours red, green and blue corresponding to a hue of 0, 120 and 240 respectively.
Secondly, this sentence set is used to visualise the sentence space. This space consists of the conceptual space of the agent, the conceptual space of the patient, and the force domain to represent the force vector, which is included in the agent space. This means that it consists of the whole colourdomain (for the colours of both the agent and patient) and the whole spatial domain (for the force vector and location of the agent and patient)
This space is then reduced to a two-dimensional visualisation using PCA. Note that this visualisation is a reflection of the multidimensional sentence space and does not hold all properties of a sentence space. It is used to compare sentence meanings more straightforwardly, and to simulate the prominence weights of the domains. Figures 9, 10, 11, 12 and 13 will highlight different aspects of the (specific) sentences, which will show the importance of the different domains used in the sentence.
At first, the whole set of sentences is evaluated. Figure 9 shows the colours of both the agents (left) and patients (right). The actual colours are represented in the two upper plots. To create a clearer distinction, the lower two plots show the same colours divided into three groups: red, blue and green. Both these plots do not show any particular clustering or distinction in the visualised sentence space. When looking at the directions of the force vectors (the pushing and pulling motions), we do see a distinction. Figure 10 shows the difference between right- and leftwards, for- and backwards, down- and upwards motions.
Secondly, the specific sentences are evaluated. Figure 11, 12 and 13 show that these sentences are clustered into 12 groups. Showing the colours of the agents and patients, as seen in figure 11, seems to indicate that the colour aspect of the agent and patient has a relative small impact on the placement of the sentences in the sentence space. Looking at the location of the patient (before and after the motion), does seem to make a bigger impact on the placement of a sen-tence, since similar locations are clustered together. This can also be said about the kinds and directions of motions, see figure 13. This figure shows that the push motions seem to be more centred, while the pull motions are located on the outside of the sentence space. The directions of motions also seems to be clustered together. In the Conclusion it will be discussed why this seems to be the case.
In this section the different aspects of the sample sentences are compared in a two-dimensional visualisation of the sentence space of push- and pull-sentences. The location of the patient and the direction of the force vector seem to have a relative big influence. This will be discussed in more detail in the Conclusion.
Figure 9: In this figure, the colours of both the agents (left) and patients (right) are shown. The actual colours are represented in the two upper plots. The lower two plots show the same colours divided into three groups: red, blue and green.
Figure 10: In this figure, the directions of the force vectors (the pushing and pulling motions) is shown. It shows the distinction between right- and leftward motions (left), for- and backward motions (middle) and down- and upward motions (right).
Figure 11: In this figure, the colours of both the agents (left) and patients (right) of the specific sentences are showed.
Figure 12: In this figure, the distinction between the locations of the patient before (left) and after (right) the motion of the specific sentences is showed.
Figure 13: In this figure, the distinction between the kind of motions (left) and the directions of the motions (right) of the specific sentences is showed.
5
Conclusion
This section will discuss the analysis and evaluation and give an answer to the research question proposed in the Introduction and note the most important findings of this thesis.
5.1
Discussion
This thesis used G¨ardenfors’s description of words and sentence meanings, to make word repre-sentations and combined these to represent sentence meanings in Conceptual Spaces.
As mentioned in the introduction, G¨ardenfors does give a systematic way of describing nouns and verbs, among other word classes. However, he does not provide clear examples of word rep-resentations and the description of how these words are combined is not so clearly formulated.
The contribution of this research is to clarify the ideas given in (G¨ardenfors, 2014) in such a way that they can be implemented computationally. To receive a clear example of word representations, G¨ardenfors’s theory about object and verb categories is narrowed down to fit the sentence set used in this research. This narrowing down meant regions, representing categories, were reduced to points, making the property of convexity trivial. Thereby, correlations between different domains are not considered, and will be discussed in the Future Research chapter. Information about meronomic (part-whole) relations was also left out.
Taking these steps, which are explained more detailed in the Analysis, did make it possible to create a clear example of word representations. However, it also discarded a lot of the more complex features of language, such as context, correlation and meronomic relations.
What the Introduction also mentioned is that G¨ardenfors uses a different way of represent-ing word meanrepresent-ing than the common Vector Space Models (VSM) (Sahlgren, 2006; Turney & Pantel, 2010; G¨ardenfors, 2014), that is with conceptual spaces. An important feature of these conceptual spaces is that they can possibly be used to represent sentences, as is tried in this research.
For this research, the sentence set is simple and clear: the sentences describe specific ob-jects and verbs, that are represented by single points and vectors, instead of obob-jects and verb categories, that are represented by regions. Since the sentence set consists of non-contextual sentences, the mapping of individual words to sentences was done with Cartesian multiplication, without discarding or emphasising any domain explicitly (the weights that were added by PCA are based on similarities, not on context). This might have led the implementation to be very similar to models that combine VSMs (Clark & Pulman, 2007; Coecke et al., 2010).
While these implementations do seem similar for these sentences, the difference is that con-ceptual spaces could possibly be used to represent other, more contextual sentences, as well. This will be elucidated further in the Future Research chapter.
As is mentioned above, an important aspect of representing categories in conceptual spaces is the convexity of the spaces that represent, for instance, object categories (G¨ardenfors, 2004). However, G¨ardenfors states that convexity is not guaranteed for representing verb categories (G¨ardenfors, 2004). This causes sentence categories to not be guaranteed convex too, which means that a sentence space can not be evaluated by confirming convexity. This has led to evaluating the sentence set on only similarities between sentence meanings.
Since this made the geometrical aspects of the sentence space was less important, comparing sentences in a two-dimensional reflection was found as sufficient evaluation. These reflection were made using Principal Component Analysis (PCA). G¨ardenfors does mention dimensionality reduction briefly (G¨ardenfors, 2004), but does not give a clear description of how to use this. The figures showed in the Evaluation should therefore be seen as a visualisation or reflection of sentence meanings, and not a representation.
While it is not a representation, it does say something about the representation. The sentence representation exists of 15 dimensions (plus five weights), in which similarities can be evaluated using a combination of Manhattan distance and Euclidean distance (G¨ardenfors, 2014). Since PCA reduces all dimensions considering similarities between dimensions, the distance between the points in the figures of the Evaluation should say something about the distance in the sentence representations. This is confirmed by the clusters that were created. In these clusters colour seemed to have less impact on the placement than location and motion. This seems to be corresponding to the emphasises of the sentences, which is the motion and replacement of the patient in the spatial domain. That PCA found this distinction could be explained by the correlation between location and motion, that both do not correlate to the colour of the objects.
5.2
Conclusion
This research used G¨ardenfors’s description of words and sentence meanings, to make word representations and combined these to represent sentence meanings in Conceptual Spaces. This section will state the most important findings that have led to an answer on the research question. G¨ardenfors’s description of words and sentence meanings can be narrowed down to the fol-lowing: nouns refer to object categories represented as convex regions and names refer to objects represented by points in these regions; verbs refer to the force or result vector (not both) of an event, and have a magnitude, a point of origin and a direction; sentences express a construal of an event, and at least contain one vector (force or result) and one object.
Analysing these descriptions leads to representing the words in the data set as follows; the aspects of the objects, coloured dots, can be represented as dots in a colour domain (existing of Hue, Saturation and Lightness dimensions) and a spatial domain (existing of a x-, y- and z-co¨ordinate); verbs, pushing and pulling motions, can be represented as a vector in a force domain (existing of a x-, y- and z-co¨ordinate).
To represent a sentence, all domains (including the prominence weights) of the agent and patient space should be multiplied with the Cartesian product, to form a point in the sentence space. To visualise this, and to mimic the relevancy weights of the domains, PCA can be used. That the sentence representation and visualisation are implemented rightly is derived from seeing similar sentences visualised near each other.
6
Future Research
This section will suggest future research regarding this thesis.
All the following suggestions are related to extending the data set used in this research. As mentioned in the Discussion, the data set that is used exists of clear objects and verbs.
At first, the data set could be extended from describing objects in certain events to object categories in general events. This means that the sentence ’Blue pushes Red rightwards’ will be represented by all possible points of blue and red, represented by convex regions instead of certain points, and all rightwards motions, also represented by a space instead of a single vector. This extension will also make it possible to add other word classes, such as adjectives and adverbs. For instance, using the adjective dark and light for colours will reduce the object category spaces to a smaller and more specific region. Using adverbs as gently and hard will similarly reduce the verb category regions to a smaller and more specific region. ding verbs to verb categories. Representing clear examples of verb categories might also give more insight in retaining convexity for verb category representation.
Extending the sentences with adverbs and adjectives will be a good first extension, since it preserves the domains that are used in this thesis.
Secondly, the sentences could be made even more complex, for instance by introducing instru-ments, multiple forces and extra aspects describing the objects. Adding an instrument means sentences like ’Red pulls Blue with a rope from the left’ are tried to be represented. For in-troducing another object that is used as instrument by the agent, G¨ardenfors’s theory should be analysed and made more clear regarding the representation of instruments. These might be represented by adding another space, next to the agent and patient space, or they could be represented by a domain in the agent space.
Adding multiple forces means sentences like ’Red pushes heavy Blue rightwards’ are tried to be represented. The heaviness of the patient, in this case Blue, will create a counter force. For introducing these counter forces, G¨ardenfors’s theory should, again, be analysed and made more clear regarding the representation of counter forces. These might be represented by adding another force domain to the patient space, which might represent what is happening more clearly, or directly be taken into account in the force domain of the agent.
Adding extra aspects means sentences like ’Round Red pulls Square Blue rightwards’. This seems to be the most simple addition, since it probably would mean another domain is added to either the agent or patient space. However, finding a good representation of aspects other than location and colour could be challenging.
Thereafter, sentences describing very different events could be added to the sentence set. This would mean the sentence space will be expanded and, if this is possible, enable the represention and comparison of different kinds of sentences within one shared sentence space, which is a crucial benefit just as in the approaches of (Coecke et al., 2010) and (Bolt et al., 2017).
Finally, further research could look into how meronomic relations between domains and con-text is represented.
G¨ardenfors does mention that meronomic relations can be represented in a different domain and should describe how, for instance, fingers are related to a hand (G¨ardenfors, 2014). He does not, however, mention clearly how these relations can be represented.
Similarly, G¨ardenfors does give examples of contextual aspects that influence the representa-tion of words and sentences (G¨ardenfors, 2014). For instance, eating an apple will make the taste domain more prominent, than painting an apple. However, he does not clearly formulate how the the representations are influenced. Further research could explore when and how prominence weights are influenced by context, or when a domain is considered fully irrelevant.
References
Bagaria, J. (2019). Set theory. In E. N. Zalta (Ed.), The stanford encyclopedia of phi-losophy (Spring 2019 ed.). Metaphysics Research Lab, Stanford University. https:// plato.stanford.edu/archives/spr2019/entries/set-theory/.
Baroni, M., & Zamparelli, R. (2010). Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 conference on empirical methods in natural language processing (pp. 1183–1193).
Bolt, J., Coecke, B., Genovese, F., Lewis, M., Marsden, D., & Piedeleu, R. (2017). In-teracting conceptual spaces i: Grammatical composition of concepts. arXiv preprint arXiv:1703.08314 .
Clark, S., & Pulman, S. (2007). Combining symbolic and distributional models of meaning. Coecke, B., Sadrzadeh, M., & Clark, S. (2010). Mathematical foundations for a compositional
distributional model of meaning. arXiv preprint arXiv:1003.4394 .
Commons, W. (2005). Color cones representing hsl color space. Retrieved from https:// en.wikipedia.org/wiki/File:Color cones.png#globalusage
Derrac, J., & Schockaert, S. (2015). Inducing semantic relations from conceptual spaces: a data-driven approach to plausible reasoning. Artificial Intelligence, 228 , 66–94.
G¨ardenfors, P. (2004). Conceptual spaces: The geometry of thought. MIT press.
Gasparri, L., & Marconi, D. (2016). Word meaning. In E. N. Zalta (Ed.), The stanford ency-clopedia of philosophy (Spring 2016 ed.). Metaphysics Research Lab, Stanford University. https://plato.stanford.edu/archives/spr2016/entries/word-meaning/.
G¨ardenfors, P. (2014). The geometry of meaning: Semantics based on conceptual spaces. MIT Press.
(n.d.). Cambridge University Press. Retrieved from https://dictionary.cambridge.org/ dictionary/english/noun
Jolliffe, I. (2011). Principal component analysis. Springer.
Jurafsky, D., & Martin, J. H. (2014). Speech and language processing (Vol. 3). Pearson London. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., & Fidler, S.
(2015). Skip-thought vectors. In Advances in neural information processing systems (pp. 3294–3302).
Koh, P. W., & Liang, P. (2017). Understanding black-box predictions via influence functions. In Proceedings of the 34th international conference on machine learning-volume 70 (pp. 1885–1894).
Sahlgren, M. (2006). The word-space model using distributional analysis to represent syntagmatic and paradigmatic relations between words. Doctor of philosophy, Stockholm University. Schubert, L. (2019). Computational linguistics. In E. N. Zalta (Ed.), The stanford encyclopedia
of philosophy (Spring 2019 ed.). Metaphysics Research Lab, Stanford University. https:// plato.stanford.edu/archives/spr2019/entries/computational-linguistics/. Sch¨utze, H. (1993). Word space. In Advances in neural information processing systems (pp.
895–902).
Smolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46 (1-2), 159–216.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Pro-ceedings of the 2013 conference on empirical methods in natural language processing (pp. 1631–1642).
Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of artificial intelligence research, 37 , 141–188.
