The automatic acquisition of a Dutch lexicon for opinion mining Maks, E.

VU Research Portal

The automatic acquisition of a Dutch lexicon for opinion mining Maks, E.

2018

document version

Publisher's PDF, also known as Version of record

Link to publication in VU Research Portal

citation for published version (APA)

Maks, E. (2018). The automatic acquisition of a Dutch lexicon for opinion mining.

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.

• You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal ?

Take down policy

If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

E-mail address:

vuresearchportal.ub@vu.nl

The aut om atic ac quisition of a Dut ch le xic o n for opi nion mi nin g Isa M

The automatic acquisition

of a Dutch lexicon for opinion mining

The automatic acquisition

of a Dutch lexicon for opinion mining

Isa Maks

Promotor:

prof.dr. P. Vossen Reading committee:

prof.dr. A. Cienki prof.dr. S. Bergler prof.dr. A. van den Bosch prof.dr. W. Martin dr. W. van Atteveldt dr. M. Marx

Copyright: Isa Maks, 2018 Photos: Gert Jan van der Wilt

VRIJE UNIVERSITEIT

The automatic acquisition of a Dutch lexicon for opinion mining

ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad Doctor aan de Vrije Universiteit Amsterdam,

op gezag van de rector magnificus prof.dr. V. Subramaniam, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de Faculteit der Geesteswetenschappen

op woensdag 27 juni 2018 om 15.45 uur in de aula van de universiteit,

De Boelelaan 1105

door

Elisabeth Maks

geboren te Amsterdam

Promotor:

prof.dr. P. Vossen Reading committee:

prof.dr. A. Cienki prof.dr. S. Bergler prof.dr. A. van den Bosch prof.dr. W. Martin dr. W. van Atteveldt dr. M. Marx

Copyright: Isa Maks, 2018 Photos: Gert Jan van der Wilt

VRIJE UNIVERSITEIT

The automatic acquisition of a Dutch lexicon for opinion mining

ACADEMISCH PROEFSCHRIFT ter verkrijging van de graad Doctor aan de Vrije Universiteit Amsterdam,

op gezag van de rector magnificus prof.dr. V. Subramaniam, in het openbaar te verdedigen ten overstaan van de promotiecommissie van de Faculteit der Geesteswetenschappen

op woensdag 27 juni 2018 om 15.45 uur in de aula van de universiteit,

De Boelelaan 1105

door

Elisabeth Maks

geboren te Amsterdam

promotor: prof.dr. P.T.J.M. Vossen

Contents

1 Introduction 11

1.1 List of publications . . . 16

1.2 Data and software . . . 17

1.3 Main contributions of this thesis . . . 17

2 Models and Resources for opinion mining 19 2.1 Introduction . . . 19

2.2 Resources in opinion mining . . . 20

2.2.1 Appraisal Framework . . . 20

2.2.2 Computational lexicons for general language . . . 22

2.2.2.1 Princeton WordNet (PWN) . . . 22

2.2.2.2 Cornetto . . . 25

2.2.2.3 FrameNet . . . 26

2.2.3 Polarity lexicons . . . 28

2.2.4 Emotion lexicons . . . 30

2.2.4.1 WordNetAffect . . . 30

2.2.5 Lexicons and multiple affect . . . 31

2.2.5.1 WordNetAffectPlus . . . 31

2.2.5.2 +/- Effect WordNet . . . 32

2.2.6 Lexicons with semantic classification . . . 33

2.2.6.1 SentiFul . . . 33

2.2.7 Corpora with annotated opinions . . . 34

2.2.7.1 MPQA opinion corpus . . . 34

2.2.7.2 Asher: opinions in discourse . . . 36

2.2.8 Summary . . . 37

2.2.8.1 Polarity . . . 38

2.2.8.2 Multiple actor attitude . . . 38

2.2.8.3 Semantic classification . . . 40

2.3 Towards a new lexicon model . . . 42

2.3.1 Polarity . . . 42

2.3.2 Multiple actor attitude . . . 43

promotor: prof.dr. P.T.J.M. Vossen

Contents

1 Introduction 11

1.1 List of publications . . . 16

1.2 Data and software . . . 17

1.3 Main contributions of this thesis . . . 17

2 Models and Resources for opinion mining 19 2.1 Introduction . . . 19

2.2 Resources in opinion mining . . . 20

2.2.1 Appraisal Framework . . . 20

2.2.2 Computational lexicons for general language . . . 22

2.2.2.1 Princeton WordNet (PWN) . . . 22

2.2.2.2 Cornetto . . . 25

2.2.2.3 FrameNet . . . 26

2.2.3 Polarity lexicons . . . 28

2.2.4 Emotion lexicons . . . 30

2.2.4.1 WordNetAffect . . . 30

2.2.5 Lexicons and multiple affect . . . 31

2.2.5.1 WordNetAffectPlus . . . 31

2.2.5.2 +/- Effect WordNet . . . 32

2.2.6 Lexicons with semantic classification . . . 33

2.2.6.1 SentiFul . . . 33

2.2.7 Corpora with annotated opinions . . . 34

2.2.7.1 MPQA opinion corpus . . . 34

2.2.7.2 Asher: opinions in discourse . . . 36

2.2.8 Summary . . . 37

2.2.8.1 Polarity . . . 38

2.2.8.2 Multiple actor attitude . . . 38

2.2.8.3 Semantic classification . . . 40

2.3 Towards a new lexicon model . . . 42

2.3.1 Polarity . . . 42

2.3.2 Multiple actor attitude . . . 43

CONTENTS

2.3.4 Illustrating the model . . . 52

2.4 Conclusions . . . 53

3 Annotation Model 55 3.1 Introduction . . . 55

3.2 Design of the annotation scheme . . . 55

3.2.1 Annotations at word sense level . . . 56

3.2.2 Annotation of attitudes and polarity . . . 56

3.2.2.1 Actor’s attitude (AC) . . . 57

3.2.2.2 Speaker/writer’s attitude (SW) . . . 60

3.2.2.3 SW and AC attitude combined (SW&AC) . . . 61

3.2.2.4 External attitude (extAtt) . . . 62

3.2.2.5 No specific attitude (noAtt) . . . 63

3.2.3 Schematic overview of the annotation scheme . . . 63

3.2.4 Examples of annotations . . . 63

3.2.4.1 Examples with verbs . . . 65

3.2.4.2 Examples with nouns . . . 67

3.2.4.3 Examples with adjectives . . . 68

3.3 Inter-annotator agreement study . . . 70

3.3.1 Composition of a representative sample . . . 70

3.3.2 Annotation task . . . 72

3.3.3 Annotations in numbers . . . 72

3.3.4 Inter-annotator agreement . . . 75

3.3.5 Analysis of disagreements . . . 79

3.3.5.1 Multiple actor attitudes . . . 79

3.3.5.2 External attitude (extATT) . . . 81

3.3.5.3 Positive vs. ’no’ polarity (noPol) . . . 82

3.3.6 Comparison with other studies . . . 83

3.4 Creating the final goldstandard . . . 85

3.4.1 Agreement on the simplified annotation schema . . . 86

3.4.2 Agreement on attitude annotations: AC, SW, noAtt . . . 86

3.4.3 Agreement on polarity annotations: positive, negative, noPol . . . 87

3.4.4 Agreement per lexicon dimension . . . 87

3.4.4.1 Lexicon dimensions and polarity . . . 88

3.4.4.2 Lexicon dimensions and attitude . . . 88

3.4.5 Agreement per semantic type . . . 91

3.4.6 Derived gold standard versions . . . 91

3.4.6.1 Gold standards for attitude . . . 93

3.4.6.2 Gold standards for polarity . . . 94

CONTENTS 4 Acquisition Methods 99 4.1 Introduction . . . 99

4.2 Background . . . 100

4.2.1 Lexicon-based methods . . . 100

4.2.2 Corpus-based methods . . . 101

4.2.3 Fine-grained classifications . . . 102

4.2.4 Multi and Cross-lingual Approaches . . . 103

4.3 Evaluation framework . . . 104

4.4 Cross-lingual transfer Method . . . 105

4.4.1 Introduction . . . 105

4.4.2 Datasets . . . 105

4.4.2.1 Sentiwordnet(SWN) . . . 105

4.4.2.2 Dutch WordNet (Cornetto) . . . 106

4.4.2.3 Gold standard . . . 106

4.4.3 Method . . . 107

4.4.4 Results . . . 107

4.4.4.1 Overall results . . . 107

4.4.4.2 Smaller selections . . . 107

4.4.5 Discussion . . . 108

4.5 Wordnet Propagation . . . 109

4.5.1 Introduction . . . 109

4.5.2 Methods . . . 109

4.5.3 Datasets . . . 111

4.5.3.1 Dutch WordNet (Cornetto) . . . 111

4.5.3.2 Seed lists . . . 111

4.5.3.3 Gold standard . . . 112

4.5.4 Results . . . 112

4.5.4.1 Baselines . . . 113

4.5.4.2 Propagation results with different seed lists . . . 113

4.5.4.3 Various wordnet relations . . . 114

4.5.4.4 Various iterations . . . 115

4.5.4.5 Synset-to-word . . . 118

4.5.5 Comparison with other work . . . 119

4.5.6 Discussion . . . 119

4.6 Lexical Feature Approach . . . 121

4.6.1 Introduction . . . 121

4.6.2 Method . . . 121

4.6.3 Data sets and features . . . 122

4.6.3.1 Lexical unit features . . . 122

4.6.3.2 Synset features . . . 125

4.6.3.3 WordNet domain features . . . 126

4.6.3.4 Ontology features . . . 127

CONTENTS

2.3.4 Illustrating the model . . . 52

2.4 Conclusions . . . 53

3 Annotation Model 55 3.1 Introduction . . . 55

3.2 Design of the annotation scheme . . . 55

3.2.1 Annotations at word sense level . . . 56

3.2.2 Annotation of attitudes and polarity . . . 56

3.2.2.1 Actor’s attitude (AC) . . . 57

3.2.2.2 Speaker/writer’s attitude (SW) . . . 60

3.2.2.3 SW and AC attitude combined (SW&AC) . . . 61

3.2.2.4 External attitude (extAtt) . . . 62

3.2.2.5 No specific attitude (noAtt) . . . 63

3.2.3 Schematic overview of the annotation scheme . . . 63

3.2.4 Examples of annotations . . . 63

3.2.4.1 Examples with verbs . . . 65

3.2.4.2 Examples with nouns . . . 67

3.2.4.3 Examples with adjectives . . . 68

3.3 Inter-annotator agreement study . . . 70

3.3.1 Composition of a representative sample . . . 70

3.3.2 Annotation task . . . 72

3.3.3 Annotations in numbers . . . 72

3.3.4 Inter-annotator agreement . . . 75

3.3.5 Analysis of disagreements . . . 79

3.3.5.1 Multiple actor attitudes . . . 79

3.3.5.2 External attitude (extATT) . . . 81

3.3.5.3 Positive vs. ’no’ polarity (noPol) . . . 82

3.3.6 Comparison with other studies . . . 83

3.4 Creating the final goldstandard . . . 85

3.4.1 Agreement on the simplified annotation schema . . . 86

3.4.2 Agreement on attitude annotations: AC, SW, noAtt . . . 86

3.4.3 Agreement on polarity annotations: positive, negative, noPol . . . 87

3.4.4 Agreement per lexicon dimension . . . 87

3.4.4.1 Lexicon dimensions and polarity . . . 88

3.4.4.2 Lexicon dimensions and attitude . . . 88

3.4.5 Agreement per semantic type . . . 91

3.4.6 Derived gold standard versions . . . 91

3.4.6.1 Gold standards for attitude . . . 93

3.4.6.2 Gold standards for polarity . . . 94

CONTENTS 4 Acquisition Methods 99 4.1 Introduction . . . 99

4.2 Background . . . 100

4.2.1 Lexicon-based methods . . . 100

4.2.2 Corpus-based methods . . . 101

4.2.3 Fine-grained classifications . . . 102

4.2.4 Multi and Cross-lingual Approaches . . . 103

4.3 Evaluation framework . . . 104

4.4 Cross-lingual transfer Method . . . 105

4.4.1 Introduction . . . 105

4.4.2 Datasets . . . 105

4.4.2.1 Sentiwordnet(SWN) . . . 105

4.4.2.2 Dutch WordNet (Cornetto) . . . 106

4.4.2.3 Gold standard . . . 106

4.4.3 Method . . . 107

4.4.4 Results . . . 107

4.4.4.1 Overall results . . . 107

4.4.4.2 Smaller selections . . . 107

4.4.5 Discussion . . . 108

4.5 Wordnet Propagation . . . 109

4.5.1 Introduction . . . 109

4.5.2 Methods . . . 109

4.5.3 Datasets . . . 111

4.5.3.1 Dutch WordNet (Cornetto) . . . 111

4.5.3.2 Seed lists . . . 111

4.5.3.3 Gold standard . . . 112

4.5.4 Results . . . 112

4.5.4.1 Baselines . . . 113

4.5.4.2 Propagation results with different seed lists . . . 113

4.5.4.3 Various wordnet relations . . . 114

4.5.4.4 Various iterations . . . 115

4.5.4.5 Synset-to-word . . . 118

4.5.5 Comparison with other work . . . 119

4.5.6 Discussion . . . 119

4.6 Lexical Feature Approach . . . 121

4.6.1 Introduction . . . 121

4.6.2 Method . . . 121

4.6.3 Data sets and features . . . 122

4.6.3.1 Lexical unit features . . . 122

4.6.3.2 Synset features . . . 125

4.6.3.3 WordNet domain features . . . 126

4.6.3.4 Ontology features . . . 127

CONTENTS

4.6.4 Results . . . 128

4.6.4.1 Results on separate features . . . 129

4.6.4.2 Combinations of features . . . 130

4.6.4.3 Results per part-of-speech . . . 131

4.6.4.4 Results on word level . . . 131

4.6.5 Comparison with other work . . . 132

4.6.6 Discussion . . . 134

4.7 Corpus comparison method . . . 135

4.7.1 Introduction . . . 135

4.7.2 Background . . . 135

4.7.3 Datasets . . . 135

4.7.3.1 Corpus composition . . . 135

4.7.3.2 Gold standard . . . 136

4.7.4 Method . . . 136

4.7.4.1 Assumptions . . . 136

4.7.4.2 Lexicon building . . . 137

4.7.5 Results . . . 138

4.7.5.1 Step1: identifying subjective words without distinguishing SW and AC attitude . . . 138

4.7.5.2 Step2: classifying words into SW and AC categories . . . . 139

4.7.6 Discussion . . . 141

4.8 Lexical Pattern method . . . 143

4.8.1 Introduction . . . 143

4.8.2 Background . . . 143

4.8.3 Datasets . . . 144

4.8.3.1 Seed lists . . . 144

4.8.3.2 Dutch N-gram corpus . . . 144

4.8.3.3 Gold standard . . . 144

4.8.4 Method . . . 144

4.8.4.1 Settings . . . 145

4.8.4.2 Finding the best association measure . . . 146

4.8.4.3 Finding the best cut-off point . . . 148

4.8.5 Results . . . 148

4.8.5.1 Results with automatically generated patterns . . . 149

4.8.5.2 Results of high ranking selections . . . 150

4.8.5.3 Results with linguistically motivated patterns . . . 152

4.8.6 Comparison with other work . . . 154

4.8.7 Discussion . . . 155

4.9 Comparing and combining methods . . . 157

4.9.1 Methods for the identification of positive and negative polarity . . . . 157

4.9.2 Methods for the identification of AC and SW attitude . . . 158

CONTENTS 5 Use Cases 167 5.1 Introduction . . . 167

5.2 Polarity in Hotel reviews . . . 168

5.2.1 Introduction . . . 168

5.2.2 Background . . . 168

5.2.3 Hotel review corpus . . . 168

5.2.3.1 Composition of the corpus . . . 168

5.2.3.2 Reviewer ratings and reader ratings . . . 169

5.2.4 Methods . . . 170

5.2.4.1 The dictionary lookup approach . . . 170

5.2.4.2 Machine-learning . . . 171

5.2.5 Results . . . 172

5.2.6 Discussion . . . 174

5.2.7 Conclusions . . . 174

5.3 Finding holders and targets in political news . . . 175

5.3.1 Introduction . . . 175

5.3.2 Background . . . 175

5.3.3 The OPeNER Corpus . . . 175

5.3.3.1 Annotations of opinion entities and relations . . . 176

5.3.4 The use of the lexicons in the opinion mining task . . . 177

5.3.4.1 Lexicon with SW and AC attitude (SWAC-lexicon . . . 177

5.3.4.2 Polarity lexicon . . . 179

5.3.5 The OPeNER opinion mining system . . . 180

5.3.5.1 Opinion entity extraction . . . 180

5.3.5.2 Opinion relation extraction . . . 180

5.3.6 Results . . . 182

5.3.6.1 Results on identification of entities (Step I) . . . 182

5.3.6.2 Results on identification of relations (Step II) . . . 183

5.3.7 Discussion and conclusions . . . 183

5.4 SW and AC words in lexicon and corpus . . . 187

5.4.1 Introduction . . . 187

5.4.2 Lexical capacity: SW and AC words in the lexicon . . . 187

5.4.3 Lexical usage: SW and AC words in a general language . . . 189

5.4.4 Distribution of SW and AC words in an opinionated corpus . . . 191

5.4.4.1 Corpus composition . . . 191

5.4.4.2 SW/AC distribution in the WNBC corpus . . . 192

5.4.5 Discussion . . . 194

5.5 Discussion and conclusions . . . 196

6 Conclusions 199

7 Bibliography 203

CONTENTS

4.6.4 Results . . . 128

4.6.4.1 Results on separate features . . . 129

4.6.4.2 Combinations of features . . . 130

4.6.4.3 Results per part-of-speech . . . 131

4.6.4.4 Results on word level . . . 131

4.6.5 Comparison with other work . . . 132

4.6.6 Discussion . . . 134

4.7 Corpus comparison method . . . 135

4.7.1 Introduction . . . 135

4.7.2 Background . . . 135

4.7.3 Datasets . . . 135

4.7.3.1 Corpus composition . . . 135

4.7.3.2 Gold standard . . . 136

4.7.4 Method . . . 136

4.7.4.1 Assumptions . . . 136

4.7.4.2 Lexicon building . . . 137

4.7.5 Results . . . 138

4.7.5.1 Step1: identifying subjective words without distinguishing SW and AC attitude . . . 138

4.7.5.2 Step2: classifying words into SW and AC categories . . . . 139

4.7.6 Discussion . . . 141

4.8 Lexical Pattern method . . . 143

4.8.1 Introduction . . . 143

4.8.2 Background . . . 143

4.8.3 Datasets . . . 144

4.8.3.1 Seed lists . . . 144

4.8.3.2 Dutch N-gram corpus . . . 144

4.8.3.3 Gold standard . . . 144

4.8.4 Method . . . 144

4.8.4.1 Settings . . . 145

4.8.4.2 Finding the best association measure . . . 146

4.8.4.3 Finding the best cut-off point . . . 148

4.8.5 Results . . . 148

4.8.5.1 Results with automatically generated patterns . . . 149

4.8.5.2 Results of high ranking selections . . . 150

4.8.5.3 Results with linguistically motivated patterns . . . 152

4.8.6 Comparison with other work . . . 154

4.8.7 Discussion . . . 155

4.9 Comparing and combining methods . . . 157

4.9.1 Methods for the identification of positive and negative polarity . . . . 157

4.9.2 Methods for the identification of AC and SW attitude . . . 158

CONTENTS 5 Use Cases 167 5.1 Introduction . . . 167

5.2 Polarity in Hotel reviews . . . 168

5.2.1 Introduction . . . 168

5.2.2 Background . . . 168

5.2.3 Hotel review corpus . . . 168

5.2.3.1 Composition of the corpus . . . 168

5.2.3.2 Reviewer ratings and reader ratings . . . 169

5.2.4 Methods . . . 170

5.2.4.1 The dictionary lookup approach . . . 170

5.2.4.2 Machine-learning . . . 171

5.2.5 Results . . . 172

5.2.6 Discussion . . . 174

5.2.7 Conclusions . . . 174

5.3 Finding holders and targets in political news . . . 175

5.3.1 Introduction . . . 175

5.3.2 Background . . . 175

5.3.3 The OPeNER Corpus . . . 175

5.3.3.1 Annotations of opinion entities and relations . . . 176

5.3.4 The use of the lexicons in the opinion mining task . . . 177

5.3.4.1 Lexicon with SW and AC attitude (SWAC-lexicon . . . 177

5.3.4.2 Polarity lexicon . . . 179

5.3.5 The OPeNER opinion mining system . . . 180

5.3.5.1 Opinion entity extraction . . . 180

5.3.5.2 Opinion relation extraction . . . 180

5.3.6 Results . . . 182

5.3.6.1 Results on identification of entities (Step I) . . . 182

5.3.6.2 Results on identification of relations (Step II) . . . 183

5.3.7 Discussion and conclusions . . . 183

5.4 SW and AC words in lexicon and corpus . . . 187

5.4.1 Introduction . . . 187

5.4.2 Lexical capacity: SW and AC words in the lexicon . . . 187

5.4.3 Lexical usage: SW and AC words in a general language . . . 189

5.4.4 Distribution of SW and AC words in an opinionated corpus . . . 191

5.4.4.1 Corpus composition . . . 191

5.4.4.2 SW/AC distribution in the WNBC corpus . . . 192

5.4.5 Discussion . . . 194

5.5 Discussion and conclusions . . . 196

6 Conclusions 199

7 Bibliography 203

CONTENTS

Dankwoord 219

1 | Introduction

People have and express their views on a sheer infinite variety of subjects. Is Rome the most beautiful city in the world? How do people feel about the Dutch king? What are the best universities in the world? Would Brexit be bad for London’s financial centre? For all sorts of reasons, people take a great interest in knowing what views other people have on subjects like these. The amount of digitized texts in which people express their opinions and attitudes give us abundant opportunities to obtain answers to these questions.

The language and style that conveys this kind of information is often diverse and com- plex. Opinions and evaluations come in many forms such as judgements, allegations, desires, intentions, beliefs and speculations (Wiebe et al. (2005)). Moreover, we can find opinions in many different texts and text genres such as news, editorials, blogs, forums, reviews and online debates.

Various tools and techniques have been developed for the automatic extraction and in- terpretation of opinionated information from text. To accomplish this, a method is needed to distinguish between opinionated and non-opinionated pieces of text. Also, a method is needed for classifying expressions into sentiment categories such as positive, negative, and neutral. Consider, for example, the following text in which a reviewer describes his visit to a museum in Rome

.

(1) A must for both locals and tourists.

This is it! This museum demands time and respect. An extraordinary example of suc- cessful conversion of a decommissioned power plant into an amazingly spacious and airy exhibition space, showcasing beautiful ancient sculptures from Rome’s imperial times as well as beautiful refined mosaics. Highly recommended, take your time.

The writer clearly wants to show his enthusiasm for the museum described. The review

offers a number of expressions that contribute to this purpose of which successful, amazingly,

beautiful, highly recommended are the most obvious ones. Expressions such as this is it!,

demands time and respect surely add to the positive opinion conveyed by this review, but

require more context for interpretation. If an automatic analysis relies on the principle of

compositionality, that is, considers the review’s meaning as the sum of its words, it will

certainly be able to classify this review as positive.

CONTENTS

Dankwoord 219

1 | Introduction

People have and express their views on a sheer infinite variety of subjects. Is Rome the most beautiful city in the world? How do people feel about the Dutch king? What are the best universities in the world? Would Brexit be bad for London’s financial centre? For all sorts of reasons, people take a great interest in knowing what views other people have on subjects like these. The amount of digitized texts in which people express their opinions and attitudes give us abundant opportunities to obtain answers to these questions.

The language and style that conveys this kind of information is often diverse and com- plex. Opinions and evaluations come in many forms such as judgements, allegations, desires, intentions, beliefs and speculations (Wiebe et al. (2005)). Moreover, we can find opinions in many different texts and text genres such as news, editorials, blogs, forums, reviews and online debates.

Various tools and techniques have been developed for the automatic extraction and in- terpretation of opinionated information from text. To accomplish this, a method is needed to distinguish between opinionated and non-opinionated pieces of text. Also, a method is needed for classifying expressions into sentiment categories such as positive, negative, and neutral. Consider, for example, the following text in which a reviewer describes his visit to a museum in Rome

.

(1) A must for both locals and tourists.

This is it! This museum demands time and respect. An extraordinary example of suc- cessful conversion of a decommissioned power plant into an amazingly spacious and airy exhibition space, showcasing beautiful ancient sculptures from Rome’s imperial times as well as beautiful refined mosaics. Highly recommended, take your time.

The writer clearly wants to show his enthusiasm for the museum described. The review

offers a number of expressions that contribute to this purpose of which successful, amazingly,

beautiful, highly recommended are the most obvious ones. Expressions such as this is it!,

demands time and respect surely add to the positive opinion conveyed by this review, but

require more context for interpretation. If an automatic analysis relies on the principle of

compositionality, that is, considers the review’s meaning as the sum of its words, it will

certainly be able to classify this review as positive.

Apart from knowing whether the overall evaluation is positive or negative, it can be in- teresting to know the holders and targets of these opinions: whose opinion is this, and what is it about? In the case of the above review this is a relatively simple task as we can safely assume that reviews express the writer’s sentiment toward the topic - in this case the museum - addressed in the message. Identifying sources or ’opinion holders’ is important especially in news articles and debates that contain various opinions of multiple persons. However, this task can be quite complex, as the following example (2), a sentence taken from a book review illustrates:

(2) And, though Jane Hawking is perfectly justified in her anger at Hawking’s second wife Elaine Mason ..., portraying Elaine as a totally cunning villainess .. is not en- tirely fair ..

In this sentence, the reviewer never explicitly refers to himself even though he expresses several attitudes such as perfectly justified and not entirely fair. In addition, he attributes attitudes, opinions and emotions to other people, namely Jane Hawkins who feels anger at Elaine Mason. However, yet another attitude holder is implied. This is the subject of portraying, the person who portrays Elaine Mason as a a totally cunning villainess (which, according to the reviewer is a bit unfair). Presumably, this implied person is the writer of the book, but this is not stated explicitly. An opinion mining system should be able to identify the various opinion expressions, their holders and targets. Also, such systems need a way of identifying unstated holders of opinion, such as the writer in this review and the subject of portraying.

A common approach to achieve this is by defining opinions as a piece of language ex- pressed by a holder towards a topic or target, representing them as triplets including holder, target and expression (see Kim and Hovy (2006) and Wiebe et al. (2005)). Holders may be persons, institutions or groups and they can be an actor in the text or the author of the text himself. Targets may be persons, institutions, or groups of persons, issues, entities and the like. The opinion holder may then be either an actor in the text, such as Jane Hawkins and Elaine in the text above or the writer such as in the example of the Rome museum. Such anal- yses also allow for the identification of multiple perspectives on the same target, as illustrated in the following example:

(3) Students incorrectly say that he is a good teacher

In this case, the students express their favorable view of their teacher, but the writer of the text clearly has a different opinion, characterizing the students’ opinions as ’incorrect’.

Another issue concerning the analysis of opinions is whether the distinction in positive and negative is sufficient to appreciate subtle nuances of opinionated expressions. Consider the following two examples:

(4) The audience was disappointed by the speech (5) He missed the time when they did things together

In example (4) the word disappointed tells us that the audience is negative about the speech. However, the feeling of disappointment is more complex: it implies that the audience was expecting or at least hoping for a better speech. In other words, before the speech was delivered they were feeling positive about it. By simply categorizing opinions in negative and positive, more subtle attitudes may not be correctly captured. Example (5) is in this respect even more complex as missing usually will be considered a negative feeling, but it is clear from this sentence that the he is actually quite positive about the time when they did things together.

People thus express their opinions in different forms and many studies focus on the iden- tification of different types of opinions.

Some studies focus on emotions in texts (Mohammad and Yang (2011)), others on insults (Foley et al. (2012)), ideological stance (Somasundaran and Wiebe (2010)), or evaluations.

Martin and White (2005) provide a rich overview framework, presenting how opinions can be expressed in text. They distinguish, for instance between moral judgements and aesthetic appreciations, the emotions that such judgements or appreciations may bring along, and the strength of the expressed opinion. The issue is even more complex, however, because of the fact that opinions are often implicit, for example:

(6) There are mice running through the office

The correct evaluative interpretation of this sentence requires context and world knowl- edge. Only when we know that mice are not supposed to be in an office, and that many people are afraid of mice, we can correctly interpret this sentence as a negative evaluation.

From this brief overview, we can conclude that the automatic analysis of opinions is a difficult task for which several approaches have been developed. Many of these approaches make use of manually or automatically constructed lexicons with words and phrases that con- vey positive or negative sentiments (Liu (2012). Rule-based applications rely completely on such lexicons, but also machine-learning approaches improve if polarity lexicons are included in the process (Yang and Cardie (2013)). For English, several lexical resources are available.

These include polarity lexicons which are lexicons that contain lists with positive and neg- ative words. Also, there are subjectivity lexicons that indicate whether a word is subjective (i.e., whether it refers to an emotion, an evaluation and the like) or not, and there are emotion lexicons which classify words into categories of emotions such as anger and fear (Valitutti and Strapparava (2010)).

This thesis aims at developing a lexical resource for opinion mining of Dutch text. The

reason for choosing Dutch is twofold. Firstly, opinion mining is a language-dependent task

and only few lexical resources for opinion mining are currently available in Dutch (Jijkoun

and Hoffman (2009), Desmedt and Daelemans (2012)). Hence, to allow for more extensive

opinion mining in Dutch, more lexical resources are needed. Secondly, apart from this ap-

plied aspect of opinion mining, there is a heuristic reason, too. New lexical resources includ-

ing those for opinion mining, are usually automatically generated relying on already available

1

Apart from knowing whether the overall evaluation is positive or negative, it can be in- teresting to know the holders and targets of these opinions: whose opinion is this, and what is it about? In the case of the above review this is a relatively simple task as we can safely assume that reviews express the writer’s sentiment toward the topic - in this case the museum - addressed in the message. Identifying sources or ’opinion holders’ is important especially in news articles and debates that contain various opinions of multiple persons. However, this task can be quite complex, as the following example (2), a sentence taken from a book review illustrates:

(2) And, though Jane Hawking is perfectly justified in her anger at Hawking’s second wife Elaine Mason ..., portraying Elaine as a totally cunning villainess .. is not en- tirely fair ..

In this sentence, the reviewer never explicitly refers to himself even though he expresses several attitudes such as perfectly justified and not entirely fair. In addition, he attributes attitudes, opinions and emotions to other people, namely Jane Hawkins who feels anger at Elaine Mason. However, yet another attitude holder is implied. This is the subject of portraying, the person who portrays Elaine Mason as a a totally cunning villainess (which, according to the reviewer is a bit unfair). Presumably, this implied person is the writer of the book, but this is not stated explicitly. An opinion mining system should be able to identify the various opinion expressions, their holders and targets. Also, such systems need a way of identifying unstated holders of opinion, such as the writer in this review and the subject of portraying.

A common approach to achieve this is by defining opinions as a piece of language ex- pressed by a holder towards a topic or target, representing them as triplets including holder, target and expression (see Kim and Hovy (2006) and Wiebe et al. (2005)). Holders may be persons, institutions or groups and they can be an actor in the text or the author of the text himself. Targets may be persons, institutions, or groups of persons, issues, entities and the like. The opinion holder may then be either an actor in the text, such as Jane Hawkins and Elaine in the text above or the writer such as in the example of the Rome museum. Such anal- yses also allow for the identification of multiple perspectives on the same target, as illustrated in the following example:

(3) Students incorrectly say that he is a good teacher

In this case, the students express their favorable view of their teacher, but the writer of the text clearly has a different opinion, characterizing the students’ opinions as ’incorrect’.

Another issue concerning the analysis of opinions is whether the distinction in positive and negative is sufficient to appreciate subtle nuances of opinionated expressions. Consider the following two examples:

(4) The audience was disappointed by the speech (5) He missed the time when they did things together

In example (4) the word disappointed tells us that the audience is negative about the speech. However, the feeling of disappointment is more complex: it implies that the audience was expecting or at least hoping for a better speech. In other words, before the speech was delivered they were feeling positive about it. By simply categorizing opinions in negative and positive, more subtle attitudes may not be correctly captured. Example (5) is in this respect even more complex as missing usually will be considered a negative feeling, but it is clear from this sentence that the he is actually quite positive about the time when they did things together.

People thus express their opinions in different forms and many studies focus on the iden- tification of different types of opinions.

Some studies focus on emotions in texts (Mohammad and Yang (2011)), others on insults (Foley et al. (2012)), ideological stance (Somasundaran and Wiebe (2010)), or evaluations.

Martin and White (2005) provide a rich overview framework, presenting how opinions can be expressed in text. They distinguish, for instance between moral judgements and aesthetic appreciations, the emotions that such judgements or appreciations may bring along, and the strength of the expressed opinion. The issue is even more complex, however, because of the fact that opinions are often implicit, for example:

(6) There are mice running through the office

The correct evaluative interpretation of this sentence requires context and world knowl- edge. Only when we know that mice are not supposed to be in an office, and that many people are afraid of mice, we can correctly interpret this sentence as a negative evaluation.

From this brief overview, we can conclude that the automatic analysis of opinions is a difficult task for which several approaches have been developed. Many of these approaches make use of manually or automatically constructed lexicons with words and phrases that con- vey positive or negative sentiments (Liu (2012). Rule-based applications rely completely on such lexicons, but also machine-learning approaches improve if polarity lexicons are included in the process (Yang and Cardie (2013)). For English, several lexical resources are available.

These include polarity lexicons which are lexicons that contain lists with positive and neg- ative words. Also, there are subjectivity lexicons that indicate whether a word is subjective (i.e., whether it refers to an emotion, an evaluation and the like) or not, and there are emotion lexicons which classify words into categories of emotions such as anger and fear (Valitutti and Strapparava (2010)).

This thesis aims at developing a lexical resource for opinion mining of Dutch text. The

reason for choosing Dutch is twofold. Firstly, opinion mining is a language-dependent task

and only few lexical resources for opinion mining are currently available in Dutch (Jijkoun

and Hoffman (2009), Desmedt and Daelemans (2012)). Hence, to allow for more extensive

opinion mining in Dutch, more lexical resources are needed. Secondly, apart from this ap-

plied aspect of opinion mining, there is a heuristic reason, too. New lexical resources includ-

ing those for opinion mining, are usually automatically generated relying on already available

English, for instance, lexical resources such as Princeton WordNet (Fellbaum (1998), Verb- Net(Kipper et al. (2006)) or FrameNet (Baker et al. (1998)) exist, equivalents of which are usually not available for other languages. Because of such constraints, new lexical resources for languages other than English, need to be developed in different ways. Interestingly, this diversity can also lead to new ideas and new insights that may have a certain generic validity, and that otherwise might not have been discovered. For Dutch, the diversity of resources is limited, but a rich semantic resource i.e. Cornetto is available (Vossen et al. (2008)).

The main focus of this thesis was to design a lexicon for opinion mining in Dutch, and to develop and test various methods that automatically generate such a lexicon. In order to achieve this, four research questions will be addressed.

Research question 1 (Chapter 2) What are the requirements for a lexicon for opinion mining in Dutch and how can these requirements be implemented?

To answer this question, we started from a general theoretical account of opinionated language (Martin and White (2005)), which can be used to distinguish opinionated from non- opinionated language and to differentiate between various types within opinionated language.

Also, we analyzed what classifications and annotations are currently used in a number of lex- icons and corpora that have been developed for opinion mining. On the basis of this analysis, we identified a number of specifications that could be useful for a lexicon for opinion mining.

We translated these requirements into an extra attitudinal layer to be integrated in Cornetto.

The results of our analysis of the existing resources are presented in Chapter 2, along with the proposed model for an lexicon for opinion mining.

Research question 2 (Chapter 3) Can the extensions of Cornetto as proposed in Chapter 2 be reliably annotated?

For this purpose, an annotation scheme was developed based on the specifications of the attitudinal layer proposed in Chapter 2. Stratified samples of entries from Cornetto were taken, and the requested annotations were performed manually by two researchers indepen- dently and tested for their inter-rater reliability. On the basis of the results, a final annotation scheme was developed, and importantly a gold standard was produced. The results of this part of the research are presented in Chapter 3.

Research question 3 (Chapter 4) What are appropriate methods to generate lexicons that meet the criteria defined in Chapter 2 automatically?

Five different acquisition methods were developed. The results of these methods were compared with the gold standard, developed in Chapter 3. Methods were compared in terms of precision and recall. The results of this part of the research are presented in Chapter 4.

Research question 4 (Chapter 5) In what way and to what extent does the use of a lexical resource that meets the specifications that were developed in Chapter 2 improve the quality of opinion mining in Dutch?

The results of the two applications of opinion mining in the Dutch language are pre- sented. One application focuses on the analysis of polarity, and consists of the sentiment classification of Dutch hotel reviews. The second application focuses on the analysis of mul- tiple attitude, and consists of the identification of opinion expressions and their holders and targets Dutch news articles. The results of this part of the research are presented in Chapter 5.

Chapter 6 summarizes the main findings of the research presented in this thesis and pro-

vides some ideas for future research.

1

English, for instance, lexical resources such as Princeton WordNet (Fellbaum (1998), Verb- Net(Kipper et al. (2006)) or FrameNet (Baker et al. (1998)) exist, equivalents of which are usually not available for other languages. Because of such constraints, new lexical resources for languages other than English, need to be developed in different ways. Interestingly, this diversity can also lead to new ideas and new insights that may have a certain generic validity, and that otherwise might not have been discovered. For Dutch, the diversity of resources is limited, but a rich semantic resource i.e. Cornetto is available (Vossen et al. (2008)).

The main focus of this thesis was to design a lexicon for opinion mining in Dutch, and to develop and test various methods that automatically generate such a lexicon. In order to achieve this, four research questions will be addressed.

Research question 1 (Chapter 2) What are the requirements for a lexicon for opinion mining in Dutch and how can these requirements be implemented?

To answer this question, we started from a general theoretical account of opinionated language (Martin and White (2005)), which can be used to distinguish opinionated from non- opinionated language and to differentiate between various types within opinionated language.

Also, we analyzed what classifications and annotations are currently used in a number of lex- icons and corpora that have been developed for opinion mining. On the basis of this analysis, we identified a number of specifications that could be useful for a lexicon for opinion mining.

We translated these requirements into an extra attitudinal layer to be integrated in Cornetto.

The results of our analysis of the existing resources are presented in Chapter 2, along with the proposed model for an lexicon for opinion mining.

Research question 2 (Chapter 3) Can the extensions of Cornetto as proposed in Chapter 2 be reliably annotated?

For this purpose, an annotation scheme was developed based on the specifications of the attitudinal layer proposed in Chapter 2. Stratified samples of entries from Cornetto were taken, and the requested annotations were performed manually by two researchers indepen- dently and tested for their inter-rater reliability. On the basis of the results, a final annotation scheme was developed, and importantly a gold standard was produced. The results of this part of the research are presented in Chapter 3.

Research question 3 (Chapter 4) What are appropriate methods to generate lexicons that meet the criteria defined in Chapter 2 automatically?

Five different acquisition methods were developed. The results of these methods were compared with the gold standard, developed in Chapter 3. Methods were compared in terms of precision and recall. The results of this part of the research are presented in Chapter 4.

Research question 4 (Chapter 5) In what way and to what extent does the use of a lexical resource that meets the specifications that were developed in Chapter 2 improve the quality of opinion mining in Dutch?

The results of the two applications of opinion mining in the Dutch language are pre- sented. One application focuses on the analysis of polarity, and consists of the sentiment classification of Dutch hotel reviews. The second application focuses on the analysis of mul- tiple attitude, and consists of the identification of opinion expressions and their holders and targets Dutch news articles. The results of this part of the research are presented in Chapter 5.

Chapter 6 summarizes the main findings of the research presented in this thesis and pro-

vides some ideas for future research.

1.1. LIST OF PUBLICATIONS

1.1 List of publications

The chapters of this thesis are based on the following papers.

Chapter 2:

Maks, I. and P. Vossen, and R. Segers and H. van der Vliet (2008). Adjectives in the Dutch Semantic Lexical Database Cornetto. In Proceedings of LREC2008, Marrakech, Mo- rocco.

Maks, I. and P. Vossen (2010). Modeling attitude, polarity and subjectivity in wordnet. In Proceeding of the 5th Global WordNet Conference (GWCâ ˘A ´Z10), Mumbai, India.

Chapter 3:

Maks, I. and P. Vossen (2010). Annotation scheme and gold standard for Dutch subjective adjectives. In Proceedings of LREC2010, Valletta, Malta.

Maks, I. and P. Vossen (2011). A verb lexicon model for deep sentiment analysis and opinion mining applications. In Proceedings of WASSA2011, Portland, Oregon, USA.

Maks, I. and P. Vossen (2012). A lexicon model for deep sentiment analysis and opinion mining applications. Decision Support Systems, 53(4).

Chapter 4:

Maks, I. and P. Vossen (2011). Different approaches to automatic polarity annotation at synset level. In Proceedings of the 1st International ESSLLI Workshop on Lexical Resources (WoLeR 2011), Ljubljana, Slovenia.

Maks, I. and P. Vossen (2012). Building a fine-grained subjectivity lexicon from a web corpus. In Proceedings of LREC2012, Istanbul, Turkey.

Maks, I., R. Izquierdo, F. Frontini, F., R. Agerri and P. Vossen, P. (2014). Generating polarity lexicons with wordnet propagation in five languages. In Proceedings of LREC2014, Reykjavik.

Chapter 5:

Maks, I. and Vossen, P. (2013). Sentiment analysis of reviews: Should we analyze writer intentions or reader perceptions? In Proceedings of RANLP 2013, pages 415â ˘A¸S419, Hissar, Bulgaria.

All chapters:

van Elfrinkhof, A., Maks, I., and Kaal, B. (2014). From text to political positions: The convergence of political, linguistic and discourse analysis. In Kaal, B., Maks, I., and van Elfrinkhof, A., editors, From Text to Political Positions : Text analysis across

1.2. DATA AND SOFTWARE

J. M. van der Zwaan and I. Leemans and E. Kuijpers and I. Maks (2015). HEEM, a Complex Model for Mining Emotions in Historical Text. In Proceedings of 11th IEEE Interna- tional Conference on e-Science, Munich, Germany,

1.2 Data and software

The data (annotations, annotated texts, and software used in this thesis can be found on https://github.com/cltl/thesisIsaMaks

1.3 Main contributions of this thesis

An account of the main corpus and lexicon resources corpus used for opinion mining A proposal for an attitudinal layer in Cornetto, including semantic class, multiple actor

attitude, and polarity

A stratified sample of the lexicon enriched with annotations by 2 annotators concerning multiple attitude and polarity

Gold standard sets for evaluating polarity Gold standard sets for evaluating attitude

Experimental results of the various acquisition methods: lexicons for polarity at word and synset level; lexicons for SW/AC attitude at word sense and word level

A set of annotated Dutch hotel reviews

1

1.1. LIST OF PUBLICATIONS

1.1 List of publications

The chapters of this thesis are based on the following papers.

Chapter 2:

Maks, I. and P. Vossen, and R. Segers and H. van der Vliet (2008). Adjectives in the Dutch Semantic Lexical Database Cornetto. In Proceedings of LREC2008, Marrakech, Mo- rocco.

Maks, I. and P. Vossen (2010). Modeling attitude, polarity and subjectivity in wordnet. In Proceeding of the 5th Global WordNet Conference (GWCâ ˘A ´Z10), Mumbai, India.

Chapter 3:

Maks, I. and P. Vossen (2010). Annotation scheme and gold standard for Dutch subjective adjectives. In Proceedings of LREC2010, Valletta, Malta.

Maks, I. and P. Vossen (2011). A verb lexicon model for deep sentiment analysis and opinion mining applications. In Proceedings of WASSA2011, Portland, Oregon, USA.

Maks, I. and P. Vossen (2012). A lexicon model for deep sentiment analysis and opinion mining applications. Decision Support Systems, 53(4).

Chapter 4:

Maks, I. and P. Vossen (2011). Different approaches to automatic polarity annotation at synset level. In Proceedings of the 1st International ESSLLI Workshop on Lexical Resources (WoLeR 2011), Ljubljana, Slovenia.

Maks, I. and P. Vossen (2012). Building a fine-grained subjectivity lexicon from a web corpus. In Proceedings of LREC2012, Istanbul, Turkey.

Maks, I., R. Izquierdo, F. Frontini, F., R. Agerri and P. Vossen, P. (2014). Generating polarity lexicons with wordnet propagation in five languages. In Proceedings of LREC2014, Reykjavik.

Chapter 5:

Maks, I. and Vossen, P. (2013). Sentiment analysis of reviews: Should we analyze writer intentions or reader perceptions? In Proceedings of RANLP 2013, pages 415â ˘A¸S419, Hissar, Bulgaria.

All chapters:

van Elfrinkhof, A., Maks, I., and Kaal, B. (2014). From text to political positions: The convergence of political, linguistic and discourse analysis. In Kaal, B., Maks, I., and van Elfrinkhof, A., editors, From Text to Political Positions : Text analysis across

1.2. DATA AND SOFTWARE

J. M. van der Zwaan and I. Leemans and E. Kuijpers and I. Maks (2015). HEEM, a Complex Model for Mining Emotions in Historical Text. In Proceedings of 11th IEEE Interna- tional Conference on e-Science, Munich, Germany,

1.2 Data and software

The data (annotations, annotated texts, and software used in this thesis can be found on https://github.com/cltl/thesisIsaMaks

1.3 Main contributions of this thesis

An account of the main corpus and lexicon resources corpus used for opinion mining A proposal for an attitudinal layer in Cornetto, including semantic class, multiple actor

attitude, and polarity

A stratified sample of the lexicon enriched with annotations by 2 annotators concerning multiple attitude and polarity

Gold standard sets for evaluating polarity Gold standard sets for evaluating attitude

Experimental results of the various acquisition methods: lexicons for polarity at word and synset level; lexicons for SW/AC attitude at word sense and word level

A set of annotated Dutch hotel reviews

2 | Models and Resources for opinion mining

2.1 Introduction

As explained in the previous chapter, there is a huge interest in opinion mining, but it poses several as yet unresolved challenges. One of the key requirements that could help resolve these challenges is a lexicon with layers of information specifically geared towards the auto- matic analysis of opinion in text.

The central question that will be addressed in this chapter is: what is the ideal design and content of such a lexicon?

To address this question, we will draw on four different sources:

1. A general theory of opinionated language, used to delineate the type of words that may be significant in the context of analysis of opinions and their function.

2. Computational lexicons for general language purpose, widely used in all kinds of nat- ural language processing tasks.

3. Several computational lexicons dedicated to opinion mining.

4. Corpora with annotated opinions and their guidelines, aiming at the identification and classification of opinions in corpus of texts.

The review of these resources allows us to see the differences and commonalities between

the various theories, lexicons and annotated corpora more clearly. We have to keep in mind,

though, that each of them were developed for slightly different purposes, making them par-

ticularly useful for some tasks, but not for others. By focusing on the strengths of the various

systems, we hope to develop an annotation scheme that is more generically useful to opinion

mining across a variety of tasks. In the second part of the chapter (see Section 2.3), we con-

sider whether and to what extent the information found in these different resources can, and

should be encoded in a lexicon for opinion mining. Based on this review, we present a novel

2 | Models and Resources for opinion mining

2.1 Introduction

As explained in the previous chapter, there is a huge interest in opinion mining, but it poses several as yet unresolved challenges. One of the key requirements that could help resolve these challenges is a lexicon with layers of information specifically geared towards the auto- matic analysis of opinion in text.

The central question that will be addressed in this chapter is: what is the ideal design and content of such a lexicon?

To address this question, we will draw on four different sources:

1. A general theory of opinionated language, used to delineate the type of words that may be significant in the context of analysis of opinions and their function.

2. Computational lexicons for general language purpose, widely used in all kinds of nat- ural language processing tasks.

3. Several computational lexicons dedicated to opinion mining.

4. Corpora with annotated opinions and their guidelines, aiming at the identification and classification of opinions in corpus of texts.

The review of these resources allows us to see the differences and commonalities between

the various theories, lexicons and annotated corpora more clearly. We have to keep in mind,

though, that each of them were developed for slightly different purposes, making them par-

ticularly useful for some tasks, but not for others. By focusing on the strengths of the various

systems, we hope to develop an annotation scheme that is more generically useful to opinion

mining across a variety of tasks. In the second part of the chapter (see Section 2.3), we con-

sider whether and to what extent the information found in these different resources can, and

should be encoded in a lexicon for opinion mining. Based on this review, we present a novel

2.2. RESOURCES IN OPINION MINING

2.2 Resources in opinion mining

2.2.1 Appraisal Framework

The Appraisal Framework (Martin and White (2005)) (AF) is a theory about how opinions and emotions are expressed in language. The theory gives a detailed description of opinion- ated language and offers a method for a qualitative analysis of speakers’ or writers’ attitudes and how they are linguistically and lexically encoded in text. It is based on the investiga- tion of individual texts, pieces of texts and groups of words. From these individual analyses, Martin and White (2005) derive a general model of opinions and attitudes in text and speech.

AF presents a general structure of how language is used to adopt or express an attitude of some kind towards some target. For example, in I found the movie quite monotonous, the speaker (the Appraiser) adopts a negative Attitude (monotonous) towards a target (e.g.

the movie. According to AF, attitudes come in different types, for example, monotonous de- scribes an inherent quality of the target, while loathed would describe the emotional reaction of the Appraiser. AF distinguishes three interacting components, i.e. attitude, engagement and graduation:

attitude is considered to be the core of the model. Three dimensions of attitude are distin- guished to wit affect, judgment and appreciation:

• affect: words that refer to personal emotions such as emotional reactions (e.g. sorrow, happy, fear, loath, cry, upset and desires (e.g. want, abhor). Affect is the most ex- plicitly subjective type of appraisal and it is concerned with registering positive and negative feelings.

(7) It was then with fury that I returned home (8) The captain wept

• judgement: words used for giving moral evaluations of people and the way they behave (e.g. fraud, swindle, intelligent, neurotic, unreliable). As with affect, they recognise positive and negative evaluations, that is traits that people admire and traits that people criticize.

(9) To see police brutally manhandling demonstrators was not only shocking but representative of more repressive regimes, such as China

• appreciation: words that refer to the value and evaluations of â ˘AŸthingsâ ˘A ´Z, perfor- mances and natural phenomena. It typically includes aesthetic evaluations (e.g. beau- tiful, ugly, shapeless). Again, they recognize positive and negative evaluations: prop- erties people value alongside those they do not.

2.2. RESOURCES IN OPINION MINING

engagement represents the expressions that authors use to express their point of view and the language they use to adopt their stances towards the opinions of other people. Examples of words classified in this category are say, believe, according to, it seems, apparently, admit- tedly, endorse, disagree, concede, confirm, proclaim, praise. Engagement is concerned with negative and positive stances.

(11) I agree with you on the following issues.

graduation Through graduation AF deals with the expressions speakers and writers use to alter the focus and strength of their appraisal and engagement. These expressions enable the speaker and writers to convey greater or lesser degrees of positivity or negativity, and to modify the appraisal expressions in terms of its intensity, quantity or temporality. Examples of words classified in this category are slightly, very, to a greater extent for strength and alleged, supposed to be, sort of for focus.

(12) I highly agree with you on the following issues.

(13) This is terribly tiresome.

According to AF, attitude can be expressed as a denoted attitude (also called directly inscribed attitude) or as an invoked attitude. Denoted attitude refers to the expressions that are included in the categories affect, judgement, appreciation and engagement and are considered the core of the attitudinal language. Examples (7,8, 9 and 10) are illustrations of denoted attitude. Expressions that ’invoke attitude’ are not considered part of the attitudinal lexicon, although they can convey feelings and attitude as illustrated by the following examples.

(14) The Spanish brought diseases to the newly conquered Americas (invoked attitude) (15) we smashed their way of life (invoked attitude)

(16) the earthquake led to over 5000 deaths (invoked attitude)

These sentences will raise an affectual response in most readers, although it is not explicitly stated in any of these sentences that something good or bad has happened. Whether affect is actually conveyed depends - more than in the case of inscribed attitude - on the context and on the background knowledge of the reader.

AF has influenced work in automatic opinion mining in particular by showing that the Affect-Appraisal-Judgement distinction is a useful one to be made (Bloom (2011)). Taboada and Grieve (2004) applied AF to categorize opinions found in text and used the attitude type (affect, appraisal or judgment) to determine whether the targets of the opinions refer to objects, emotions or behaviors, thus not only identifying different expressions but also different types of targets. They showed, for example, that reviews of products contain a high degree of appreciation words (referring to costs and quality) whereas book and movie reviews also contain a fair amount of judgement words. In addition, various studies have shown that aspects of AF could be translated into guidelines for the annotation of news paper articles Martin (2004) and book reviews Read and Carroll (2012).

What makes AF interesting for our study, is that it is lexically based and that the authors

2

2.2. RESOURCES IN OPINION MINING

2.2 Resources in opinion mining

2.2.1 Appraisal Framework

The Appraisal Framework (Martin and White (2005)) (AF) is a theory about how opinions and emotions are expressed in language. The theory gives a detailed description of opinion- ated language and offers a method for a qualitative analysis of speakers’ or writers’ attitudes and how they are linguistically and lexically encoded in text. It is based on the investiga- tion of individual texts, pieces of texts and groups of words. From these individual analyses, Martin and White (2005) derive a general model of opinions and attitudes in text and speech.

AF presents a general structure of how language is used to adopt or express an attitude of some kind towards some target. For example, in I found the movie quite monotonous, the speaker (the Appraiser) adopts a negative Attitude (monotonous) towards a target (e.g.

the movie. According to AF, attitudes come in different types, for example, monotonous de- scribes an inherent quality of the target, while loathed would describe the emotional reaction of the Appraiser. AF distinguishes three interacting components, i.e. attitude, engagement and graduation:

attitude is considered to be the core of the model. Three dimensions of attitude are distin- guished to wit affect, judgment and appreciation:

• affect: words that refer to personal emotions such as emotional reactions (e.g. sorrow, happy, fear, loath, cry, upset and desires (e.g. want, abhor). Affect is the most ex- plicitly subjective type of appraisal and it is concerned with registering positive and negative feelings.

(7) It was then with fury that I returned home (8) The captain wept

• judgement: words used for giving moral evaluations of people and the way they behave (e.g. fraud, swindle, intelligent, neurotic, unreliable). As with affect, they recognise positive and negative evaluations, that is traits that people admire and traits that people criticize.

(9) To see police brutally manhandling demonstrators was not only shocking but representative of more repressive regimes, such as China

• appreciation: words that refer to the value and evaluations of â ˘AŸthingsâ ˘A ´Z, perfor- mances and natural phenomena. It typically includes aesthetic evaluations (e.g. beau- tiful, ugly, shapeless). Again, they recognize positive and negative evaluations: prop- erties people value alongside those they do not.

2.2. RESOURCES IN OPINION MINING

engagement represents the expressions that authors use to express their point of view and the language they use to adopt their stances towards the opinions of other people. Examples of words classified in this category are say, believe, according to, it seems, apparently, admit- tedly, endorse, disagree, concede, confirm, proclaim, praise. Engagement is concerned with negative and positive stances.

(11) I agree with you on the following issues.

graduation Through graduation AF deals with the expressions speakers and writers use to alter the focus and strength of their appraisal and engagement. These expressions enable the speaker and writers to convey greater or lesser degrees of positivity or negativity, and to modify the appraisal expressions in terms of its intensity, quantity or temporality. Examples of words classified in this category are slightly, very, to a greater extent for strength and alleged, supposed to be, sort of for focus.

(12) I highly agree with you on the following issues.

(13) This is terribly tiresome.

According to AF, attitude can be expressed as a denoted attitude (also called directly inscribed attitude) or as an invoked attitude. Denoted attitude refers to the expressions that are included in the categories affect, judgement, appreciation and engagement and are considered the core of the attitudinal language. Examples (7,8, 9 and 10) are illustrations of denoted attitude. Expressions that ’invoke attitude’ are not considered part of the attitudinal lexicon, although they can convey feelings and attitude as illustrated by the following examples.

(14) The Spanish brought diseases to the newly conquered Americas (invoked attitude) (15) we smashed their way of life (invoked attitude)

(16) the earthquake led to over 5000 deaths (invoked attitude)

These sentences will raise an affectual response in most readers, although it is not explicitly stated in any of these sentences that something good or bad has happened. Whether affect is actually conveyed depends - more than in the case of inscribed attitude - on the context and on the background knowledge of the reader.

AF has influenced work in automatic opinion mining in particular by showing that the Affect-Appraisal-Judgement distinction is a useful one to be made (Bloom (2011)). Taboada and Grieve (2004) applied AF to categorize opinions found in text and used the attitude type (affect, appraisal or judgment) to determine whether the targets of the opinions refer to objects, emotions or behaviors, thus not only identifying different expressions but also different types of targets. They showed, for example, that reviews of products contain a high degree of appreciation words (referring to costs and quality) whereas book and movie reviews also contain a fair amount of judgement words. In addition, various studies have shown that aspects of AF could be translated into guidelines for the annotation of news paper articles Martin (2004) and book reviews Read and Carroll (2012).

What makes AF interesting for our study, is that it is lexically based and that the authors