WHAT DO THE PICTURES ADD?

(1)

WHAT DO THE PICTURES

ADD?

A quantitative study of the effects that enriched multimodal letters have on

retention and understanding in comparison to less enriched multimodal letters.

First version of the bachelor’s thesis

BA Communication- and Informationsciences

Student: Rob S. Hölscher

Student number: S2951223

Address: Otto Eerelmanstraat 2a

Phone number: +31613780916

Mail address: r.s.holscher@student.rug.nl

Faculty of Arts

University of Groningen, Groningen

Supervisor: Dr. J. Wildfeuer

Second reader: Dr. N. Van Schepen

Date of completion: 30-06-2020

(2)

Abstract

Recently, the municipality of Rotterdam has started enriching their letters by implementing pictures. The goal of these enriched multimodal municipal letters is to increase the level of understanding of their letters. We know that properly designed enriched multimodal artefacts can help people with understanding certain types of information (Bateman, 2014; Mayer, 2005). However, there has been no research yet on whether enriched multimodal municipal letters improve understanding and/or retention. So, this paper focuses on the effects of enriched multimodal municipal letters on understanding and retention and does so using the following research question: “To which extent do enriched multimodal municipal letters improve its receiver’s retention and understanding of the message in comparison to less enriched multimodal municipal letters?”.

To measure these effects, a comparison has been made with the use of two research conditions, being two enriched multimodal letters and two less enriched multimodal letters. Respondents of this quantitative study received one of the four research conditions, after which they answered a questionnaire. The questionnaires consisted of a Likert-scale part in which the understanding was assessed and multiple-choice questions in which the retention of the respondents was examined. The outcomes of these questionnaires were statistically processed and compared. Both the questions on understanding and retention were focussed on the text-image relations in the multimodal documents.

The results of this study teach us that the enriched multimodal municipal letters do not have an effect on retention. This implies that the respondents who received the enriched multimodal artefacts did remember the message to roughly the same extent and were able to recall the information as good as the respondents in the research group that received the less enriched multimodal artefact. The results also show that the enriched multimodal municipal letters have an effect on the concept of understanding. On average, the respondents that received the enriched multimodal letters felt that they understood the letters better than the respondents that received the less enriched multimodal letters.

A recommendation for further academic research would be to develop a clearer, and universal, framework for examining enriched multimodal letters, so that academic research on this phenomenon can be applied to more research fields and specific subjects concerning multimodality.

(3)

1. Introduction

Since 2019, the municipality of Rotterdam has made a change in the letters that are addressed to its inhabitants. The change is the addition of pictures to their, previously only textual, letters which makes these new letters enriched multimodal artefacts. The reason for the municipality to do this, is that according to Leene Communicatie (n.d.) the previously in Rotterdam used textual letters led to an 80% rate of understanding among the receivers. This implies that 20% of the readers did not understand the letters that the municipality sent. However, there is no research yet on whether the implementation of these new, enriched municipal letters has risen the level understanding. So, the subject of this study is the effect of enriched multimodal municipal letters on the level of understanding and retention in comparison to less enriched multimodal municipal letters. Within this scope, ‘less enriched multimodal municipal letters’ are letters that do not contain pictures. These less enriched multimodal letters cannot be defined as monomodal letters, as they have more than one ‘mode’. The enriched letters are called this way due to the fact that they are relatively the same as the less enriched letters, but have an additional mode, which is ‘images’.

Within the genre of multimodal municipal letters, the study focuses on what effect text-image relations have on retention and understanding, in comparison to municipal letters in which there is only textual information given.

As discussed before, this study intends to find effects of two different degrees of multimodality in municipal letters. The practical reason for this study, is to assess if the enriched multimodal letters do increase the receivers’ understanding and retention of the given information. this might be the case, over time more municipalities could implement this way of designing their letters to help their inhabitants understand and remember the information in the letters better.

Although the genre of municipal letters in comparison with multimodality is not widely discussed in the academic field, the amount of research on multimodality and its text-picture relations has been on the rise the last few decades (Hiippala, 2015). Maat (2006) mentions a rise in research on text-picture relations as well, but also states that there is no comprehensive theoretical framework for text-image research yet.

The goal of this study is to create a clear view on whether the addition of pictorial elements to municipal letters improve understanding and retention. The current amount of research on municipal letters in combination with text-image relations is not comprehensive, so it is also a clear goal to add academic knowledge of this field. The addition of research to this field will benefit municipalities and help designing their letters in order to help their inhabitants understand and remember the letters. So, this paper focuses on the effects of enriched multimodal municipal letters on understanding and retention and does so using the following research question: “To which extent do enriched multimodal municipal letters improve its receiver’s retention and understanding of the message in comparison to less enriched multimodal municipal letters?”. As is done conform Glenberg and Langston (1992); in this study there is no use of pictures that give extra information, only pictures that visualize an exact depiction of information stated in the text.

(5)

According to Glenberg (1979, as cited in in Glenberg & Langston, 1992), explicit repetitions have large effects on memory. That means that by only using pictures that do not contain extra information, like in the enriched multimodal municipal letters, the enriched letters should have a greater effect on retention. Therefore, the hypothesis concerning retention is that the enriched multimodal letters help respondents to remember the information in the text better.

In this quantitative study, two enriched multimodal letters of the municipality of Rotterdam and two less enriched multimodal letters as research conditions are used. The latter two conditions are modified to have the same information as the Rotterdam letters, but do not contain pictures. That brings it to a total of four research conditions, in which there are two different subjects of the letters. The reason for doing research with four different conditions, and not two, is to enhance the external validity. In this study, each respondent receives a randomly assigned letter and afterwards answers a questionnaire. The survey contains questions designed to examine different scores on the concepts of understanding and retention. The next chapter that will be presented, is about the theoretical background that is necessary to be appointed within the framework of this study. Following, the method-section is presented, wherein the materials and methods are disclosed after which the results of the study are displayed. The data is statistically processed, and its outcomes will be discussed in the conclusion. Ultimately, the shortcomings of this study and recommendations for future research are presented in the discussion.

(6)

2. Theoretical framework

For centuries, people have been designing artefacts in which texts and pictures are used together. These artefacts that combine multiple ‘modes’, such as text and pictures, are called multimodal artefacts. However, these multimodal artefacts have only been a subject of academic research for the last few decades (Hiippala, 2015). Previously, the different modes were studied more separately, without considering the relation between these modes. The general goal of constructing a multimodal artefact, is the idea of ‘meaning multiplication’ (Bateman, 2014. P.6). According to Bateman (2014, p.6), this means that the idea is that “under the right conditions, the value of a combination of different modes of meaning can be worth more than the information that we get from the modes when used alone”. This existence or non-existence of meaning multiplication in municipal multimodal letters is what this study is searching for, and especially for its effects on retention and understanding. Retention is, in short, according to Karpicke and Roediger (2007) the ability to remember certain information, in combination with the ability to reproduce that information. The concept of understanding can be briefly defined as an “active process that requires the connecting of facts, the relating of newly acquired information to what is already known, the weaving of bits of knowledge into an integrated and cohesive whole” (Nickerson, 1985, p. 234, as cited in Helmstad, 1999).

The right conditions of multimodal artefacts are often discussed and there is no framework to create these right conditions yet in terms of text-image relations. However, there are several studies that look into these conditions and its complications. Fillappatou and Pumfrey (1996) state that pictures improve general comprehension of communicated information, but the exact workings of these pictures are complex and differ among academic research. According to Fillappatou and Pumfrey (1996), pictures may also distract readers because of useless details or integration of information that was misunderstood. This means that the addition of pictures to a text could be either beneficial or not for the meaning multiplication depending on multiple factors (Bateman 2014, p.6). According to Van Leeuwen (2005b, as cited in Bateman, 2014, p. 249), there is not yet a detailed framework for discussing and examining multimodal documents, but the assessment of multimodal artefacts should at least contain a description of text-image relations using the concept of conjunctive relations. These conjunctive relations of Van Leeuwen (2005b, as cited in Bateman, 2014, p. 249) aim to describe what the exact function is of the information that is constructed in text-image relations.

While discussing retention of information that is given in multimodal artefacts, Mayer et al. (2001) states that “people learn ‘more deeply’ from text and image instead of from text alone”. Mayer et al. (2001) mentioned this in the ‘cognitive theory of multimedia learning’. The deeper learning would be caused by the dual-channel assumption. This assumption implies that people use different cognitive channels to process pictorial elements and textual elements, which would decrease the total cognitive load. The cognitive load is defined as the amount of effort a person has to give in order to process the given information. It is important to note, that in this cognitive theory of multimedia learning, people are described to be studying information in the given multimodal artefact. This notion of studying is not necessarily the case with the multimodal municipal letters. Also, Van Hooijdonk and Krahmer (2008) state that text and pictures do not always help each other, their reason is that

(7)

a reader has to shift constantly between the two information sources. This constant shifting would impair the ability to take in information.

If an artefact consists of the mode ‘text’ and no further audio-visual modes are used, people have to translate the textual instructions, according to Kaschak and Glenberg (2000). In this context, the concept of translation can be defined as the displacement or transfer of textual information to real world concepts. The hypothesis of Kaschak and Glenberg (2000), “readers associate words and phrases with objects and actions in ‘the real world’, which should facilitate understanding”, makes it easier to understand why low-literate people would benefit from the addition of pictures, when they don’t understand the text itself, or have trouble translating the information. Glenberg and Langston (1992) conducted two experiments on the comprehension and retention of their students, using illustrated texts. They mention that in illustrated texts, in which the pictures do not contain any extra information than is stated in the text, the pictures help people to comprehend and remember texts. The reason for this is, according to Glenberg and Langston (1992), that the pictures help build certain ‘mental models’ that would help understand and remember the text itself.

However, whether all people benefit from this, is still the question in terms of understanding and retention of multimodal municipal letters. This study will be conducted to assess whether multimodal municipal letters improve the construction of the abovementioned mental models, and therefore help comprehension and retention (Glenberg and Langston, 1992). Because this exact kind of study has not been conducted previously, it might be the case that the enriched multimodal municipal letters reduce or simply do not improve understanding. That could eventually be due to the notion in Van Hooijdonk and Krahmer (2008), which argues that the reader of a multimodal document has to constantly shift between the sources of information, or modes, which reduces understanding. All of these different arguments and conclusions about multimodal artefacts, are therefore put to the test in this study concerning multimodal municipal letters. The conclusions of studies that are given in this theoretical framework mainly focus on a particular set of concepts or specific sub-areas, and that might be a reason why they differ so much from each other. In this study, the available frameworks and perspectives that are created in the abovementioned studies will be used to assess multimodal artefacts in the specific sub-area of municipal letters. Bateman (2014) and the abovementioned theories make up for a rather detailed framework to assess and analyse the multimodal enriched letters. They also give a research start in which the breadth of multimodality is displayed with the intention to avoid involuntary inclusions. The need for this study is eventually to assess whether enriched multimodal municipal letters improve its receivers’ understanding and retention. The final goal is to create a theoretical understanding on why municipalities should transform their standard letters into enriched multimodal letters or not.

(8)

3. Method

Before addressing the method-section of this report of the study, it’s useful to once again visit the research question, hypotheses and the concept definitions. The research question is defined as “to which extent do enriched multimodal municipal letters improve its receiver’s retention and understanding of the message in comparison to less enriched multimodal municipal letters?”. Because there are two concepts that will be examined, being both understanding and retention, there are two different hypotheses. The hypothesis concerning the concept of retention is that the enriched multimodal letters help respondents to remember the information in the text better. This is because the enriched multimodal letters contain pictures which depict words that are stated in the text next to them and according to Glenberg and Langston (1992), this repetitive nature of presented information helps to better remember. The hypothesis concerning understanding, is that enriched multimodal letters help the readers understand the information in the letters better than the less enriched multimodal letters. This hypothesis about the concept of understanding arises from the fact that Carney and Levin (2002) studied text-image relations and came to the conclusion that well-arranged images in multimodal artefacts provide an overall growth in information uptake and understanding, due to the relative simplification that the images bring. This simplification is constructed due to the easier translation of written information into ‘the real world’ by depicting images (Kaschak & Glenberg, 2000). In rest of the method section, the design of the experimental study will be discussed firstly. Hereafter, the variables and materials will be displayed, after which the operationalisation is drawn up. Next, the respondents are described, and the method section will follow with a delineation of the overall procedure within this study. Finally, there will be a section in which the data-analysis will be discussed.

3.1 Design

As stated before, this study has an experimental design. The experiment has four conditions, being two conditions that are original multimodal artefacts (appendix 1) and two conditions that are manipulated (appendix 2). Within the four research conditions, there are two different subjects. The first subject is a letter of the municipality of Rotterdam that contains the request to extend a passport, and the second is a letter of the municipality of Rotterdam that has a remission of waste costs denial as a subject (table 1).

Table 1. The four different research conditions.

Subject Original multimodal artefact

Manipulated multimodal artefact

Receiver of the letter has to extend a passport Research condition 2: named as paspoortconditie B Research condition 1: named as paspoortconditie A

Receiver of the letter has been denied remission of waste costs Research condition 4: named as kwijtscheldingsconditie B Research condition 3: named as kwijtscheldingsconditie A

(9)

The reason that there are four different research conditions instead of two, is to raise the internal validity as well as the eventual external validity of the study.

The two concepts, that are understanding and retention, are measured with the answers of respondents on a survey (appendix 3). The participants were randomly assigned to a survey with one of the four research conditions to increase the external validity of this study. The first part of the questionnaire had an introduction as its subject to instruct the respondents. The second part consisted of demographical questions and the third part was a request for an extensive review of the multimodal document, after which one of the four research conditions followed. Following, statements about understanding were presented to the respondents, which they answered using a Likert-scale. The retention was tested using multiple choice questions about what the respondents remember of the randomly assigned multimodal document.

It is important to mention that in the design of this study there has been no pre-test, so this experiment aimed to study only the understanding and retention of respondents at one given time. This means that there is no comparison between a progression of the respondents concerning understanding or retention. Also, because the original multimodal documents (research conditions 2 and 4) were in Dutch, only Dutch-speaking respondents could participate in the study. To accommodate these Dutch speaking respondents, the questionnaire (appendix 3) was entirely presented in Dutch as well.

3.2 Variables

A questionnaire was used to measure the respondents’ understanding and retention of the multimodal artefacts. The concept of the degree of understanding is measured using a Likert-scale. This 5-point Likert scale varied from ‘completely disagree’ to ‘completely agree’, with a neutral option in the middle. That means that the variable ‘understanding’ was measured on an ordinal level, because the variable consists of multiple values that have differences between them, but the values are not particularly easy to point out.

The concept of the degree of retention is measured using multiple-choice questions about recall of the multimodal letter by the respondents. Important to note, is that the questions were based on information that was stated in text-image combinations that the multimodal document possessed. Because the answers of the respondents can only be ‘correct’ or ‘wrong’, the variable of retention is measured on a nominal level. Within this nominal level, the variable has different values but the differences between these values is not expressible in numbers.

Moving on with the two independent variables of this experimental study. The first independent variable of this study is the fact that in research condition 2 and 4 (appendix 1), there are pictures within the letters. The second independent variable is that in research condition 1 and 3 (appendix 2), there are no pictures in the multimodal documents. These latter stated research conditions are the manipulated documents.

(10)

3.3 Material

The material that has been used in this study is constructed via different sources. However, the original multimodal instructions (research conditions 2 and 4, appendix 1) both have the same source, namely the municipality of Rotterdam. These two letters were created by Leene Communicatie, a communication agency that designed the letter specifically for the municipality of Rotterdam, because of its vast amount of analphabetic and low-literate inhabitants. The goal of these two letters is to improve the degree of understanding amongst the low-literate and analphabetic receivers by adding pictorial elements next to written text. This would help explain the information or give clear examples of it. Before constructing two different manipulated versions of the letters, the text-image relations of the original letters were analysed using methods in Bateman (2014). This analysis is widely discussed in appendix 1. Using the analysis, there has been expressed that within both of the original multimodal documents (also named ‘enriched multimodal documents’), the text-image relations lack Barthes’ linguistic messages, but do have differences in coded iconic and uncoded iconic messages (as cited in Bateman, 2014). When analysing these multimodal documents using Barthes’ text-image analysis methods, it also became clear that no combination of the text and pictures was equal. The framework of Van Leeuwen (2005b, as cited in Bateman, 2014) for assessing text-image relations in multimodal artefacts lined up with the conclusions stated above.

To analyse the effects that these text-image relations have in terms of retention and understanding, there have been made two different manipulated multimodal documents (research conditions 1 and 3, appendix 2). These documents are exact copies of the original letters that the municipality of Rotterdam uses, but have only one manipulated difference. This difference is that the manipulated letters do not contain any pictorial elements, so there are no text-image relations within the letter to interfere with the respondents understanding and recall of the given information.

When describing the multimodal documents briefly, it is necessary to state the subject of the letters that are used in this study. Research conditions 1 (appendix 2) and 2 (appendix 1) are letters by the municipality of Rotterdam that have the ‘renewal or extending of a passport’ as subject. Research conditions 3 (appendix 2) and 4 (appendix 1) are also letters by the municipality of Rotterdam that are sent to its inhabitants and have ‘the denial of remission of waste costs’ as a subject.

Another piece of material that is used in this study, is the questionnaire (appendix 3) that was presented to respondents in Qualtrics.com. Not all respondents received the same questionnaires, because the questions that they received depended on what research condition the respondents got randomly assigned to. This means that for each research condition there is a special set of questions, which is be clarified above every part of the questions beneath in appendix 3. As a result, the two subjects of the multimodal letters, being a passport-renewal and a remission of waste costs, each ensure a segregation in the questions and the two conditions (original letter and manipulated letter), make up for the need to adjust and divide different questions. Also, an important note; the questions are in Dutch, but halfway appendix 3 the questions are translated and displayed in English. As has been discussed before, the questions have multiple differences to measure different variables. To measure the retention that

(11)

the respondents had, there were multiple-choice questions asked that measured their recall. This way of ‘testing’ the respondents made it relatively easy to assess whether the respondents that received one research condition, had more or less recall than the respondents that received another research condition. To measure the understanding that the respondents had concerning the multimodal document that they were randomly assigned to, there were questions asked based on a 5-point Likert-scale. This way of assessing the respondent’s opinions on certain statements, allowed for statistically processing the answers and finding differences in answers between the research conditions.

The statistical processing of all the data that were gathered using a survey via Qualtrics.com, was done in Microsoft Excel and Rstudio.

3.4 Operationalisation

To be able to answer the research question “To which extent do enriched multimodal municipal letters improve its receiver’s retention and understanding of the message in comparison to less enriched multimodal municipal letters?”, it is required to establish an exact description of both retention and understanding. The term ‘understanding’ is within this study seen as a synonym of comprehension, like discussed and debated in Alvermann and Wilson (2011, p. 119). The overarching definition of understanding according to Alvermann and Wilson (2011, p.119), is defined as “the reconstruction of the intended meaning of a communication, where the presumption … is that meaning resides in the message awaiting interpretation, and that the message received is congruent with the message sent” (Harris & Hodges, 1955, p. 38-39, as cited in Alvermann & Wilson, 2011, p 118). According to Nickerson (1985, p. 234, as cited in Helmstad, 1999), understanding can be defined as an “active process that requires the connecting of facts, the relating of newly acquired information to what is already known, the weaving of bits of knowledge into an integrated and cohesive whole”. Therefore, understanding will be seen and discussed as a concept that describes the degree of ability to reconstruct, link and relate new information to previously acquired knowledge. However, this concept is not the same as retention. The second concept that is being examined within this multimodal study is retention. Retention is, according to Karpicke and Roediger (2007), the ability to remember certain information, in combination with the ability to reproduce that information. In this study it must clearly be stated that the difference between understanding and retention, is that within the concept of retention there first has to be the ability to remember the given information in order to reproduce the information.

Measuring and examining the variables of retention and understanding within the four research conditions will be done using the answers of respondents to a questionnaire.

(12)

3.5 Respondents

In total, there were 229 respondents that finished the questionnaire. Among these 229 respondents, there were 75 male respondents (32.75%), 152 female respondents (66.38%) and two respondents that tick the option ‘other’ (0.87%). The sexes of the respondents were divided between the research conditions as stated in table 2. The average age of all the respondents is 34.83 years of age (table 2).

Also, the highest level of education was questioned in the demographical questions of the survey. These education levels were divided in three levels of education in which relatively many Dutch people have obtained their diplomas or are still planning to obtain them. If there was a question that a respondent did not have one of these three Dutch education levels, the respondents could tick the option ‘other, namely…’. 57 (24.89%) of the respondents have enjoyed the highest level of education in the Netherlands, wetenschappelijk onderwijs (WO). The second highest level of education in the Netherlands is hoger beroepsonderwijs (HBO), and 108 (47.16%) of the respondents have enjoyed this level of education. The third level of education, middelbaar beroepsonderwijs (MBO) was enjoyed by 52 (22.71%) respondents (table 2).

The last demographical question that was asked was about the place of residence of the respondents (table 2). Relatively seen, a lot of the respondents live in Groningen and Emmen. 64 (27.95%) of the respondents live in Groningen, 92 (40.17%) of the respondents live in Emmen, and 73 (31.88%) respondents live scattered across other parts of the Netherlands.

All of the respondents were Dutch natives or could at least speak the Dutch language. This requirement was specifically created, because the original multimodal documents (condition 2 and condition 4), were written in Dutch.

(13)

Table 2. Summary of the respondent’s answers to the demographical questions. Research condition Age

(m)

Sex Education level Place of residence Paspoortconditie A 29.95 years of age Male: 20 (36.36%) Female: 34 (61.82%) Other: 1 (1.82%) WO: 15 (27.27%) HBO: 27 (49.09%) MBO: 9 (16.36%) Other: 4 (7.27%) Groningen: 18 (32.73%) Emmen: 18 (32.73%) Other: 19 (34.55%) Paspoortconditie B 39.35 years of age Male: 29 (33.33%) Female: 57 (65.52%) Other: 1 (1.15%) WO: 16 (18.18%) HBO: 35 (39.77%) MBO: 30 (34.09%) Other: 7 (7.95%) Groningen: 22 (25.29%) Emmen: 40 (45.98%) Other: 25 (28.73%) Kwijtscheldingsconditie A 38.86 years of age Male: 13 (30.23%) Female: 30 (69.77%) WO: 11 (25.58%) HBO: 22 (51.16%) MBO: 9 (20.93%) Other: 1 (2.33%) Groningen: 7 (16.28%) Emmen: 17 (39.53%) Other: 19 (44.19%) Kwijtscheldingsconditie B 28.05 years of age Male 13 (29.55%) Female: 31 (70.45%) WO: 15 (34.09%) HBO: 24 (54.55%) MBO: 4 (9.09%) Other: 1 (2.27%) Groningen: 17 (38.64%) Emmen: 17 (38.64%) Other: 10 (22.7%)

Total of all respondents 34.83

years of age Male: 75 (32.75%) Female: 152 (66.38%) Other: 2 (0.87%) WO: 57 (24.89%) HBO: 108 (47.16%) MBO: 52 (22.71%) Other: 13 (5.68%) Groningen: 64 (27.95%) Emmen: 92 (40.17%) Other: 73 (31.88%)

3.6 Procedure

To allow respondents to participate in this study, a message was posted on various social media pages (Facebook, LinkedIn and Instagram). This message began with an informal greeting, after which followed an explanation of why this study is being conducted and a short invitation to participate. This short invitation to participate was supplemented by the possibility to win a gift voucher from a popular Dutch webstore through a draw among the respondents. Also, the respondents were informed that the study would take a maximum of ten minutes of their time. Following, the people who were willing to participate, were asked to click one in four links randomly, that led them to one of the four research conditions in Qualtrics.com.

(14)

On the first webpage in Qualtrics, which was the same for everyone, people were yet again briefly informed about the intentions of the study and the tasks that they were expected to perform. On this page the respondents were also informed that they had the possibility to stay completely anonymous and that their data would be handled discreetly. Hereafter came some demographical questions, one of the four research conditions that they were asked to study, and a questionnaire that they filled in after studying the letter that was randomly assigned to them. In the end of the online questionnaire, the respondents were thanked and informed to seek contact via the stated credentials if they had any questions or comments about the study.

As has been stated before, the respondents could participate in the study online. This minimized the time to conduct the study, and the respondents could fill it in whenever they wanted. Off the record, some respondents had additional comments on the size of the multimodal artefacts that they were shown within the study. These comments were all about the size of the depictions of the letters and they therefore stated that these were a tad on the small side. Unfortunately, an enlarged version of the letters could not be included when drawing up the questions in Qualtrics.com.

3.7 Data analysis

As mentioned above, the data was collected using a questionnaire in Qualtrics.com. Using this questionnaire, data of the effects on retention and understanding of the multimodal artefacts were received. The data of the effects of the four research conditions on retention/recall were received because of the answers on the multiple-choice questions and the data of the effects on the four research questions on understanding were received because of the answers on the statements with a Likert-scale answering option. The data were statistically processed using the programme Microsoft Excel and Rstudio. The standard deviations, the means and other data were calculated using these programmes.

When assessing the retention/recall of the respondents, the data on whether participants answered the questions correctly or wrong was used to assess if the type of multimodal document had any effects on retention. The concept of understanding was measured quite differently, because it was not simply counting the wrong or false answers, but this data had ordinal values. The scores that were assessed to investigate the understanding of the respondents of the multimodal artefacts are classified as follows; 1 is strongly agree and a score of 5 is strongly disagree. That means the higher the score, the less respondents agree with the given statements. After processing the data to get the means of each condition, the results of each research condition were examined to assess similarities, differences and other details that are worth mentioning. Because there are two independent variables and to dependent variables in this study, the data were statistically processed using two Wilcoxon Signed Rank tests in Rstudio. The first Wilcoxon Signed Rank test was used to examine the differences between research conditions 1 and 2 with the data that was gathered on the concept of understanding. The second Wilcoxon Signed Rank test was used to examine research conditions 3 and 4, to examine the differences between them on understanding as well. Ultimately, the results of these tests were analysed

(15)

and processed, after which clear statements could be made with which the final research question could be answered. The data on differences in the effect of the multimodal letters on retention were processed simpler, because they were gathered using a multiple-choice test. The amount of good and bad answers was counted and summarized by calculating the means of the number of good answers per research condition.

(16)

4. Results

The results will be discussed in this chapter of the study. The layout will be as following. Firstly, the reliability of sets of questions will be widely discussed, after which the results of the study on the concept of understanding is set out. In the end of this chapter, the effect on retention will be shown and explained accordingly.

4.1 Reliability

Because of the fact that there were multiple question sheets for multiple research conditions, the reliability of the questions was tested per subject. The first subject, the concept of understanding by the respondents, was universal amongst all four different research conditions. This concept was measured using questions 5, 6, 7, 8 and 9 (appendix 3). These questions were statistically processed to measure reliability with Cronbach’s alpha. The Cronbach’s alpha (𝛼 = 0,63) showed that the reliability of these Likert-scale questions was questionable. However, there were more questions concerning the understanding of the multimodal documents (part 3 and part 4 in appendix 3). These questions had also had to be processed to measure reliability with Cronbach’s alpha. The Cronbach’s alpha in part 3 of the questions (appendix 3) was assessed and was 𝛼 = 0.50. This is classified as poor reliability, but after deleting question 13 (appendix 3), the Cronbach’s alpha rose to 𝛼 = 0.65, which is questionable to acceptable. The reliability of part 4 of the questionnaire was 𝛼 = 0.68, so this part 4 is also questionable to acceptable.

However, when assessing all the questions that consisted out of Likert-scales in research condition 1 and 3 (they had the same questionnaire because the research conditions were both less enriched multimodal artefacts), the Cronbach’s alpha was 𝛼 = 0.88, which is reliable due to its classification as ‘good’. The same process was executed for the Likert-scale part of the questionnaires of research conditions 2 and 4 (they had the same questionnaire because the research conditions were both enriched multimodal artefacts) and it reached the reliability of 𝛼 = 0.78, which is considered to be acceptable. Therefore, the Likert-scale parts of the questionnaires can both be considered to be acceptable in terms of reliability.

4.2 Understanding

Questions 5-9 (appendix 3) were specially designed to assess the concept of understanding of the multimodal documents. Within these questions, the 5-point Likert-scale varied from ‘completely disagree’ to ‘completely agree’, with a neutral option in the middle. This means that the lower the scores, the more the respondents agreed with the statements that they were given. The mean of all answers to questions 5-9 is 2.15 (SD = 0.91). The mean to all answers to questions 5-9 that were asked in enriched multimodal artefacts (research conditions 2 and 4) was 2.15 (SD = 0.91). The mean to all answers to questions 5-9 that were asked in less enriched multimodal artefacts (research conditions 1 and 3) was also 2.15 (SD = 0.91). In table 3 the means of all questions and the total means are presented.

(17)

Table 3. The means of questions 5-9 and the total means.

Condition Question 5 Question 6 Question 7 Question 8 Question 9 Total Research condition 1 1.97 1.53 1.57 2 2.77 1.97 Research condition 2 2.37 2.02 2.11 2.42 2.27 2.3 Research condition 3 3.38 2.62 2.45 2.24 2.43 2.42 Research condition 4 2.07 1.89 2.04 1.96 2.07 2.01

In table 3 the assessment of the understanding by the respondents on the entire research condition that they received is presented. The respondents that were randomly assigned to the enriched multimodal artefacts (research condition 2 and 4) were asked specifically in questions 10-13 (appendix 3) if the pictures enhanced their understanding and the mean of their answers was 2.26 (SD = 1.01). The respondents that were randomly assigned to the less enriched multimodal artefacts (research condition 1 and 3) were asked in questions 14-16 (appendix 3) if they thought that pictorial elements or images would enhance their understanding of the multimodal artefact. The mean of their answers was 2,70 (SD = 0.98).

To examine the data more deeply, two different Wilcoxon Signed Rank tests were executed. The reason for using Wilcoxon Signed Rank tests, is because of the goal to test if the results, that are processed in the data of the questionnaire, differ from each other per research condition. The Wilcoxon Signed Rank test is an alternative to the more known T-test, but it does not need normally distributed data to be processed and executed. Another reason for the Wilcoxon Signed Rank test is because of the fact that this test is able to process ordinal values, which the answers in the questionnaires on Likert-scales are. The reason that research conditions 1 and 2, and research conditions 3 and 4 are compared separately, is because research condition 1 and 2 have the same subject and research condition 3 and 4 have the same subject. Due to the fact that they have the same subject, they can be compared to assess the differences in understanding.

The first Wilcoxon Signed Rank test was conducted to find differences between understanding in research conditions 1 and 2. Part 2, part 3 and part 7 of the questions (Appendix 3) were used to assess the differences amongst the research conditions, so these were deferred to the two-tailed Wilcoxon Signed Rank test. H0 in this test was ‘H0 = there is no difference between understanding in research condition 1 and 2’. H1, the alternative hypothesis was ‘H1 = there is a difference between understanding in research condition 1 and 2’. In the process, α = 0.05. With research condition 1 and 2, the test statistic was 118 and n = 47. At n = 47 and α = 0.05, the critical value is 378. This means that H0 is rejected, because 118 < 378. There is

(18)

sufficient evidence to suggest that there is a difference between the additives of research conditions 1 and 2 in terms of understanding the multimodal documents. As said before, the Wilcoxon Signed Rank test was also conducted to assess if there was any difference between research condition 3 and 4 in terms of understanding. As was the same in the comparison between research conditions 1 and 2, H0 in this test was ‘H0 = there is no difference between understanding in research condition 3 and 4’. H1, the alternative hypothesis was ‘H1 = there is a difference between understanding in research condition 3 and 4’. The number of respondents is n = 27. With n = 27, the α = 0.05 that is used, gives the critical value in Wilcoxon Signed Rank

tests of 107. The test statistic of the Wilcoxon Signed Rank test between research

conditions 3 and 4 is 69. That means that the test statistic is less than the critical value and H0 is rejected (H0 is rejected because 69 < 107). There is sufficient evidence to suggest that there is a difference between the additives of research conditions 3 and 4 in terms of understanding the multimodal documents.

4.3 Retention

To assess the retention of respondents in every research condition, multiple-choice questions about the content of the letter had to be made (appendix 3). In the research conditions 1 (appendix 1) and 2 (appendix 1) there were four multiple-choice questions that tested the respondents recall of information that could be obtained in parts of the letter that was stated within a text-image relation in research condition 2. In table 4 the total number of answers on the questions that belongs to research condition 1 are given, along with the total number and percentages of correct answers in that particular question. This process was repeated for research condition 2 in table 5. Table 4. measuring recall in research condition 1 with questions designed for recall.

Answers Question 17 Question 18 Question 19 Question 20 Total number of answers on this question 31 32 31 31 Total of correct answers 10 (32.26%) 23 (71.88%) 26 (83.87%) 25 (80.65%)

Table 5. measuring recall in research condition 2 with questions designed for recall.

Answers Question 17 Question 18 Question 19 Question 20 Total number of answers on this question 48 48 48 48 Total of correct answers 16 (33.33%) 33 (67.75%) 37 (77.88%) 29 (60.42%)

(19)

When assessing table 5 and table 6 above, it seems that there is a difference in percentages of correctly answered answers. The respondents that were randomly assigned to research condition 1 answered 67.2% correctly, and the respondents that were randomly assigned to research condition 2 answered 59.9% of the questions that had retention as a subject correctly.

Here the assessment of the results of the part on retention within the questionnaires in research conditions 3 (appendix 1) and 4 (appendix 1) will be discussed. There were three multiple-choice questions that tested the respondents’ recall of information in research condition 4. This information could only be obtained in parts of the letter that present a text-image relation. In table 6 the total number of answers on the questions that belongs to research condition 3 are given, along with the total number and percentages of correct answers in that particular question. This process was repeated for research condition 4 in table 7.

Table 6. measuring recall in research condition 3 with questions designed for recall. Answers Question 22 Question 23 Question 24 Total number of answers on this question 21 21 21 Total of correct answers 14 (66.67%) 16 (79.19%) 12 (57.14%)

Furthermore, table 7 gives the fact that within research condition, the respondents that were randomly assigned to this condition, answered 66.67% of their questions correctly.

Table 7. measuring recall in research condition 4 with questions designed for recall. Answers Question 22 Question 23 Question 24 Total number of answers on this question 28 28 28 Total of correct answers 23 (82.14%) 20 (71.43%) 12 (42.86%)

Table 7 shows that the respondents of research condition 4 answered 65.48% of their questions correctly.

(20)

5. Discussion

The goal of this study was to assess whether the newly implemented, and enriched, multimodal letters of the municipality of Rotterdam improved the retention and understanding among its receivers. There were multiple reasons to conduct the study. The first reason was simply that the municipality of Rotterdam did not test the effects of their enriched multimodal letters, which means that we did not know if the enrichment of the multimodal letters actually improved the transmission of

information. The second reason for conducting the research was to create a clear picture of what multimodality in a letter form does in terms of retention and

understanding. Although letters are an everyday phenomenon for many people, they have not yet been examined from this perspective and there has been composed a structured framework for these multimodal artefacts yet (Hiippala, 2015).

Primarily, the effect that enriched multimodal letters have on the concept of retention will be discussed. Based on a previously stated study by Glenberg (1979, as cited in in Glenberg & Langston, 1992), the hypothesis was stated as follows: “the enriched multimodal letters help respondents to remember the information in the text better”. However, this research shows that the enriched multimodal letters have no improved effect on retention compared to the less enriched multimodal letters. Therefore, the previously prepared hypothesis rejected. Research conditions 2 and 4 were the enriched multimodal letters, and on average, the respondents that received these documents scored not as good on the questions that were designed to examine their retention compared to the respondents did that received research conditions 1 and 3. The average score on correctly answered questions in research condition 1 was 67.2%, and the average score on correctly answered questions in research condition 2, which had the same subject and text, was 59.9%. A similar effect was found in the comparison of correctly answered questions between research condition 3 (66.67% correct answers) and research condition 4 (65.48%). With these data and results, it is possible to conclude that enriched multimodal letters have no positive effects on retention, compared to less enriched multimodal letters. This might be due to the fact that the people who receive the letters are not actively studying the given information as in the study of Mayer (2005). This is in line with the conclusions of Van Hooijdonk and Krahmer (2008), which stated that the risk of a multimodal artefact is that the reader/viewer has to constantly shift between information sources, and this may impede the recording of extra information. The analysis of the multimodal documents (Appendix 1) have shown that all the text-image relations in the letters are unequal in terms of Barthes’ three messages (Bateman, 2014). So, these results mean that enriched multimodal letters that contain only unequal text-image relations, have no positive effect on retention in comparison with less enriched multimodal letters. The implication of this conclusion means that, when purely focused on retention, a municipality should not bother to create a multimodal variant of their letters.

Secondly, the effect that enriched multimodal letters have on the concept of understanding will be discussed. The hypothesis was stated as follows: “enriched multimodal letters help the readers understand the information in the letters better than the less enriched multimodal letters”. To examine the concept of understanding and the differences among the research conditions, Likert-scale questions from part 2, part 3 and part 7 (appendix 3) were used. The reason for this, is that the goal of these questions was to measure understanding and they occurred in every questionnaire,

(21)

regardless of what research condition the respondent was randomly assigned to. This made it possible to examine the effects on understanding of al research conditions in the same way. The research shows that enriched multimodal letters have an effect on understanding. The different perceptions of the enriched multimodal research conditions (conditions 2 and 4) were compared with the perceptions of the less enriched multimodal research conditions (conditions 1 and 3) using two Wilcoxon Signed Rank test. This test showed that there is a substantial difference between the multimodal artefacts. The difference is that the enriched multimodal letters, the original research conditions, are found to be easier to understand according to the respondents. This conclusion stems from multiple parts in the results, but mostly from the fact that in the comparison of research condition 1 and 2, there was sufficient evidence to suggest that there is a difference between the additives in terms of understanding the multimodal documents. The comparison between research condition 3 and 4 has evenly contributed to the conclusion, because it had a similar result, namely that there was also sufficient evidence to suggest that there is a difference between the additives in terms of understanding the multimodal documents. Therefore, there is no data to refute the hypothesis concerning understanding. These results could be explained by the previously mentioned conclusions of Bateman (2014) and Fillappatou and Pumfrey (1996). Fillappatou and Pumfrey (1996) stated that pictures within a multimodal document have the power to improve general comprehension of the to be communicated information. The study also mentioned that pictures may distract readers, because of useless details or integration of information that was misunderstood. However, in the enriched multimodal letters that we use in this study, the text-picture relations are all unequal according to the analysis of the artefacts. Because the pictures give a less detailed form of communication than the text that is related to it, it is likely and plausible that the unequal relation ensures the positive effect on understanding of the enriched multimodal letters.

To repeat Bateman’s (2014, p.6) idea of meaning multiplication once again; “under the right conditions, the value of a combination of different modes of meaning can be worth more than the information that we get from the modes when used alone”. The fact that the respondents believe that they understood the enriched multimodal letters better than the less enriched multimodal letters, can be appointed as a form of meaning multiplication. This means that the multiple modes possessed by the letters, work together to obtain greater overall informational value in terms of understanding. However, this does not necessarily mean that all enriched multimodal letters have effect on the overall understanding. It must be said that the enriched multimodal letters should have the same components as have been analysed in Appendix 1. These include concepts such as unequal text-image relations. Based on these conclusions, one cannot give a verdict on other types of multimodal letters, even when they have the same subject. In addition, one can doubt whether full meaning multiplication has been achieved with the use of these letters. This is due to the fact that the effects of the original research conditions (the enriched multimodal letters) had, relative to the less enriched multimodal letters, virtually no positive effect on the retention by respondents of the given information.

Methodological limitations

The fact that people could skip questions in the survey, led to a rather big loss of data. This is because only the fully completed questionnaires were considered to be subject of examination. Because of the loss of data, it would be better in the future to add the

(22)

restriction in the questionnaire where respondents can only make progress when answering all of the questions on a certain page. It could have been the case that respondents who did not complete the questionnaire did not do this due to incomprehension of the idea that they no longer knew the answers.

Also, within this research there was only time to test two original documents, the municipality of Rotterdam has issued and published more versions of enriched multimodal letters than the two that are discussed. Furthermore, although there were a relatively large number of participants in this study, the respondents all came from a social network, which may have influenced the results. In addition, low literacy is more common among the elderly than among the younger population. Because the respondent group was relatively young, it may be that the enriched multimodal letters have different effect within a group with different ages. A textual questionnaire was also administered, making it difficult or impossible to reach low-literate people that were able to respond to this study. These implications create a potentially lower external validity of the study.

Recommendations for future research

The first recommendation for future research is to conduct this study elsewhere, and with a different group of respondents, with the goal to find if the same conclusions can be drawn up. In addition, there is a recommendation to carry out the same type of study, in which the enriched multimodal documents contain content that have equal text-image relations. The reason for this, is to investigate whether the effects on retention and understanding will be different from the results revealed in this study. The third recommendation for follow-up research is to test letters that are similar, which means that they have the same content and presentation as analysed in appendix 1, in other areas. These areas could be different research-areas or areas that have another subject than municipal notices.

(23)

6. Conclusion

This study was conducted to assess the effects that enriched multimodal documents have on retention and understanding. The enriched multimodal letters were compared to less enriched multimodal letters to examine the differences between the two versions of municipal letters. The research question that was leading in this study was designed as follows; “To which extent do enriched multimodal municipal letters improve its receiver’s retention and understanding of the message in comparison to less enriched multimodal municipal letters?”. This research question was connected to the assessment of two enriched multimodal letters, the analysis and eventually the data gathered from a questionnaire. The data collection was carried out by having respondents fill in questionnaires about their randomly assigned research condition. The data that was collected, was statistically processed in Microsoft Excel and RStudio. Hereafter, the comparisons and effects were collected and written down in the results-section. Eventually, a detailed discussion and this conclusion were inserted. The intention of this study was to gain new insights in the workings and effects of the information commanded in text-image relations of enriched multimodal municipal letters.

The results of this study teach us that the enriched multimodal municipal letters have no effect on retention in comparison to less enriched multimodal municipal letters. This means that, on average, the respondents that were randomly assigned to the enriched research conditions answered the same amount of questions correctly in comparison to the respondents that were randomly assigned to the less enriched research conditions. The results of this study also teach us that the enriched multimodal municipal letters have an effect on understanding. On average, the respondents that were randomly assigned to the enriched multimodal letters, indicated that they better understood the letters than the respondents that were randomly assigned to the less enriched multimodal letters.

These conclusions mean that the future of the enriched multimodal letters from municipalities depend on the goals that municipalities want to achieve with them. If a municipality wants to increase retention, the enriched multimodal letter is not an immediate recommendation to use. However, when it comes to the improvement of understanding, the use of enriched multimodal letters can be recommended. Although it is necessary to use enriched multimodal letters as they have been analysed (appendix 1). And the final conclusion about this study; the analysis of the multimodal documents can be used as a framework to construct enriched multimodal letters, whenever a municipality has the goal to affect understanding. That means that this study has assessed and examined the effects of multimodal letters on retention and understanding, as well as contributed a part to the background of municipal policy making.

(24)

Bibliography

Alvermann, D. E., & Wilson, A. A. (2011). Comprehension strategy instruction for multimodal texts in science. Theory into Practice, 50(2), 116-124.

Bateman, J. (2014). Text and image: A critical introduction to the visual/verbal divide. Routledge.

Carney, R. N., & Levin, J. R. (2002). Pictorial illustrations still improve students' learning from text. Educational psychology review, 14(1), 5-26.

Filippatou, D., & Pumfrey, P. D. (1996). Pictures, titles, reading accuracy and reading comprehension: A research review (1973-95). Educational Research, 38(3), 259– 291

Glenberg, A. M. (1979). as cited in Glenberg, A. M., & Langston, W. E. (1992). Comprehension of illustrated text: Pictures help to build mental models. Journal of memory and language, 31(2), 129-151.

Glenberg, A. M., & Langston, W. E. (1992). Comprehension of illustrated text: Pictures help to build mental models. Journal of memory and language, 31(2), 129-151.

Helmstad G. (1999). Understandings of Understanding. An Inquiry Concerning Experiential Conditions for Developmental Learning. Goteborg Studies in

Educational Sciences, Acta Universitatis Gothoburgensis, Department of Education, Gothenburgh University. (Doctoral diss.).

Hiippala, T. (2015). The structure of multimodal documents: An empirical approach. Routledge.

Houts, P. S., Doak, C. C., Doak, L. G., & Loscalzo, M. J. (2006). The role of pictures in improving health communication: a review of research on attention,

comprehension, recall, and adherence. Patient education and counseling, 61(2), 173-190.

van Hooijdonk, C. M. J., & Krahmer, E. (2008). Information modalities for procedural instructions: The influence of text, static, and dynamic visuals on learning and

executing RSI exercises. IEEE Transactions on Professional Communications, 51(1), 50-62.

Karpicke, J., & Roediger, H. (2007). Repeated retrieval during learning is the key to long-term retention. Journal of Memory and Language, 57(2), 151-162.

Kaschak, M. P., & Glenberg, A. M. (2000). Constructing meaning: The role of affordances and grammatical constructions in sentence comprehension. Journal of memory and language, 43(3), 508-529.

(25)

Maat, H. P. (2006). De effectiviteit van tekst en beeld. Tijdschrift voor taalbeheersing, 28(2).

Mayer, R. E. (2005). Cognitive theory of multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 31-48). Cambridge:

Cambridge University Press.

Mayer, R. E. & Moreno, R. (2003). Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychologist, 38, 43–52.

Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents?. Cognition and instruction, 19(2), 177-213.

Nickerson, R. S. (1985). Understanding understanding. American Journal of Education, 93(2), 201-239.

Van Der Sluis, I., Eppinga, A. N., & Redeker, G. (2017). Text-Picture Relations in Multimodal Instructions. In Proceedings of the IWCS workshop on Foundations of Situated and Multimodal Communication.

Leene Communicatie. (n.d.) Dit was ook voor mijzelf een eyeopener. Consulted on 2nd of April from https://www.leenecommunicatie.nl/beeldbrieven-rotterdam/

(26)

Appendix

Appendix 1: Analysis of original multimodal artefacts

Analyzing the multimodal artefacts that are used in the study, will be done using the frameworks that Bateman (2014) describes. In this book, John Bateman critically describes the fundamentals of how, if, and when images and textual components make meaning.

(27)

Firstly, we discuss the ‘three messages’ of Barthes. These consist of 1) linguistic messages, 2) coded iconic messages and 3) uncoded iconic messages.

A linguistic message is “simply any words or other linguistic material that might appear”.

A coded iconic message is “what the image is portraying”

An uncoded iconic message is “describing or accounting for what the image actually shows.”

Under here there’s a table in which each image is described, using Barthes’ distinctions.

Firstly, in the analysis using Barthes’ distinctions on the images that are on the multimodal artefact, we have to consider the fact that these pictures do not contain text, and therefore the only linguistic message are textual components of the letter that are not in the images.

Table 1. description of the images using Barthes’ three messages

Image Linguistic Coded iconic Uncoded iconic Picture 1: man

holding laptop

x This image portrays a

man visiting the website of Rotterdam.nl/passport This image contains a man holding a laptop Picture 2: man holding phone

x This image portrays a

man calling 14 010 to set an appointment This image contains a man holding a phone Picture 3: four pass photo’s of woman

x This image portrays

an example of the pass photo that the person reading the letter has to bring

This image contains four pass photos of a woman Picture 4: three different debit cards

the fact that in most municipal stores, the only means of paying for a new passport is with a debit card, and not cash This image contains three different types of debit cards Picture 5: passport and identification card

the fact that the person reading the letter, has to bring both a passport and identification card to the appointment This image contains a passport and identification card

In the multimodal artefact, the pictures are interrelated with the textual components. You might think that this means that in all 5 pictures, the text plays a role of fixing the interpretation. This is defined as anchorage by Barthes: “the text replies – in a more or less direct, more or less partial manner to the question: what is it?”. The message of the multimodal would not be understood if there were only the pictures and no linguistic components. On the other hand, the message of the multimodal letter can be understood if there were to be no images. Therefore, the relation between text and

(28)

image is unequal. However, not all the 5 textual components that are interrelated to the images in the multimodal artefact can be defined as amplifying the image. Some of the images might reduce the textual components.

Barthes speaks of a systemic classification network for defining the relations between text and images. The relationship in this network is set up as follows in image 1:

Image 1.

This systemic classification network helps us define the relationship between the images and the interrelated textual/linguistic components in table 2.

Table 2: relationships of pictures with their linguistic elements defined by Barthes’ systemic classification network

Image Equal/unequal Amplifying/reducing Picture 1: man holding laptop Unequal due to difference in coded and uncoded iconic message

Image reduces text, due to the fact that there is a man in the picture and that he uses a laptop to visit the website. the visiting of the website can be done on all other kinds of devices and the person reading the multimodal letter and thus, in need of a new passport doesn’t have to be a man

Picture 2: man holding phone Unequal due to difference in coded and uncoded iconic message

Image reduces text, again, because of the fact that there is a man that is in the picture and the person reading the multimodal letter and thus, in need of a new passport doesn’t have to be a man.

Picture 3: four pass photo’s of woman Unequal due to difference in coded and uncoded iconic message

This is a special case, because the text next to the image says that the person that has an assignment only needs to bring one pass photo. However, the picture portrays four different pass photos of a woman. If we assume that the multiplication of photos signifies ‘pass photos’, the image reduces the text, due to the fact that there is a woman and the person reading the multimodal letter and thus, in need of a new passport doesn’t have to be a woman. Also, the picture doesn’t show the necessary action that the reader has needs to be carried out with the pass photos. However, the multiple photos are amplified by the text.

Picture 4: three different debit cards Unequal due to difference in coded and uncoded iconic message

The image reduces the text, due to the fact that there are only three debit cards. The text however, states that payment by card is an option and doesn’t say which cards so credit cards could also be available, as well as

(29)

debit cards of other banks. Also, the picture doesn’t show the action that should be done with the cards.

Picture 5: passport and identification card Unequal due to difference in coded and uncoded iconic message

The picture reduces the text, because the picture only shows the image of the passport and id-card and it doesn’t show the action to be done with it. The picture also reduces the text, because of the fact that the identification card and the passport are both Dutch. Bateman (2014, p. 249) also addresses the multimodal conjunctive relations that are stated in Van Leeuwen (2005b). According to Bateman (2014, p. 249) the multimodal conjunctive relation framework “takes the diversification of potential realizations illustrated by the examples under a step further so as to allow non-realizations also”. Van Leeuwen (2005b) builds upon the framework of Barthes (as cited in Bateman, 2014), like described above. The image-text relations then could be defined like in table 3.

Table 3: Image-text relations described by Van Leeuwen (2005b: 230, figure 11.4)

Image-text relations

elaboration 1) Specification 2) Explanation

1) a) The image

makes the text more specific

b) The text makes the image more specific

2) The text

paraphrases the image (or vice versa) 1a) illustration 1b) anchorage extension 1) Similarity 2) Contrast 3) Complement

1) The content of the text is similar to that of the image

2) The content of the text contrasts with that of the image 3) The content of the

text adds further information to that of the text and vice versa

3) relay

In terms of Van Leeuwen (2005b), the image-text relations can be defined as follows: • Picture 1: Because the image makes the text more specific, due to the fact that the person on the photograph is a man and makes use of a laptop. This means that in terms of Van Leeuwen (2005b), picture 1 is defined using the path elaboration à specification à illustration.

• Picture 2: Because the image makes the text more specific, due to the fact that the person on the photograph is a man. This means that in terms of Van

WHAT DO THE PICTURES ADD?