Author
Loes van Bemmel
s4574249
Supervisor
Hanna K. Schraffenberger
July 1, 2018
Faculty of Social Sciences Radboud University
Abstract. Several musea already use AR to guide visitors around their exhibitions, in addition to written guides or audio guides. Although these passive ways of receiving information are commonly used in musea, it has been shown that learning via interaction improves memory. Interacting with paintings is generally not possible, since most paintings cannot be touched without causing damage. AR provides a new way to look or interact with such art pieces. In this thesis, it is examined whether in-teracting with art via AR will enhance the recollection of said art. This is done with an app, the RecolourAR app, and a visual recollection test, both free recall and cued recall. No significant effect of the mode (iAR, AR, no app) was found on the recollection scores.
Keywords: Augmented Reality · Interactive Augmented Reality · Vi-sual Recollection
1
Introduction
Augmented Reality, or AR, is a technique to enhance the so-called real world with virtual additions. AR is described by Milgram et al. [14] as being somewhere on the Reality-Virtuality Continuum that ranges from the Real Environment and a Virtual Environment. Virtual Reality (VR), which is often confused with AR, would be on the Virtual Environment end of this continuum. AR, contrary to VR, is not a fully immersive technology. Instead of replacing the real world with a virtual world like VR does, AR enhances the real world with virtual elements. The user is still able to see the real world with AR. AR is described as a Mixed Reality, which combines real world elements with a virtual overlay. This overlay could be several things, such as audio, visual elements or a combination. Azuma [2] mentions that ideally, the virtual aspects of AR would seem to coexist and blend in seamlessly with objects in the real world. Azuma argues that this seamless transition would would require the virtual elements to be photorealistic, so that they do not feel out of place in the real world.
The topic of Augmented Reality, or AR, has been researched a lot in the last years. In an AR survey done by Azuma [2], it is mentioned how AR applications were developed up to 1997 and some of the difficulties that the technique faced in that time. At this time, registration and sensing errors were the biggest prob-lems that AR applications faced. If Azuma’s survey from 1997 [2] is compared to a more recent survey from 2015 by Billinghurst et al. [4], it is obvious that tech-niques have advanced in the time. The survey of Billinghurst et al. summarizes almost 50 years of AR development, and thus provides a clear overview of the techniques. The survey also discusses recent solutions to the registrations and sensing errors mentioned by Azuma. Billinghurst et al. also give an overview of the past and current applications of AR. While Azuma focused on broad appli-cations of AR, such as medical or manufacturing appliappli-cations, Billinghurst et al. also discuss the novel personal applications of AR. Examples of these personal applications are personal information systems, entertainment and education.
In the years between the two surveys, the processing power of handheld devices has increased drastically. Because of this, common handheld devices
this thesis. Then I illustrate the app that was developed to perform the exper-iment and explain the experexper-iment itself. After that, the results are shown and discussed in the conclusion and discussion section. At the end there is a section for future research where I consider other possible uses of the RecolourAR app and things I would change in possible future research.
It is important to note that initially, the experiment was meant to take place in a primary school and the participants would be children between 10 and 12 years old. However, because of circumstances we had to resort to performing the experiment at the Radboud University with adult students. Because of this un-foreseen switch between target subject groups, some of the choices were initially meant for children and later adapted to fit adults.
1.1 AR and art
Some musea or exhibits already experiment with AR apps to provide a new way for visitors to view art. Examples of these exhibits are described in the papers of Billinghurst et al. [4], Sommerauer et al. [21], and Chang et al. [6]. Each of these papers show how the AR apps enhance the user experience of the museum, as people like the additional information they can get with the AR in combination with the exhibition pieces.
Sommerauer et al. [21] shows how AR can be used in informal learning en-vironments. In their paper, the AR app is used on a mathematical exhibition. The AR is activated by special markers near the exhibits, and the app shows a video or animation that gives extra information about the exhibition piece.
In the paper of Chang et al. [6], an AR app was developed to guide visitors in an art museum. When the AR app was pointed at an artwork, the app gave both an audio and a written description of the artwork via the app. The app had two functions in terms of the description. In one of the functions, the description is factual, a description of the impressions of the painting. In the other function the description is more of a interpretation of the art, which stimulates the user to form an opinion about the art.
Chang et al. compared their AR app to the conventional audio-guide that was used in the museum. They found a significant effect on the learning performance for the AR-guided group, which suggests that AR can enhance learning in musea. In the same experiment, users commented that they like the interaction with the app and the exhibition piece. Also the fact that an AR app can provide other modalities than traditionally provided in a museum, such as audio or visual guides, appeals to the visitors.
In the survey of Billinghurst et al. [4], several AR applications for musea are named. Whereas the other papers are mostly limited to AR guides in musea, Billinghurst also mentions AR being part of exhibitions. AR can also be used to show virtual artifacts in museums.
An example of this is the exhibition of Studio Drift [13] that was displayed in the Stedelijk Museum Amsterdam while the pilot test of this thesis was being held. The exhibition was called “Concrete Storm”, and consisted of little pillars standing in a room. When the visitor looks through the Hololens headset that is part of the exhibition, an AR animation can be seen of the pillars falling over. While the AR in this exhibition was part of the art itself instead of a novel way to view the art, it is apparent that musea are interested in the relatively new AR technology in combination with art.
AR can be part of an art piece, and AR can also be used to create art or augment our experience of art. ‘Augmented Creativity’ is a term first used by
Z¨und et al in their paper Augmented creativity: bridging the real and virtual
worlds to enhance creative play [27]. Augmented Creativity is described as a way to use AR to enhance real-world creative activities. One of the ways to combine AR and creativity is the ability to dynamically recolour a painting with AR, used by Ryffel et al. [20] in their app “AR Museum”. The user is able to point a tablet with a camera at a painting, and by clicking on the painting on the tablet screen, they are able to recolour certain parts of the painting. The concept of augmented creativity is interesting, since it suggests interaction between a user and a program in a new creative way. Augmented creativity essentially provides a new new way to create art.
1.2 Learning with AR
Other than being used in musea for several reasons as discussed above, AR is also used in classrooms to enhance learning in children. For instance, Billinghurst [4] mentions several applications for young children to learn with AR in the classroom. Those applications mostly focus on motivating children to engage in the class material. For example, after interacting with a drawing with AR, the children’s own written story about that drawing became significantly longer than before the interaction with the AR app. This is a clear example of AR enhancing the creative processes of children.
One of the papers that clearly shows improvement in learning when using AR is the previously mentioned paper of Sommerauer [21]. In this paper, the learning is applied to a mathematical exhibition. The participants remembered significantly more about the AR exhibits than the non-AR exhibits. However,
or audible elements such as a voice explaining certain aspects of the art. Which modality of learning is best for a child is very individual and is different for every child. Learning with AR could simply provide more options, so that each child can choose the method that best suits them.
Chang et al. [6] shows that AR does not only provide a new learning modality which improves learning, but also enhances the participation and motivation of people. According to a review done in 2014 by Bacca et al. [3], this effect of enhanced motivation in people has also been found in other papers. The review states that most of the applications of AR on education were done in science, and mainly tested on Bachelor students.
1.3 Interactive learning in musea
While musea are generally not focused on learning, it could improve art education and general cultural education if visitors would learn from museum exhibits. According to Mrs Jacobs [12], an art teacher who was consulted for this project, art education is most effective when there is interaction between student and art. High school art teachers are encouraged to let their students interact with art as much as possible. This positive learning effect for interaction can also be found in academic papers. Hall et al. [10] suggest that the option to play with museum pieces would greatly enhance museum visits for children in particular. Not only would the visits be more interesting and engaging, the children also learn more from the interaction.
Falk et al. have researched recollection from museum experiences in their paper Recalling the Museum Experience [9]. Children have a better recollec-tion of museum visits if the museum visits were interactive and engaging. One child was asked to participate in a demonstration and thus remembered it very clearly after the visit. Falk also suggested that museum visits, in particular for interactive museums, persist in long term memory. In one study mentioned by Falk et al, it is shown that out of all the things in a museum, visitors remember objects the best. Out of all the things the participants remembered, 51% of the remembered things were objects in museum [9]. The interactive videos shown at
one particular museum exhibition were particularly memorable for the visitors. Another memorable object in the museum was a mask that the visitors could try on, and thus have interaction with. Falk suggests that interaction enhances the recollection and memory of objects in musea.
This effect of interaction is also shown in Rosier’s paper [19]. When specif-ically researching performance in memory tasks, Rosier concluded that people score better in visual memory tasks when engaging with art beforehand. How-ever, his study suggests that just paying attention to art does not have the same significant effect that a creative process does have. Rosier suggests that cre-atively engaging in art enhances memory more than passively viewing art. This suggests that creativity could play an important role in the interaction between art and viewer.
To summarize, Chang et al. [6] and Sommerauer et al. [21] both suggest that using AR to learn from museum exhibits improves the memory of these exhibits.
Not only does AR provide a new modality to learn, as D¨unser [8] has shown,
but according to Chang et al. it also improves the users’ motivation to learn. Billinghurst et al. [4] have shown that AR is already being used to teach children in classrooms. Falk et al. [9], Hall et al. [10] as well as Mrs Jacobs [12] stress the possible positive influence of interaction when learning.
Ryffel et al. [20], Z¨und et al. [27] and Rosier [19] have shown the importance of
creative processes in children, and suggest ways that AR can improve creative processes. Rosier also concludes that being creative enhances memory.
AR and interaction have been shown to both enhance memory separately, but a combination has not been researched extensively yet. While all these papers look very promising for the future of AR in education, I did notice that there were very little papers that compared interactive AR to non-interactive AR and the influence of the interaction on learning.
In musea, AR seems to be mostly used as a new modality for a guide, com-pared to audio-guides, human guides and written guides. AR is also used in musea as part of an art piece. However, AR is not used to creatively look at art in musea a lot, while it is very suited for this purpose because it is non-invasive for the artworks.
That is why, given all the literature discussed earlier and the text above, I wanted to research interactive and non-interactive AR in regard to memory or recollection of paintings. The interactive AR will also have a creative element to it, as it has been shown to enhance memory in Rosier’s paper [19], and the
importance of creative processes in children has been highlighted by both Z¨und
et al. [27] and Ryffel et al. [20].
The research question for this thesis is “Does interaction with art via Aug-mented Reality improve the recollection of art?”.
by the comments of Mrs Jacobs and previous research.
A doubly multivariate repeated-measures ANOVA analysis was used to anal-yse the data.
2.1 Research questions and hypothesis
The research question for this experiment is “Does interaction with art via Aug-mented Reality improve the recollection of art?”.
My hypothesis is that interaction via augmented reality will result in better recollection than a non-interactive AR program. Although interactive AR has not been researched in regards to visual recollection specifically, both interaction and AR have been shown to improve learning in children separately [9, 19, 10, 8, 21]. I think these results will extend to visual recollection of art in adults and that the combination of interaction and AR will improve this recollection. I also think that both the interactive and the non-interactive mode of the app will result in better memory scores compared to the no-AR mode of the app.
The null hypothesis is that there is no effect of mode on the recollection of art. So, in the population, the means of the recollection scores would be the same for interactive AR, non-interactive AR and no AR.
2.2 Pilot test
For the pilot test, 4 adults were asked to test an interactive AR app that was developed to dynamically change a painting’s colours similar to the app used in the paper of Ryffel et al. [20]. This app was created with Unity [25] and Vuforia [26]. It was implemented on a smartphone.
The stimuli used for this pilot test was a painting by Picasso called “Femme assise au chapeau en forme de poisson”(seated woman with fish) [17], which was at the time of the test displayed in the Stedelijk Museum Amsterdam. Screenshots of the pilot test are shown in figure 1.
The participants were asked to look at the painting with the interactive AR app and to recolour the painting on the screen. Afterwards, they were asked to write down everything they remembered from the painting and any comments they had about the app. Immediate feedback in the form of comments during the experiment were also taken note of. These immediate comments were mostly regarding the user interface(UI) of the app.
After the pilot test, it became apparent that an abstract painting like the Picasso used in the pilot was not suitable for a recollection test. Especially the more abstract paintings would be difficult to test for recollection, because they do simply not contain a lot of concrete things to recollect.
Another thing that changed after the pilot test was the way that subjects were able to interact with the painting via the app. In the prototype app, users had to click on the round buttons on the left of the screen and change the colour the painting by changing the colour of those buttons. Some of the subjects commented that this way was not very intuitive, and also did not really feel like interaction with the painting. Because of this, the app was changed to click the painting directly to recolour objects.
(a) Painting without any mod-ifications
(b) The painting after interac-tion with the app
Fig. 1: Picasso’s “Femme assise au chapeau en forme de poisson” with the pro-totype app for the pilot test
changing the subject group for my experiment.
Additionally, because she is a teacher she has inside knowledge in how school classes conduct their tests and how they teach art. This knowledge helped me in creating appropriate questions for the recollection test and a fitting scoring measure.
During our interview [12], Mrs Jacobs proposed to use realistic paintings for the test, as she thought they would be most suitable. She also argued that using Genre Paintings from the Dutch Golden Age would possibly facilitate recollection for students that are not very acquainted with art, since Genre Paintings contain everyday situations like simple house life that anyone can recognize. This is similar to laboratory recall studies, where stimuli are usually familiar words [16]. In this experiment, the recollection stimuli will be somewhat familiar objects in paintings. Other than being familiar, the Genre Paintings also contain a lot of concrete objects that can be used for a recollection task.
Mrs Jacobs and I also discussed how to grade the participants for the test they will fill in. We collaboratively came up with a grade that somewhat resembles a grade on a high school art test, as confirmed by Mrs Jacobs who made many of these tests herself.
She was very enthusiastic about this experiment and the app that was de-veloped. A lot of suggestions regarding use in classrooms or musea were made by her, and although they were not relevant for this particular experiment, they were very interesting in the way they offered a new perspective on the use of the RecolourAR app beyond recollection experiments.
Art education is not just about remembering the art, this is actually a very small part of the education and not always relevant in exams. What is more in-teresting from an education perspective, according to Mrs Jacobs, is the meaning and themes behind the art. These are not always clear from just looking at the art, but require more background information and knowledge about art history. Generally, descriptions of paintings in musea contain some background informa-tion, so I decided to also add a description of the painting to each painting in the experiment.
2.4 RecolourAR: the AR app
An app was developed that allows the user to interact with paintings via Aug-mented Reality, and also allows users to look at the paintings with non-interactive AR. The app developed for this thesis is called the RecolourAR app.
The app and was made in Unity 2017.4.0f1 [25]. The AR platform used to de-velop the app is Vuforia [26]. RecolourAR app has three ‘modes’: the interactive mode, the non-interactive mode and the no-app mode.
Because of these modes, there are 6 different versions of the app. For each version of the app, all three of the modes are assigned to the paintings. The modes are shuffled between the versions as can be seen in table 1.
For this experiment, the ColourAR app was designed to work on 3 different paintings. The AR modes for these 3 paintings were manually implemented for the application. This effectively means that the ColourAR only works for those three paintings. In the future, this could be expanded for more paintings.
Table 1: Distribution of app modes over versions
Version Papegaaiekooi Dansles Luitspeler
1 iAR AR no 2 iAR no AR 3 AR iAR no 4 no iAR AR 5 AR no iAR 6 no AR iAR
Interactive AR mode The interactive mode is meant to engage with the user and enable them to creatively view the painting. The key component of this mode is the interaction with the user, and this interaction has to provide engaging creative ways for the user to look at the art.
This has been implemented by allowing the user to recolour the paintings. This works in the following way:
Some of the objects in the paintings allow the user to click on them to select them. Then the users can choose a colour in the colour options on the left on the screen. After they clicked on a colour, the object that they selected in the painting becomes that colour. This has been heavily influenced by the app and research of Ryffel et al. [20], and screenshots of the app in the interactive mode can be seen in figure 2.
While this interactive mode has been influenced by Ryffel et al. [20], it is different in user interface. For the app of Ryffel et al., the user has to tap the screen and the app will randomly choose the colour to recolour the painting with. For the RecolourAR app, the user is able to choose the colour from an array of colours on the screen. While this does restrict the colour-choice of the user, it does give the user a choice rather than randomly assigning a colour.
(a) Painting without any modifications
(b) The painting after in-teraction with the app
Fig. 2: interactive AR mode of the app displayed on “De papegaaiekooi”
AR mode The AR mode, or non-interactive AR mode, has AR elements but no interactive elements. It is still AR in the sense that it provides a new way to look at the painting.
The way RecolourAR app achieves that, is by putting some of the elements of the painting in 3D in front of the painting. For example, a window at the back of the painting is in front of the painting, but still less in front compared to a dog that is in the front of the painting. Screenshots of the AR mode of the app used in the experiment can be seen in figure 3.
In this way, the app can provide a better idea of the placement of certain objects in the painting, as well as highlight some of the objects in the painting, by letting it jump out of the painting.
The objects that are in 3D in this version are the same objects that allow recolouring in the interactive version of the app.
No-app mode In the no-app mode, the app does not provide anything extra to the painting. This mode is meant to be a control condition. For this mode, the users should observe the art like they would in a museum. Screenshots of the no-app mode can be seen in figure 4.
In the experiment, the participants were not instructed on whether they should still look through the screen or not. Most participants lowered the smart-phone after they found out the device was displaying the no-app mode, and they looked at the painting without the device.
(a) AR from an angle (b) AR from the front
(c) AR from an angle
Fig. 4: The app in the no-app mode, displayed on “Luitspeler”
2.5 Painting stimuli and experiment setup
The paintings chosen for this experiment are “De dansles” [24] and “De pape-gaaiekooi” [23] of the Dutch artist Jan Steen and “De luitspeler” [22] of Hendrick Sorgh. All of these paintings are currently displayed in the Gallery of Honour in the Rijksmuseum. The jpg files used to print the posters of the paintings for the experiment and the descriptions of the paintings were also downloaded from the Rijksmuseum site. All three of these paintings are displayed adjacent to each other in the same room in the Gallery of Honour, and the experiment setup reflects how the paintings are displayed in the museum.
The paintings have to be similar so that there is no bias. As Mrs Jacobs suggested during our interview [12], the paintings chosen for this experiment are all Genre Paintings from the Dutch Golden Age. Additionally, two of the paintings are from the same painter.
These paintings are specifically chosen because they contain several objects that can easily be used for the recollection test. However, these particular paint-ings do not include too much objects, because viewers will probably not be able to remember a lot about the paintings. In fact, Miller [15] has argued that a person is only able to remember around 7 things at a time. In choosing the paintings, there was a trade-off between simplicity and being able to make an complete recollection task.
I specifically wanted to use paintings by Dutch artists, and preferably paint-ings that are also displayed in a Dutch museum. In that case, there was a possi-bility that some students have seen the paintings before, and that would improve recollection. Since the chosen paintings are displayed in the same room in the Rijksmuseum, it is likely that if the student has seen one of the paintings, they have also seen the others. This recognition would not influence the scores since the scores are compared within-subject for the paintings.
In the information form provided before the experiment, the students were told that after the viewing of the paintings they would fill in a question list about the app and the paintings. The participants were asked to at least remember the title of the paintings, so that they could answer questions about them later. It was not explicitly stated that they would have to remember the paintings or the description of them. This was intentional, as the experiment was meant to resemble a museum setting in which the viewer would not necessarily try to memorize the paintings, but rather observe the paintings. That is also the reason that the participants were not asked to look at the paintings in a certain order or for a certain time, but rather take their time and look at the paintings like they would in a museum. The order of the paintings on the wall (so which one is most left, most right, and in the middle) was changed in between the experiments.
A picture of one of the experiment setups can be seen at figure 5. As Mrs Ja-cobs [12] suggested, a description of the painting was added on the wall beneath the painting. The descriptions can be read at appendix G.
Because of the three different modes of the ColourAR app, all three modes had to be applied to all three paintings. This resulted in 6 versions of the app, as mentioned before. For each version, there were 4 participants. In total, there were 24 participants. The participants were all students from the Radboud University Nijmegen. The RecolourAR app was installed on a 5.5 inch android phone for the experiment.
Fig. 5: One of the experiment setups with the paintings and the descriptions displayed on a wall
2.6 Recollection test
As Miller concluded in his paper The Magical Number Seven, Plus or Minus Two Some Limits on Our Capacity for Processing Information [15], people are only
questions that were asked about the app can be seen in appendix A.
For each of the three paintings there were questions about how beautiful and interesting they thought the painting was, as well as technical questions about the painting. These questions can be found in appendix B.
After the app questions, there was a checklist for each painting, consisting of 85 words. The subjects were asked to tick off the words corresponding to the objects they remembered being in the painting. The checklist can be seen at appendix C. For each painting, around 20 of the objects in the checklist were actually present in the painting, so there were 25 distractor objects in the list. For ‘de Papegaaiekooi’ 23 of the objects were present in the painting. For ‘de dansles’, this is 18. For ‘de luitspeler’, 19 of the objects in the checklist were present. These 60 objects have been selected by comparing the results of participants of the pilot test, when they were asked to describe the paintings as accurately as possible while looking at it. Mrs Jacobs [12] has also approved this checklist to be a complete description of the objects in the paintings. The distractor objects on the checklist are selected from other paintings of the artists Jan Steen, so that the painting style of the objects is the same. Because this checklist had to be complete, the amount of correct answers are not the same for all three paintings. This is taken into consideration with how the data is analysed.
The checklist is meant to slightly resemble a free recall test. According to Ozubko [16], free recall is done unaided and without any hints. This is not exactly the case in the checklist, as there are objects that the participant can choose from. However, I argue that it is also not exactly a cued recall test, as there are also many distractor objects.
After the checklist there was another test about each of the three paintings, the image test. In this test, the subject is shown 2 different pictures and asked which one was in the painting. This test is meant to resemble a cued recall test. Ozubko [16] mentions that cued recall encourages guessing. To combat this effect, a third option of “neither/I don’t know” has been added to the images test. This encourages the participants to only choose the pictures that they are certain were in the painting, and choose the third option otherwise. The experiment setup of this experiment is similar to the experiment done by Castelhano and Henderson
in their paper “Incidental visual memory for objects in scenes” [5]. Because the participants were not asked to explicitly remember the paintings, one could argue that the recollection in this experiment is also incidental visual memory. The image test of this thesis was inspired by their experiment. To mimic their experiment, the two stimuli used in one question in this test were very similar. An example of the two images shown in this test can be seen in figure 6. The full image tests can be seen at appendices D, E and F for the ‘papegaaiekooi’, ‘dansles’ and ‘luitspeler’ respectively.
(a) Stimulus 1 (b) Stimulus 2
Fig. 6: Example of the images test stimuli
2.7 Statistical design
The statistical analysis done on the data is a multivariate repeated-measures ANOVA done in SPSS 25 [11].
Dependent variable 1 = checklistScore (quantitative, the F1 score) Dependent variable 2 = imageScore (quantitative, 0-10)
Dependent variable 3 = questionScore (quantitative, 0-5) Dependent variable 4 = testScore (quantitative, 0-10) Dependent variable 5 = beautyScore (quantitative, 0-10) Dependent variable 6 = interestScore (quantitative, 0-10) Within-subject factor = mode (qualitative: iAR, AR, no) Between-subject factor = version (qualitative: 1, 2, 3, 4, 5, 6)
The F1 score was chosen as a measure of the checklistScore. The F1 score is a harmonic mean of precision and recall. The definitions below are taken over from Powers [18], and how the True Positives(TP), False Positives(FP) and False Negatives(FN) are defined can be found in table 2.
P recision(P ) = T P
either, as False Negatives are not counted in that measure, and any missed checks would have been ignored. The F1 score is a harmonic mean of these two measures and thus suited for the checklist score. F1 is defined as
F 1 = 2 ∗ P ∗ R
P + R
However, the F1 score is not a perfect measure of any checklist score. Ac-cording to Powers [18], F1 is advocated as a single measure to capture the ef-fectiveness of a system, but still completely ignores the True Negatives. I argue that this is actually good for my checklist score, as the true negatives vary per painting and by not including them in scores, this variation is also ignored. Addi-tionally, since the participants were only asked which of the objects were present in the painting instead of also indicating which objects were not present in the painting, ignoring the True Negatives seems valid.
The imageScore is simply the amount of correct answers at the image test. This score can vary from 0 to 10 since there were 10 questions. Note that there was also an option “I don’t know/neither of the two images were in the painting”, but in truth one option was always present in the painting. This is done so that if the participant chooses one of the two image options, they are more certain of their answer.
The questionScore is the amount of technical questions that the subject an-swered correctly about the painting. These are the artist, the year in which the art was painted, etc. All this information could have been found in the descrip-tion under the painting.
The testScore is a separate score that was also calculated three times (for each mode) for each subject. This score was based on a grade that a student would get if the recollection test was a test in art class. Mrs Jacobs [12] suggested adding this score. These scores are calculated by multiplying the F1 score for the checklist by 20, and then take the average of that and the images score.
testScore = (F 1 ∗ 20) + imagesScore
The values for these testScores were between 0 and 10. This is different than the standard university grade range, but reflects the grading range used in high school according to Mrs Jacobs [12]. Mrs Jacobs suggested this score calculation based on the tests that she has created herself for her students.
The beautyScore is the answer to the question “How beautiful did you think this painting was?” on a scale from 0 to 10. The interestScore is the answer to the question “How interesting did you think this painting was?” on a scale from 0 to 10.
3
Results
The descriptive statistics of the checklistScores, imageScores, questionScores, testScores, beautyScores and the interestScores can be found in table 3. What stands out is the fact that the means for the questionScores are exactly the same for each of the modes.
Table 3 also shows us that for a lot of the scores, the minimum was 0, which means that some participants did not get one single question of that score correct. Or they scored beauty and interest very low on some paintings.
The doubly multivariate scores are significant if p <0.05. For this analysis, the score for version is F(30, 57)=1.175 with p=0.296, the score for mode is F(12, 7)=1.051 with p=0.495 and the score for version x mode interaction is F(60, 27)=1.376 with p=0.057.
The elementary report for the doubly multivariate repeated-measues ANOVA can be found in appendix H.
Ahead of the recollection test, the participants were asked questions about the app itself. The scores the participants gave to the questions regarding the app can be seen in the boxplot in figure 7. The answer to the question “Would you be more interested in musea if they had an AR app like this?” is represented by the score app interested. The answer to the question “Did you feel like the app added something to the artworks?” is represented by the score app added. The answer to the question “How well did the app work?” is represented by the score app wellworking. The answer to the question “How much fun was it to play with the app?” is represented by the score app fun.
4
Conclusion and Discussion
For the doubly multivariate scores, the score for version is F(30, 57)=1.175 with p=0.296 which is not significant. For the mode, the score is F(12, 7)=1.051 with p=0.495 which is also not significant. The version x mode interaction score is F(60, 27)=1.376 with p=0.057, which is small but not significant.
This means that the general null hypotheses for these scores are not re-jected, so we cannot conclude that version or mode had any significant effect on the scores. We can also not conclude that there was any interaction between version and mode in this experiment. Because the general null hypotheses are not rejected, we do not need to evaluate the univariate tests on significance.
Min 0,077 0 0 Max 0,6857 0,65 0,5909 imageScore Mean 4,2917 4,5417 4,5417 Median 5 4,5 5 SD 2,1362 1,8411 2,1055 Min 0 0 0 Max 7 8 8 questionScore Mean 0,875 0,875 0,875 Median 1 1 1 SD 0,9469 0,7409 0,8502 Min 0 0 0 Max 3 2 3 testScore Mean 5,8 5,9 5,4 Median 5,5 5,8 5,8 SD 2,4486 2,411 2,5482 Min 1,3 0,7 0 Max 10 9,9 9,2 beautyScore Mean 6,875 6,75 6,875 Median 7 7 7 SD 1,1539 1,1516 1,985 Min 4 4 0 Max 9 8 10 interestScore Mean 6,9583 6,3333 6,5417 Median 7 6 7 SD 1,3345 1,4039 2,1865 Min 5 3 0 Max 10 9 10
Fig. 7: Boxplot of the scores about the app questions
The research question for this experiment was “Does interaction with art via Augmented Reality improve the recollection of art?”. The results of this experiment have not confirmed our expectations, so I have to conclude that I did not find that interaction via Augmented Reality improved the recollection of art.
It is possible that 24 participants were not a big enough subject group to get a valid result. Because the participants in this experiment were all students, and most of them were not interested in art (reflected by their interestScores on the paintings and the comments they wrote), it could be that these participants are not a valid reflection of the population. All students were below the age of 30, which could have also influenced the scores.
Some of the participants have stated, after filling in the question list, that they did not pay a lot of attention to the painting itself but rather tried to find errors in the app. This is not necessarily a bad thing, because recollection does not require the participant explicitly trying to remember the painting, and this experiment is meant to resemble the incidental visual memory test of Castelhano [5]. The experiment setup was meant to resemble a museum setting for the participants, so a choice was made to go for ecological validity rather than control in this experiment. This is also why the students were not instructed to spend a certain amount of time observing the paintings, but rather choose how long they wanted to look themselves.
Other than not paying a lot of attention to the painting itself, a lot of par-ticipants had not read the description of the paintings that was provided below the painting. Because of this, they could not answer the open questions about the technical aspects of the paintings. In a lot of cases they only knew one of the
if its dimensions were more sophisticated”, which could explain the low scores on the AR mode.
Chang et al. [6] stated that AR could distract from the real world objects. This has also been pointed out by Dunleavy and Dede in their paper “Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning” [7]. They argue that AR could cause people to pay too much attention to the virtual aspects of the app, and ignore the surrounding real world. I speculate that this is also the case in the experiment done in this thesis: the participants might have been so distracted by the app that they did not pay any attention to the painting itself. One participant experienced this distraction and also recognized it, as they answered the question “Would you be more interested in musea if they had an AR app like this?” with “Yes, it will be more fun. An unwanted effect might be (as I notice for myself) is that you are more busy with playing around with the app and not actually taking up the information. This differs per person, of course, but in general people like pictures better than reading.”
While the app used in this experiment resembles the app created by Ryffel et al. [20], the experiment is completely different. Ryffel et al. did not test recol-lection or memory performance with their app at all. Rather, they focussed on the technical aspects of generalizing an algorithm for painting recolouring. The target group for the two apps is also not the same, as Ryffel et al. created their app for children, while the final target group for my experiment was adults.
The answer to the question “Would you be more interested in musea if they had an AR app like this?”, represented in the score app interested varies a lot, as can be seen in figure 7. A lot of participants did agree with this, but others indicated that they would never be interested in musea, so they scored a 0. The answers for this particular question varies between 0 and 10, which is indicative of how diverse the participants were in art appreciation and interest.
When asked the question “How well did the app work?”, some participants indicated that the app was sometimes very slow on the smartphone, or that the screen was too small to see the details of the painting clearly.
The scores of app fun, which is the answer to the question “How much fun was it to play with the app?” was a little bit lower than expected with an average of 6,8. I suspect this is because the paintings were not particularly beautiful or interesting to someone who is not interested in art in the first place. The paintings used in the experiment are not particularly colourful, and because of their realistic painting style they could be perceived as boring.
In the open questions, a lot of participants noted that they can imagine the RecolourAR app being used in a museum by children. Some of the comments by participants are: “I think it is a nice idea which could be used to connect the smartphone generation to classical art.”, “By incorporating an interaction effect, this can make viewing art a more dynamic experience. This can especially be useful for children I think, who tend to find staring at art for longer periods at a time dull and boring.”, “It could work really well as a way to ‘lure’ young people to musea they might otherwise find boring.”.
Personally, I am very happy with these comments as the initial user group was supposed to be children.
5
Future research
As mentioned before, the initial test group consisted of children between the ages of 10 and 12 years old. While this could not be done for this thesis, I would still recommend testing with children in future research. The interaction with the painting would probably be more fun for children than for adults, as many of the participants of this study also suggested. While not all paintings would be interesting for children, I argue that more abstract paintings with bright colours, like the Picasso painting used in the pilot test, would be more suited for recolouring than the realistic paintings used in this thesis. Some participants mentioned that recolouring the realistically coloured painting with bright colours felt out of place. I did not choose abstract paintings as stimuli for this experiment, because it would be difficult to create a complete recollection test. However, I argue that recolouring colourful paintings would be more fun and could possibly result in better scores because the participants would interact with the painting more if they thought it was interesting.
While the recall tests used in this experiment were interesting, the results might have been more significant if more accurate tests were used. Perhaps a real free recall test would get different results. Additionally, in this experiment the subjects themselves could decide how long to spend on each painting, and whether they read the descriptions or not. Perhaps if these conditions were more controlled, the experiment would be more valid, as a more controlled experiment allows less dependent factors. If the participants were all instructed to spend at exactly 2 minutes inspecting each painting, it could have influenced their performance on the test.
Other than another test subject group and new questions, the app could also be improved. Implementation on a tablet instead of a smartphone would
I expect that children will have even more fun than students with interactive AR combined with paintings. Though this particular app does not necessarily result in a significantly enhanced recollection of paintings, it could enhance interest in the paintings and art in general.
One subject’s comments on the experiment perfectly sum up my own thoughts on the experiment, the shortcomings, and possible future research: “I think the idea of looking at a painting in 3D could be fun. This will be easier to enjoy on a larger screen or other environment. Probably people will develop tools for this. Being an art lover I do think that an app is distracting a bit from the real work. For now, it was interesting! ”.
References
1. Apple Inc.: Arkit, https://developer.apple.com/arkit/
2. Azuma, R.T.: A survey of augmented reality. Presence: Teleoperators & Virtual Environments 6(4), 355–385 (1997)
3. Bacca, J., Baldiris, S., Fabregat, R., Graf, S., et al.: Augmented reality trends in education: a systematic review of research and applications (2014)
4. Billinghurst, M., Clark, A., Lee, G., et al.: A survey of augmented reality. Foun-dations and Trends in Human–Computer Interaction 8(2-3), 73–272 (2015)R 5. Castelhano, M., Henderson, J.: Incidental visual memory for objects in scenes.
Visual Cognition 12(6), 1017–1040 (2005)
6. Chang, K.E., Chang, C.T., Hou, H.T., Sung, Y.T., Chao, H.L., Lee, C.M.: Devel-opment and behavioral pattern analysis of a mobile guide system with augmented reality for painting appreciation instruction in an art museum. Computers & Ed-ucation 71, 185–197 (2014)
7. Dunleavy, M., Dede, C., Mitchell, R.: Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. Journal of science Education and Technology 18(1), 7–22 (2009)
8. D¨unser, A.: Supporting low ability readers with interactive augmented reality. An-nual Review of CyberTherapy and Telemedicine 6(1), 39–46 (2008)
9. Falk, J.H., Dierking, L.D.: Recalling the museum experience. Journal of Museum Education 20(2), 10–13 (1995)
10. Hall, T., Bannon, L.: Designing ubiquitous computing to enhance children’s learn-ing in museums. Journal of Computer Assisted Learnlearn-ing 22(4), 231–243 (2006)
11. IBM: Spss,
https://www.ibm.com/analytics/data-science/predictive-analytics/spss-statistical-software
12. Jacobs, C.: Interview about art and education (2018)
13. in partnership with Microsoft Hololens, Studio Drift.: Concrete storm. commisioned by Artsy Projects for Armory Show 2017, New York (US). Displayed in Stedelijk Museum Amsterdam (2017)
14. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: A class of displays on the reality-virtuality continuum. In: Telemanipulator and telepresence technologies. vol. 2351, pp. 282–293. International Society for Optics and Photonics (1995)
15. Miller, G.A.: The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review 63(2), 81 (1956) 16. Ozubko, J.D.: Is free recall actually superior to cued recall? introducing the
recog-nized recall procedure to examine the costs and benefits of cueing (2011) 17. Picasso, P.: Femme assise au chapeau en forme de poisson (1942)
18. Powers, D.M.: Evaluation: from precision, recall and f-measure to roc, informed-ness, markedness and correlation (2011)
19. Rosier, J.T.: Art and memory: An examination of the learning benefits of visual-art exposure (2010)
20. Ryffel, M., Z¨und, F., Aksoy, Y., Marra, A., Nitti, M., Aydın, T.O., Sumner, B.: Ar museum: A mobile augmented reality application for interactive painting recoloring (2017)
21. Sommerauer, P., M¨uller, O.: Augmented reality in informal learning environments: A field experiment in a mathematics exhibition. Computers & Education 79, 59–68 (2014)
22. Sorgh, H.M.: De luitspeler (1661)
23. Steen, J.H.: Interieur met een vrouw die een papegaai voert, bekend als ‘de pape-gaaiekooi’ (1660-1670)
24. Steen, J.H.: Kinderen leren een poes dansen, bekend als ‘de dansles’ (1660-1679) 25. Unity Technologies: Unity. https://unity3d.com/ (2017)
26. Vuforia: Vuforia augmented reality sdk. https://www.vuforia.com/ (2018) 27. Z¨und, F., Sumner, R., Ryffel, M., Magnenat, S., Marra, A., Nitti, M., Kapadia,
M., Noris, G., Mitchell, K., Gross, M.: Augmented creativity: bridging the real and virtual worlds to enhance creative play pp. 1–7 (11 2015)
Did you feel like the app added something to the artworks?
10
9
8
7
6
5
4
3
2
1
0
yes
no
Please elaborate on that
---Would you be more interested in musea if they had an AR app like this?
10
9
8
7
6
5
4
3
2
1
0
yes
no
Please elaborate on that
---Do you think art could be more fun when using an app like this?
10
9
8
7
6
5
4
3
2
1
0
yes
no
If you have any suggestions or comments on the app, please state them here.
---Appendix B
Please answer the following questions about the painting. Try to be as accurate as possible, but do
not worry if you do not remember, you can also leave the question blank.
What is the full name of the painting?
---Who is the artist?
---What materials are used?
---In what year was it painted?
---What were the dimensions of the painting?
---Have you seen this painting before?
yes
no
How beautiful did you think this painting was?
10
9
8
7
6
5
4
3
2
1
0
very beautiful
ugly
How interesting did you think the painting was?
10
9
8
7
6
5
4
3
2
1
0
1 man 1 boy pineapple
barrel pipe painting
violin 1 cat hair ribbon
tin plate smile with teeth lamp
2 dogs pencil open door
1 woman parrot 2 boys
chair apple wooden floor
coin cards earring
fire stove necklace
water tiled floor grapes
jug/can moustache 2 girls
booklet clock fruit bowl
pear duck flute
crown board game pan
1 girl bread closed door
bone sword plate
closet chicken 3 boys
basket soup spoon 2 women
window bridge boat
Appendix D
Next, you will see 2 different images. These are all zoomed in parts of paintings.
Please choose the picture that you think belongs to the papegaaiekooi.
There is also a possibility that neither of the pictures were in the painting.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Appendix F
Next, you will see 2 different images. These are all zoomed in parts of paintings.
Please choose the picture that you think belongs to de luitspeler.
There is also a possibility that neither of the pictures were in the painting.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1
2
neither
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
the background suggest that harmonious love is the subject.
Yet the work may also be a warning against impulsive lust.
Both interpretations are equally valid.
Legaat van L. Dupper Wzn., Dordrecht, 1870
SK-A-495
---Kinderen leren een poes dansen, bekend als ‘De dansles’
/ Children Teaching a Cat to Dance, Known as ‘The Dancing Lesson’
Jan Havicksz Steen (c. 1625-1679)
oil on panel, 1660-1679
h 68,5cm x b59cm
These children are up to mischief: they are teaching a cat to dance to the
music of a shawm, a 17th-century wind instrument. While they are clearly
enjoying themselves, the cat screeches in protest, joined by a barking dog.
The old man at the window angrily rebukes the children: should they not be
learning something rather than giving dancing lessons to a cat?
Legaat van jhr. J.S.H. van de Poll, Amsterdam, 1880
SK-A-718
---/ Interior with a Woman Feeding a Parrot, Known as ‘The Parrot Cage’
Jan Havicksz Steen (c. 1625-1679)
oil on canvas, laid down on panel, c. 1660-1670
h50cm x b40cm
The wealth of motifs and humorous vignettes in Jan Steen’s paintings often
overshadow his great artistic flair. Steen’s mastery lies both in his subtle colour
combinations and his rendering of different kinds of textures. The colour scheme
of his painting is fairly drab and grey: only the clothing of the woman in the middle
is rendered in delicate shades of violet and green.
Aankoop, 1809
SK-A-386
---Within-subject factor = mode (qualitative: iAR, AR, no) Between-subject factor = version (qualitative: 1, 2, 3, 4, 5, 6)
Classification of Measurements
Measure mode
Dependent Variable
checklistScore iAR F1_iAR
AR F1_AR
no F1_no
imageScore iAR images_iAR
AR images_AR
no images_no
questionScore iAR questions_iAR
AR questions_AR
no questions_no
testScore iAR testScore_iAR
AR testScore_AR
no testScore_no
beautyScore iAR beauty_iAR
AR beauty_AR
no beauty_no
interestScore iAR interest_iAR
AR interest_AR
Dependent
variable version iAR AR no average checklistScore 1 0,2375 0,3374 0,3155 0,2968 2 0,228 0,3866 0,2113 0,2753 3 0,3482 0,2901 0,3898 0,3427 4 0,4393 0,4452 0,3601 0,4149 5 0,461 0,2616 0,3469 0,3565 6 0,472 0,4388 0,2379 0,3829 average 0,3643 0,3600 0,3103 imageScore 1 5,5 5,5 3,25 4,75 2 2,25 5,25 4,75 4,08 3 4,25 3,5 5,25 4,33 4 4 4,75 3,25 4,00 5 4 3,5 5 4,17 6 5,75 4,75 5,75 5,42 average 4,29 4,54 4,54 questionScore 1 1 0,75 0,5 0,75 2 2 1,25 1,75 1,67 3 0,25 0,75 0,75 0,58 4 1,25 0,75 0,75 0,92 5 0,5 1,25 1 0,92 6 0,25 0,5 0,5 0,42 average 0,88 0,88 0,88 testScore 1 5,1 6,1 4,8 5,3 2 3,4 6,5 4,5 4,8 3 5,6 4,7 6,5 5,6 4 6,4 6,8 5,2 6,2 5 6,6 4,4 6,0 5,7 6 7,6 6,8 5,2 6,5 average 5,8 5,9 5,4 beautyScore 1 6,5 6,25 4,75 5,83 2 7 7 8 7,33
6 6,75 5,5 5,5 5,92
average 6,96 6,33 6,54
Hypotheses
Dependent variable contrast version mode
version x mode checklistScore iAR+AR+no μ1•= μ2• = μ3• = μ4• = μ5• = μ6•
iAR vs AR μiAR• = μAR•
μiAR1 - μAR1 = μiAR2 - μAR2 = μiAR3 - μAR3 = μiAR4 - μAR4 = μiAR5 - μAR5 = μiAR6 - μAR6 AR vs no μAR• = μno• μAR1 - μno1 = μAR2 - μno2 = μAR3 - μno3 = μAR4 - μno4 = μAR5 - μno5 = μAR6 - μno6 general All of the H0 above are true
All of the H0 above are true
All of the H0 above are true
imageScore iAR+AR+no
μ1•= μ2•
= μ3• = μ4•
= μ5• = μ6•
iAR vs AR μiAR• = μAR•
μiAR1 - μAR1 =
μiAR2 - μAR2 =
μiAR3 - μAR3 =
μiAR4 - μAR4 =
AR vs no μAR• = μno• μAR1 - μno1 = μAR2 - μno2 = μAR3 - μno3 = μAR4 - μno4 = μAR5 - μno5 = μAR6 - μno6 general All of the H0 above are true
All of the H0 above are true
All of the H0 above are true
questionScore iAR+AR+no
μ1•= μ2•
= μ3• = μ4•
= μ5• = μ6•
iAR vs AR μiAR• = μAR•
μiAR1 - μAR1 = μiAR2 - μAR2 = μiAR3 - μAR3 = μiAR4 - μAR4 = μiAR5 - μAR5 = μiAR6 - μAR6 AR vs no μAR• = μno• μAR1 - μno1 = μAR2 - μno2 = μAR3 - μno3 = μAR4 - μno4 = μAR5 - μno5 = μAR6 - μno6 general All of the H0 above are true
All of the H0 above are true
All of the H0 above are true
testScore iAR+AR+no
μ1•= μ2•
= μ3• = μ4•
= μ5• = μ6•
iAR vs AR μiAR• = μAR•
μiAR1 - μAR1 = μiAR2 - μAR2 = μiAR3 - μAR3 = μiAR4 - μAR4 = μiAR5 - μAR5 = μiAR6 - μAR6 AR vs no μAR• = μno• μAR1 - μno1 = μAR2 - μno2 = μAR3 - μno3 = μAR4 - μno4 = μAR5 - μno5 = μAR6 - μno6 general All of the H0 above are true
All of the H0 above are true
All of the H0 above are true
beautyScore iAR+AR+no
μ1•= μ2•
= μ3• = μ4•
interestScore iAR+AR+no
μ1•= μ2•
= μ3• = μ4•
= μ5• = μ6•
iAR vs AR μiAR• = μAR•
μiAR1 - μAR1 = μiAR2 - μAR2 = μiAR3 - μAR3 = μiAR4 - μAR4 = μiAR5 - μAR5 = μiAR6 - μAR6 AR vs no μAR• = μno• μAR1 - μno1 = μAR2 - μno2 = μAR3 - μno3 = μAR4 - μno4 = μAR5 - μno5 = μAR6 - μno6 general All of the H0 above are true
All of the H0 above are true
All of the H0 above are true
checklistScore and
imageScore and
questionScore and
testScore and
beautyScore and
interestScore Doubly general
All of the H0 above are true
All of the H0 above are true
All of the H0 above are true H0(no main effect of version) =
H0(no main effect of version on checklistScore) and H0(no main effect of version on imageScore) and H0(no main effect of version on questionScore) and H0(no main effect of version on testScore) and H0(no main effect of version on beautyScore) and H0(no main effect of version on interestScore) H0(no main effect of mode) =
H0(no main effect of mode on questionScore) and H0(no main effect of mode on testScore) and H0(no main effect of mode on beautyScore) and H0(no main effect of mode on interestScore) H0(no version x mode interaction) =
H0(no version x mode interaction for checklistScore) and H0(no version x mode interaction for imageScore) and H0(no version x mode interaction for questionScore) and H0(no version x mode interaction for testScore) and H0(no version x mode interaction for beautyScore) and H0(no version x mode interaction for interestScore)
Results
Dependent variable Contrast version mode
version x mode checklistScore iAR+AR+no F(5,18) = 0.612 p = 0.692 eta2 = 0.145 iAR vs AR F(1,18) = 0.010 p = 0.920 eta2 = 0.001 F(5,18) = 1.434 p = 0.260 eta2 = 0.285 AR vs no F(1,18) = 1.298 p = 0.270 eta2 = 0.067 F(5,18) = 0.956 p = 0.470 eta2 = 0.210 multivariate imageScore iAR+AR+no F(5,18) = 0.594 p = 0.705 eta2 = 0.142 iAR vs AR F(1,18) = 0.292 p = 0.596 eta2 = 0.016 F(5,18) = 1.712 p = 0.183 eta2 = 0.322 AR vs no F(1,18) = 0.298 p = 0.592 eta2 = 0.016 F(5,18) = 2.148 p = 0.106 eta2 = 0.374 multivariate questionScore iAR+AR+no F(5,18) = 1.748 p = 0.175 eta2 = 0.327 iAR vs AR F(1,18) = 0 p = 1 eta2 = 0 F(5,18) = 2.8 p = 0.048 eta2 = 0.438
multivariate beautyScore iAR+AR+no F(5,18) = 1.80 p = 0.164 eta2 = 0.333 iAR vs AR F(1,18) = 0.172 p = 0.683 eta2 = 0.009 F(5,18) = 0.310 p = 0.901 eta2 = 0.079 AR vs no F(1,18) = 0 p = 1 eta2 = 0 F(5,18) = 1.161 p = 0.366 eta2 = 0.244 multivariate interestScore iAR+AR+no F(5,18) = 1.676 p = 0.191 eta2 = 0.318 iAR vs AR F(1,18) = 2,668 p = 0.120 eta2 = 0.129 F(5,18) = 0.932 p = 0.484 eta2 = 0.206 AR vs no F(1,18) = 1.2 p = 0.288 eta2 = 0.062 F(5,18) = 1.805 p = 0.163 eta2 = 0.334 multivariate checklistScore and imageScore and questionScore and testScore and beautyScore and interestScore doubly multivariate F(30, 57)= 1.175 p =0.296 F(12, 7) = 1.051 p = 0.495 F(60, 27) = 1.376 p = 0.057
