Effects Of Augmented Reality On The Knowledge Gain
SUBMITTED IN PARTIAL FULLFILLMENT FOR THE DEGREE OF MASTER
OF SCIENCE
Michael Kern
11821450
M
ASTER
I
NFORMATION
S
TUDIES
H
UMAN-
C
ENTERED
M
ULTIMEDIA
F
ACULTY OF
S
CIENCE
U
NIVERSITY OF
A
MSTERDAM
18/07/2018
1st Supervisor 2nd Supervisor
Dr. Robert Belleman Dr. Frank Nack
Effects Of Augmented Reality On The Knowledge Gain
Analyzing the knowledge gain with augmented reality in educational astronomy
Michael Kern
University of Amsterdam Amsterdam, The Netherlands michael.kern@student.uva.nl
ABSTRACT
While the technological level, availability, and usability of aug-mented reality are increasing, AR is rarely being used in a wide-spread educational context. Current augmented reality applications focus mostly on entertainment or on highly specific professional and academical use cases, but already show how promising this tech-nology is and what kind of learning experiences can be built with it. This research examined how augmented reality affects the knowl-edge gain in the hypothetical academic use case of basic astronomy. The experiment provided test subjects with traditional study mate-rial and also with study matemate-rial enhanced by augmented reality but with the exact same information, type of media, and instructional approach being used. Through a comparison of assessment scores after studying with each type of material, the data showed that the augmented reality technology does not have an overall positive effect on the knowledge gain. However, this experiment provided evidence for a positive relation between augmented reality and spatial knowledge gains, a negative relation between augmented reality and declarative knowledge gains, and a clear preference of students for studying with augmented reality. Through additional qualitative research, some of the negative factors that influenced the knowledge gains were analyzed and expanded with suggestions on how to balance these issues. Although statistically significant results in favor of educational AR could not be found, this research shows the positive and negative effects of augmented reality on the knowledge gain and displays the current research gap as well as the required future research in this promising area.
KEYWORDS
Augmented Reality, Knowledge Gain, Education, Astronomy
1
INTRODUCTION
The study material that is typically being used in today’s schools and universities does not differ much from the material that was being employed one or two decades ago. Texts, images, audio, and video still dominate the study materials and are mainly provided to the students through traditional technologies. Although technol-ogy is starting to play an increasing role in education and allows for new ways to present media, one of the most promising tech-nologies, augmented reality or AR, is rarely utilized. While some ways to implement AR into common study fields, such as chemistry, mathematics, and many more[15][34], have been suggested, many are either niche tools developed for a specific context, such as AR applications for a creative design course[30], or have at least not been transferred into the common study and teaching practices yet, such as AR science books[9]. A widespread change of teaching and
learning practices comes with inherent organizational or financial challenges though and is therefore not easily implemented. Study material where AR is used to present the information and me-dia to the students can have a positive effect on student engagement and increase the intrinsic motivation of students[8]. Additional benefits that research often attributes to augmented reality are in-creases in spatial abilities and practical skills[6]. Since a school or university curriculum typically focuses on theories and factual knowledge, rather than spatial and practical knowledge, this could explain the lack of AR being currently used in common study mate-rials. Problematically, many of the benefits attributed to the imple-mentation of augmented reality are also based on experiments that analyze changes in the knowledge gain where technology is not the only varying factor, but where information, media, or instructional approach also differ. So does augmented reality only offer benefits for spatial and practical knowledge, and are these benefits actually gained solely by implementing the technology?
2
RELATED WORK
The way certain learning approaches influence the knowledge gain, be it through the choice of media, technology, or instructional method, has been a frequently researched topic. Since the emer-gence of usable augmented reality technology, experiments have been made to see how a learning method that implements aug-mented reality would influence the learning outcome. The follow-ing sections will define the theoretical framework for this paper and focus on research about general factors influencing the learning process and the knowledge gain, use cases of augmented reality in an educational context, and more specific use cases of AR in the field of astronomy.
2.1
Influences On The Knowledge Gain
The knowledge gain of a student in a certain topic can depend on many factors that are often specific to the student’s individual and social background, such as personal interest or aspiration[19]. Nev-ertheless, research shows multiple factors that have a generalizable impact on learning. One of these factors is the combined use of multiple types of media, showing to have a positive influence on learning activities, leading to increased knowledge transfer and a deeper understanding of the topic[21]. Nevertheless, some re-search argues that the instructional method chosen to deliver the information is crucial to learning, rather than the medium[7][3]. But there is also research linking the media richness not directly to the knowledge gain, but to a higher motivation and perceived effectiveness[17], while it can indeed also increase the distraction of students[20]. Additional influential factors are the expertise of the instructor, practice habits, and motivation[22]. Modeling or
using a simulation as a didactical method can prove beneficial to the learning and is often used in areas where practical skills are required[18][1][11]. Computer-based simulations predominantly prove useful concerning procedural knowledge[25][13]. Especially through the implementation of technologies like AR, such simula-tions can enhance the learning experience even further by being more immersive and engaging, and by providing richer visualiza-tions.
2.2
Augmented Reality In Education
Using augmented reality in an educational context is not a recent idea. In fact, the first applications providing augmented reality experiences were developed as early as the 1960s[29]. Although the term augmented reality was established later than that, these early AR applications already had the same fundamental functionality as today’s implementations of AR, adding (digital) information to the real world. Since then, through the increase in computational power and technological level overall, AR applications became more immersive and easier to create and use. From the beginning, many of the use cases that involved AR were in an education and training context, either at academical institutes or companies[29]. Most benefits that are attributed to the use of AR in education are an increase in student engagement and motivation[2][31], and an improved knowledge gain when topics with spatial components are concerned[6]. Evidence of such a higher engagement with the study material can also be found when using AR to enhance books, leading to an increase in story recall, reading comprehension skills, as well as an increase in the understanding of spatial concepts[4][9]. Depending on the educational field that AR applications have been developed for, other benefits of AR like an increase in creativity[30] or motor skills[14] can be found. While many of these applications of AR have proven to be beneficial to the knowledge gain, there is little comparative research between a system with AR and a system without AR but with the exact same influential factors like informa-tion presentainforma-tion, use of media, or instrucinforma-tional approach. In most cases, new AR applications are being developed that have multiple varying factors, not just the additional technology, compared to the traditional approaches. Therefore some of the benefits that are assigned to AR could actually be the result of other factors that can influence the knowledge gain.
2.3
Augmented Reality In Astronomy
One of the study fields where augmented reality is more commonly found in is astronomy. Astronomy seems to be an ideal area to evaluate augmented reality study material since it requires both spatial and declarative knowledge. In amateur astronomy, for ex-ample, there are AR applications that enhance stargazing activities with additional information about celestial objects to increase the connection between the observed visual information and the corre-sponding factual information[16][27]. Astronomy in an education context focuses mostly on modeling celestial objects and their rela-tions. Augmented reality applications have been created as didactic tools to enrich the conceptual understanding of astronomical topics. Especially younger students often struggle to understand astronom-ical models and relations[5][12]. To counter these issues and to enrich the learning experience, augmented reality has proven to
be an effective tool[10]. Nevertheless, these benefits again relate to spatial and practical knowledge, rather than factual and declarative knowledge. If augmented reality should therefore ever be used in mainstream education, there needs to be sufficient research sug-gesting a clear benefit towards declarative knowledge gain among students.
3
RESEARCH QUESTION AND HYPOTHESES
The theoretical framework shows that there is a widespread agree-ment that augagree-mented reality has the potential to benefit the learn-ing process in one way or another. However, most of the research attributes these benefits to increased engagement, intrinsic moti-vation, and perceived effectiveness, or only in regards to spatial knowledge and practical skills. In such research, a traditional ap-proach without the use of AR and an apap-proach that is enhanced through augmented reality technology are compared. The com-parison usually focuses on either the qualitative feedback of the students about their learning experience or on a quantitative knowl-edge assessment. An inherent fault of such experiments is that the augmented reality approach often includes a newly developed sys-tem or material, and thus differs in more than the aspect of the employed technology from the traditional approach. If the instruc-tional method, use of media, or even the information within the approaches differ, any effect on the learning process can not with-out a doubt be attributed to the use of AR, but might be caused by the other differing factors. Therefore, the main goal of this research is to find out if the positive effects that have been attributed to the use of AR technology can be reproduced in an experiment where the only varying factor is the technology. Additional goals of this research are to observe the effects that the use of augmented real-ity has on learning with declarative knowledge and to find out if the order in which the two approaches are used can influence the learning process.
Based on the problem definition resulting from the theoretical framework, the following research question will be examined:
RQ What effect does augmented reality have on the knowledge gain?
To address this, the following alternative hypotheses are tested: H1 The overall knowledge gain is higher after using AR study
material than after traditional material.
H2 The declarative knowledge gain is higher after using tradi-tional study material than after AR material.
H3 The spatial knowledge gain is higher after using AR study material than after traditional material.
H4 The perceived effectiveness of AR study material is higher than that of traditional material.
H5 The preference of students for AR study material is higher than for traditional material.
4
RESEARCH METHOD AND DESIGN
To be able to answer this research question, a mixture of quantita-tive and qualitaquantita-tive research methods were used in a hypothetical case where basic astronomy topics were being taught to students. Since astronomy is a field of study where usually a combination of spatial and declarative knowledge is required, this use case was chosen.
For this experiment, two equally sized groups of test subjects were used, each studying with traditional material and augmented reality enhanced study material in a lab setting. The order in which the study material was presented to the test subjects differed in both groups. Therefore there were two experimental conditions, one TM-to-ARM condition (group A) where the participants first stud-ied with the traditional material (TM) and then with the augmented reality material (ARM), and one ARM-to-TM (group B) condition where the test subjects first receive the ARM and then the TM. After each study time, the test subjects received 10 closed quan-titative questions in a knowledge assessment. Because the only difference between the two types of study material is the use of augmented reality to present the information, a difference in score of the knowledge assessments would suggest that AR indeed affects the knowledge gain in education contexts. Additionally, a qualita-tive interview was lead after the experiment to gain deeper insights and gather possible explanations for differing scores. This reverse order experimental setup is commonly used to inspect the effects of two varying conditions and was recently used in research by Oh et. al.[23] to analyze, among other factors, the learning outcomes in two different AR approaches. Figure 1 displays a systematic overview of the experimental setup and each step of the experi-ment will be explained in further detail in the following sections.
Figure 1: Experiment procedure, starting with the general introduction and ending with the post-experiment inter-view for group A (TM-to-ARM) and group B (ARM-to-TM)
4.1
Participants
The requirements for the participants of the experiment were an advanced knowledge of the English language, no prior experience using augmented reality devices, and a low level of knowledge about astronomy. Advanced English language skills were necessary for the test subjects to understand the information in the study materials, as well as to understand the questions and answers in the knowledge assessments. If this research should provide an indi-cation that augmented reality as a technology is beneficial to the knowledge gain, applying AR to learning material would include
introducing students to the technology. To take the effects of be-ing introduced to a new technology into consideration, another requirement for the test subjects was no prior experience with AR applications. Finally, participants with advanced knowledge of astronomy were excluded from the experiment. Since the ques-tions in the knowledge assessments are aimed at students with low knowledge about astronomy, students with advanced astronomy knowledge would be able to answer them without relying on the provided information. Through convenience sampling, 26 test sub-jects could be found that were bachelor or master students in the field of information studies and aligned with the requirements. Each participant was randomly assigned to group A or group B, leading to an equal group size of 13 test subjects per experimental condition. While group A was first introduced to the traditional study material (TM), group B received the study material enhanced with augmented reality (ARM) first. Through convenience sam-pling, the random enrolment in one of the two groups, and the requirements, this approach allowed to simulate the scenario of introducing augmented reality enhanced study material in an edu-cational setting.
4.2
Study Material
In order to relate potential score differences in the knowledge as-sessments to augmented reality technology, the two different types of study material were created so that the only difference is the technology. Both types of study materials focused on our solar system in the form of textual, auditory, and visual information. Before starting each study time of 10 minutes, the test subjects were instructed to carefully listen to all available audio snippets and take in the visual information as they see fit. The order in which the audio snippets were listened to and what aspects of the visual information were observed could be freely chosen by the par-ticipants. By using the same types of media, the same instructional approach, and displaying the same information, the only difference between the augmented reality enhanced study material (ARM) and the traditional material (TM) is the use of AR.
Figure 2: Simplified overview of the solar system showing celestial objects and their orbital paths as used in the TM. Used in ARM as 3D hologram without the black background.
4.2.1 Augmented Reality Study Material (ARM). The Galaxy
Ex-plorer1application for the Microsoft HoloLens2was developed by
Microsoft as one of the first applications for the HoloLens. While the application includes information about other parts of the Milky Way galaxy, like nebulas and black holes, the study material en-hanced by AR only considered the information about our solar system. When the user selects our solar system, the first audio snip-pet starts playing and the view changes with an animated transition to a simplified overview of our solar system as depicted in figure 2. For each of the 8 planets in our solar system, as well as the Sun and Pluto, there is an individual audio snippet that starts playing when the celestial object is selected. The view transitions then to a more detailed and zoomed-in viewpoint, as shown in figure 3, which shows unique visual features that are being mentioned in the audio snippets. Additionally, there is an audio snippet relating to the dimensions in the solar system and a more realistic overview of the solar system. The 12 audio snippets provide the auditory information about our solar system, while the visual information can be observed in a simplified or realistic overview of the solar system, as well as in a more detailed view for each celestial object. Finally, the only textual information that is provided by the ARM is the name and orbital period of the currently selected celestial object in the overview of the solar system.
A different part of the galaxy, the galactic center, was used as an introduction to the HoloLens. The test subjects received funda-mental instructions on how to interact with the HoloLens and the application. There is no overlap between the provided information related to the galactic center and the information related to our solar system which is the focus of this experiment. Thus, select-ing the galactic center and listenselect-ing to the audio snippet was an ideal way to introduce the test subjects to controls and settings before the study time could commence. While the application does allow for additional controls, like rotating and zooming, these were prohibited for the students to keep the interaction simple, to pre-vent possible problems during use, and to display the media with a consistent viewpoint for every participant.
Figure 3: Detailed view of a celestial object (Jupiter) showing its unique features (red spot) as used in the TM. Used in ARM as 3D hologram without the black background.
1Galaxy Explorer in the Microsoft Store:
https://www.microsoft.com/en-us/p/galaxy-explorer/9nblggh4q4jg?activetab=pivot%3Aoverviewtab
2Microsoft Website about the HoloLens: https://www.microsoft.com/en-us/hololens
4.2.2 Traditional Study Material (TM). While the ARM did only
need to be adapted slightly for the sake of this experiment, the traditional study material had to be created from scratch. A website was created that makes use of the same images, audio snippets, and textual information as the Galaxy Explorer application3. While the audio snippets could be extracted from the GitHub repository of the application4and the textual information reproduced identically, the images used in the TM were screenshots from the application as depicted by figure 2 and 3. The actual three-dimensional models of the celestial objects that are being used in Galaxy Explorer could not be reproduced in the traditional material. Through the use of screenshots, the most important visual information could still be observed in the TM, and the missing 3D visuals can be seen as part of the AR technology. Therefore the only difference between ARM and TM is the augmented reality technology. The traditional study material was presented on a tablet and allowed for the same controls as the ARM, also excluding zooming and rotating func-tions. Additionally, test subjects were asked to use their personal earphones for the TM in order to increase the similarity to the way the audio snippets are played in the HoloLens and to assure that the test subjects could hear the audio snippets well enough. Through the high familiarity with tablet computers, a special in-troduction as for the ARM was not necessary for the traditional study material. The test subjects were only informed about where to change the audio volume and that, as for the ARM, zooming and rotating functionalities have been intentionally disabled.
4.3
Knowledge Assessments
After studying with a certain type of study material, each test subject received a corresponding knowledge assessment. Both, the knowledge assessment after the traditional material (KATM) and the knowledge assessment after the augmented reality material (KAARM), include 10 closed questions each that have been extracted and derived from the Test Of Astronomical STandards or TOAST[26]. While the TOAST is the most common standardized test in the field of astronomy, many of its questions are too much reliant on astrophysics and other specific information that is not included in the study materials for this experiment. Around half of the KATM and KAARM questions could be extracted identically from the TOAST. The other questions in the knowledge assessments were derived from the TOAST as closely as possible, focusing on style and syntax, as well as the kind of information that is being queried. Research suggests that augmented reality might provide a benefit to the knowledge gain when spatial knowledge is concerned, like the different sizes, arrangements, and interactions of objects. Because of this, the 10 questions of each knowledge assessment have been split into 5 questions involving declarative knowledge and 5 questions involving spatial knowledge. Questions that can be answered by factual information that is provided explicitly and directly to the test subject, either via audio snippet or text, are considered declarative knowledge in this context. On the other hand, questions that can only be answered by observing spatial properties of the celestial objects, e.g. distances, sizes, and colors,
3Website that was used as traditional study material:
https://michael-kern.info/thesis/app.php
4GitHub repository of the Galaxy Explorer application:
https://github.com/Microsoft/GalaxyExplorer
are considered to be spatial knowledge.
In order to eliminate answers that were not affected by the provided study material, test subjects were asked to mark the questions to which they would have known the answer before starting the experiment. If a test subject marked a question and answered it correctly, the response was not counted towards the overall score of the test subject. If the question was marked but answered falsely, the response did count towards the overall score. While in some fields of study one could certainly argue that some information needed to answer a question might have been gained subconsciously, this is not the case for this experiment. Through the very specific nature of the questions and the uncommon study field of astronomy, the test subjects were able to reliably mark a question if they were sure they already knew the answer. Both knowledge assessments had predefined answers and therefore a quantitative measurement of the score for each participant could be made.
4.3.1 Knowledge Assessment after TM (KATM). The knowledge
assessment that was being handed to the test subjects after the traditional study material considered of the following 10 questions. Questions are named D + number if they rely on declarative knowl-edge and S + number if spatial knowlknowl-edge is required:
D1 What will happen to our Sun in 5 billion years? D2 What is our Sun mostly made of?
D3 What is the ’Goldilocks Zone’?
D4 Why does Jupiter have a large red spot that can be seen from earth?
D5 Why is Pluto not considered a planet anymore?
S6 If you were in a spacecraft near the Sun and began traveling to Pluto you might pass…
S7 Which of the following statements about the orbits of plan-ets in our solar system is correct?
S8 Which of the following ranks locations from closest to Earth to farthest from Earth?
S9 Which of the following is the best ranking (from smallest to largest) for the size of these objects?
S10 Which of the following statements about orbits in our solar system is correct?
4.3.2 Knowledge Assessment after ARM (KAARM). After
study-ing with study material enhanced by augmented reality, the test subjects were handed the following 10 questions:
D1 What is Pluto mostly made of?
D2 On which planet in our solar system can it rain sulfuric acid?
D3 How big is the Sun compared to Earth? D4 Why does Saturn have a pale yellow color?
D5 On which planet can you find frozen CO2at the planet’s
poles?
S6 Between which of the following to planets in our solar system is an asteroid belt?
S7 Which of the following planets has a blue color when seen from space?
S8 Which of the following distances between two planets in our solar system is the smallest?
S9 Which of the following ranks planets, from closest to the Sun to farthest from the Sun?
S10 Which of the following is the best ranking (from smallest to largest) for the size of these objects?
4.4
Post-Experiment Interview
The final step of the experiment was a qualitative semi-structured interview with open-ended questions. After the test subjects studied with both materials and received both knowledge assessments, they were asked the following 4 questions:
INT1 During the study time with the ARM, did you move around in the solar system overview and the detail view? INT2 Which of the two methods (studying with ARM and
study-ing with TM) do you think is more effective and why? INT3 Which of the two methods (studying with ARM and
study-ing with TM) do you enjoy more and why?
INT4 Do you have any remarks on the augmented reality study-ing method?
The purpose of these questions was to gather additional feedback on the studying methods in order to explain individual results. Ques-tion 1 was asked to see if the test subjects actually moved around while wearing the HoloLens and thus made use of the spatial dimen-sion. To figure out the relation between the perceived effectiveness and the enjoyability of the two studying methods, questions 2 and 3 were included in the questionnaire. While research already sug-gests that many students would favor the use of augmented reality technology in an educational context[33], these two questions aim to find out if the preference is based on a perceived higher effec-tiveness or other factors. Finally, test subjects were asked for their feedback on studying with ARM and their suggestions on how to improve augmented reality enhanced studying in this context.
5
RESULTS
After 26 iterations of the experiment, 13 of them with the TM-to-ARM condition and 13 of them with the TM-to-ARM-to-TM condition, the knowledge assessment scores were evaluated and compared. To answer the individual hypotheses and the overall research question in the following sections, the scores were handled as percentages and rounded to the nearest integer value. Figure 4 shows the over-all average scores of group A and group B, as well as the average declarative and spatial scores for each group and each knowledge assessment. A score of 100% would mean that all the questions that were answered and not skipped, were answered correctly. For the following sections, unless otherwise stated, the average values for the specific conditions will be used. All standard deviations range between a 19% maximum (for KATM A) and a 14% minimum (for KATM B) and should be considered when reading the following results to give a better understanding of the mean values. Further-more considered should be the variances that range from 2% to 11% and are taken as a reference for when a score difference is considered small if its value is close to the variance. The scores are normally distributed with more than 95% of the scores extending within two standard deviations from the mean. Additionally, if the median values show a distinct difference from the mean it will be applied to compensate for outliers and to provide an understanding of which direction the scores are leaning towards.
Figure 4: Overview showing mean scores of group A and group B in KATM and KAARM. The bold value is the overall score and the D and S values are the declarative and spatial mean scores of each group in each knowledge assessment.
5.1
Hypotheses Testing
For the set hypotheses that result out of the research question and the problem definition, the following sections each explain all relevant findings for each hypothesis and finish with a statement to whether the hypothesis can be adopted or rejected.
5.1.1 H1: The overall knowledge gain is higher after using AR
study material than after traditional material. In the KATM, the average score of both groups was 82%. Group A, which had the TM-to-ARM condition and therefore received the KATM as first knowledge assessment, had an average score of 74%. Group B, which had the ARM-to-TM condition and therefore received the KATM as second knowledge assessment, had an average score of 91%. This means that group A scored 17% lower than group B. Group B first received the augmented reality study material, therefore test subjects of group B received the information for the second time before the KATM. Additionally, group B first received the KAARM, meaning that they could have used the KAARM to gain a better un-derstanding of what the questions focus on and afterward adopted their study behavior in the study time with the TM accordingly. In the KAARM, the average score of both groups was 68%. Group A scored 63%, while group B scored 74% and therefore 11% higher than group A. Considering that group A first received the TM and the KATM, and therefore could study the information for the second time before the KAARM and also had an idea of the questioning style from the KATM, the lower score is unexpected and could indicate an increasing level of distraction from the first study time iteration to the second.
This suspected learning effect that results out of studying the infor-mation for the second time is 17% for group B from the KAARM to the KATM, but -11% for group A from the KATM to the KAARM. This means that for group A the expected learning effect is actually reversed and they performed 11% lower in the second knowledge assessment they received. Combined with the score differences for KATM and KAARM, this suggests that the distracting abilities of the AR technology are increased for group A since it adds to the
effect of paying less attention due to hearing the audio snippets for the second time.
Finally, the hypothesis H1 cannot be adopted since both groups have lower average scores in the KAARM than in the KATM. There-fore the experiment failed to reject the null hypothesis that there is no difference between the knowledge gain with TM and with the ARM.
5.1.2 H2: The declarative knowledge gain is higher after using
traditional study material than after AR material. The combined av-erage score in KATM and KAARM with declarative questions was 74% for both groups. Group A had an average score of 71% with declarative questions and group B an average score of 78%. The difference of only 7%, therefore, indicates that test subjects scored similar regardless of their condition.
The overall average score of declarative questions in the KATM was 87% with the median being 100% while the average score in the KAARM was only 62% with a 75% median. This 25% difference indicates that the declarative knowledge gain after using the tradi-tion study material is higher than after using the augmented reality study material and indicates the distracting factor of AR. Group A answered 83% of the declarative questions in the KATM correctly, while group B scored 7% higher with an average score of 90%. In the KAARM, group A answered 58% of the questions correctly and group B scored 65%. Group A’s score on declarative questions decreased by -25% from 83% in the KATM to 58% in the KAARM, while group B’s score increased by 25% from 65% in the KAARM to 90% in the KATM. The small 7% difference in declarative KATM questions compared to the 18% standard deviation means that the learning effect did not influence the AR approach. To summarize, the null hypothesis that there is no difference in regards to the declarative knowledge gain can be rejected and hy-pothesis H2 adopted, since test subjects answered more declarative questions correctly after studying with the TM than after studying with the ARM.
5.1.3 H3: The spatial knowledge gain is higher after using AR
study material than after traditional material. Questions related to spatial knowledge in KATM and KAARM were on average answered 78% correctly. Group A scored 67% and thus 21% lower than group B with 88% mean and 100% median. This large difference indicates that the order in which the test subjects received the study material influenced the spatial knowledge gain. The condition ARM-to-TM seems more effective when it comes to gaining spatial knowledge. This can be seen as a beneficial indication for the easier conceptu-alization of the spatial model in the ARM, followed by a compact overview of spatial topics in the TM.
Group B answered 91% of spatial questions in the KATM correctly and thus 23% higher than group A with 68%. The higher score of group B might indicate the above-mentioned learning effect based on the combination of TM and ARM. In the KAARM, group B scored 85%, therefore the learning effect was 6% between KAARM and KATM. Group A scored 67% in the KAARM and 68% in the KATM which doesn’t indicate a learning effect. The median for spatial questions of group A in the KAARM is 60% and thus 7% lower than the mean, suggesting that the differences would even be signifi-cantly larger when removing the outliers.
Overall, the gathered data does not indicate a rejection of the null
hypothesis, being that there is only a small difference in the spa-tial knowledge gain between ARM and TM. In fact, the spaspa-tial knowledge gain for group B was slightly higher after studying with the traditional material than after the augmented reality mate-rial. However, the order in which the two types of study material are observed does indeed seem to have an influence on the spa-tial knowledge gain and generates a new alternative hypothesis, that ARM-to-TM is more effective than TM-to-ARM when spatial knowledge is concerned.
5.1.4 H4: The perceived effectiveness of AR study material is
higher than that of traditional material. As part of the qualitative post-experiment interview, participants were asked which of the two approaches they thought was more effective. 65% answered that they believe the traditional approach to be more effective to learn such information as presented in the experiment. With minor exceptions, the majority of participants argued that the traditional material is less distracting than the augmented reality. More than half of the participants were aware of this negative effect that the AR technology might have on their knowledge gain. The second main argument is the better overview and compactness of informa-tion in the tradiinforma-tional material. Through the immersive nature of AR technology and the ARM, participants felt it is easier to gain a better overview of the information in the traditional approach. The results in regard to hypothesis H2 and H3 give a further indication of this notion.
On the other hand, the immersive capabilities of AR, combined with the higher level of detailed visualizations and 3D capabilities, were also seen by 62% as the main reason why the augmented re-ality approach might actually be more effective. Especially when knowledge is concerned that relies more on visual information and an accurate representation of an object, the perceived effectiveness of AR was higher because the participants could more easily under-stand and conceptualize the spatial attributes of the information. The majority of participants did not perceive the effectiveness of augmented reality as higher than that of the traditional approach. Students were more aware of the possible downsides of new tech-nology and media-richness which research has shown can decrease the knowledge gain. Therefore the null hypothesis that there is no difference in the perceived effectiveness cannot be rejected and H4 cannot be adopted.
5.1.5 H5: The preference of students for AR study material is
higher than for traditional material. Every single participant an-swered that they indeed enjoyed the augmented reality approach more than the traditional method. This indicates a clear prefer-ence towards this approach and was explained by the participants with the same main arguments as for the perceived effectiveness. Students enjoyed the immersive aspects of the AR technology, espe-cially being able to control their perspective by physically moving around in the hologram of the solar system. The superior and more detailed visualizations of the HoloLens, as well as the general cu-riosity towards a new technology, additionally contributed to this preference and the result is not surprising. It is worth mentioning though that out of the 100% that expressed their preference for AR, 65% also said that they perceive AR to be less effective and still stated their preference for AR just moments later.
This leads to the conclusion that the clear majority of test subjects
prefers the AR method over the traditional material and thus the null hypothesis, that there’s no difference in preference of ARM and TM, can be rejected and hypothesis H5 can be accepted. Although half of those test subjects were also aware of the downsides that the technology might cause, it did not affect their preference.
5.2
Additional Findings
Apart from the findings that directly relate to the hypotheses, addi-tional findings were discovered that mostly result out of the qualita-tive post-experiment interview and also outliers in the knowledge assessments. Although most of these findings cannot be directly correlated to the main hypotheses, some still contribute to the overall research question and are thus explained in the following sections before coming back to the research question.
5.2.1 Improving the AR approach. Asked about any additional
remarks on the AR method, 38% replied with hardware issues of the HoloLens that affected their learning experience. These issues were mostly focussing around the small field of view that reduces the immersion and the quality of the visualizations, as well as the weight and structure of the device which lead to test subjects getting uncomfortable after wearing it for a few minutes. To increase the knowledge gain, 42% also suggested that the audio snippets should be repeatable or that the key points of the snippets are additionally shown as textual information. This was perceived to lead to a higher knowledge gain since it could counter the distracting effects of AR technology that were expressed by 65% of the participants. Adding textual information could potentially also contribute to a more compact overview of the information, which was also one of the main deciding factors of perceived effectiveness. Only 2 out of the 26 test subjects did not move around during the study time with the ARM. Interestingly, both did not express a higher perceived effectiveness of the TM due to increased distractions in the AR approach. Instead, they argued that the ARM was more effective due to the above-mentioned benefits. This might be explained with physical movement adding another aspect to the multitasking that participants did during the ARM study time, and therefore physical movement would increase the distraction levels. But due to this particularly small sample size, no conclusions can be drawn at this point and only future research could determine how physical movement affects the knowledge gain.
5.2.2 Outlier Questions. Question D3 in the KAARM is the most
prominent outlier among the questions in both knowledge assess-ments. Exactly 1 out of the 26 total participants managed to answer this question correctly, and after being asked how the participant chose the answer, they admitted it was just a guess. The answer to this question about the relation of Earth’s size and the size of the sun was explicitly mentioned in the audio snippet for the sun, as were the answers to all other declarative questions. After half of the experiments had passed without a correct answer, it became clear that something was exceptional about this question. From then on students were asked why they chose their incorrect answer. None of the students could recall on hearing the required information in the audio snippet of the ARM and just guessed the correct answer. With a 1 in 5 chance to guess the correct answer, only 1 correct an-swer in 26 participants means that some other factor played a role.
When examined, participants revealed that they were thinking of the relation between the sun’s mass to Earth’s mass and supposed that the ratio for the size must be an equally large number. This influenced the test subjects not to pick the low correct number. But no matter the reasoning, the fact that the correct answer was indeed explicitly mentioned in the audio snippets goes to show that the AR technology does have a distracting effect which causes this outlier.
Outliers among the questions in the KATM, where more than one-third of the test subjects skipped the question and more than 90% of the answers were correct, were the questions D1, D4, and S8. Mean-ing that more than one-third of the participants knew the correct answer and did not rely on information from the study material to answer them, these questions can be left out in a future experiment. In the KAARM there were no outliers with a larger skipping quota than one-third. The most skipped question there was the question S9 with 27%, which could therefore also be considered to be left out in a future experiment. Additionally, KAARM questions D2 and S8 were disproportionally less often answered correctly by participants of group A. The average of correct answers is more than 30% higher in group B who received the ARM first. Without additional research, this can only be taken as a further indication that the students paid less attention to the audio snippets in the second study time and thus group A was more distracted while studying the ARM.
5.3
What Effect Does Augmented Reality Have
On The Knowledge Gain?
An overall positive effect on the knowledge gain just by implement-ing augmented reality into the study material is not supported by the data gathered from this experiment. The total average scores in both experimental conditions, as well as the average scores for declarative and spatial questions, were lower in the KAARM than in the KATM. The main factors that influence these results are the distractions resulting from the unfamiliarity with AR technology and from the immersive visualizations. None of the test subjects was familiar with the HoloLens or extensive use of augmented reality before the experiment. Implementing new technologies in an educational context has proven to have a distracting effect on the students and negatively impact the knowledge gain[32][20]. Especially a media-rich technology like augmented reality with a lot of visual information can add to the extraneous cognitive load that is imposed upon the students[28][24]. The data of this experi-ment furthermore suggests that the distraction level with the ARM was even higher for group A since they heard the audio snippets for the second time during the ARM approach and therefore paid less attention. Due to these negative influences on the knowledge gain, the augmented reality approach has also not proven to be more effective when declarative knowledge is concerned. The data clearly indicates that the traditional approach was more benefi-cial for the declarative knowledge gain, whereas the effects on the spatial knowledge gain are less certain. While the overall average score in spatial questions was just slightly higher in the KATM, the ARM-to-TM condition led to higher scores and thus indicates that the spatial knowledge gain could benefit from augmented reality. For the TM-to-ARM condition, the knowledge gain after the ARM
was not significantly higher, therefore indicating that the ARM-to-TM condition is more effective. Even though more than half of the test subjects were aware of the possible downsides of augmented reality, 100% still preferred the augmented reality method over the traditional approach.
So to summarize, this research indicates that augmented reality can indeed have a positive effect on the spatial knowledge gain and is the preferred method of learning. But due to the negative effect in regards to declarative knowledge and the potentially distracting abilities of the technology, an overall positive effect on the knowl-edge gain just by implementing the technology cannot be claimed. On the other hand, the preference for the implementation of AR into the study material can lead to a higher engagement and moti-vation, and the additional remarks made by the test subjects can be used to counter possible downsides. Therefore, if a study material is created while focusing on the AR technology and considering its positive and negative attributes, as was done in many other pieces of research, a higher knowledge gain can be achieved.
6
DISCUSSION
As expected, the gathered data does not support a positive effect on the knowledge gain just by implementing augmented reality technology. Especially when declarative knowledge is concerned, the negative effects of augmented reality become more apparent. Mainly the distracting effect due to unfamiliarity with the technol-ogy and the shift of focus towards the visual information explain why the declarative knowledge gain does not benefit from AR. Nev-ertheless, the research also shows that the spatial knowledge gain can be positively affected by the media-richness and visualization capabilities of AR. Being immersed in a holographic model, com-bined with the ability to move around in it and to control viewpoints, allows the students for a better conceptualization of the system that is modeled. Other research has already indicated these correlations between visualization, modeling, and learning effect[3][1]. With the clear preference of the students for using AR in an education context, it is worth analyzing the best way to implement it into study material.
Other factors that contribute to the knowledge gain like the instruc-tional method or information presentation were eliminated in this research for the purpose of attributing the results to AR as the only varying factor. But the different intrinsic difficulties of the questions in the two knowledge assessment could have influenced the results of the research. Even though the questions all were adopted from a standardized astronomy test and followed the same schema, 5 declarative questions explicitly answered in the audio snippets plus 5 spatial questions that could be answered with the presented visual information, nuances like wording or answer options can alter the results. Performing such an experiment again and reversing the knowledge assessments can be a promising way to observe differ-ences in the intrinsic difficulties of the questions. Due to the small total number of questions which could be even further lowered by correctly skipping questions, answering a single question wrong can already have a significant impact on the overall score. Because of this and the small sample size of 26 participants, statistically relevant conclusions cannot be deducted and would only result in very high p-values and thus very low statistical significance. With
a bigger sample size, it can be expected that the trends stay the same, as related research suggests and the qualitative results of this research show. Statistically proving a positive effect on spatial and a negative or neutral effect on declarative knowledge can direct future research towards a path where effects on the knowledge gain are more accurately attributed to the technology.
7
CONCLUSION
When augmented reality in an educational context is regarded in research, that research typically focuses on the perceived benefits that AR has on the knowledge gain. Learning approaches are being created that have implemented AR and are being compared to the previous methods. As this research shows, the implementation of the technology itself is unlikely to solely cause the reported effects to the knowledge gain. Other factors like an instructional approach that is optimized for AR or a different presentation of the information are required to achieve the best possible effects on the knowledge gain. Especially the distracting effect of AR in education is often disregarded, even though students recognize this as a potential negative effect and this research further indicates this downside. But to fill the research gap of how AR affects the knowl-edge gain, further research is necessary which can eliminate flaws like intrinsic difficulty differences of the knowledge assessments and statistically prove the findings indicated by this experiment with a larger sample size. Future research could, for example, recre-ate the experiment and switch KATM and KAARM questions, or increase the sample size enough so that each test subject gets the same questions, but half of the sample solely uses the ARM and the other half only studies with the TM.
Seeing how technology is progressing and educational approaches are starting to more and more adjust, sooner or later augmented re-ality will be also found in a wide range of educational contexts. The clear preference for students for a technology like AR over tradi-tional learning methods can lead to increased student engagement and motivation. It is therefore worth pursuing to further investigate how the technology can be used to benefit the learning experience to the fullest extent. For this, it is crucial that the precise effects of a technology are analyzed and separated from other factors of the learning method. Possible downsides of implementing a technology can be counteracted if it is known what these downsides are, why they are caused by the technology, and how it affects the knowl-edge gain. A combination of different technologies and learning approaches might prove as an effective way of utilizing the benefits and countering the downsides of each. For the hypothetical edu-cational use case of this experiment, a combination of traditional material to study the declarative information and augmented reality enhanced material to study the spatial and visual information will most likely lead to the best results.
Technologies can bring benefits to a learning method which are often unique to the technology and couldn’t be accomplished oth-erwise. But to allow a wide-range implementation it needs to be proven beyond a doubt what the specific benefits are and which aspect of the technology causes it. Only this allows for the benefits to be examined individually, generalized to a broader scale, and finally implemented into current learning approaches.
REFERENCES
[1] Stephen Alessi. 2000. Building Versus Using Simulations. Integrated and Holistic Perspectives on Learning, Instruction and Technology(2000), 175–196. DOI:http: //dx.doi.org/10.1007/0-306-47584-7{ }10
[2] Misty Antonioli, Corinne Blake, and Kelly Sparks. 2014. Augmented reality applications in education. In Journal of Technology Studies, Vol. 40. IEEE, 96–107. DOI:http://dx.doi.org/10.1109/IWADS.2000.880913
[3] Anthony R. Artino and Steven J. Durning. 2012. ’Media will never influence learning’: But will simulation? Medical Education 46, 7 (2012), 630–632. DOI: http://dx.doi.org/10.1111/j.1365-2923.2012.04270.x
[4] Mark Billinghurst and Andreas Duenser. 2012. Augmented Reality in the Class-room. Computer 45, 7 (7 2012), 56–63. DOI:http://dx.doi.org/10.1109/MC.2012. 111
[5] T. G. K. Bryce and E. J. Blown. 2012. The Novice-Expert Continuum in Astronomy Knowledge. International Journal of Science Education 34, 4 (3 2012), 545–587. DOI:http://dx.doi.org/10.1080/09500693.2011.642325
[6] Kun-Hung Cheng and Chin-Chung Tsai. 2013. Affordances of Augmented Re-ality in Science Learning: Suggestions for Future Research. Journal of Science Education and Technology22, 4 (2013), 449–462. DOI:http://dx.doi.org/10.1007/ s10956-012-9405-9
[7] R. E. Clark. 1983. Reconsidering Research on Learning from Media. Review of Educational Research53, 4 (1983), 445–459. DOI:http://dx.doi.org/10.3102/ 00346543053004445
[8] Angela Di Serio, Mara Blanca Ib´a ˜nez, and Carlos Delgado Kloos. 2013. Impact of an augmented reality system on students’ motivation for a visual art course. Computers and Education68 (2013), 585–596. DOI:http://dx.doi.org/10.1016/j. compedu.2012.03.002
[9] Andreas D¨unser, Lawrence Walker, Heather Horner, and Daniel Bentall. 2012. Creating interactive physics education books with augmented reality. Proceedings of the 24th Australian Computer-Human Interaction Conference on - OzCHI ’12 (2012), 107–114. DOI:http://dx.doi.org/10.1145/2414536.2414554
[10] Stephanie Fleck and Gilles Simon. 2013. An Augmented Reality Environment for Astronomy Learning in Elementary Grades. Proceedings of the 25th ICME conference francophone on l’Interaction Homme-Machine - IHM ’13(2013), 14–22. DOI:http://dx.doi.org/10.1145/2534903.2534907
[11] Ernest Timothy Goetz, Lauren Cifuentes, Wendy Keeney-kennicutt, and Trina J Davis. 2015. Effectiveness of virtual reality-based instruction on students fi learning outcomes in K-12 and higher education : A meta-analysis. 70, JANUARY 2014 (2015), 29–40. DOI:http://dx.doi.org/10.1016/j.compedu.2013.07.033 [12] Triin Hannust and Eve Kikas. 2007. Children’s knowledge of astronomy and its
change in the course of learning. Early Childhood Research Quarterly 22, 1 (1 2007), 89–104. DOI:http://dx.doi.org/10.1016/j.ecresq.2006.11.001
[13] W. Lewis Johnson. 2010. Using immersive simulations to develop intercultural competence. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)6259 LNCS (2010), 1–15. DOI:http://dx.doi.org/10.1007/978-3-642-17184-0{ }1
[14] Aaron Kotranza, D. Scott Lind, Carla M. Pugh, and Benjamin Lok. 2009. Real-time in-situ visual feedback of task performance in mixed environments for learning joint psychomotor-cognitive tasks. In 2009 8th IEEE International Symposium on Mixed and Augmented Reality. IEEE, 125–134. DOI:http://dx.doi.org/10.1109/ ISMAR.2009.5336485
[15] Kangdon Lee. 2012. Augmented Reality in Education and Training. Linking Research and Practice to Improve Learning56, 2 (2012), 13–21. DOI:http://dx.doi. org/10.1007/s11528-012-0559-3
[16] A. Lintu and M. Magnor. 2006. An Augmented Reality System for Astronomical Observations. In IEEE Virtual Reality Conference (VR 2006). IEEE, 119–126. DOI: http://dx.doi.org/10.1109/VR.2006.24
[17] Su Houn Liu, Hsiu Li Liao, and Jean A. Pratt. 2009. Impact of media richness and flow on e-learning technology acceptance. Computers and Education 52, 3 (2009), 599–607. DOI:http://dx.doi.org/10.1016/j.compedu.2008.11.002
[18] Anu Maria. 1997. Introduction to modelling and simulation. Winter Simulation Conference(1997), 7–13. DOI:http://dx.doi.org/10.1109/WSC.1997.640371 [19] Mia Mari´c. 2014. Individual and Social Factors Related to Students’ Academic
Achievement and Motivation for Learning. 17 (2014), 63–79.
[20] Giacinto Matarazzo and Abigail Sellen. 2000. The value of video in work at a distance: Addition or distraction? Behaviour and Information Technology 19, 5 (2000), 339–348. DOI:http://dx.doi.org/10.1080/014492900750000045 [21] Richard E. Mayer. 2003. The promise of multimedia learning: Using the same
instructional design methods across different media. Learning and Instruction 13, 2 (2003), 125–139. DOI:http://dx.doi.org/10.1016/S0959-47520200016-6 [22] National Academies Press. 2000. How People Learn. National Academies Press,
Washington, D.C. DOI:http://dx.doi.org/10.17226/9853
[23] Seungjae Oh, Hyo Jeong So, and Matthew Gaydos. 2018. Hybrid Augmented Reality for Participatory Learning: The Hidden Efficacy of Multi-User Game-Based Simulation. IEEE Transactions on Learning Technologies 11, 1 (2018), 115– 127. DOI:http://dx.doi.org/10.1109/TLT.2017.2750673
[24] Fred Paas, Juhani Tuovinen, Huib Tabbers, and Pascal W. M. Van Gerven. 2010. Cognitive Load Measurement as a Means to Advance Cognitive Load Theory. Educational Psychologist1520, 38 (2010), 43–52. DOI:http://dx.doi.org/10.1207/ S15326985EP3801
[25] Nico Rutten, Wouter R. Van Joolingen, and Jan T. Van Der Veen. 2012. The learning effects of computer simulations in science education. Computers and Education58, 1 (2012), 136–153. DOI:http://dx.doi.org/10.1016/j.compedu.2011. 07.017
[26] Stephanie J. Slater. 2015. The Development And Validation Of The Test Of Astronomy STandards (TOAST). Journal of Astronomy & Earth Sciences Education (JAESE)1, 1 (1 2015), 1. DOI:http://dx.doi.org/10.19030/jaese.v1i1.9102 [27] Masato Soga, Koji Matsui, Kazuki Takaseki, and Kohe Tokoi. 2008. Interactive
Learning Environment for Astronomy with Finger Pointing and Augmented Reality. In 2008 Eighth IEEE International Conference on Advanced Learning Tech-nologies. IEEE, 542–543. DOI:http://dx.doi.org/10.1109/ICALT.2008.144 [28] John Sweller, Jeroen J.G. Van Merrienboer, and Fred G.W.C. Paas. 1998. Cognitive
Architecture and Instructional Design. Educational Psychology Review 10, 3 (1998), 251–296. DOI:http://dx.doi.org/10.1023/A:1022193728205
[29] The New Media Consortium. 2010. The Horizon Report. Vol. 3. 1–40 pages. DOI: http://dx.doi.org/10.1227/01.NEU.0000280154.19237.36
[30] Xiaodong Wei, Dongdong Weng, Yue Liu, and Yongtian Wang. 2015. Teaching based on augmented reality for a technical creative design course. Computers and Education81 (2015), 221–234. DOI:http://dx.doi.org/10.1016/j.compedu.2014. 10.017
[31] Marcin Wichrowski. 2013. Teaching Augmented Reality in Practice: Tools, Workshops and Students’ Projects. Proceedings of the International Conference on Multimedia, Interaction, Design and Innovation(2013), 19:1–19:10. DOI:http: //dx.doi.org/10.1145/2500342.2500362
[32] Jennie Winter, Debby Cotton, Joan Gavin, and Jon D. Yorke. 2010. Effective e-learning? Multi-tasking, distractions and boundary management by graduate students in an online environment. ALT-J: Research in Learning Technology 18, 1 (2010), 71–83. DOI:http://dx.doi.org/10.1080/09687761003657598
[33] Hsin Kai Wu, Silvia Wen Yu Lee, Hsin Yi Chang, and Jyh Chong Liang. 2013. Current status, opportunities and challenges of augmented reality in educa-tion. Computers and Education 62 (2013), 41–49. DOI:http://dx.doi.org/10.1016/j. compedu.2012.10.024
[34] Steve Chi-Yin Yuen, Gallayanee Yaoyuneyong, and Erik Johnson. 2011. Aug-mented Reality: An Overview and Five Directions for AR in Education. Journal of Educational Technology Development and Exchange4, 1 (2011), 119–140. DOI: http://dx.doi.org/10.18785/jetde.0401.10
APPENDIX A
KNOWLEDGE ASSESSMENT DATA
The following tables show the gathered data from the experiment and the different calculations. While figure 5 shows an overview of the combined scores and calculations, figure 6 shows the raw data that was being gathered. Symbols and abbreviations are defined as the following:
• A stands for group A with the TM-to-ARM condition • B stands for group B with the ARM-to-TM condition
• KATM stands for the knowledge assessment after the traditional material
• KAARM stands for the knowledge assessment after the augmented reality material • D stands for the declarative questions
• S stands for the spatial questions • Ø stands for the mean scores • MED stands for the median scores
• SKIP stands for the percentage of skipped questions • Green 1 stands for a correct answer
• Red 0 stands for an incorrect answer
• Yellow X stands for a correct but skipped answer • STDEV stands for standard deviation
• VARIANCE stands for variance
• Learning Effect stands for the difference between the first knowledge assessment, which was KATM for group A and KAARM for group B, and the second knowledge assessment, which was KAARM for group A and KATM for group B
Figure 5: These tables show the various calculated scores from the gathered data. The largest mean score of each row is highlighted with a dark green, the lowest mean score of each row is white. To show where the difference between median and mean score is significant, red highlighting means that the median is lower than the average score, green highlighting means that the median is higher than the average score.
Figure 6: This table shows the individual scores with each question for every test subject, average scores, and skipping rates.
APPENDIX B
POST-EXPERIMENT INTERVIEW DATA
The following table shows the coded replies from the post-experiment interview. For the open questions INT2, INT3, and INT4, the answers were categorized according to the main arguments of each response. Additional symbols and abbreviations are defined as the following:
• INT stands for the post-experiment interview questions
• Green 1 stands for ’yes’ in INT1 or answering with a certain argument in INT2, INT3, and INT4 • Red 0 stands for ’no’ in INT1 or answering without a certain argument in INT2, INT3, and INT4 • MOVING stands for whether or not the test subjects were moving during the ARM study time • TM DISTRACT stands for believing the TM to be more effective due to fewer distractions
• AR IMMERSIVE stands for perceiving the ARM as more effective in INT2 or as more enjoyable in INT3 due to immersion • AUDIO + TEXT stands for test subjects advising for a combination of audio and text in the ARM
• HARDWARE stands for test subjects who had hardware issues or think the hardware needs to be improved
Figure 7: This table shows the individual data that was gathered in the post-experiment interview for each test subject and the average scores. The answer categories can be found in the first column.
