Digital Examination Support
for Students with Dyslexia
Submitted in partial fulfilment of the requirements
for the degree of Master of Science
Stavros Kanellopoulos
11394048
Master Information Studies
Human-Centred Multimedia
Faculty of Science
University of Amsterdam
July 14, 2017
1st Supervisor: 2nd Supervisor:
Dr. André Nusselder Dr. Daniel Buzzo
INTRODUCTION 2
BACKGROUND 3
Defining Dyslexia 3
Examination and Dyslexia 3
Reading and Understanding 3
Answering 4
Dyslexia and Examination Support 4
Dyslexia and Assistive Technology 4
METHODOLOGY 4
Interviews 5
Interview Settings 5
Dyslexic Student Interviews 5
The effects of Dyslexia According to Students 5
Extra Time as Accommodation 6
Coping Mechanisms 6
Expert Interview 6
Prototype Description 7
Interface Description and Functionality 7
Development 8
Experiment 9
ANALYSIS AND RESULTS 9
Data and Analysis Strategy 10
Quantity-Based Data 10
Quality-Based Data 10
Analysis and Results 10
High Level Analysis 10
Medium Level Analysis 12
Low Level Analysis 13
DISCUSSION 14 Analysis Strategy 14 Inconclusive Data 15 Concept Decisions 15 Applications in Real-life 16 CONCLUSIONS 16 FUTURE WORK 16 AKNOWLEDGEMENTS 17 REFERENCES 17 APPENDICES 18 1
Digital Examination Support
for Students with Dyslexia
Stavros Kanellopoulos
University of Amsterdam 1012 WX Amsterdam +31 06 2731 3590Kanellopoulos.s@gmail.com
Keywords
Digital Exam; Dyslexia; Learning Disabilities; Education; University Students; Examination Support; Text-to-Speech;
ABSTRACT
The examination process is a big part of higher education; yet, in special cases it is not considered always fair. Dyslexia for example is the most common learning disability that can affect the performance of a student and ultimately the accuracy of an exam assessment. While support can be provided to students with dyslexia, it is not always effective. In many cases, students with dyslexia are required to use much more of their mental capacity than regular students do, in order to perform equally on an exam. This makes the process of exam-taking much more demanding. This paper focuses on giving an alternative supporting mechanism to students with dyslexia during an exam, using digital technology, that aims to increase the performance of dyslexic students while reducing the extra mental capacity required. After conducting an interview with dyslexic students and an expert on the field, we identified a supporting method that uses sound and has the potential to achieve our goals. By creating a prototype that makes use of this method, we tested it to students with dyslexia in order to see it’s gain. The results of this study were promising, suggesting that there is an increase in performance and a lot of work to be done in this field.
1.
INTRODUCTION
Examination is the process in which experts can measure accurately and reliably the learning outcomes attained by a person with regards to their knowledge or skills acquired in the past. Nowadays, examination plays a big role in higher educational environments however, in some special cases it is not considered fair. Usually, these are cases in which learning disabilities might occur.
Dyslexia is the most common disability declared at higher educational institutions [14], and it only grows with each passing year [11]. While that is the case, there are several misconceptions with regards to the characteristics of dyslexia that can be mislead people that are not experts in
the field. There is a number of people, working in the academic field, who suggest that dyslexia is non-existent [7] or that dyslexia affects people only during their childhood years [6].
This usually occurs because the characteristics of dyslexia do not affect people externally, thus, making them hard to be observed by people that are not actively concerned about that issue. In other words, when people have autism or visual impairment, they can be easily distinguished by people around them, however dyslexia has undetectable to the eye characteristics that makes it an invisible disability. According to Mullins et al., having a disability with invisible external symptoms, such as dyslexia, can create an environment in which dyslexic people are treated “normally”, when they shouldn’t, thus the validity of dyslexia as a disability can be biased and the full extent of dyslexia symptoms can become unclear [15].
Many universities in different countries register students with all kinds of different learning disabilities under the same general category, ‘learning disabilities’ [13], without any subcategorization. This includes dyslexia, dyscalculia, ADHD and other difficulties under the same umbrella. While dyslexia is indeed a type of learning disability, it has its own unique symptoms that affect people in a much different way compared to symptoms affecting people with other learning disabilities, meaning that the support provided in an educational environment should also reflect on the specific type of disability that affects each individual student.
This research aims to identify the weaknesses of students with dyslexia compared to regular students, their strengths, and provide an alternative supporting method with which dyslexic students could accomplish a much better exam performance, through the help of digital examination environment.
In this research we will answer the question “What is required by a digital examination system in order to provide the right kind of support to students with dyslexia and help them perform according to their fullest potential”. In order to accomplish that, we will need to answer 2 questions with regards to current trends, “what types of
support are currently used to support students with dyslexia during exams” and “how capable are these methods in supporting students with dyslexia perform”. In addition, we will identify mechanisms that could potentially support students with dyslexia, by answering questions such as
“what are the advantages and disadvantages of people with dyslexia”, “what types of support mechanisms can help students with dyslexia perform better”, and “how these
support mechanisms could be added into a digital examination system while maintaining an equal fairness between dyslexic and regular students”.
To support the answers to the questions above, this paper presents a literature background on what are the characteristics of students with dyslexia; where do students with dyslexia find difficulties when taking an exam; what kind of supporting methods are currently being used to support dyslexic students; and what kind of technologies are being used officially by the educational system to support students with dyslexia. In addition, it discusses the findings of an interview conducted to students with dyslexia and an expert on the field, that aimed to identify the issues and strong points of people that have dyslexia. Following the findings of the literature and the interviews, we identified and tested the effects of a supporting method that uses text-to-speech technology during an exam, in order to support students with dyslexia. The paper presents and discusses the results of the experiment, while it also suggests further research steps on the subject that could further benefit the field.
2.
BACKGROUND
2.1
Defining Dyslexia
In order to start answering all our questions and accurately prototype and test a system that supports students with dyslexia to perform during exams, we need to understand what dyslexia really is.
The issue however is that there is no universal definition for dyslexia. Currently, the American Psychiatric Association uses the term‘specific learning disorder’ as a generic expression towards reading, writing or mathematical difficulties [2]; however, this is too broad to help us identify dyslexia as a specific kind of deficiency. This generalization and lack of a specific definition troubles researchers since when dyslexia was first diagnosed. Back in 1995, Lyon Reid stated that the absence of a fitting definition for dyslexia had a negative effect on research, as it resulted on an unclear selection of criteria for the sample being examined [3], developing in a vague understanding. More recently (2016), Gavin Reid accepts that dyslexia still does not have a clear definition, due to the various dimensions that characterize it, and argues that the diversity of factors that contribute to dyslexia can influence our understanding [1]. He also raises the point that the definitions of dyslexia usually reflect on the
particular purpose they serve. For example, the definition of dyslexia when it comes to support in education (provide special provision) is different than a definition suitable for when it comes to plainly understanding dyslexia as a concept (help people co-op with dyslexia) [1], and so on. In other words, different definitions can influence our perception towards dyslexia and thus cause confusion, ultimately leading to limited guidance towards assessment or intervention and poor development on that area. In this paper, we try not to define dyslexia as a concept since it will give us a rather narrow view of what dyslexia actually is. However, we try to define dyslexia as a set of characteristics that several people might share.
For this paper, dyslexia is a difficulty in the information process of an individual, which can affect reading, writing and spelling, while also can have an impact on memory, speed processing, or automaticity [1, 6, 7, 8, 9, 13]. This definition clearly identifies the most common characteristics of dyslexia. It is important to note here that people with dyslexia neither manifest only the symptoms described above, nor the symptoms are of the same scale of importance for everyone. The scale and the number of the symptoms of dyslexia can be identified by experts of the field when the individuals are being examined. However, these symptoms are commonly shared among a large number of people that are identified as dyslexic.
While individuals with dyslexia can manifest some disadvantages, such as the ones aforementioned, they also have certain advantages. According to Reid Gavin, individuals with dyslexia can comprehend their environment better when interacting with it through various senses and that would allow them to experience success more often [19]. In other words, multisensory interaction, such as audio, visual or kinesthetic stimulation, is an essential aspect for students with dyslexia.
2.2
Examination and Dyslexia
Exams can have different forms, such as multiple choice, mathematical problems, short answer, long essay, and so on [4]. In this paper we will mainly focus on exams that require a written answer and how dyslexic students perceive them.
To make it simpler, we will divide the examination procedure into 2 different parts that describe the whole process. These parts are: (1) reading and understanding the exam questions, (2) writing an answer to each question.
2.2.1 Reading and Understanding
During this step, the student needs to read a question and understand its requirements in order to be able to give an answer or solution. However, there are significant differences in reading and understanding, between students with dyslexia and students without [5, 8].
The study of Henriette Folkmann Pedersen and the experiment of Rosa Kwok et al. showed that students with dyslexia had difficulties with decoding information when 3
reading a text and comprehending textual information provided to them.
This results to a slower reading process and the making of more reading errors or spelling errors compared to regular students [5, 8]. In some cases, students with dyslexia even produced reading errors in which the meaning of a sentence was semantically altered [5]. In addition, students with dyslexia have a difficulty making inferences or rephrase information from a written text [5].
The results of Henriette Folkmann Pedersen’s study suggested that students with dyslexia in higher education focus their attention on either decoding the text (reading and spelling correctly) or understanding its meaning, since engaging in both might be mentally too demanding [5].
2.2.2 Answering
In the “answering” part of the test, the student is called to give a written answer to a question. For students with dyslexia, writing can be challenging too. At this point, the research that performed, with regards to writing skills of students with dyslexia at a university level is limited, thus, not much is known about the writing-based consequences of dyslexia in higher education. However, we can predict that people with dyslexia manifest issues with writing. As aforementioned, dyslexia has a negative effect on spelling, and spelling is a major component of the writing process [1, 7, 8, 9, 11].
Vincent Connelly et al. compared dyslexic and regular student performance based on spelling errors and handwriting fluency (words per minute). The results suggested that dyslexic students performed slightly poorer with regards to spelling errors, while they were significantly disadvantaged on handwriting fluency [11]. Yet, according to Noel Gregg et al., the disadvantage of handwriting fluency has a major negative effect to students with dyslexia, only when placed on timed writing tests [12].
2.3
Dyslexia and Examination Support
During examination, universities can provide extra time to students with dyslexia [17, 21], as a supporting mechanism. In rare occasions they might be able to provide sheltered conditions [18] such as isolated rooms.
While extra time is the default support for dyslexic students, sometimes, it can be perceived by the students as unsatisfactory [17]. Noel Greg et al. [10] conducted a meta-analysis in which they compared the test scores of students with dyslexia that received extended time and regular students that received no accommodation. The results of the study raised more questions with regards to this type of accommodation, as it was revealed that regular students outperformed students with dyslexia even though they had less time [10].
Moreover, in order to receive the extra time, dyslexic students need to bargain with the personnel of the
university that is accounted responsible [18]. This might occur due to concerns, by the academia, that adjusting the examination process for dyslexic students might disadvantage others [17, 18]. However, denying support to a student that was identified as dyslexic by an expert is similar to denying support to a blind person. For example, lecturers might question why adjustments should be made only for dyslexic students and not for students with socially disadvantaged backgrounds or foreign students that study abroad [18]. Nevertheless, foreign students can be regular students that are not very good at performing in a different language, and thus providing them with extra time can possibly benefit them. On the other hand, dyslexic students have a disability that does not allow them to decode textual information as successfully as other students, disregarding the given language in which the information is on.
2.4
Dyslexia and Assistive Technology
Since smart phones and tablets emerged there was a tremendous development of applications that aim to support reading and writing. Yet, the research is still insufficient with regards to their effects on students with dyslexia [20, 16]. A smart phone application was used in an experiment conducted by Linda Fälth et al., to study its effects on students with dyslexia in “learning”. The application provided various assistive functions that proved to be successful in increasing student grades in the long run [20]. However, applications like this are used by people with dyslexia to support their learning process, in home settings or in everyday life. None of these are used officially in educational environments as either examination support or learning support.
3.
METHODOLOGY
While the literature is able to give us a theoretical understanding of what dyslexia is and what kinds of issues students with dyslexia usually have, we need more practical evidence.
This research follows an inductive approach of reasoning. Our goal was to collect information from interviews that would allow us to identify a possible way to help students with dyslexia perform better during an exam and eventually test it and form a hypothesis.
Our first step was to interview students with dyslexia and an expert on the field of dyslexia, in order to reinforce our understanding and collect valuable information with regards to the experiences of dyslexic students. The aim of the interviews was to allow us have a more holistic view with regards to information provided by the literature and a more confident standing on our final hypothesis.
By understanding the literature and analysing the interviews, we concluded that the sense of sound, as a channel of communication, can be helpful to students with dyslexia during an exam. Finally, we embedded this idea into a digital prototype that was later tested upon students 4
with dyslexia in order to determine its effects.
3.1
Interviews
Following the literature it was indicated that students with dyslexia can comprehend information better through multisensory interaction. This gave us an indication that audio and visual stimulation could enhance their decoding skills and possibly their performance. Moreover, the information derived from the interviews clearly suggested that “audio” plays an important role in the information decoding process of students with dyslexia. Sound has the potential to benefit dyslexic students when they decode textual information that is either provided to them or generated by them.
3.1.1 Interview Settings
We conducted a number of semi-structured interviews with dyslexic students and an expert neuropsychologist. Interviews were semi-structured in order to allow for an open discussion to develop therefore, increasing the possibility of providing us with valuable non-anticipated information.
In total, four dyslexic students from the University of Amsterdam were interviewed in order to understand how they experience the effects of dyslexia. All students were diagnosed with dyslexia by an expert. Only one submitted the documents of her diagnosis to the UvA, in order to be eligible to receive “extra time”.
In addition, Dr Chrysostomos Karpathiou was also interviewed due to his background in Neuropsychology and Neurolinguistics and his focus in the area of dyslexia. The interview was conducted at his office, in which he helps students with different types of dyslexia understand what they have and develop coping mechanisms when studying. These interviews helped us enrich our knowledge with regards to what dyslexia is and how it affects students. In addition, they provided us with valuable information with regards to difficulties students with dyslexia face at an educational environment and how we might be able to overcome them.
3.1.2 Dyslexic Student Interviews
After conducting 4 interviews with dyslexic students, we collected enough information with regards to the effects of dyslexia in each individual, their point of view with regards to extra time, and possible coping mechanisms that they can use in order to overcome their difficulties.
It is important to note that ¾ students had similar characteristics while the fourth student provided us with a few extra different characteristics that we later on identified, with the help of our expert, as part of a different type of dyslexia. This 4th student didn’t allow us to reach data saturation with regards to the characteristics of dyslexia in general. We are confident that we managed to reach data saturation in one of the main different types of
dyslexia identified by the expert. Due to the time limitations of this project, we were unable to find more dyslexic students willing to be interviewed before we had to move on the next phase of our research.
The effects of Dyslexia According to Students
The most common effect of dyslexia we will discuss is related to information decoding and text understanding. In the following analysis, we will have to do some minor grouping of students that will allow us to see the scale of dyslexia in some of them. For that, we will name the four participants as D, N, M and G respectively.
All four participants indicated that they have a general difficulty in understanding a text when they read it. They stated that due to this difficulty, they tend to read much slower (compared themselves to their colleagues) and in many cases, they revisit a sentence 2 or 3 times in a row, until they clearly understand its meaning.
Students M and G had more extensive difficulties when reading a text. M stated that it is common for her to read the words wrong in a sentence. G expressed that when reading a text he tends to skip punctuation, making the decoding of a sentence sometimes impossible. We can argue that both cases present a more distinctive example of a text’s meaning alteration that can lead to a harder information decoding experience. This comes to parallelization with Henriette Folkmann Pedersen study, discussed in 2.2.1.
In addition, all four students identified that they had a difficulty with spelling. Students stated that they had difficulties remembering words that have uncommon spelling such as, double or silent letters. Students D and G also stated that they would even confuse letters with each other, based on the letter’s sound. The most common example for that is the confusion between the letters “b” and “d” that are similar in both form and sound.
Students M and G had extensive difficulties with spelling as well. Both students indicated that they often write words as they hear them since they can’t either recall the appropriate spelling or stay and spend much time in one word while in some cases, such as an exam, the time tends to be limited. In more detail, M suggested that this happens because she is completely unable to recall the proper spelling. In the case of G, he indicated that he might be able to recall the proper spelling but he has to spend at least 2-3 seconds on the spelling of that word. While he has the potential to correct it, he often loses his train of thought thus, preferring to write it wrong.
Moreover, 3/4 students (D, N and G) stated that it is hard for them to memorize information that they read from a text. They expressed a difficulty in memorizing information presented to them in a form of a larger text, such as an essay, while G also expressed a difficulty in memorizing the order of directions, when more than one are provided to 5
him. Consequently, it is necessary for them that they separate the information into smaller parts in order to be able to follow.
Furthermore, a less common characteristic that was noted, was related to the perception that students D and M had with regards to their own texts, during revision. The students stated that they are unable to find their spelling errors during revision; yet, it is common for their professors to see missing words in their texts. This can sometimes lead to a difficulty in understanding a specific sentence, by the professor, or the creation of a sentence with a completely different meaning. Additionally, D indicated that he is unable to see his own mistakes when reading in his head what he wrote, yet he has no trouble finding them when reading out loud.
Finally, participant G stated that he has difficulties in decoding the meaning of mathematical forms that he has to read. No other participant indicated such a difficulty, however we will see in section 3.1.3 that it is a characteristic of reversed dyslexia. This information did not allow us to come to a data saturation with regards to the characteristics of students with dyslexia; thus, we only focus to text related issues, and not mathematical related issues, in this paper.
Extra Time as Accommodation
While most of the difficulties mentioned by the students could be considered time related, when asked about extra time, not all students replied positively.
Since students need to read a text slow and multiple times in order to understand it, we would expect that by providing them with extra time would have helped. Similarly to spelling, if students need more time to think the correct spelling of a word, giving them extra time should suffice.
However, out of all four students, only student N supported the idea of extra time being helpful. She stated that extra time would allow her to read a text slow and multiple times in order to fully understand it. On the other hand, all other 3 students were not positive about the effects of extra time. All three students stated that they didn’t even submit their dyslexia papers to the university since extra time won’t allow them to further understand what they read. The students explained that they can get really tired by spending too much time and brainpower in understanding a text, finding the correct spelling of a word or follow an exam questions, leading to similar bad results. In other words, if a dyslexic student had to spend the regular amount of time on a text and poorly understand it, the end results would have been similar to when the student would have to spend more time and brainpower to understand the text and ultimately get tired in the middle of the process.
Having that in mind, in our understanding, students with dyslexia don’t necessarily need a supporting method that
will allow them to work much harder than regular students in order to achieve similar results. What they need is a supporting method that will allow them to achieve the same results as any other student would, by using the same mental capacity. This does not mean that the exam should be easier, however, it should offer enough support to allow students with dyslexia understand it more efficiently. Coping Mechanisms
In order to find possible ways to help students with dyslexia, we asked them how they cope with their difficulties in an everyday life situation. Students stated that they had developed coping mechanisms that would allow them to decode more efficiently the information received. In the case of better understanding, memorizing and revising a text, all students made use of some kind of audio. All four students indicated that reading out loud would significantly help them understand what they read. Student D also stated that he makes use of audio versions of books when available. Nevertheless, during an exam this is not possible, bringing them in front of a difficult situation.
On the other hand, the spelling was a more complex issue and students would try to avoid it. Most of the students would use the computer’s or the mobile’s spell checking function to make sure a word is correct rather than spend time on it. In rare cases where two words have similar spelling but their meaning and pronunciation is different, they would prefer using sound, if available. For example the words “though”, “through”, “thorough” and
“thought”. All these words have a very similar spelling and when written can be easily confused by people with spelling difficulties; thus, hearing them can make clear which word they meant to use.
3.1.3 Expert Interview
In order to cross check the information received by the dyslexic students, it was necessary that we ask the opinion of an expert in dyslexia. Dr Chrysostomos Karpathiou was able to help us by answering questions with regards to the common characteristics of dyslexia and possible ways that could help a dyslexic person overcome these difficulties. We started the interview by asking Dr Karpathiou what exactly is dyslexia. He stated that dyslexia occurs when some parts of the brain are not as developed as they should be, and thus disturbing some brain functions. Since the beginning of our conversation he made clear that our understanding of dyslexia changes constantly since the time it was first diagnosed and that we have an enormous increase in dyslexic people throughout the years.
Different Types of Dyslexia
He argued that dyslexia can be diagnosed in people of all ages and that it can affect both a person’s brain functionality and psyche. In order to make this clear, he shared with us the 3 most common types of dyslexia. The 6
typical, the reverse and Type-A dyslexia.
Typical Dyslexia: In this form of dyslexia, people have difficulties decoding information that they only read by themselves. In addition, students confuse letters with each other and generate incorrectly spelled words that sometimes can be really strange.
Reversed Dyslexia: In reverse dyslexia people have issues with decoding mathematical formulas. Essentially, they can’t see the relations between the symbols that are being used in the mathematical formula. While during the interviews we came across student G, who had a difficulty with mathematical formulas, this paper does not focus on this type of dyslexia, hence we won’t mention it again later. Type-A Dyslexia: The characteristics of people with this type of dyslexia are more psyche related, compared to the previous ones. More specifically, people with Type-A dyslexia are getting tired easier and they have a harder time concentrating, when studying.
In addition, they tend to be more insecure, have low self-esteem and be more sensitive. They also tend to study much harder. Unfortunately though, this is not enough, since the person studies more than a regular student but still gets a lower grade. In an example of an extreme case, assuming that the maximum grade for an exam could be 10, the student would study for a grade of 15 and in the end he would receive a grade of 5. People with strong Type-A dyslexia tend to quit study related activities after school. Moreover, a distinctive characteristic of dyslexia type-A is that it usually coexists with one of the previously mentioned types. In other words, people with typical or reverse dyslexia can also have traits of dyslexia type-A. Supporting Dyslexic Students
At the beginning of the interview, Dr Karpathiou stated that due to all the difficulties that students with dyslexia have, providing them with extra time would allow them to have a higher performance. Yet, this does not mean that they would perform based on their fullest capabilities because they still have issues in information decoding. The only difference is that they can spend more time and brainpower in decoding the given information.
While he was listing the different types of dyslexia for us, he mentioned that, for students with regular dyslexia, hearing the information would help them decode as effectively as a regular student would decode a written information. Following this information, we asked him if by listening the information would allow a dyslexic student to perform differently. He stated that hearing it while simultaneously being able to see it, would help significantly. Although, the dyslexic student should not hear it from another person because this might generate anxiety and activate all the issues that dyslexia type-A describes.
At this point, we asked about the use of technology and if a recorded voice can be more suitable. He stated that this would be indeed more suitable for students with Type-A. In his personal opinion, being able to hear a text or a question multiple times would be really useful in performance. However, he made a note about using computers, that was not anticipated. Typing in a computer is much easier than writing on paper, for students with dyslexia, because decoding icons is easier than decoding symbols. For example, generating the letter “a” by typing a button on a keyboard is considered an icon, while writing the same letter is considered a symbol. This ultimately means that the brain is not trained properly by using a computer, and thus Dr Karpathiou is against any use of computers until a certain age. Although this note was interesting, in this paper we don’t analyse the continuous use of a computer system thus, the long term effects of weak brain training is not a concern for this research.
3.2
Prototype Description
Based on the analysis of the information collected from the literature and the interviews, we concluded that sound is the most valuable tool for students with dyslexia when it comes to information decoding. In addition, we also understood that students have a difficulty locating their spelling mistakes during revision. Based on that, we developed a digital prototype of an examination environment that would allow us to test, on students, the effects of text-to-speech as a supporting method and the visual highlight of spelling errors (see figure 1).
Figure 1: A representation of the enhanced version of the prototype.
3.2.1 Interface Description and Functionality
As we will see later on in section 3.3, for the experiment we used both regular and dyslexic students, therefore it is important to note here that we created and used 2 versions of our prototype. One simple version without any support and an enhanced version with text-to-speech and spelling errors underlining, for regular and dyslexic participants respectively.
Figure 2: A representation of all the different areas of the prototype.
The prototype interface was divided into three main areas (see figure 2). The left side is where the main content is provided and it is common for both the simple and the enhanced version. On this area, the student can find the question that he/she needs to answer, the requirement guidelines (e.g. number of words and time available to complete the exam) and two readings that contain all the information needed in order to answer the question, are provided (see figure 3).
Figure 3: A close-up of the left side. All content necessary to answer one question, are located here.
The right side is where students can submit their answer. On this side, there is a text area where the student can write his/her answer, a word counter that the student can refer to in order to see if the word limit is exceeded or not, and a submit button (see figure 4).
In addition to that, the enhanced version also has a spell checking mechanism that is used to highlight spelling errors. Its goal is not to provide the correct answer to the students. It is used as a visual stimuli that could notify the dyslexic student that he/she had a spelling error, in case he/she wanted to give it more attention.
Figure 4: A close-up of the right side. The text area (blurred for privacy reasons), the word counter (bottom-left) and the
submit button (bottom-right) are located here. On the top side two smaller areas can be located. The upper area is where generic information is being presented to the student. This information is based on the exam as a whole and not for each individual question. For example, a timer, the number of the current question, or the number of all answered questions out of the total, can be placed here (see figure 5).
Figure 5: A close-up of the top side. Generic information and functional tools are placed here.
While the upper area is visible to all students, the lower area is only available to dyslexic students. In this area two buttons, (“audio” and “pause”) that can be used to access and manipulate a text-to-speech function, are located (see figure 5).
In order to use this function, the student needs to select the part of the text he/she wishes to listen to. This can be either from the right side (provided text) or the left side (generated answer). After selecting the text, the student can click the “audio” button. Then the selected text is being generated to speech. The student can listen to the artificially generated sound through a headphone set. If the student wishes to stop the speech at any point before finishing naturally, the “pause” button must be clicked. This process can take place as many times as the student requires it to.
3.2.2 Development
The prototype was developed with the use of HTML language. It was created in a way that would allow us to test our supporting method to the student while store the results of each student online, so to process them later. The left side of the prototype is only text in a div tag (a 8
section of HTML). The right side is an HTML form with 4 components; a text body and a submit button for everyone, and a word counter and a spellchecking function for the dyslexic students. The spellchecking function could be enabled/disabled through an on/off option located in the
body text area of the form. The word counter was created with the use of javascript in order to count the words in real-time. The code used for theword counter function can be located online at the countable Javascript library . 1
The text to speech function was also developed with the use
of javascript. A Web Speech API provided by Google
Chrome, named Speech Synthesis , was used. This would2
limit us in testing the prototype successfully only through Chrome browser; however, the experiments that we will discuss in section 3.3 were conducted in a controlled environment. Thus, we were able to overcome this limitation by using only Google Chrome.
With the help of a programmer named Pasquale Muscettola, we were able to set it up and place it in an online server that he provided. Since it was connected online, every time participants would submit their answers, all information needed, such as their answer, time spend on test and word count, would be stored there for later retrieval.
3.3
Experiment
The prototype was tested on 16 students from the University of Amsterdam. Since our goal was to compare the performance between dyslexic and regular students, the test had an A/B form. Out of these 16 participants, 8 were regular students, acting as the control group, and 8 were dyslexic students, acting as the experimental group. Initially, in order to find participants, we contacted the people of the UvA responsible for providing support to dyslexic students. However, they stated that they had technical difficulties extracting data of dyslexic students out of their database. In order to overcome this issue and collect the participants required, we used convenience sampling. All participants were located by the author of this article and associates of his.
The experiment was divided into 3 different parts. (1) One short interview at the beginning that was addressed only to dyslexic students, (2) the prototype testing, and (3) another short interview with regards to the experience of each student during testing.
The questions for the first short interview were of the same nature as the questions of the previous semi-structured interview we discussed in section 3.1.2. This time, questions with regards to the dyslexic student’s coping mechanisms and their opinion in what would be helpful, were excluded.
The aim of the first part was to ensure that the dyslexic
1Countable Javascript Library
2Introduction to the Speech Synthesis API
participants had difficulties similar to the ones discovered during interviews. This way we would ensure that all dyslexic students would be suitable for the prototype testing and the results would be reliable. Out of the 8 dyslexic participants, 3 had already participated in the interviews of section 3.1.2. Consequently, the first part of the experiment was conducted only to the other 5 dyslexic participants.
The prototype testing was based on a reading-writing exam situation. Using the prototype, students had to answer 2 separate questions that corresponded to one reading and one transcribed lecture each. Both the reading and the transcribed lecture focused on the same topic; yet, the information was contradicted between each them. All students had 20 minutes to answer each question. Each question required from them to summarize the points made in the transcribed lecture and explaining how each point casts doubt on the points made in the reading.
Regular students used the simple version of the prototype to answer both given questions. On the other hand, dyslexic students used the simple version to answer one question and the enhanced version to answer the other. Out of 8 dyslexic students, 4 received the enhanced version during the first question and 4 during the second question. This separation would allow us to preclude the possibility that one reading would be easier than the other. In a case like that, students would perform better in one question rather than another due to the difficulty of each reading and not due to the effects of the provided support.
In order to ensure the reliability of the exam, the reading materials were selected from real-life examples of the TOEFL placement testing. TOEFL testing is a standardized test, accepted by the majority of English-speaking institutions. It measures the abilities of non-native English speaking students that aim to enroll in an English-speaking university [22]. This way, we would ensure that the readings we used were approved by a board of experts, as appropriate exam material in a university level.
The second short interview of the experiment was addressed to all participants, since it aimed to extract information related to their experience during the test. Questions with regards to test difficulty, note keeping, performance and prototype functionality/interface, were asked.
These questions were necessary in order to increase our understanding in how participants perceived our prototype, and help us identify and evaluate its possible advantages and disadvantages.
4.
ANALYSIS AND RESULTS
The experiment showed that, when given text-to-speech support, there was an increase in understanding and performing of every dyslexic student, resulting to an increase in grades.
The data collected from the experiment could be translated 9
into two different data categories; quantity-based and
quality-based. Using them, we performed different analyses in 3 different levels; a high, a medium and a low level analysis that showed us similar results from a different perspective and an idea of a bigger analysis of the subject. For thehigh level analysis we used descriptive statistics in order to observe the differences in performance between regular and dyslexic students. In addition, we used it to observe differences between the two different sets of dyslexic students. The sample size was not enough to ensure statistical significance and be able to generalize our findings into a bigger group, however the results can indicate an image of a future analysis with bigger sample size.
The medium level analysis takes into account the overall performance of all students in each group. Using that, we made more targeted observations that focused on the performance of each individual student. That way we could observe the fluctuation in performance of every single student and understand the effects of our method in a deeper level.
The low level analysis had a more narrow focus since it emphasised on the performance of each dyslexic student separately. This analysis showed the specific gain of each individual dyslexic student separately.
4.1
Data and Analysis Strategy
4.1.1 Quantity-Based Data
The quantity-based category contains numerical data with regards to each participant’s performance. This category includes, the accurate points raised, the number of spelling and grammar errors made, the reading time needed and the overall grade received, by every participant for each of the two questions.
With the exception of “reading time needed”, all other numeric data derived with the help of a native english speaking, senior english literature student (ELS) at the University of Oxford, Clio Takas. The ELS received all the answers provided by each student, anonymously. This way we would eliminate potential biases, since the ELS is not aware which student is dyslexic and which is not. However, she knew which 2 answers corresponded to which participant, since every answer was marked by each participant’s identification code.
To grade the answers, the ELS followed the official Integrated Writing Rubric for TOEFL testing. The grade3
scale of the Rubric is from 0 to 5. In addition to the grades, she was requested to make notes with regards to the number of accurate points raised and the number of spelling and grammar errors made, in each given answer. The maximum number of points that could be made in each
3TOEFL Integrated Writing Rubric
reading was 4, and the number of spelling/grammar errors started from 0 as the best and did not have a maximum limit.
Overall, the results provided us with performances from 6 different student sets: (1) the performance of regular students during the first reading, (2) the performance of regular students during the second reading, (3) the performance of dyslexic students during the first reading without support, (4) the performance of dyslexic students during the first reading with support, (5) the performance of dyslexic students during the second reading without support, and (6) the performance of dyslexic students during the second reading with support (see table 1). # of Students Regular Students
Reading 1 Reading 2 8 Regular Non-Enhanced
results (set 1) Non-Enhanced results (set 2) Dyslectic Students
Reading 1 Reading 2 4 Dyslexic Non-Enhanced
results (set 3) Enhanced results (set 5)
4 Dyslexic Enhanced results
(set 4)
Non-Enhanced results (set 6) Table 1: A representation of all 6 performance groups.
4.1.2 Quality-Based Data
The quality-based data were collected from the second short interview that was conducted after the prototype. This data collection includes (1) the difficulty of each of the two readings, (2) the time limit pressure and (3) the best performing reading; all based on the student’s opinion. In addition, the notes of the ELS were used to determine the quality of each provided answer. The information provided by the ELS is based on the student’s overall text understanding, writing style, structure of a given answer, confidence in answering and more (see section 4.2.3)
4.2
Analysis and Results
4.2.1
High Level Analysis
Due to the number of participants, we are not able to do an accurate statistical analysis that will allow for generalization; however, we will present and discuss the performance of both, regular and dyslexic students, in numbers and see trends that would allow us to predict a possible outcome in a bigger sample size.
Comparing different combinations of performance measurements (see table 1), results to different insights. More specifically, comparing the number of accurate
points raised, thenumber of spelling/grammar errors made
and the overall grade between different sets from table 1, can show us an indication of (1) the difference in difficulty 10
between the two questions (2) the difference in performance between regular and dyslexic students, and (3) the difference in performance between the two groups of dyslexic students.
In order to make things more clear, we will start our analysis by focusing only on the overall grade (OG) as a comparison attribute, since it is correlated to the number of accurate points risen (NAPR) and the number of
spelling/grammar errors made (NSGEM). For a full table with all dimensions of performance, see Appendix A. We performed a spearman’s rho correlation test between the 2 dimension NAPR and NSGEM, and the OG dimension, in both readings.
While the sample size is small, the test showed that there is a significant correlation in both readings, between the
NAPRand theOG, while there is no significant correlation between the NSGEM and the OG (see figure 6 and 7). Although not significant, the correlation betweenNSGEM and OG is negative which indicates that the more spelling/grammar errors a student had the lower his grade becomes. It is important to note that NAPR has a higher weight on the OG than NSGEM has. It is something we would expect since, if a person has no spelling/grammar errors but failed to accurately raise any points, would have a much lower grade compared to a person who raised all points accurately but had a huge number of spelling/grammar errors.
Figure 6: Spearman’s Rho correlation between the dimensions of the first reading.
Figure 7: Spearman’s Rho correlation between the dimensions of the first reading.
By comparing sets 1 and 2 we observed that both questions/readings were of similar difficulty. As we can see in table 2, the mean of performance for all regular students, in all 3 dimensions fluctuates only slightly, between the
two parts.With a maximum of 5 on OG, we observe an increase of 0.38 points between the first and the second exam part (See table 2). Sets 1 and 2 are the two parts of the control group, meaning that a huge difference in performance would suggest a significant difference in the difficulty level between the two questions/readings. Having two text of similar difficulty suggests that if a comparison between a student’s performance in question 1 and 2 would be made, a significant difference in performance would suggest that another parameter was responsible. Since the only other strong parameter we have is the text-to-speech function, we are allowed to make assumptions with regards to its effects.
Table 2: The mean in performance of regular students, in all dimensions, for both readings 1 and 2. With a quick look in table 3, we observe that the fluctuation of grades in the two dyslexic groups was more notable. Especially for the group of dyslexic students that received support during the first question/reading (sets 4 and 6, table 1). We can clearly observe that upon losing their support, the mean in overall grade decreased by 2 full points (from 2.75 to 0.75). On the other hand, dyslexic students that received support during the second question/reading (sets 3 and 5, table 1) had no significant fluctuation in performance, since we only have an increase of 0.25 points (from 1.50 to 1.75).
Since regular students performed similarly in both exam parts, we would expect that dyslexic students would also perform similarly in the exam parts where the same parameters were provided (when having support and when not). In other words, we would expect that the mean in OG of set 4 (Mean=2.75, table 3) would be similar to the mean in OG of set 5 (Mean=1.75, table 3), and the the mean in
OG of set 3 (Mean=1.50, table 3) would match the mean in
OG of set 6 (Mean=0.75, table 3) accordingly.
Table 3: The mean in grades, of all groups of students, for both readings 1 and 2.
This result suggests that the order in which dyslexic students received support can have an impact. Our hypothesis is that by receiving support during the first part, the strategy, of dyslexic students, in answering the question included all available tools. During the second question, in which they had to follow the same process without provided support, they got confused. On the contrary, dyslexic students that received support during the second question, had already developed a strategy in answering the question by not using any additional supporting tools. This suggests that they didn’t fully take advantage of the support provided to them during the second question.
Having that in mind, when considering a scenario in which dyslexic students would have to take an exam using this method, we would expect that theiroverall gradewould be similar to the overall grade of dyslexic students who received support during the first part. Assuming that this is accurate, we can observe that the mean grade of set 4 [dylsexic students (Mean=2.75, table 3)] is really close to the grade of set 1 [regular students (Mean=3.25, table 3)]. This suggests that, at the same amount of time, dylsexic students have the potential to perform similarly to regular students, when provided with the support we suggest. Of course in order to support this assumption we would need bigger samples of regular and dyslexic students that would only answer 1 question. In this scenario, regular students would have no support in answering this question, while dyslexic students would.
Someone could argue that due to the difference in group size, the group of regular students (8 students) is not comparable to the two groups of dyslexic students (4 students each). In order to compare the regular with the dyslexic students, we have to break the 8 regular student group into two groups of 4. It is important to note that one regular student performed significantly worse than all other students thus, she would be considered an outlier. More specifically, the sum of accurate points risen, in both questions, for all other regular students was in the range of 5-8, while her’s was 2. This can also be observed in the number of spelling/grammar errors made and, inevitably, in the total grade. In both questions, she made a total of 20 spelling/grammar errors while all other students made 1-13 spelling/grammar errors. The sum of her overall grades in both questions was 2, while all other students had a sum of 5-10.
To allow for a normal distribution of data and to be able to divide the regular students into two groups comparable to the two dyslexic student groups, we removed the worst and the best performing students from both regular and dyslexic categories. This action decreased our sample to 6 regular students and 3 dyslexic students per dyslexic group, but it would allow us a better understanding of the data trends. As we can see in table 4, the fluctuation of the mean in OG
between each question, in both groups of regular students, is more distinct. However, it never exceeds 1 point. In such small groups, we ought to consider this a not significant difference, as the mean can fluctuate remarkably only by slightly changing the grade of one person. While that is the case, we can observe the same difference between the two sets of dyslexic students as before. Again, students that had support during the first exam part had a huge drop in grade during the second exam part, while students that had support during the second part, didn’t had a significant increase.
Table 4: The mean in overall grade, for 2 groups of regular students and 2 groups of dyslexic students, in
both readings 1 and 2.
Finally, It is important to note that all data presented above, should not be considered appropriate for generalization. During this analysis, we were able to make hypotheses, and see a glimpse of the possibilities of a future quantitative research with a significant sample size.
4.2.2
Medium Level Analysis
For themedium level analysis, we will continue focusing on the performance of the students based on their OG. However, this time we will analyse them in an individual level.
Table 5 could be really helpful for understanding this analysis, since it presents an overview of the performance of all students. For a clearer view of the table, see Appendix B. In table 5 we can see a comparison in student performance, between the two questions. Each row represents the results of each student in each reading. The vertical line in the middle of the table separates the results of each student, in each question. The horizontal double thick line separates regular and dyslexic students, and the horizontal single thick line separates dyslexic students that had support during the first and second question. The different colors represent the fluctuation in performance, in every dimension, between the two questions. Black, Green and Red represent stability, increase and decrease respectively.
By observing the data presented on table 5, we can see a balanced performance in regular students. This is something we expected and something that was also shown in the high level analysis. Since regular students act as a 12
control group and the two readings are of equal difficulty, student performance will remain stable with some students performing better during the first part and some during the second.
Table 5: A comparison of performance of all students, between the two readings, in all dimensions. However, we can observe that all dyslexic students, in both groups, performed better when provided with support. By taking a look in the performance of dyslexic students, in all dimensions, it becomes clearer that the group of dyslexic students who had support during the first part performed much better. This is also supported by the high level analysis where it showed a significant difference in grade. Nevertheless, all dyslexic students that received support during the second exam part, performed better in at least one dimension, while they stayed stable in other dimensions. This indicates that, even though the gain from the provided support was not that major, there was still an increase in the performance of every student. Participant D had a slightly bigger value in spelling/grammar errors however he managed to raise more accurate points. The high level analysis indicated that there was no significant increase in the performance of dyslexic students that received support during the second question. After conducting the medium level analysis, we can see that all dyslexic participants benefited from the support provided to them.
In addition, we can conclude to the same hypothesis as before. When providing students with two similar questions, their strategy in answering them is developed during the first.
4.2.3
Low Level Analysis
The low level analysis is based on the perception and experience of individual dyslexic students and a comparison of their documented performance, as perceived by the ELS. While the medium level analysis showed an increase in performance of every dyslexic student, the low
level analysis can support that the increase was not
accidental and had a deeper factor that is related to the provided support.
Student Experiences
To better understand the experience that dyslexic students had during the exam, we first need to understand how they perceived the test. When asked, 5/8 dyslexic students perceived the exam as challenging, compared to only 1/8 regular student that found the exam to be a challenge. In addition, 4/8 dyslexic students stated that the time limit was pressuring, compared to only one regular student who stated the same.
Dyslexic students were not always sure if the text-to-speech would be helpful. However, in the end of the experiment, all 8 dyslexic students stated at some point that the text-to-speech function helped at some point.
In more detail, three students stated that the text-to-speech was helpful during the revision of their text. They noticed missing words in sentences and wrongly used words (with similar spelling to the words they intended to use) that were not identified by the spellchecker. On the other hand, one student stated that the text-to-speech helped him understand the simplicity of the language he uses when writing; yet, he was confused with hearing what he wrote, thus he didn’t revise his text at all. Moreover, three students stated that the text-to-speech was really helpful with the text understanding. The students expressed that they managed to decode more information from the text when they could hear it and reading it simultaneously. Two of them stated that once the text-to-speech was not available during the second exam, they felt lost and they couldn’t perform. Furthermore, one student stated that the robotic voice of the text-to-speech made things difficult and that a natural voice would be more clear; nevertheless, she stated that if she still managed to understand the supported reading better. Finally, one student suggested that hearing the readings helped him keep better notes since he usually struggles when jumping from reading to note keeping and back. One observation that was made during the analysis of their experience was that students showed an increase in confidence. At the end of the experiment, all students were asked in which part they believed that they performed better. All dyslexic students stated that they performed better on the supported question. While not all of them gave a confident answer, when comparing these statements with their results, we can see that all of them managed to speculate correctly. It is important to note here that only half regular students managed to guess correctly.
Based on these statements, we can argue that students perceived a gain in their overall performance and their confidence level was increased with the help of the text-to-speech function. However, it doesn’t seem to have the same confidence boost effect in all students.
English Literature Student: Note Comparison
With the help of the ELS, we managed to see the gain in individual performances. Our goal was to compare her notes between the two answers of all dyslexic students 13
individually and identify changes in their answer development. After analysing her notes, we managed to see an increase in every dyslexic student’s performance. A useful observation that can support the increase of OG in
medium level analysis and the confidence of dyslexic
students.
By observing each dyslexic student case separately, we can detect an increase in text understanding and/or a betterment in the structure of their given answers. In total we will present the results of four students that had an increased understanding of the sources provided to them, one student that optimized his structure and three students that managed to do both, during the supported exam part.
Participant D: had support during the second exam part. During the first exam part she managed to raise a few points but failed to compare them with each other. Additionally, half of the information provided in her answer was made up, since it was nowhere in the reading. While she followed the same strategy during the second part, this time she accurately noted information relevant to the text and to the points described. However, she failed again to compare them with each other thus, resulting in a low overall grade.
Participant R3: had support during the second exam part. During the first exam part she had a decent structure but she didn’t raise any relevant points. The ELS suggests that R3 was aware that she had to compare points made in the two readings; yet, she failed because she seemed not to be able to understand the reading and eventually comparing irrelevant things together. During the second part, she managed to clearly identify all the information with regards to the points raised in the text. However this time she failed to make a comparison between the points. An observation during the experiment led us to believe that she failed to do so due to the fact that she skipped the question during the second part. She was preoccupied in using the text-to-speech that she failed to follow the directions provided to her. Nevertheless, she managed to decode the information provided by the text.
Participant N2: had support during the first exam part. According to the ELS she made a clear comparison in two points and she had a very good, but not excellent, awareness of all the text sources. During the second part her structure became weak, her answer was unclear and she only managed to mention one point.
Participant D2: had support during the first exam part. His text was difficult to understand in places due to some mistakes or non-sequiturs that lead nowhere. It was a badly written text but a lots of points were accurately risen and compared. During the second part he had a confused train of thought, he included irrelevant material that was not in the text and he only accurately compared one point. Participant R2: had help during the second exam part and
he was the overall the best performing dyslexic student. However, during the first part he was struggling with tenses and had a lot of spelling errors, making his answer unclear in parts. During the second part, he managed to have more confidence in his comparisons and optimize his structure. Participant L2: had support during the second exam part. during the first exam part, he had a lot of spelling errors and her answer was vague and sloppy due to limited understanding of the text. In addition he included information that was not presented in the readings. During the second part L2 managed to decrease the number of spelling errors and optimize the structure of his answer. Moreover, while he failed to compare more accurate points, he showed that he clearly understood the reading.
Participant M: had support during the first exam part. M managed to have a clearly written answer, while she identified and understood each point made in the reading. She accurately compared two of them. During the second part, she used awkward phrasing and had multiple errors. Additionally, she failed to fully understand at least one point made in the text.
Participant G: had support during the first exam part. He had a clear and fluent structure. He managed to raise two points successfully and a third one that did not come to fruition. He showed that he was aware of what the reading was about. During the second part, he had a weak structure and not a single accurate point was risen.
At this point, it is not a question that dyslexic students have a struggle in performing during an exam. However, with the text-to-speech function inclusion in exams, we can see a betterment of dyslexic student performance. All different types of analyses pointed out that the text-to-speech function can help dyslexic students decode more information, improve their structure and be more confident in their results.
5.
DISCUSSION
We presented how dyslexia works, the flaws and advantages of students with dyslexia, and a text-to-speech method that can support and benefit these students during an online examination. Nonetheless, it comes to our understanding that some of the decisions we made provide us with interesting findings, some with limitations, and a few more slightly interrupted our results at some level.
5.1.1
Analysis Strategy
By combining three different levels of analysis we aimed to overcome the weaknesses and problems that can be generated by each single level. Someone could argue that the low level analysis is the one that should be deemed important and the other two can be classified as supportive, since the lowest level shows the true gain in every student’s case. While it is true that the low level analysis provide us with enough information to guarantee us a statement, it is 14
