Design-based goal setting application development: a
Framework of User Needs and Requirements and
Goal-Setting Theory
AbstractMany students in higher education are lacking motivation to accomplish tasks that are required of them. Introducing technology to assist students in setting and committing to goals could help them accomplish their goals or tasks more effectively. A goal-setting webapplication is being developed, so (UvA) students can create, maintain, and monitor learning goals in the academic setting. The purpose of this study was to research user needs and requirement, measure user engagement and see how goal-setting theory can contribute to the development of the webapplication. Literature about goal-goal-setting theory was analysed to determine moderators (with respect to the functionality of the application), which could influence goal commitment. Design-based research was conducted to gain insight in these user needs and requirements. When determining what kind of visualization would help students to set and achieve goals and help them reflect on their goal-setting activity, there is no real clear consensus among participants. Some of the participants liked the visuals of the screens; others stated that the layout and visuals needs to be improved. The quantitative research part focused on the measurement of user engagement via a questionnaire (Qualtrics). Statistical analysis indicated that there is still some room for improvement within Perceived Usability (6 − 𝑖𝑡𝑒𝑚, α = 0.919, 𝑀 = 4,907, 𝑆𝐷 = 1,22), Control & Challenge (6 − 𝑖𝑡𝑒𝑚, α = 0.933, 𝑀 = 5,370, 𝑆𝐷 = 1,111), Aesthetics (5 − 𝑖𝑡𝑒𝑚, α = 0.933, 𝑀 = 4,511, 𝑆𝐷 = 1,327 ) and Endurability ( 6 − 𝑖𝑡𝑒𝑚, α = 0.793, 𝑀 = 4,519, 𝑆𝐷 = 0,719).
Victor Tilon - 10116362
Supervisor: Vladimer Kobayashi
Date of submission: July 15, 2015
University of Amsterdam Faculty Economics and Business
Statement of originality
This document is written by student Victor Tilon, who declares to take full responsibility for the contents of this document.
I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.
The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.
Table of contents
1. Introduction ……… 4
2. Literature Review ………8
2.1 Goal-Setting Theory…..……….8
2.2 User Engagement with Technology ……… 11
2.3 Measurement of User Engagement………13
3. Conceptual Framework………15 4. Methodology………17 4.1 Design-based research………...17 4.2 Proposed iterations………..………..18 4.2.1 First Iteration………..………20 4.2.1.1 Research Design………...………20
4.2.1.2 Evaluation results of User Needs and Requirements…....22
4.2.2 Second Iteration…...29
4.2.2.1 Research Design...29
4.2.2.2 Evaluation results of User Needs and Requirements…...30
4.2.2.3 Evaluation results of User Engagement………...36
5. Discussion………41
5.1 Linking results to existing theory………...……41
5.2 Limitations of methodology………...…………43
6. Conclusion………...………45
7. References………...46
8. Appendix….….….….….….….….….….….….….….…..….….….……….….52
A: Questionnaire first iteration….….….….….….…..….….….….….….…….52
B: Questionnaire second iteration (Qualtrics)……….………..60
C: Comparison Table Goal-Setting applications………...71
D: Interview transcript 1………74
E: Interview transcript 2……….………81
F: Interview transcript 3……….………87
1. Introduction
Nowadays it is not really surprising to see that many students are unmotivated to accomplish tasks in the academic setting. Students in higher education find themselves in a state of not carrying out the tasks that are required of them. Having a lack of motivation for those tasks, could lead to feelings of frustration and dissatisfaction with their own being, which in turn could reinforce the feeling that they are not capable of achieving in an academic setting (Legault, Green-Demers & Pelletier, 2006). Many research has been done on this topic, describing how cognitive factors, ineffective study habits, inadequate study experience, dysfunctional student-teacher interaction and medical condition could explain why students have difficulty in achieving in the academic setting (Hendricson & Kleffner, 2002). Some students are considered as high-achieving learners and others as underachieving learners. High-achieving learners are more persistent in efficient behaviour, have good impulse control, are active learners and have a social network of good students for peer support (Hendricson & Kleffner, 2002). They are more likely to have success (e.g. high grades) in the academic setting. Underachieving students have low impulse control, are mobile, have conflicts between social and academic priorities, have a poor student network and a distracting study environment (Hendricson & Kleffner, 2002). They are more likely in not achieving success in the academic setting. Others discuss how an explanation why students have difficulty in achieving in the academic setting should be sought in self-determination theory (Legault, Green-Demers & Pelletier, 2006). Self-determined motivation, which is mainly intrinsic, has been associated with higher academic performance and better information processing. However some discussed how academic performance could partly be explained by self-efficacy theory (Schunk, 1991). It proposes how self-self-efficacy could affect choice of activities, persistence and effort from an individual towards a goal. Some people who would have low-efficacy would give up easier on tasks, where people with high-low-efficacy would keep going to achieve their goals. They are more capable of pursuing goals when encountering difficulties. All students differ in how they perceive themselves: they could differ in the ability to acquire knowledge and skills and how they handle attitudes (Schunk, 1991). Together with other factors as goal-setting and situational factors, it could affect students in how they perform.
A lot of research has been exploring the motivating power of goals. It was stated that students would perform better if they know what goals they are seeking and if they know whether these goals are important to them (Forsyth & McMillan, 1991). These goals should be positively formulated, specific, realistic and personally important for the students (Forsyth & McMillan, 1991). Goal-setting theory explains why some people perform better than others
when given they have the same set of knowledge or ability, thus the difference should be explained by different motivations. This theory (Locke & Latham, 1991) is based on the assumption that human behaviour has a purpose and it is regulated by goals of the individual. There are many ways to improve motivation of students to accomplish goals and technology could have a contribution to it. Several applications have been suggested ranging from private communication between teacher and student, writing personal blogs to track their learning achievements and creating peer support systems (Zakrajsek, 2005). Many IT applications have been written to help people accomplish different kind of goals. Some of them are specifically been developed to help people lose weight and set calorie goals (MyFitnessPal – “MyFitnessPal, n.d., Fitbit – “Fitbit”, n.d.) and others are more ‘generic’, helping people achieving their life goals or accomplishing simple tasks (e.g. GoalsOnTrack – “GoalsOnTrack”, n.d., Rewire – “Rewire”, n.d., stickK – “stickK”, n.d., Achieve – “Achieve Productivity Timer”, n.d. ). While MyFitnessPal has a large food database, it relies too much on calories, which is not the only contributor to losing weight, and the database always has to keep being updated (Virginia Tech Dining Services, n.d.). The application ‘Achieve’ has a very basic design and is not suited for complex goal tracking (Micu, 2015). Both applications ‘stickK’ and Coach.me (“Coach.me”, n.d.) have an online community. This online community can assists people with achieving their goals. One distinct feature of ‘stickK’ is a penalty system..When an user doesn’t reach his or her goals, an indicated amount of money from the user’s bank account will be donated to a charity organization. Another application, Coach.me, has a very interactive community where people could comment on goals of other people, next to a possibility to ‘hire’ an online coach for a fee whom supports you when trying to reach your goals. One possible downside of this application is that all goals are somehow predetermined; you can only select goals from a database. A comparison table of some popular existing goal-setting application can be found in appendix C instead of a table underneath this section, because of the large size of the comparison table.
While many setting applications exist, there are only a limited amount of goal-setting application specifically developed for students in higher education. Two examples of goal-setting applications for education are Studygrasp and IMD3, however Studygrasp was considered a fail and it shut down (“Studygrasp”, n.d.). IMD3 is an goal-setting application funded by the Virginia Department of Education and has a focus on compulsory education and youth with disabilities (“New IMD3 App Available”, 2015). This thesis will focus on mapping user needs and requirements, measuring user engagement and the use of goal-setting theory and its applied use to the development of a software application (figure 1). This
software application is a goal-setting program for (UvA) students to create, maintain, and monitor learning goals in the academic setting. Therefore this practiced-oriented thesis should have more practical and societal relevance than contributing to an academic audience. This research and application should be relevant for students who would like to engage in more effective ways to study and achieving goals. If you want to look at developing an effective interface for a goal-setting application it is necessary to study the interaction between people and machines. Because this application is already in development, it is important to provide potential users with a role as co-designers of the application. This application will use an user-centered approach in the process of development. Users would use this application to set goals for themselves and thus improve their performance in the academic setting. Especially with a user-centered approach, it is important to test alongside potential users whether it meets the expectations of the user, as a developer can not always predict how well it will function in a real world setting. Therefore the knowledge of potential users will be used to optimize the application. An important concept to investigate is user engagement, this mainly refers to the quality of the user experience, especially with respect to the positive properties of the interaction with technology (Attfield, Kazai, Lalmas & Piwowarski, 2011). Although there are more methods found in literature to actually measure user engagement, there are only a few attempts whereby these methods could be combined to increase the generalizability to measure user engagement. Measuring user engagement is often context dependent.
In determining what to research for this thesis, it is necessary to understand the goals of the project and the contructs to look for. A goal-setting application will be developed, whereby students will use it. These students will be monitored in how they are making progress toward their goals. This application is already in development and applies a user-centered design: user information is used to improve application. The goal of this study is to solicit user needs and requirements, measuring current user engagement and investigate how existing theory about goal-setting could inform and improve the app’s design interface. Goal-setting theory is used to search for contructs of goal commitment and other theoretical constructs, which could benefit the actual goal-setting in the application. Using goal-setting theory should be necessary, since it would justify a theoretical base for specific design choices, which contribute to the effectiveness of the goal-setting application; e.g. goals formulated in the application should be specific and difficult (but not too difficult). Sometimes designers of educational products and processes draw too heavily on imagination and intuition instead of theory (Dede, 2005). This often leads to design choices that are not
grounded in theory and often contribute to ‘design creep’ or ‘feature creep’: an on-going expansion of corrections in the design. Every implementation difficulty is dealt by increasing the sweeping designs (Dede, 2005), instead of relying on bounded research beforehand. When developing the application, it is necessary to figure out what users want, how to let them successfully engage with the application and how to let them help reach their own goals. Therefore my main research question will be: How to combine user needs and goal-setting
theory in creating an effective interface for a goal setting application in education. Subquestions are:
-‐ How should we keep users engaged in goal-setting with regards to moderators which
influence goal commitment?
-‐ What kind of visualization could motivate students to set and achieve goals and help
them reflect on their goal-setting activity?
-‐ Which part(s) of user engagement could be improved with respect to usability &
functionality, overall experience of the user and visuals?
To conduct this research, literature about goal-setting, user engagement and design-based research will be analysed and reviewed. Next, design-based research will be conducted where two iterative sessions are used to gain insight in user needs and requirements and user engagement. These two iterations will consist of qualitative and quantitative research.
Figure 1: Explanation of the position of this thesis: the research should close the gap between theory, user requirements and possible design principles (developer)
2. Literature review
2.1 Goal-Setting Theory
Goal-setting theory explains why some people perform better than others given they have the same set of knowledge or ability, thus the difference should be explained by different motivations. This theory (Locke & Latham, 1991) is based on the assumption that human behaviour has a purpose and it is regulated by goals of the individual. Goals therefore motivate action, so goals should motivate students to start working on their tasks. Two elements of goals are content and intensity. Content refers to what the person is seeking and intensity refers to the scope or focus of the choice process. Multiple studies suggested that task performance is postively related to goal difficulty: the harder the goal, the higher the performance. This is suggested because of the effort you should put in to achieve the harder goal. Locke and Latham (1991) described that not only a goal has to be difficult, but also it has to be specific in order to maintain high level of performance. When an external person or application suggests these specific goals, this could represent extrinsic imposition on task engagement and this could lead to decreased task interest (Elliot & Harackiewicz, 1994). However goals have the power to increase intrinsic motivation, because it promotes task involvement and provide on-going competence feedback (Elliot & Harackiewicz, 1994). Although difficult and specific goals should lead to higher performance, it could have some drawback: a person could focus on reaching the goal instead of learning some important kind of skill. Setting specific, difficult learning goals (e.g. learning how to apply the t-test of my statistics course) would have a greater effect on performance of students than setting only performance goals (e.g. achieve a higher grade for next exam). It is believed that a learning goal enhances planning, monitoring and evaluating progress toward goal attainment (Locke & Latham, 2006).
This suggests that there are different orientations toward goals. Bell & Kozlowksi (2002) describe, according to their research, that you could have a learning or performance orientation towards tasks. A learning orientation is associated with positive outcomes on performance, while a performance orientation is associated with negative outcomes. Having an attitude towards learning orientation increases self-efficacy, performance and knowledge of the individual.
A goal has no use if the person is not trying to commit to this goal. A major aspect, which is widely researched in goal-setting theory, is goal commitment. Goal commitment is described as the person’s attachment or determination to reach a goal
(Locke, Latham & Erez, 1988). Therefore there should be some relationship between levels of goal commitment and performance. Earley (1985) reported that there was some relationship between performance, goal commitment and goal difficulty. He found out that goal difficulty had a significant impact on the relationship between goal commitment and performance. Locke and Latham (1991) in another paper defined goal commitment more clearly. It refers
“to the degree to which the individual is attached to the goal, considers it significant or important, is determined to reach it, and keeps it in the face of setbacks and obstacles” (p. 17). When goals became more difficult, it would raise performance up to a certain point, but it
would have negative effects on performance after that point. Hard goals are generally less accepted than easy goals, so the harder the goal became, the lower the goal acceptance. Locke, Latham & Erez (1988) suggested that when researching goal commitment, you should make a distinction between external, internal and interactive factors. External factors described how legitimate authority, trust, peer (group) influence and values, incentives and
rewards could contribute to goal commitment. Assigning a goal to an individual by a
legitimate authority implies that the individual is capable of successfully achieving a goal, which should increase the self-efficacy of the individual. People usually would like to follow what an authoritative person has to say and they would judge his goal-setting advice as more legitimate. Trust is an important factor to consider, because people would not always accept what an authoritative person has to say (Locke, Latham & Erez, 1988). Peers could influence goal commitment by serving as role models for the individual. Internal factors were more-or-less based on the expectancy of success, self-efficacy and self-administered rewards (Locke, Latham & Erez, 1988). Self-efficacy for example, has been an important concept in the social-cognitive theory by Bandura (1986). It refers to the self-confidence of an individual at performing well at a task. It could be measured by asking participants whether they could reach different kind of performance levels and by letting them rate their own confidence on a scale whether they could reach those different kinds of performance levels. Interactive factor was based on participation; whether people set his or her goals themselves or it is assigned by someone of something else. A lot of research has been done on whether assigned goals or self-set, participative goals are more effective. Latham concluded in his own papers that they are both equal in effectiveness, whereas Erez didn’t think assigned goals were as effective as self-set participative goals (Locke, Latham & Erez, 1988). Schunk (1990) described how he found out that preference for self-set or assigned goals when dealing with college students depends on whether they are high or low in achievement motivation. Students with high achievement motivation performed equally well with both kinds of goals conditions, while
students with low achievement motivation performed only well when they had self-set goals. When a supervisor (or another authoritative person) participates in shared goal setting with an inferior, then it leads to higher commitment than just telling people what to do without an explanation.
One important concept which is related to goal-setting is feedback. Feedback is a piece of information and therefore should not necessary have effects on performance. However, it could have a meaning when this information is appraised and understood what decisions
could be made with respect to the information.
Feedback normally consists of information about the outcomes of the performance of the individual. According to Locke & Latham (1991), feedback has no direct motivational effect on performance. However, Erez (1977) thinks that goal-setting would not be very effective without receiving feedback. Therefore it is concluded that feedback should moderate the effects of goal on performance. Schunk (1990) and Earley, Northcaft, Lee and Lituchy (1990) both support this view. In one of Schunk’s (1990) own experiments, children were assigned to commit to a product-goal, process-goal, or process-goal plus feedback on progress condition. He concluded that the group with process-goal plus feedback on progress condition had a higher performance. If goals were combined with feedback, it would raise performance and would increase the self-efficacy for goal commitment and effort towards the goal. Positive feedback should raise self-efficacy, but it should not be concluded that feedback always raises performance. Positive feedback could give the impression that someone’s performance is ‘alright’, and therefore an individual should not see any reason to improve (Locke & Latham, 1991). Earley, Northcraft, Lee and Lituchy (1990) stated that two forms of feedback could be formulated; outcome feedback, information about performance outcome, and process feedback, information about how an individual carry out a work strategy. This work strategy could be described as how an individual tries to accomplish goals. They concluded how both kind of feedback could differ in how they moderate the effects of goal-setting on performance.
When discussing the generalizability of goal setting research, I would like to refer to a more recent study of goal-setting theory by Locke and Latham (2006) and Locke (1996): goal setting is essentially moderated by feedback, commitment to the goal, task complexity and situational constrains. Effects of goal setting have been found in generous studies involving more than 80.000 participants in 88 different tasks across multiple countries (Locke and Latham, 2006). Even when differentiating for methodologies, group vs. individual studies or
other dependent variables, the robustness of goal setting effects is quite high: a success rate of 90%. This suggests that in many cases, goal setting is effective.
2.2 User Engagement With Technology
When defining whether people would use an application, how to motivate users keep using the application and how to improve it, it should be very useful to look for engagement in human-computer interaction. An interface, which is boring to use and does not hold human attention is quickly discharged as non-useful. One of the earliest and influential theory about interaction between humans and information systems had been described by Davis (1989). He suggested the Technology Acceptance Model (TAM) for which could be explained under what conditions a user intends to use an information system. It suggested that two factors are mainly important in accepting or rejecting technology. The first factor is Perceived Usefulness, which is the degree to which a user believes that a particular system will improve his or her job performance (Davis, 1989). The other factor is Perceived Ease of Use, which is the degree to which the user believes that the system is free from difficulties (Davis, 1989). User acceptance is an important predictor whether a system will succeed. In addition, Perceived Ease of Use has a causal effect on Perceived Usefulness, after all, if a system is easier to use, thus users are more likely to understand the value of it (Davis, 1989). In addition, the appearance of a system, mainly the design features of an interface affects the Perceived Usefulness and Perceived Ease of Use. Until today, much research has been done about the TAM model, making the TAM model more wider with theoretical constructs till it forms the TAM2 model (Venkatesh & Davis, 2000). In recent studies, these two factors, are but a small number of components of user engagement.
To investigate user engagement you need an interface which allows interaction with a user. Such interfaces might, in fact, must be able to adapt to needs and choices of the user. The term "User Engagement" is rather difficult to define, but it has many applications in scientific and commercial fields. Severals concepts are linked to user engagement. For example, 'User Engagement' can be seen as a process, a part of a process, the overall process or as an experience (Peters, Castellano & Freitas, 2009). If you look at user engagement, many researchers would like to know exactly ‘what’ people are engaged with. This is pretty hard to measure, because it has several dimensions. Suppose you would like to measure the involvement of an individual with a system by checking whether he directs his gaze on the system, this would not necessarily mean that he really gives attention to the system. The person could be thinking about other things. Peters, Castellano and Freitas (2009) also
describe some broader concepts of engagement. They suggest, although it is an oversimplification, that perception, cognition, experience and action play a role in it. When perception is assumed, if you want to measure 'attention’, it really depends on how you want to define ‘attention’. It is thus a matter of perception. Cognition is seen the mental state of the user. Experience is seen as the degree to which an individual loses himself completely in the experience of a system. Finally, action is a set of clear signals from which you can see that someone is really involved in the system. This could manifest itself in the form of someone who actually uses a system and from which you could see he is engaged through his body language and spoken words. Engaging systems are systems which entrain users in their activity, are interesting to use and are gripping and retaining the attention of the user. (Chapman, Selvarajah & Webster, 1999). This is quite similiar in how Quesenbery & Design (2003) described it: it stated how an engaging product draws the user in the experience and encourage the interaction between human and machine. Visual design, choice of language, the media used and the style of interaction are all playing a part in the experience which creates user engagement (Quesenbery & Design, 2003).
An important concept that is closely related to user engagement is flow theory of Csikszentmihalyi (2000). This theory is mainly a concept of psychology and it has received a lot of respect in research of interactions between humans and computers (Webster & Martocchio, 1995). When having flow, an individual will be so involved with his experience that he is completely engaged in the things he is doing. This theory is used to understand the motivation and the responses of individuals who use an application. This theory has a particular focus on intrinsic motivation, while interacting with an application, there might be something which keeps you working or ‘engaged’ without any intrinsic motivation. This is the case if there are rewards or points you can score if you use an application. An individual who is in a state of 'flow', will primarily have a specific focus, is intrinsically curious and interested in the application (Csikszentmihalyi, 2000). Another concept related to user engagement is aesthetics. It means the degree to which a particular visual design meets certain design principles (Beardsley, 1982). This also determines a part of user engagement, since aesthetics, according to research from Overbeeke, Djajadiningrat, Hummels, Wensveen & Frens (2005), is linked to user convenience and needs. More recent research was done on user engagement. O’Brien & Toms (2008) described how prior research indicated that engagement consists of several components: intrinsic interest, curiousity, motivation, goals and mental
models and user activities and attitudes. This was based on surveys in a variety of sectors,
difficult to assess the generaliziblity of this definition. Another study describes how engagement characteristics are defined by focused attention; focusing attention on the subject only, positive affect; users should be affectively involved and should experience emotions regarding the subject, aesthetics; it has to be visual appealing, endurability; the likelihood of remembering the experience, novelty; experiences should be engaged because it is novel or surprising, richness and control; more variety and possibilities is better, reputation and trust; the interface has to do what it should do, user context; user’s own motivation and benefits affect the experience (Attfield, Kazai, Lalmas, Piwowarski, 2011). They based this on their own literature study and some of their own suggestions. O’Brien & Toms (2010) proposed a model of user engagement based on prior research about user engagement and incorporated it with elements of Aethetic, Flow and Play theory. They tested this model in an exploratory study where participants in four different information systems were questioned; web searching, shopping, video games and online learning. It turned out that user engagement would consist of: Aesthetics, Affect, Focused Attention, Challenge, Control, Feedback,
Interest, Motivation, Novelty and Perceived Time. After Exploratory Factor Analysis, 6
factors would appear: Focused Attention, Perceived Usability, Aesthetics, Endurability,
Novelty and Involvement.
2.3 Measurement of User Engagement
Measuring user engagement is quite a subject of discussion. There are several methods and it is often very context-dependent. Researching user engagement in gaming is different than researching user engagement of webapplication. Since there must be mapped how users think about certain things, it is not logical to use general methods for measuring user engagement. Like measuring the time how long it takes participants to fulfil certain tasks or how often an internet user visits a particular site. Of all attempts to measure user engagement, questionnaires are the commonly used methods. Webster & Ho (1997) attempted to measure user engagement in multimedia presentations. They assumed that user engagement is very similar to the concept of 'playfullness’. From this perspective, they wanted to see how multimedia technologies could influence the engagement of listeners. This allowed them to look at what areas a multimedia presentation could be improved. After holding a pilot and two research studies they came to a 7-item construct of user engagement: challenge, feedback,
control, variety, attention focus, curiosity, intrinsic interest. The first four items in particular
were about influence on commitment and the latter defined user engagement. The fact that user engagement is still very context-dependent and can be measured differently also emerged
in a similar study of Chapman, Selvarajah & Webster (1999). They wanted to investigate whether there was a difference in user engagement between three kinds of types of multimedia systems. These multimedia systems are mainly used for training purposes. The multimedia systems could consist of a text-based format, an audio format or video format. They tried to measure user engagement based on three constructs: focus attention, curiosity,
and intrinsic interest. The questions they asked participants were typically measured on a
7-point Likert-scale.
One of the latest attempts to develop a good reliable and valid scale was done by O’Brien & Toms (2010). They built upon earlier research on user engagement and came to the conclusion that if you want a good view on user engagement, more constructs should be considered. After holding a large-scale study and statistical analysis (e.g Exploratory Factor Analysis), they discovered that their "User Engagement Scale" (UES) consisted of six constructs: Aesthetic Appeal, how does the application visually looks like, Endurability, this reflects the overall evaluation of the experience again. It also shows the extent to which users would recommend this to other people. Felt Involvement, being part of the experience and having fun, Focused Attention, concentration and it refers to the extent which a user turns his attention to application, Novelty, feelings of interest in the tasks required and curiosity generated by the application itself and Perceived Usability, the affective and cognitive assessments of the application (O'Brien & Toms, 2013).
3. Conceptual Framework
Literature about goal-setting and user engagement is quite broad. A lot of constructs and results from goal-setting experiments can be found which have an influence on goal commitment. An important implication beforehand, is that internal factors of goal commitment such as self-efficacy are not considered. The measurement of user engagement is quite context-dependent. The results that are found in this research are, therefore, probably only valid in this study. Not all operationalized constructs from literature will be measured, because of certain requirements of the project leader of the application and the features of the application at this time. The literature of goal-setting can be used to investigate some external variables that are also reflected in the application itself like assigned goals vs. self-set goals (Locke, Latham & Erez, 1988). Some of the questions of the questionnaire are based on those findings. It makes no sense to examine internal factors (e.g self-efficacy), because it is outside the scope of this research. User engagement can be used to look at a part of constucts within user engagement. The focus of this research is actually meant to investigate the user needs and requirement than actually contributing to the literature of goal-setting theory and user engagement. After looking to the features of the application, some construct can be found (figure 3)
Proposed factors of interest (Goal Setting)
Figure 3: A few of the external and interactive constructs which affects goal commitment. The definition of Legitimate Authority, Trust and Participation are derived from Locke, Latham
and Erez (1988), Feedback derived from Locke (1996)
Legitimate Authority
Trust
Participation Feedback
Legitimate Authority indicates that an authoritative person somehow already establishes the
goals for you. The application allows a teacher to check your goals or establish classgoals.
Trust indicates the degree to which you can trust someone when the goal is made by an
external person, but compulsory to must follow. In the application, for example, you can use goals of other students, although the question is whether students are more likely to commit to goals of other students or teachers. Participation indicates that there is an interaction in the creation of goals. You can see this by the fact that teachers can assess goals via a rating.
Feedback is a way to give positive or negative comments on your goal-setting progress. In the
application, feedback can be given by the teacher to students via email or a rating system to assess your goal efficiency whether it is SMART (Conzemius & O'Neill, 2005). At suggestion of the project leader of this application it would be sufficient to investigate the visuals of the application, how users experience the application, how likely they would think it’s sufficient for goal-setting and the usability (and features) of the application. Based on these suggestions, some factors of interest are proposed which are based on the UES 6-item scale by O’Brien and Toms (2013).
Proposed factors of interest (User Engagement)
Figure 4: A few of the constructs which are part of user engagement.
Perceived Usability describes the affective and cognitive responses to the system (O’Brien &
Toms, 2013). Aesthetics describes the perception of the visual appearance of a system interface (O’Brien & Toms, 2013) and Endurability describes the overall evaluation of the experience and its perceived success (Webster & Ahuja, 2006; O’Brien & Toms, 2010).
Perceived Usability
Aesthetics
Endurability
4. Methodology
Since it is not sure what to look for specifically and new concepts could come up or existing concepts could be rejected during the research, design-based research would be preferred. As an example, Schunk (1990) described how students with high achievement motivation performed equally well with both kinds of goals conditions, while students with low achievement motivation performed only well when they had self-set goals. However this performance could be influenced by the actual use of the application; students could indicate whether assigned goals or self-set goals would help them better in goal-setting. Design-based research consist of iterative cycles of design, engagement, analysis and redesign (Herrington, McKenney, Reeves, & Oliver, 2007). This design is conducted in a real-world setting whereby it is theory-driven and based on well-studied relevant research on goal-setting theory and user engagement. According to Reeves (2006), the best approach should be to start with analysis of the problem, followed by the development of design principles based on existing principles, iterative cycles of testing and refinement of solutions in practice and finally reflection to produce the (recommended) design principles that should be developed for the software application.
4.1 Design-Based Research
In doing research, design-based research will be prefered. Design-based research copes with the idea that the real world of education is messy in variables: subjects will act different in different situations. So an experimental approach, where variables are controlled, cannot always correspond to a realistic outcome of the real world. Design-based research could challenge this problem by offering a new approach: learning in context through the systematic design and study of strategies and tools (The Design-Based Research Collective, 2003). With design-based research it is possible to look how a particular design plays out in practice, and how social and contextual variables interact with cognitive variables (Dede, 2005). It provides flexible methodologies aimed to improve educational practices through iterative analysis, design, development and implementation (Wang & Hannafin, 2005; Herrington, McKenney, Reeves, & Oliver, 2007). This approach should be ideally for his user-centerd application, since it is possible to cope with unexpected results when determining user needs and requirements even though it does not correspond with researched literature.
One study, for example, which applied design-based research for the development and evaluation of a webapplication is a study of Santos, Govaerts, Verbert & Duval (2012). They developed a dashboard for engineering students where students can reflect on their progress
of a course compare results with other students. The development of this application was done by means of four iterations wherein the usability, use and usefulness were examined and improved. Another study, which did not conduct design-based research, but made an assessment of user needs is the study of Lindquist and Long (2011). Their goal was to
develop a digital educational application to make students more engaged with online primary sources by using an user-centered approach. Students were interviewed for needs and
requirements thus contributed to the development of the application.
When researching user needs and requirement and combining it with goal setting theory to create an effective interface, an interface design has to be developed which should be improved at several iterations. Attfield et al. (2011) suggested that the best way to start when investigating engagement and user interaction is to start with exploratory, qualitative research to discover engagement characteristics for the interaction.
4.2 Proposed Iterations
For this research, two iterations involving qualitative research and quantitative research will be used.
The qualitative research primarily consist of exploratory open questions which are mainly used to identify user needs and requirements. Participants will also the opportunity to speak freely and give explanations for answers. The qualitative part will mainly look at the current features that this application already has, possible features that are still missing, comparisons with similar applications, the degree to which users experience the visual aspect and how they experience the ease of use and endurability. Some questions in the qualitative research are also related to goal commitment. This is done to substantiate some questions with goal-setting theory.
The quantitative research will consist of an evaluation of the current state of the measured user engagement with respect to the proposed constructs. These two kind of research are primarily self-reports, that is, the answers and results which are generated are particularly subjective data. Self-reports are very useful to investigate behavior and motivation of users. A disadvantage is that users may not want to give answers in full detail because of a breach of privacy (Marshall & Rossman, 2014). In addition, participants would be inclined to give less extreme preferences for quantitative data which is based on a scale. So the answer given does not always correspond to what they really mean (Marshall & Rossman, 2014), and participants could be inclined to give answers which are socially desirable (Kelly, Harper & Landau, 2008).
At suggestion of the project leader of this application it would be sufficient to investigate the visuals of the application, how users experience the application, how likely they would think it’s sufficient for goal-setting and the usability (and features) of the application. Therefore a questionnaire will be used which is based on UES but with some alterations. A questionnaire is considered as a good option to investigate the attitudes of potential users, because it delivers a high face validity (Kelly, 2009). However, it is not always reliable or valid data and it is dependent on researcher consensus (Kelly, 2009). To determine Aesthetics, the five questions of the UES questionnaire are used. To determine Perceived Usability, the same questions from the questionnaire are used with the addition of extra questions related to control and challenge of the application. Control refers to the extent in which users feel in control over their experience of the application. (O’Brien and Toms, 2010). An example of this question was ‘I would find the app to be flexible to interact with’, indicating whether the participant is able to let the application do what he wants. Challenge refers to the amount of effort someone experiences when he have to complete online tasks (O’Brien and Toms, 2010). An example of this question was ‘I would find the app easy to use’, indicating whether the participant believes it does not cost a lot of effort to use this application.
To determine the overal experience of the application, the term ‘Endurability’ (as proposed in O’Brien & Toms, 2010) is used. Endurability describes the overall evaluation of the experience and its perceived success (Webster & Ahuja, 2006; O’Brien & Toms, 2010). These questions were related to whether using the goal-setting application would be rewarding and whether the experience worked out as planned. An example question of Endurability was: Using the app would improve my goal-setting achievement. This would indicate whether the goal-setting would be a success. Because of these alterations a reliability analysis with Cronbach's Alpha (α) is conducted to measure the internal consistency of this multi-item scales. In addition, because of likert-type data, descriptive statistics will be displayed of the individual questions. Using an exploratory factor analysis could be done to confirm whether the multi-scale items represent a specific ability or trait, but this was outside the scope of this research.
4.2.1 First Iteration
4.2.1.1 Research Design
For this first iteration, semi-structured interview were used with a retrospective think-aloud protocol. This is different from a regular think-aloud protocol where participants must tell what they are currently thinking or feeling while interacting with a system. A think-aloud protocol is also used at usability testing of software or other computer systems (Guan, Lee, Cuddihy & Ramey, 2006). However, this can cause a number of implications in practice: the talking during the interview could have a negative impact on the performance of users on a task. It can have an effect on attention and concentration and there is a chance that participants will perform tasks differently (Guan, Lee, Cuddihy & Ramey, 2006), and mostly the generated data represent a reflection of actual real-life use of an application, instead participants’ review of usability (Van den Haak, De Jong & Schellens, 2003). To avoid these problems, a retrospective think-aloud protocol can be of great use. When using a retrospective think-aloud protocol, participants are expected to perform the tasks required of them and afterwards they can verbalize their thoughts in retroperspective. An important advantage of retrospective think-aloud protocol is that participants are more likely to give explanations and to give suggestions (Van den Haak, De Jong & Schellens, 2003).
On this first iteration, four higher educated students participated. These participants were mainly found by using personal contacts of the researcher. These participants were asked in a personal one-on-one conversation whether they wanted to participate. All the participants were between 22 and 25 years (3 male and 1 female). Initially, teachers could be asked to serve as participants in this iteration, but given the time pressure and the fact that several teachers had busy schedules and therefore could not been interviewed on time, they are not included in this iteration. Besides the usual questions about the features, visualization, ease of us, overall experience and suggestions by participants, there were also questions about possible comparisons between the goal-setting application and other applications used by the participants. Questions about features involved asking whether they like it if there was a tutorial to make SMART goals or asking what they think of the assignment or ratings by teacher (What do you think about the assignment of ratings to goals?, Would you like to have
some tutorial inside the application which could guide and/or remind you how to create SMART (Specific, Measurable, Attainable, Relevant, and Time-bound) goals?). Participants
were also asked what they think of the visuals of a specific section of the site (Task: check
effectively? Explain.) or what they think of ease of use of the application (When you made a wrong decision in the application, did you find a way to correct that decision? Explain briefly).
The interview started as follows: the researcher gave a brief introduction and explained what the participant was about to see and it was stated that due to the confidential nature of this application, they may not speak of this application outside of this research. The participant accessed the goal-setting application at the url http://goal.ihatestatistics.com/ via a personal computer or the computer of the researcher. They were asked to create an account and log in. The researcher gave an introduction about the features of the application and explained and showed it to the participant. The webapplication consists mainly of three screens: Goal
Overview, Inspiration and Dashboard. In addition, a screen can be opened of 'View details',
where you could see more specific details on your created goal. Participant could also create a new goal through a screen. On the homescreen (Goal Overview), you can see which goals are pending, expired or finished. You could sort your goals in groups via a drag-and-drop possibility of the interface. On the 'Inspiration' screen you can commit to a goal which is created by another student and approved by the teacher. At the ‘Dashboard’ screen, you can see a line graph about how many goals are still pending, how many have been completed, how many you committed from other students (Inspiration) and how many goals are aborted. After explaining the application, the participant could test the application freely. The researcher explained that the participant had to check all three menu options, try to create and delete a (sub)goal, check goal details, let them observe what will happen if you commit to a goal of the ‘Inspiration’ screen, let them understand what is the purpose of the graph displayed in ‘Dashboard’. Through a combination of explanation of the researcher and the opportunity to let participant test out the application themselves, they could have an idea what is the use of the application. Hereafter, the actual interview started. All interviews were recorded with permission of the participants through a sound device, so a transcript could be made later. All interviews were conducted in Dutch, so that the participants were able to explain themselves more clear. It was thought beforehand that the interviews costed 20-25 minutes. In addition, all interview data was made anonymous for analysis. Some of the questions were related to a specific interaction at a certain time. The participant was given the opportunity to test out the application during the interview. At the end, they were asked to fill in an electronic questionnaire, made in Qualtrics, which was based on the UES scale (O'Brien & Toms, 2008). Participants had to answer whether they agree or disagree on some statements
and had to point this out on a 7-point likert scale. This scale ranged from 1 (Strongly
Disagree) to 7 (Strongly Agree).
4.2.1.2. Evaluation Results of User Needs and Requirements
The purpose of the interview was to map user needs and requirements. The structure of the results is based on the topics of the questions.
Expectation of features
The researcher asked participants what they expect from a goal-setting application. This question was stated to determine whether they are somehow familiar with a goal-setting application or what they guess the features are. All participants responded differently, a goal-setting application was seen as a tool to make some sort of planning or as an agenda with notifications. Another participant said that it should be some sort of to-do list. Which is quite similar to a goal-setting application. A goal-setting application should be clear and easy to use and you can fill goals at the right way. One participant referred to Khan Academy, which is an educational organization to provide free world-class education for people around the world. It was stated that he really liked the incentive system from Khan’s Academy,
something like earned ‘badges’. When he thought about goal-setting, he thinks about getting positive feedback on your goals.
Past experience with goal-setting or to-do lists
The researchers asked whether the participant has ever used a goal-setting application of to-do list application before or whether he is using one at the moment. Most of the participants didn’t use or used a goal-setting/to-do list application before. Someone pointed to the use of
Datumprikker.nl. Most of them did work with to-do lists on paper. A participant stated that he
tried to make a schedule for the goals on a to-do list, but he ended up not following that schedule.
Missing features
The researcher asked whether the participant felt he missed a particular feature at a moment when interacting with the application. He could suggest what he would like at that particular moment. The researcher gave an example like whether he would a search function. All participants gave different answers, someone said that he didn’t like the visuals and someone stated that he missed some sort of agenda or overview of a month. So you can see how many
days are between goals. Someone was quite satisfied and didn’t felt the need for an extra feature. A participant suggested that it would be nice if you could connect the goal-setting application with courses at the UvA. So you can select goals from a list, which are course-specific. So a student can get insight what is requested of them when doing the course. He stated that he just followed the class and tutorials and it would be nice if you have things to be done and you could look it up in the application
Checking goals by teacher
The researcher asked whether they liked it if a teacher could check their goals. Most of them were quite in doubt about this topic. Someone did state that it would be useful but didn’t thought teacher would spend time on it. Half of the participants stated that they didn’t like the feeling if they would get ‘checked’. A participant stated that he wanted to know for himself whether he wants to get something done and that nobody should watch him whether he is achieving or failing his goals. Another one had concerns for breach of privacy but he did thought it could work when a teacher can give feedback on his progress and how to improve the formulation of your goals. Another point of concern was that you should not get an advantage or disadvantage if you deliberately formulate your goals better than reality.
Students should not formulate goals deliberately well to impress the teacher. How a student is performing in the application should not be linked with his actual progress in a course.
Seeing goals of other students
The researcher asked whether the participant would like to see the goals of other students. Most of them liked the idea, stated that it could be useful and nice. Seeing how other students are tackling their goals. It could give insight in how a (good) goal is formulated. However you should not compare yourself with others. A participant stated that it could work both ways: you could get motivated to do something or when you see that other students are not trying hard to excel in their scores (like people only striving for a 6), then you won’t be that motivated to work hard.
Sharing of own goals
The researcher asked whether the participant would like it if you could share your own goals, which might be beneficial for other students. Most of the participants had mixed answers. It was stated that someone would like it if you could share it in little groups (e.g. 10 friends) or something like a Facebook group. Two of the participants were quite positive, stating that it
could be some bit of extra motivation to work on your goals. A participant stated that he was quite neutral about this topic: it is good to share when it’s fun to use, but you should not get demotivated (or insecure) when you’re seeing that the rest of the class if far ahead of your own progress.
Assigned goals vs. self-set goals
The researcher asked which statement described the participant the best. They could prefer teacher assigned goals, self-set goals or both. All participants said that both applied to them. It could be useful if a teacher assigns it if you are not doing your own planning. A participant stated that you don’t need a lot of other goals except goals for your exam and some deadlines. A participant stated that a teacher already assign goals to you during college, a teacher already makes expectations what a student should do next week. A teacher could give input and the student could decide what he wants to do with it.
Assignment of ratings to goal
The researcher asked what they thought of the possibility to assign a rating to formulated goals by teachers. Participants were quite mixed about this topic. Someone stated that he preferred feedback instead of a rating. He does not know how he would interpret such a rating. Every student is making a personal schedule for him- or herself. Someone stated that a teacher couldn’t always know whether a goal is good for you personally. He could have a different perspective of seeing things. However, it is understandable that people would like it if you formulated a good goal, got a good rating and a teacher approved it. Another
participant stated that it is hard to give a rating to a goal, however when the teacher gets interactive, it could be quite useful. But this same participant had some privacy concerns whether the teacher would know the name of the person who is the creator of the goal.
Committing to goals of students
The researcher asked what the participant thought of committing to goals created by other students. Most of them were quite positive about it. It could be useful for inspiration and students don’t really have to make a planning themselves. You can get an idea whether it’s a good goal or not. A participant liked it that you could see how many people already
committed to that goal, so you could be triggered to commit to it, even if you’re still in doubt. A participant stated that he stubborn: he would like to do things his own way. He preferred to go to college and summarize information to learn instead of committing to other goals.
However if the goal was realistic, it could motivate him a bit more to actually do it. Correcting of wrong decisions in application
The researcher asked whether the participant found a way to correct a decision if he/she made a mistake in the application. If they didn’t come up with an idea, the researcher gave an example: they could try to edit their goals and see what happens (there was not ‘edit’ button in the application). A participant stated that the goal overview was a bit chaotic and not so clear. Three of the participants stated that there should be a possibility to have an ‘edit’ option, so you could change the date or name of the created goals.
Email reminders
The researcher asked how the participant how often he/she would like to receive email reminders. These reminders will be sent to the mail of the user and consists of a summary of pending goals. It is not a reminder when they are close to a deadline. Every participant
responded quite differently, someone stated that he would like to get it via Facebook, because he uses it more often than email and he would want to get those reminders whenever he wants to (custom option). Another participant agreed that he didn’t read email very often and would like to get notifications on mobile screen. It was preferred to set those reminders whenever the participant wants to. Another participant would want to do it custom and stated that is
depends on the amount of goals: if you have 2 deadlines in a month, you don’t want to get reminders of pending goals 2 times a week. One participant suggested that this goal-setting application should have a mobile application. Students should decide for themselves when they wanted to get reminders. The researcher suggested that you could get reminders in Facebook if that was possible. However, the participant stated that he is not checking reminders very often on Facebook and he always sends them to ‘trash’ in Facebook. Feedback from teacher
The researcher asked the participants whether they liked it if a teacher gives them feedback about how well they are doing in terms of setting and achieving goals. Two of the participants explicit stated that they would like to have it in the application itself (not email). A participant stated that the ratings are already enough for feedback and was not sure to have it in the application in the first place. This participant stated that it wasn’t sure whether the teacher has enough time to actually do this and does not exactly understand what a teacher could say when he gives feedback. Another participant was quite positive, stating it could be useful,
even if it’s bad feedback. However, the frequency of feedback given should not become too obstrusive. Another participant stated that he would not really benefit of this feedback. He noticed that every student works differently and some of them always try to accomplish tasks last-minute. He wouldn’t prefer to get constantly feedback, because he is used to doing things his own way. However this same participant raises some concerns: feedback should not be given, because it is ‘feedback’. Feedback should really serve a purpose. Not all information from a student has to be sent to the teacher.
SMART tutorial
The researcher stated whether the participant would like it if there were a tutorial how to make SMART (Specific, Measurable, Attainable, Relevant, and Time-bound) goals. All participants responded very positive towards that idea.
Displaying of own progress vs. aggregate performance
The researcher stated that the user is the only one to see how he/she is making progress. The researcher asked whether the participant would like a feature where the user can see how he/she is making progress with respect to the aggregate performance of the whole class. Three participants responded quite positive. One of the three did stated that it probably could work a bit demotivational. When you’re far behind the aggregate performance of the class, you could stress about your own performance. A participant stated that he studies differently than most people; therefore his own goals could not be seen well or appreciated by others. This should not mean he is doing a bad job. He also stated that goals should be formulated realistic.
Students should not fill in goals that are more positive formulated than he/she is about to do in real life.
Suggestions (fun and cool)
The researcher asked whether the participant has some suggstions for this application to make it more fun and cool to use. One participant stated that it should somehow be linked to
Facebook, because many students are on Facebook. Another participant stated that if you created a goal, it’s given a high rating and the teacher approves it, then the teacher should be able to comment on it. This should make it more interactive. A teacher could comment on why it’s a good goal. Another participant stated that he didn’t like the long white blocks on the goal overview screen. It should be made more ‘fun’. He referred to Khan Academy, stating that you have to be stimulated in a playful way when you want to use this application
Visualization
The researcher asked whether the participant thought the total visualization was helpful and whether it displayed the information effectively. This question was asked 4 times, each at seperate screens of the goal-setting application. These seperate screens are titled Goal
Overview, Inspiration, Dashboard and View Details. The researcher explained what is the use of the particular screen before asking the question.
Goal Overview
All of the participants had remarks on the visuals. One of the participants stated it was sloppy, unclear and chaotic. Another participant stated that it missed an agenda or calendar feature. This same participant stated that he missed an option to finish the goal without going to the ‘View details’ screen. In addition, goals should be moved to a separate tab/screen when they are finished, instead of scrolling down. Another participant suggested making it nicer visually and reducing the size of those white blocks so you have to scroll less. However he stated that the whole interface does look well informed. Another participant suggested that it should be more compact and manageable. This same participant also suggested some possibility to ‘magnify’ your created goals, so it is not necessary to go to a separate ‘View details’ screen.
Inspiration
Two of the participants were not able to see the Inspiration screen correctly. One of the participants could not create a new account during the interview and therefore had to work on the account of the researcher where all goals of Inspiration were already used. Another participant didn’t see any ‘Inspiration’ goals during the interview, probably because the functionality wasn’t active. The other two participants both stated that it is clear and simple. However, it could be inconvenient if you have to scroll down far.
Dashboard
One participant stated that it was a bit unclear and maybe not a good idea. Another participant thought the visuals were addressed well, however he sometimes couldn’t see differences between two lines because they both had the value ‘1’, therefore no difference could be seen. Another participant said it was clear and suggested an option to switch between different kinds of graphs (e.g. pie chart). But he does not know the best way to display the statistics.
View Details
One participant didn’t like the layout and pointed out how some elements are not very logical. He referred to the green ‘checkmarks’ on this screen. This was a point of improvement. Another participant referred to the tags and didn’t think it was of much use if you can’t search on tags. This participant suggested an option where you can give your own goal a challenge rating (something like Easy, Medium, Hard goal). Another participant stated that he didn’t see room for improvement except for a feature to search through your goals by using tagwords. Another participant didn’t understand the use for those tags aswell. This same participant found some elements not very logical; the green ‘checkmarks’ next to subgoals were not very clear. It was first thought it was a feature to turn on/off your subgoals, but it was a feature to ‘finish’ your subgoals.
Comparison with other applications
The researcher asked whether the participant see the same usability and/or features, which were also present in other existing or past application that he/she has ever used. This could provide useful information to look at other popular applications. Two of the participants didn’t have an answer, one of the participants referred to Datumprikker.nl and another one to Khan Academy. Khan Academy already displays your goals. It is more about seeing progress, then actually setting goals.
Attempt to try the application
The researcher asked whether they would give this application a try. Two of the participants were quite positive, stating a positive answer and another one saying it’s useful and could really help. Someone stated it was a bit unclear and should be improved. Another participant said it could help in the form of an agenda, but he/she wasn’t sure.
Recommendation to colleague or student
The researcher asked how likely they would recommend this application to a collegue or student based on a scale from 1-10. One participant gave it a 7, another one gave it an 8 because of the interaction with teacher and student. One participant stated the application was still a bit unclear and gave it a 4.
