1
Final version: 30 June 2015 Supervisor: dhr. D. Heinhuis
dhr. D. Heinhuis dhr. drs. A.W. Abcouwer
………. ……….
Kevin Achterberg – 10617744
Thesis Master Information Studies – Business Information Systems University of Amsterdam Faculty of Science
Integrated information systems:
The influence of the use of mobile- and web
apps on perceived usability and user
2
Abstract
In recent years the rise of mobile technology has changed the way people live and work and software developers have to cope with these changing user expectations. To what degree this influences users of on-premises business information systems is still unknown and increases uncertainty of how and what to develop. This study seeks out what kind of influence the use of mobile- & web apps is having on how people perceive integrated information systems. A questionnaire was distributed among a sample of 497 users of AccountView, an ERP system, which measured their perceived usability and user satisfaction. The results indicate that while perceived usability and user satisfaction are correlated, the use of mobile- & web apps is not influencing or changing these constructs and their relation. This tell us the perceptions of users of on-premises integrated information systems is not changing due to increased use of mobile- and web apps. This leaves the question whether users will be more inclined to use mobile- & web apps for business purposes in the future.
3
Acknowledgements
I would first like to thank my supervisor dhr. D. Heinhuis who made me focus on what is important and helped me greatly during the thesis process. He always helped me in a very timely manner and provided valuable insights. I wouldn’t be able to deliver this thesis on time and with the same quality as it is today. Also I would like to thank my second examiner dhr. drs. A.W. Abcouwer for the interesting lectures throughout my master period. Also I would like to thank dhr. prof. dr. T.M. van Engers for his initial insights for my master thesis and suggesting a supervisor for me and ofcourse his insightfull lectures. Furthermore I would like to show my deep appreciation for Visma Software. They made it possible for me to conduct my research with customers of their AccountView software. In particular I would like to thank Jurriaan Wijnberg who was my supervisor during my thesis and Evert Mens for helping and supporting my endeavors within the company. They both inspired, guided and helped me effortlessly. In the end I would like to thank my family, in particular my father for supporting me when I was stressed but most importantly my sister who brainstormed a lot with me about possible directions and research methods. Without their loving support I might have stayed clueless at times. In general I would also like to thank the University of Amsterdam for giving me a great learning experience and everybody at the Faculty of Sciences for being orderly, friendly and available.
4
Table of contents
Cover page………….……… 1 Abstract……….. 2 Acknowledgements……….. 3 Table of contents……….. 4 1. Introduction……… 6 1.1 Mobile applications……….. 6 1.2 Influence on business……….. 61.3 Software development trends……… 7
1.4 Issue’s with component development and usability……… 7
1.5 Problem statement and research question……… 8
2. Literature study……….. 10
2.1 Usability……….. 10
2.1.1 Usability in general and selection of literature……… 10
2.1.2 Perspectives on usability……… 11
2.1.3 Usability definitions in software quality models………... 12
2.1.4 Dimensions chosen for perceived usability………. 13
2.2 User satisfaction……… 14
2.2.1 User satisfaction in Delone & McLean model of information systems success... 14
2.2.2 User satisfaction and user attitudes……… 16
2.2.3 Perceptions that influence attitude: Technology Acceptance Model……. 16
2.2.4 Expectancies that influence behavior: Unified Theory of Acceptance and Use of Technology 17 2.2.5 Integrated model: Relation of quality, satisfaction and attitude…………. 18
2.2.6 User satisfaction/attitudes and perceived usability………. 19
2.3 Dimensions for usability, hypotheses and research model………. 19
2.3.1 Relation perceived usability dimensions and user satisfaction………….. 19
2.3.2 Differences in perceived usability……… 20
2.3.3 Differences in user satisfaction……… 21
2.3.4 Moderating effect of use of mobile- and web apps……… 22
2.3.4.1 Operability……… 23 2.3.4.2 Understandability……… 23 2.3.4.3 Learnability……….. 24 2.3.4.4 Attractiveness………. 25 2.3.4.5 Helpfulness……….. 25 3. Methodology……… 27
5
3.1 Usability evaluation practices………... 27
3.2 Case study sample group……….. 28
3.3 Questionnaire design………. 28
3.3.1 Adapting questionnaire items………. 29
3.3.2 Questionnaire development process………. 30
3.3.3 Questionnaire development in this study……….. 31
3.4 Sample group details………. 32
4. Results……….. 33
4.1 Tests and analyses………. 33
4.2 Investigation in questionnaire batteries……… 34
4.2.1 Investigation perceived usability questionnaire……… 34
4.2.2 Investigation user satisfaction Questionnaire………... 35
4.3 Differences in use of mobile- and web apps ……… 35
4.3.1 Differences in perceived usability of AccountView……… 35
4.3.2 Differences in user satisfaction of AccountView ……….. 36
4.4 Correlation and moderation in relation between perceived usability and user satisfaction……. 36
4.4.1 Relation perceived usability and user satisfaction……… 36
4.4.2. Moderating effect of use of mobile- & web app on relation perceived usability and user satisfaction…… 37
4.5 Post hoc power analysis………. 39
5. Discussion………. 40 5.1 Interpretation of results……… 40 5.2 Additional analysis……….. 40 5.3 Implications……….. 42 5.4 Limitations………. 42 5.5 Future research……… 43 5.6 Conclusions………. 43
Literature reference list……… 44
Appendix A……… 52
Appendix B……… 53
Appendix C……… 54
6
1. Introduction
1.1 Mobile applicationsIn recent years we’ve seen an exponential growth of mobile software applications, the largest on mobile device platforms, which are called ‘native apps’ and web based applications, like web shops, social media sites and web games. More and more people are interacting with a lot of different small apps for mobile devices and internet browsers nowadays, and this trend is only expected to be growing (‘Number of mobile app’, 2015). Each of the applications have different graphical user interfaces and little to no documentation, but this doesn’t seem to hinder users figuring out how to use them. Due to increased exposure with different kinds of interfaces people have increasingly less difficulty working with different user interfaces because they are getting more familiar with the diversity between them. Studies haven’t shed a light on what this exactly means for software developers, but it helps people develop digital and media competencies (Mihailidis, 2014).
In contrast with traditional computer software, which are operated by mouse and keyboard, native mobile apps are operated by the touchscreen of the mobile device which acts to send information, but also as input device. In between are web based applications that are operated in a browser, both on mobile devices and PC’s. This requires new ways of thinking about and designing the user interface. Recently the use of mobile technology to access the internet has surpassed the use of desktop PC’s and is increasingly becoming the preferred tool for work and communication (Murtagh, 2014). The advances in how these modern technologies can be accessed by consumers are having an impact on how software is used and perceived.
1.2 Influence on business
Can this have far stretching consequences for corporate life? Some academics say enterprises are recognizing that the IT consumption in private life changes the expectations and behavior in the working life of people using desktop applications (Schreiber et al, 2012). The adoption of smartphones and explosive growth of its applications and consumption can have major implications for businesses. 10 years ago it was out of the question to use a PDA to receive work e-mails but today companies are embracing Bring-Your-Own-Device policies and integrating that with work processes. People adapt the use of all these different interfaces and will prefer the newer possibilities over the older. Businesses should adapt to deal with these new perceptions of users. It’s getting increasingly noticeable that no longer software companies have customers that are satisfied with off the shelf software packages, but rather want the possibility to add or access additional services in order to customize the package to their changing needs and demands (Park, Geum & Lee, 2012). This is a concept users are getting more and more familiar with through the offerings of different app stores enabling them to customize their smartphone. In this regard usability is key as it entails the human factors when interacting with computing systems. When these systems are poorly designed and lack usability this can have major costs for the manufacturer, either directly in reduced sales or indirectly in lowered customer satisfaction and a
7
tarnished image (Chapanis, 1991). These new developments in mobile- and web app technology and consumption could very well mean that we’re entering a new era of software development.
1.3 Software development trends
These developments could be good news for software developers that don’t know what their customers want and expect with software. Some advances made in software engineering and development could already be aligning with customer preferences. As first the increasing use of mobile devices in private life is pushing the mobile web. Applications are already increasingly written for the Web as services, to be run inside a web browser or some other web runtime environment. Second, industries and technologies move faster than before and requirements, needs and expectations change faster than traditional software development methods can handle. In recent years software developers have been adopting agile development methods, leaving the traditional ‘waterfall’ method behind them (Cohen et al, 2003). Customers are increasingly unable to define their needs up front while expecting more from the software, making the highly planned and specific requirements driven traditional methods not optimal for successful development in the current IT landscape. Since there are so many options available to customers it’s hard to prioritize what they exactly really need and just want (Bakshi & Sing, 2013). Long development times and stipulated plans make it hard to deal with changes during the development process, which could result in a less usable product. With software running on the web, it is hosted in the ‘cloud’ and versioning and updating becomes centralized for all users which makes it easier to rollout improvements. This in turn makes it logical to decrease the length of the development cycle in order to easily and more regularly facilitate users with what they want. With agile development the life-cycle of a project can be reduced up to 75% which consequently means going through these cycles more often making it much more viable to use in the current IT landscape. Agile development methods work best in small to medium sized teams (Popli & Chauhan, 2013). Thirdly we see the trend to develop software in components that can be integrated in to a larger software suite, e.g. an ERP system. Component development makes the development process more manageable and updates more controllable (Sprott, 2000). New cloud based delivery platforms are instigating the trend in which web-based components are integrated with on-premises legacy system. The concept provides new exciting opportunities for enterprises to have more flexible and easy-running business models, which allows access anywhere, cost reduction and is seen as a requisite to survive in the future of the information systems landscape (Alshamaila et al, 2013). This is also what smartphone users are coming to expect from applications.
1.4 Issue’s with component development and usability
There are however some disadvantages with component development regarding the usability of products. These components are similar to apps as they offer a specific small functionality, but as components are to be widely reusable, they must be sufficiently general, scalable and adaptable resulting in a more complex to use product (Crnkovic, 2001). Interfaces and work processes might not align with the different components, because the teams working on these components aren’t necessarily properly aligned and each build toward their own product. Consequently if the development team of an
8
ERP system wants to integrate another team’s software component they are faced with some problems. They can manage to get data exchanged between the two, but have difficulty integrating and properly aligning usability aspects, like the design aesthetics or working processes, of the component with the existing ERP system. Next to that, implementing the integration seamlessly most often implies writing code specifically for the components that are supposed to interact with each other and leads to code-tangling and systems that are hard to maintain (Paulheim, 2010). Additionally the legacy system, which are usually based on much older architecture principles, may not allow for the source code to be altered making seamless integration impossible, leaving interface usability issues. The risk the developers and vendors ultimately face by not adapting to these changes is not being able to satisfy their end users. This problem arises from their inability to keep the interface of their larger software systems consistent with the new components they introduce. Throughout the development cycle of a software product, time and effort are put in to make it usable for customers, but with trends regarding use of mobile phones and their applications usability requirements are transforming. In this regard software applications from the mobile consumer market normally do not offer the interfaces or use the standards in professional use of IT (Weiß & Leimeister, 2012). However they are satisfying their end users as we can deduct from the explosive growth of the consumer mobile application market. In mobile apps we see inconsistency of screen design and gestures used to operate different apps. This means that once users figure out how something works they can’t transfer skills from one app to the next (Kelly & Schrape, 2010). This does however add to their competence to figure out and work with different user interfaces and solve problems with operating different applications.
1.5 Problem statement and research question
The growing trend mentioned before, where people seem to have no difficulty switching between different interfaces to work with different applications, raises a question. Is it still important to focus on minimizing differences in usability aspects, like aesthetics, consistency and flow, when integrating new functionality in the form of a software component or service? With these components being increasingly more web based they look and feel more like applications people use on their smartphones all the time. When media competence is increased by using more different interfaces on mobile phones and the internet, do people subjectively value the usability of software in their work setting differently? Is the private use of IT having an impact on how software developers can be successful? Businesses face the question how important it is to focus on usability, which aspects of usability are most important when developing software and how this relates to users that deal with apps more and more frequent. Does software developing need to embrace different strategies to satisfy their end users? What usability entails is constantly changing over time and that makes it of importance to validate whether we’re entering a new era of studying and perceiving usability. New believes and customs people develop by new technologies could potentially shake the research and development field to its very core.
The questions imposed in the last section make it interesting to conduct a study in what kind of influence new mobile technology and software has on usability perceptions of users using information systems in the work setting. Also are the perceptions influencing the user’s overall satisfaction with those systems?
9
Literature shows that the upcoming trend of mobile devices is changing and shaping our perceptions of how we live our lives and how we manage the balance between work and personal life (Duxbury & Smart, 2011). This suggests that it could also be changing the perceptions about how we want to work and what we want to work with in business computing. This in combination with the changes in software development and the growing difficulty in offering consistent usability make it interesting to study if mobile applications are changing our mindset. This study will be in the topic of usability evaluation and focus on the influence the use of mobile- and web apps has on usability perceptions. The main research question imposed in this study will be:
To what extent leads the use of mobile applications to a change in usability perceptions of information systems in work settings?
In order to accurately address this question it will be divided into smaller sub questions, of which the first two will serve to better understand usability and how to study it. The last two will serve to make expectations about how usability is changing and whether we can validate that. These sub questions are:
What determines perceived usability?
How can we operationalize perceived usability?
Which changes in usability perceptions do we expect with increased use of mobile- and web applications?
Can we empirically validate our predictions in regard to the users changing usability perceptions in information systems in work settings due to the use of mobile- and web applications?
To be able to properly answer the research question the first section will assess the literature on usability. What does usability lead to and which are its dimensions, attributes and characteristics as described by several authors in the field of usability testing, development and evaluation. Subsequently the dimensions selected for this study are defined. From a conceptual research model we will ground our hypotheses. The second section describes the methodology for our case study by first describing the software system that will be the subject of the analysis, the user groups that will be studied and the method of gathering data. Subsequently the data will be analyzed and results will be presented. The third section will discuss the process and results of the research followed by a conclusion.
10
2. Literature study
In this chapter we will answer the first two sub questions: ‘What determines perceived usability?’ and ‘How can we operationalize perceived usability?’ It will first show what perceived usability is and how it can be operationalized in the context of information systems used in the working environment. Then the link between perceived usability and satisfaction is made which leads us to how this has been studied in information systems before. Furthermore it will show in which ways it has been operationalized and what impact this has had on usability evaluation and business practice. After that the different conceptual research models will be presented and the hypotheses this leads to will be grounded.
2.1 Usability
2.1.1 Usability in general and selection of literature
Usability is used in many contexts, from household apparatus to buildings, and is a term that can have different meanings depending on the context. Usability is important for a product as it makes the product easier and more predictable to use and it is generally accepted that good usability leads to greater satisfaction with the user (Doll & Torkzadeh, 1988; Delone & McLean, 2003; Flavián et al, 2006). In this study we will focus on the usability of software and in a more narrow sense information systems. But what is usability in software and in which fields has it been studied?
Software usability research is a key interest in the field of Human Computer Interaction (HCI) which focuses particularly on the interfaces between people and computers. HCI as a research field is placed at the intersection of, most notably, computer science and behavioral science, but consists of many more fields like design-, media-, and cognitive science (Long & Dowell, 1989). The strength of HCI is that theory and practice lie closely together (Dix, 2010).
In order to get a deeper understanding of usability several of the different definitions and contexts of the concept will be reviewed, with as initial point the literature review by Madan & Dubey (2012). They did an extensive survey of usability concepts and usability evaluation, which provided an initial starting ground for the search of literature on usability. Additional literature, in the form of articles, conference papers and books, was also found by searching in the Google Scholar database on the terms ‘usability’, ‘usability evaluation’, ‘usability characteristics’ and ‘software usability’ from the period 1980 till present day. Respectively this amounted for 954.000, 557.000, 321.000 and 587.000 hits, which was a much too large amount of literature to be able to go through with the given time. For that reason we sorted the selection on descending relevance and investigated the hits until pages with search results no longer presented useful literature. Furthermore articles and papers were examined according to a list from Microsoft Academic Search (‘Top journals in Human Computer Interaction’, 2013) which gave us 26 notable scientific journals in usability. Articles were selected from ‘Behaviour and Information Technology’, ‘Interacting with Computers’ and the ‘International Journal of Human-Computer Interaction’. Valuable references from articles from these journals were also studied. There was
11
considerable overlap in articles in using these methods to find and select literature. From these references it becomes clear that usability is an important factor that determines software quality. The quality of software can be judged from three different perspectives that will be discussed next.
2.1.2 Perspectives on usability
Usability is seen as an important factor contributing to software quality. When taking a closer look at software quality models, which exist in different forms, every single one of them regards usability as a vital part of software quality (Xenos, 2001). We can distinguish three perspectives of looking at usability in software products. The first is usability as an internal quality of the software, for instance the time it takes the program to generate a result (execution speed) (ISO/IEC 9126-1, 2001). This is referred to as the supervisory perspective by Paithankar and Ingle (2010, p. 582) as this viewpoint is mainly taken by software developers in order to evaluate usability concerns. The developers supervise the development and maintenance of the software. In this regard a function that produces proper qualitative outcomes but takes way too much time to generate the outcome is deemed lacking usability. The second perspective is usability as part of the external quality of the software, for example is the program comprehensible to users (understandability). Paithankar and Ingle (2010) call this the direct perspective as it concerns users interacting with the software directly and emphasizes requirements from the perspective of software usage. As for an example, the software may produce quality results in a timely manner but the function cannot be found or comprehended by the user which harms the usability of the product. Finally usability can be viewed as a quality of the software product in use, which refers to what a usable program leads to when users utilize it, for example being able to perform a task (effectiveness). This can also be seen as the indirect perspective and it is of importance when reviewing or assessing a software product, to see if it’s appropriate to use in various contexts. In this regard a manager can look at whether the software makes the processes involved more effective than other products or the original situation.
The three perspectives to usability either influence or depend on each other. Internal quality influences the external quality while external quality depends on internal quality. The same holds for the external quality influencing the quality in use (Fitzpatrick & Higgins, 1998). In usability research researchers are most interested in external quality and quality of use. The first is important because the external quality of the software is what people notice and experience and is the outcome of those aspects that contribute to internal usability. The second one, quality in use, researchers find interesting because it shows how external usability influences the outcome of interacting with the software in regard of usability goals (Winter, Wagner & Deissenboeck, 2008). In order to study people’s attitudes about the usability of a software product it’s necessary to take the direct/external quality perspective. This perspective concerns the interaction of users.
This leads us to answer our first sub question by stating that perceived usability is determined by the interaction of users with a software program. It is best viewed from a direct/external quality perspective, but it must be noted that it influences the quality in use. In the next section we will answer the second
12
sub question, ‘‘How can we operationalize perceived usability?’’ by investigating what interaction with a software program means in conceptualizing software usability.
2.1.3 Usability definitions in software quality models
Each quality model has different conceptualizations of usability, but they differ more by which perspective it is looked at. In table 5 in appendix A different conceptualizations of usability are matched with the viewpoint they take in how usability affects quality of software. In the next section these will be further elaborated and the ones most appropriate for this study highlighted. McCall et al. (1977) were the first to produce a software quality model and they deemed usability as one of the eleven quality factors of a software product. For the factor usability they proposed three criteria that fit it, namely: operability, training and communicativeness. This viewpoint looks for a very small degree at internal characteristics and focusses mainly on external characteristic. Those aspects in the software program the user really comes in contact with. Nielsen (1993) states in his book Usability Engineering that usability is not a single, one-dimensional property of a user interface, but goes beyond the interface. It is not only the presentation on how to interact with the software, but also the processes that make up the interaction. He elaborates by mentioning that usability is traditionally associated with five attributes, which he describes as: learnability, efficiency, memorability, errors and satisfaction. Preece et al. (1993) made a classification in which they attributed safety, effectiveness, efficiency and enjoyableness to usability. These are all goals of what a software product with good or improved usability should have and can be deemed as quality in use attributes. Quesenbery (2001) included five attributes to usability derived from the usability definition of the ISO 9241 standard. The ISO 9241 standard (1998) defines usability as ‘the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use’. This definition looks at usability in the context of outcomes. Quesenbery takes a perspective focusing more on the external qualities inherent to usability and expanded this by adding the five E’s, which reflect on the software product being: effective, efficient, engaging, error tolerant and easy to learn. While the ISO 9241 standard takes a quality in use perspective there is also a standard that takes the external viewpoint.
In the international standard ISO/IEC 9126-1 (2001) an external quality model for software products is presented and internal and external metrics for measuring this are proposed. One of the six qualities a software product has in this description is usability. Usability to their standards is the capability of the software product to be understood, learned, used and attractive to the user when used under specified conditions. For the quality characteristic of usability five attributes are formulated which are: understandability, learnability, operability, attractiveness and usability compliance. The model encapsulates quite well the external characteristics that make up the quality in use characteristics (Azuma, 2004). The quality model for software products that is made to replace the ISO 9126 standard was released in 2011 and is called the ISO/IEC 25010 standard (2011). The standard offers a product quality model composed of eight characteristics where they replaced the term usability by operability to avoid conflicting terms in their standards. Usability in this standard consists of the following external quality attributes: appropriateness, recognisability, learnability, operability, user error, user interface
13
aesthetics and accessibility. While the ISO/IE 25010 standard was supposed to replace the ISO 9126 standard still the majority of practitioners refer to the old standard (Alonso- Ríos et al., 2009; Veenendaal, 2014). Along with the definitions of usability through standards committee’s and boards there have also been researchers that have conceptualized usability in order to research it with surveys, like the SUMI (Kirakowski & Corbett, 1993).
The Software Usability Measurement Inventory (SUMI) defined the concept of usability along five sub-scales, which are: efficiency, affect, helpfulness, control and learnability. These attributes were the guidelines for developing a questionnaire for analyzing the degree of end user satisfaction and the quality of use of the information system being studied. It is concerned with the users’ perceptions and attitudes toward a piece of software (Kirakowski & Corbett, 1993). Interestingly instead of looking at error tolerance or errors they take a more positivistic stand saying usability consists of helpfulness for when errors occur, which can be considered an inevitability when working with a system. Alonso- Ríos et al. (2009) made an analysis of existing usability concepts and described a detailed and exhaustive taxonomy of all its characteristics and sub-characteristics. They made this taxonomy from all three perspectives and incorporated and conjoined usability dimensions from several different studies. They described the main characteristics of usability as: knowability, operability, efficiency, robustness, safety and subjective satisfaction. They propose that this composes all of the aspects depicted in the various usability conceptualizations and is a mixture of looking at the internal quality, external quality and quality in use of software usability.
2.1.4 Dimensions chosen for perceived usability
Table 5 in appendix A shows an overview of the different classifications of usability based on the literature presented above. As this overview shows there exists no universal consensus of what makes up usability, but we can acknowledge there seems to be somewhat overlap between each of the definitions and dimensions. The selection of dimensions to make up the concept of usability seems so often to be arbitrary to the researcher’s preferences or becomes overly complex to be able to address all of what seems to be associated with it. In order to choose how to operationalize usability in this study we have to accurately appoint what we want to find out. We’re interested in which way the use of mobile- and web apps influences usability perceptions as certain aspects of these mobile applications have similarities with information systems. This means we have to look at the dimensions of usability as an external quality of software as perceived by the user. In turn we have to look at how this relates to how the user finds the information system acceptable and satisfactory in the context of use (Seffah et al., 2006).
The dimensions chosen to reflect usability in this study are largely based on the ISO 9126 standard (2001) which is considered to provide a comprehensive structure for the role of usability as part of software quality. It reflects most of the attributes that are encountered by users and are more characteristic of usability in interaction than the capabilities or outcomes of usability (Bevan, 2009). The difference with the ISO 9126 standard and what is used in this study is that we decided to dismiss
14
usability compliance since this perception is different over time and differs with contexts. Rather we’ve chosen to incorporate the concept of helpfulness, which adds the dimension of how to deal with errors. Dealing with errors or software being error tolerant is a usability dimension depicted in many other quality models (Nielsen, 1993; ISO 9241-11, 1998; Kirakowski & Corbett, 1993; Quesenbery, 2001). It makes up a substantial part of what the user visually encounters when interacting with the software when dealing with problems and is deemed appropriate in our conceptualization of perceived usability. This leads us to the following attributes of usability: operability, understandability, learnability, attractiveness and helpfulness.
There is however one outcome of usability that we don’t share under the dimensions of our external perspective on usability which is satisfaction. Satisfaction has been used in several quality models as part of usability (Nielsen, 1993; ISO 9241-11, 1998; Alonso-Ríos et al, 2009; Preece et al, 1993). Satisfaction can be seen as an outcome of usability for a user and in order to check the quality in use of an information system it can be an appropriate measure (Flavián et al, 2006). End-users are more likely to be satisfied with an information system when they believe that using it will increase their productivity and performance. Perceived usefulness, which relates closest to operability and understandability in our usability dimensions, and learnability are determinants of end-user satisfaction. Also end-users rate systems as less useful if they find them difficult to use (Calisir & Calisir, 2004). Because we’re interested to study the effect the use of mobile- and web apps has on perceived usability we should also consider if it affects the relationship with satisfaction. From the three perspectives on usability satisfaction is the one factor that reflects how the user feels about the interaction with the software that is a consequence of usability, among other quality factors. It could be the case that the use of mobile- and web apps only marginally changes how users perceive the usability of the system they work with, however dramatically changes how these perceptions relate to their user satisfaction. User satisfaction correlates strongly with the expectations people have and how these expectations relate to actual use (Fornell, 1992). Perceived usability might stay the same, but it could be that expectations of usability are different resulting in lower user satisfaction. The relation between satisfaction and usability is more complex than satisfaction being part of usability as we will further elaborate in the next paragraphs by reviewing theories that study user satisfaction and user attitudes.
2.2 User satisfaction
2.2.1 User satisfaction in Delone & McLean model of information systems success
While satisfaction has been shared under usability in some software quality models, computer user satisfaction in research has been a prevalent measure of Information System (IS) success. DeLone and McLean (1992) introduced their IS success model in which user satisfaction depends on system quality and information quality and ultimately influences the use of the system. After the model had been widely used they updated it ten years after its introduction on the base of research contributions to better match with the changed role and management of information systems. Their updated model suggests that usability is part of the quality dimensions which influence user satisfaction and intention to use. The
15
different kinds of quality are different categorizations as in most software quality models mentioned before. Usability, among other qualities, is spread out over the three dimensions of quality in the updated model since it effects the information quality, e.g. completeness and understandability, the system quality, e.g. ease of learning and intuitiveness, and service quality, e.g. help accuracy and technical competence. These three contribute to user satisfaction in their model which in turn influences the net benefits of the system (Delone & McLean, 2003). As is depicted in figure 1 the updated DeLone & McLean model of information systems success indicates that usability, as part of system-, information- and service quality, indirectly contributes to the net benefits of the system.
Figure 1: updated DeLone & McLean model of information systems success (DeLone & McLean, 2003 p.24)
While in this study we’re not especially interested in IS success but more in the actual satisfaction of the end user using the application in relation to the perceived usability, it is worth noting that user satisfaction is an indicator of IS success. Ultimately the development of software does have the goal for the software to be successful which makes the influence of usability on user satisfaction very relevant. Usability characteristics that influence user satisfaction could potentially harm the perceptions of that user about the software. This in turn can lead to dissatisfaction which can result in negative evaluations of the software from users to potential new customers. This form of negative publicity can strongly hinder the distribution and eventual IS success, which shows that high perceived net benefits are very important. Bolton & Lemon (1999) found that overall satisfaction at current times is positively related with perceived value of usage in the subsequent period. This means that when people were satisfied with a service they will perceive it as better after that. This effect comes from the attitude they develop during use and changes their beliefs for future evaluation.
16
User satisfaction is a type of attitude formed by the negative and/or positive feelings a user has towards an information system (Melone, 1990). Ajzen (1988, p. 241) states ‘an attitude is an individual’s disposition to respond favorably or unfavorably to an object, person, institution, or event, or to any other discriminable aspect of the individual’s world’. These dispositions can be numerous, both positive or negative, and the weighted accumulation of these make up someone’s attitude. Attitudes consist of a cognitive, affective and conative part that form the judgement. However attitudes don’t necessary directly reflect behavior that is performed since prior to acting out behavior the attitudes will first be reflected on the subjective norms that hold for the situation and the perceived behavioral control someone feels (Ajzen, 1991). Computer user satisfaction is described in many closely aligned terms and has been defined in many ways. To illustrate Ives et al. (1983, p. 1) defined user information satisfaction as ‘the extent to which users believe the information system available to them meets their information requirements. Doll & Torkzadeh (1988) point out that user information satisfaction instruments have not been designed for measuring end-user satisfaction and developed a tool for measuring end user computing satisfaction. They define end user computing satisfaction as ‘the affective attitude towards a specific computer application by someone who interacts with the application directly’ (Doll & Torkzadeh, 1988, p. 261). They explicitly state that it’s about the affective part of the attitude that culminates towards user satisfaction about a specific computer application. Melone (1990, p. 80) states that a slightly broader construct which encompasses the ideas embodied in user satisfaction is the construct user attitudes.
2.2.3 Perceptions that influence attitude: Technology Acceptance Model
Two important and widely accepted theories that studied the user’s attitudes about and intention to use an information systems are the Technology Acceptance Model, referred to as TAM (Davis et al, 1989) and the Unified Theory of Acceptance and Use of Technology, referred to as UTAUT (Venkatesh et al., 2003). TAM deals with the prediction of the acceptability of a new information system. The model suggests that two main factors determine this acceptability: perceived usefulness and perceived ease of use. Perceived usefulness is defined as being the degree to which a person believes the use of a system will improve his performance. Perceived ease of use refers to the degree to which a person believes that the use of a system will be effortless (Davis, 1989). Perceived usefulness and perceived ease of use are results of what users perceive of the software quality. As can be seen in the model in figure 2 these two factors form the attitude toward using the information system which together with the perceived usefulness will determine the behavioral intention to use the information system. With two systems offering the same features, a user will be more accepting towards the system that he perceives easier to use. This implies that a system that is easier to use will have higher perceived usability which will lead to more satisfaction from the user.
17
Figure 2 Technology Acceptance Model (Davis et al, 1989, p. 985)
2.2.4 Expectancies that influence behavior: Unified Theory of Acceptance and Use of Technology
UTAUT unifies several theories that try to explain information systems usage behavior. UTAUT seeks to explain the user’s behavioral intentions and subsequent usage of an information systems. It holds that performance expectancy, effort expectancy and social influence are direct determinants of usage intention and behavior and a fourth construct facilitating conditions as direct determinant for use behavior. As is shown in figure 3 the model proposes that gender, age, experience and voluntariness of use moderate the impact of these four constructs on behavioral intention and use behavior. The UTAUT model takes in to account the external variables that TAM largely neglects in forming and moderating the behavioral intention towards using an information system. UTAUT doesn’t encompass attitudes or satisfaction but does depict that expectations and how they are lived by users influences their behavior.
18
2.2.5 Integrated model: Relation of quality, satisfaction and attitude
Wixom and Todd (2005) notice that user satisfaction is a poor predictor of actual usage of an information system while the attitude construct in TAM has been widely applied to understand the attitude one holds about the use of technology. User satisfaction is better utilized to find out which aspects of a system are not satisfactory to users. They propose an integrated model that combines satisfaction and technology acceptance theories. As is shown in figure 4 system characteristics (the object) form object-based beliefs on the information quality and system quality of the information system. These beliefs form the attitudes in the form of information satisfaction and system satisfaction. And these object-based attitudes shape the behavioral beliefs that ultimately form the attitude and intention to use. This integrated model also shows and proves that in the end the system characteristics indirectly lead towards the attitudes and beliefs that make out whether the system will be used and accepted. They also analyzed different structures of their model and concluded that the further removed a factor is from the actual usage behavior construct, the less predictive it will be of use. Their findings support that the mediating roles of quality and satisfaction are critical in bridging the gap between system characteristics and intention to use and attitude about the information system. Their model also clearly shows that quality is an object-based belief which influences the object-object-based attitudes of satisfaction
19
2.2.6 User satisfaction/attitudes and perceived usability
As we’ve shown in the previous paragraphs the system characteristics ultimately lead to whether an information system is accepted, used and can become successful. Usability is part of the characteristics that make up the quality of a system. In the Delone & Mclean model (2003) quality and system use influence user satisfaction. The perception of the quality is reflected in user satisfaction. In TAM (Davis et al, 1989) perceived usability is a large factor in forming the attitude towards the system as perceived ease of use and usefulness correlate with the attitude towards use. The attitude then determines the intention to use. Here we see that the perception of the system in use reflects on the attitude. The UTAUT model (Venkatesh et al, 2003) doesn’t deal with attitudes but uses the expectations users have to predict behavioral intention. In this regard people whose expectations have not been met will tend to form a negative attitude. In this model the difference between perceptions and expectations make up the attitude that influences behavior. These relationships are moderated by user- and usage characteristics. In the end Wixom & Todd (2005) integrated these theories in one model where the beliefs about quality formed by the expectations and perceptions of quality characteristics influence attitudes about satisfaction. Changes in how usability is perceived in relation to what is expected will have a direct impact on what people think of the quality of a system, their satisfaction with the system and how willing they are to use it. This in turn could lead to a difference in satisfaction which makes it important to study whether usability perceptions or user satisfaction are changing these days because of increased use of mobile- and web apps. In the next section we will propose several models and hypotheses that aim to find these changes.
2.3 Dimensions for usability, hypotheses and research model
Based on the findings in the previous section we will proceed to answer the second sub question: How can we operationalize perceived usability? Subsequently the operationalization also allows us to incorporate the predictions we have regarding the research question. This will lead to the answer of the third sub question of this thesis which is: Which changes in usability perceptions do we expect with increased use of mobile- & web applications?
2.3.1 Relation perceived usability dimensions and user satisfaction
We will introduce several hypotheses derived from our expectations. While satisfaction has been shared under usability by many researchers we conclude that it is an object based attitude that correlates with the beliefs a user has on usability itself. In order to see if this connection exists the first hypotheses will investigate in whether the perceived usability aspect correlate with system user satisfaction. In figure 5 the dimensions of usability in this study and the influence we expect them to have on user satisfaction are shown.
20
Figure 5 Dimensions of usability and their relation with user satisfaction
We believe that each of these dimensions that make up usability correlate with how high the user satisfaction is. So our first hypothesis will be:
H1: Understandability, learnability, operability, attractiveness and helpfulness correlate with user satisfaction
2.3.2 Differences in perceived usability
The next set of hypotheses we present come from the increased component integrations, that offer (app-like) functionalities, introduced in information systems. As mentioned in the introduction we suspect that there is a shift in how usability is perceived. Users that make more use of mobile- & web apps will have a higher perceived usability with an information system that has functional integrations opposed to users who make less use of mobile- & web apps. This comes forth from the assumption that they are more familiar with different interfaces and are custom to switching between how functionalities are presented and operated. This makes that they can more easily work with the functional integrations in the information system rating its usability higher. In contrast we believe that the opposite holds for information systems without integrations as the users might find the information system lacks functionalities they want and their expectations aren’t met with what they are custom to in their everyday life. They see the system without integrations as not modern enough and lacking usability. Figure 6 shows a diagram in which these relationships are shown.
21
Figure 6 Diagram with hypotheses about differences in perceived usability between user groups (heavy, medium-light or non-use of mobile- & web apps and information systems (with or without integrations)
The figure shows that we expect that mostly the people that make heavy use of mobile- & web apps
or don’t use it at all will have a different perceived usability from the average. This makes for the
following two hypotheses
H2: Users that make more use of mobile- & web apps perceive usability of an information system with
integrations as higher than users that make less use of mobile- & web apps
H3: Users that make more use of mobile- & web apps perceive usability of an information system
without integrations as lower than users that make less use of mobile- & web apps
2.3.3 Differences in user satisfaction
We expect that the same relationships exist for system user satisfaction as perceived usability is a part
of the direct factors that influence user satisfaction. Figure 7 shows these relationships
Figure 7 Diagram with hypotheses about differences in user satisfaction between user groups (heavy, medium-light or nonuse of mobile- & web apps and information systems (with or without integrations)
22
From figure 7 we can derive the following hypotheses from the expectations:
H4: Users that make more use of mobile- & web apps rate their user satisfaction with an information
system with integrations as higher than users that make less use of mobile- & web apps
H5: Users that make more use of mobile- & web apps rate their user satisfaction with an information
system without integrations as lower than users that make less use of mobile- & web apps
2.3.4 Moderating effect of use of mobile- and web apps
In the previous section we’ve shown our expectations about the differences in perceived usability and user satisfaction between user groups and information systems. Further we want to research which dimensions of usability influence user satisfaction. Previously we’ve studied the literature and we’ve chosen for the ISO 9126 definition of usability as it relates to the external software characteristics of usability that users encounter. We assume that each of the dimensions we’ve selected, i.e. understandability, learnability, operability, attractiveness and helpfulness influence user satisfaction as follows from the updated DeLone & McLean model of information systems success. With H1 we want to study which of the dimensions of perceived usability has the largest influence on user satisfaction. For the next hypotheses we assume that these influences are moderated by certain characteristics users have. As is shown in the conceptual research model in figure 8 we believe these are: use of mobile- & web apps, age, experience with software and use of software. With software we mean the information system with or without integrations. Since the scope of this study is to investigate the influence of the use of mobile- & web apps we will focus our hypotheses around the use of mobile- & web apps. The relationships of this effect are highlighted in green and are made bold in the model in figure 8.
23
In order to explain why we expect these relationships to be moderated we will have a closer look at the dimensions of usability we’re using in this study. Most of the dimensions have been adapted from the ISO-9126 standard (2001) except for helpfulness which is provided by Kirakowski & Corbett (1993).
2.3.4.1 Operability
Operability is defined as ‘the capacity of the system to provide the functionalities necessary to implement the tasks intended by the user’ (Alonso-Ríos, 2009, p.58). Dubey et al. (2012, p. 431) in their taxonomy of software usability define it as ‘the degree to which the software provides the users with necessary functionalities that help them perform tasks correctly’. It is considered with both the precision of the offered functionalities to help perform the task as well the completeness of the result generated by performing the task. Another indicator from these definitions lies in the part mentioning ‘provides’ which we interpret as the location from which the functions can be operated within the software. Is the function easy to find, and therefore operate, or does the system provide the necessary functionality but in a way that it is hard to find and use. This can be the case when functionality is hidden deep within different menu structures. When looking at the influence of operability on user satisfaction we argue that there are two constructs that moderate this effect. The first one is hypothesized to be experience, since experience with the software product will allow the user to perform their intended use more easily resulting in a higher perceived operability and user satisfaction. The second one is regarded as the use of mobile apps since it is to be believed that people who make more use of apps on their mobile phones have less difficulty finding and utilizing the software functionalities to their intended use since they’ve encountered more different applications through mobile devices and web browsers. The perceived operability will influence their satisfaction stronger because they tend to complete their tasks more completely and precisely. Although we think that experience with the software product will moderate the effect operability has on system user satisfaction to stay within the scope of the study we will only incorporate the use of mobile- & web apps in the hypothesis
H6: The influence of operability on user satisfaction is moderated by personal use of mobile apps, such that the effect is stronger for workers who make more personal use of mobile apps and have more experience with the software product.
2.3.4.2 Understandability
Understandability is defined by the ISO 9126 standard (2001) as ‘the capability of the software product to enable the user to understand whether the software is suitable, and how it can be used for particular tasks and conditions of use’. It entails product description and demonstrations on whether the software product is suitable for the intended use as well with the interface functions, e.g. menus, are easy to understand. From this we may derive that software with more functionality has more interface functions which in turn can make it harder to understand. Bertoa et al. (2006) found that understandability of software components depends on the structure and organization of the manual. With more functionality it is logically that the manual also gets more elaborate, or contrary incomplete, which in turn can strongly
24
affect the understandability of the manual and its product. This makes it harder to find out if the software is suitable and how it can be used. We do not think that the use of mobile- & web apps influences the effect understandability has on user satisfaction, because we don’t think it will help them in evaluating whether the information system is suitable and figuring out how it can be utilized. However as figure 8 shows as workers gain more experience with the software product the manual is less needed, the interface is comprehended and understandability has less influence on user satisfaction. The same goes for workers who use the software more frequently than others because in both situations they rely less on the software’s capability to make them understand the suitability of tasks but more on their own previous experiences and interactions. For the dimension understandability we do not expect the use of mobile- & web apps has any effect which we will try to validate by the following hypothesis:
H7: The influence of understandability on user satisfaction is not moderated by personal use of mobile apps.
2.3.4.3 Learnability
Grossman et al. (2009) addressed that learnability doesn’t have a widely agreed definition and this comes from different conceptualizations of the term. They found that different researchers look at different aspects of learnability like Nielsen (1993) who highlights the initial learning curve to allowing users to reach a reasonable level of proficiency. They found that defining learnability based on the initial user experience is common in usability research. Another way of looking at learnability rather than initial learning is extended learning. Dix et al. (2003) give a definition that implies both initial as extend learning. Their definition composes of the ease at which new users can begin effective interaction which indicates initial learning, followed by the addition, and achieve maximal performance. This indicates that the user is usually always able to learn more about, or new ways to deal or operate the software. We argue that users working with complex software seldom reach the state of mastering the entire application since there may be things that will always remain outside the user’s scope. Bertoa et al. (2006) found that learnability depends on both the quality of the manual and the complexity of the design. ISO 9126 standard (2001) looks at learnability as the capability of the software product to enable the user to learn its application. This way of looking at learnability goes beyond mere initial learning and also takes into account learning at a prolonged stage of working with the software especially with software suites that are large and continuously enhanced. According to assimilation theory a person links information related to the context with new information to be learned. In this process the learner recognizes appropriate anchoring concepts from long-term memory and manipulates them accordingly to fit and connect with the new information. After meaningful learning a person should be able to handle problems different from the original context since they are able to extend their knowledge (Davis & Wiedenbeck, 1998). Based on this theory people who have used different apps and learned how to interact with them and utilize them can extend this knowledge to new contexts and solve problems they encounter with information systems in the work setting. This may mean that people who make more personal use of mobile- & web apps have less difficulty dealing with a larger more complex system and are less
25
sustainable to the quality of the manual and therefore are less inclined to let learnability influence their user satisfaction. That is why we propose that:
H8: The influence of learnability on user satisfaction is moderated by personal use of web- and mobile apps, such that the effect is weaker for workers who make more personal use of mobile apps.
2.3.4.4 Attractiveness
ISO 9126 standard (2001) defines attractiveness as the capability of the software product to be attractive to the user. This can be divided in to two categories. The first being the aesthetics of the design being visually appealing to the user and the second being the capacity of the software to let the user adapt the interface appearance. The attractiveness of the aesthetics of the design is concerned with the feeling the user experiences when looking at the interface. This can be influenced by the color, shape and composition of entities on the screen, as well the consistency between them. When more functionalities are added to a software product it becomes harder to maintain a visual appealing interface, especially when integrated components were developed outside the reach of the development team, making them inconsistent with the rest. Also when the software becomes more elaborate it may become harder for the user to adapt the interface appearance, because the screen is already more crowded leaving no room to customize. We believe that age plays an important role in the influence perceived attractiveness has on satisfaction. Older people will have less tendency to let the attractiveness of the software product influence their satisfaction with the system. Another moderating effect on this relationship is hypothesized to be personal use of mobile- and web apps as exposure to mobile apps makes these people deal with lots of different looking interfaces and designs which in turn makes them more indifferent to the aesthetics and inconsistencies in aesthetics of the software. As is shown in figure 8 we do expect that age moderates the effect attractiveness has on user satisfaction to stay within the scope of the study we will only incorporate the personal use of mobile- and web apps in the hypothesis.
H9: The influence of attractiveness on user satisfaction is moderated by personal use of mobile- and web apps, such that the effect is weaker for people who make more personal use of mobile- and web apps
2.3.4.5 Helpfulness
The ISO 25010 standard (2011) tells us that helpfulness is the degree to which the software product provides help when users need assistance. It notes that this includes help that is easy to find, comprehensive and effective. Helpfulness reflects the user’s perception that the software communicates in a helpful way and assists in the resolution of operational problems (Cavallin et al, 2007, p. 230). We believe that age and use of software moderate the impact of helpfulness on user satisfaction. Younger people tend to be more resourceful when looking for solutions and will depend less on help that the software offers. In turn older people will be more satisfied with the system when the helpfulness is perceived high. People who make more use of the software are likely to encounter more problems and therefore are expected to attach more value on helpfulness as an effect on user satisfaction. We
26
however do not expect that the personal use of mobile- and web apps plays a role in influencing the effect helpfulness has onuser satisfaction since both people who use them a lot or little will have the same effect the helpfulness of a system has on their satisfaction with that system. To test our expectations in the research model shown in figure 8, we propose the following hypothesis:
H10: The influence of helpfulness on user satisfaction is not moderated by personal use of mobile apps.
Investigating in how usability has been conceptualized and operationalized has led us to answer our second sub question, ‘How can we operationalize perceived usability?’. There isn’t an easy answer to this question as it comes down to which perspective the study requires to research usability. In this study the factors understandability, learnability, operability, attractiveness and helpfulness are chosen to represent usability. We have devised hypotheses for perceived usability, user satisfaction and the relation between the dimensions of perceived usability and user satisfaction which answers our third sub question. The following chapter will describe the chosen methodology and the steps taken to test the above stated hypotheses.
27
3. Methodology
In this following two chapters we will try to answer the fourth sub question which is: Can we empirically validate our predictions in regard to the users changing usability perceptions in information systems in work settings due to the use of mobile- and web applications? To be able to validate our predictions we have to set up a research effort for collecting data that we can then use to analyze. In order to test the hypotheses from the previous chapter we will conduct a case study with the users of a single software program. A quantitative method for obtaining data is chosen of which the instrument will be a questionnaire. Questionnaires are a valid method for obtaining information about opinions and perceptions of people by tapping into their subjective feelings. By investigating a certain sample of people it will allow generalizations of a larger group of people. An important consideration to be able to generalize is that the sample is randomly chosen, meaning every potential respondent has an equal chance of being selected to participate (Fowler, 2008).
3.1 Usability evaluation practices
Throughout the practice of evaluating usability different methods and metrics have been developed to measure whether a software product is usable. In general there have been three ways of researching this. The first one tries to objectively measure actual performance by observing users interacting with the product and analyzing metrics from these observations. In the second method practitioners collect subjective data on user opinions and attitudes. Finally the third method uses experts to evaluate the software product according to guidelines and expert experiences and opinions (Madan & Dubey, 2012). The first is generally conducted in laboratory settings on prototypes or products still in development and can reveal a lot about the internal quality of the software product in order to better facilitate usability engineering. This method needs an elaborate setting and is time consuming as each user is studied while interacting with the software product. Some well-known examples for this are the ‘Metrics for Usability Standards in Computing (MUSiC)’ as used by Macleod et al. (1997) and ‘Quality in Use Integrated Measurement’ (QUIM) (Seffah et al, 2006). The second way of researching usability can be conducted in more general settings and is much less expensive and time consuming. With survey data collection through questionnaires we can find out more about how the user experiences usability as an external quality or quality in use. Several questionnaire methods have been developed regarding usability measurement of which the most notable are SUMI (Software Usability Measurement Inventory (Kirakowski & Corbett, 1993)), QUIS (Questionnaire for User Interaction Satisfaction (Chin et al, 1988)) and CSUQ (Computer System Usability Questionnaire (Lewis, 1995)). In the third method inspection of experts is used to identify weak points in the usability of the software product which should be improved in order to improve the external quality of the software product. This method is generally used in an early phase with practitioners that develop software (Madan & Dubey, 2012). In our study a questionnaire seems to be the most appropriate method for obtaining data as the scope doesn’t allow for an elaborate laboratory testing method and the observation by experts will not give us any insight on the influence of mobile- and web apps. Next to that a questionnaire is efficient, saves time and costs and can tell us enough about user attitudes and opinions.
