Why do information systems in housing
associations not satisfy expectations?
By
Niels Winkel s1737732 n.winkel@student.rug.nl
November 23, 2009
Master thesis, MscBA, specialization Business & ICT University of Groningen, Faculty of Economics and Business
University of Groningen
Faculty of Economics and Business
Supervisor: Prof. dr. H.G. Sol Examiner: Prof. dr. E.W. Berghout
CEPO
2
TABLE OF CONTENTS
Abstract... 3
1
Introduction ... 4
2
Theoretical background ... 7
2.1 Dimensions of information system success... 7
2.1.1 Evaluation of information system success dimensions ... 8
2.1.2 Conclusion concerning relevant dimensions of information system success .... 10
2.2 Measures of information system success dimensions ... 11
2.2.1 Measuring stated and implied expectations ... 12
2.2.2 Measuring information system product quality ... 12
2.2.3 Measuring realization of net benefits ... 14
2.3 Research question and sub-questions ... 19
3
Case research ... 21
3.1 Research approach ... 21 3.2 Case 1: HAX... 24 3.2.1 Interpretation of findings... 26 3.3 Case 2: HAY... 35 3.3.1 Interpretation of findings... 364
Conclusions and discussion... 46
4.1 Why do information systems in housing associations not satisfy expectations? ... 46
4.1.1 Quality conform expectations?... 46
4.1.2 Benefits conform expectations? ... 48
4.1.3 Conclusion regarding how information systems can satisfy expectations ... 48
4.2 Recommendations ... 49
4.3 Discussion... 51
References ... 52
3
ABSTRACT
4
1
INTRODUCTION
Reliability of the information services within a housing organization is crucial. The organization takes important operational, tactical, and strategic decisions based upon the information retrieved from their information system. Therefore, an information system is of high importance for a housing association. Due to mergers, the number of housing associations is decreased from 767 in 1997 to 455 in 2008. All housing associations together own 2.4 million houses, which is 32 percent of all houses in the Netherlands (VROM, 2009a). The main aim of housing associations is to provide social rental housing, essentially for lower income groups. To cope with the social responsibilities and aforementioned mergers, housing associations demand more sophisticated information systems. They need better management information for managing the organization and answering questions from the government and other stakeholders (VROM, 2009b; DSA-Vision, 2007). In addition, a frequently occurring change from activity-based working towards process-based working to improve performance requires new information systems. Furthermore, housing associations intend to improve performance by moving from different small discrete information systems to one integrated information system. To keep up with these changing needs, research by PriceWaterhouseCoopers (2008) showed that 86 percent of the Dutch housing associations state to reconsider their current information system within five years. Software vendors have developed packaged information systems to fulfill the changed and future needs of these housing associations. For example, Itris (2009) states they deliver an all in one solution that is ready for the customer’s future needs. SAP (2009) states that with their solution, a housing association acquires a complete and integrated information system that allows the housing association to manage, control, and analyze its real estate portfolio and its internal processes. Despite of the promised benefits of the new information systems, CEPO beyond ICT1 beholds that housing associations have difficulties with gaining benefits. Functions are not adequate for the required task, management information cannot be generated, or users need to increase effort for the same work, are examples of issues stated by consultants of CEPO.
The aim of the acceptance phase of a project is to identify these issues and allowing the organization to take appropriate actions before going live (Koomen, Aalst, Broekman, & Vroom, 2006). However, within the limited time and budget, not all aspects of the information system can be tested. In addition, the users performing the acceptance tests have only limited experience with the new information system and changed organizational processes. Consultants (e.g., CEPO consultants) can use their experience from former projects to improve this process; however, every project has its own unique aspects (Pollack, 2005; Rosqvist, Koskela, & Harju, 2003). In addition, a thorough analysis of the issues after going live is limited because project evaluations are rarely undertaken (Berghout & Nijland, 2002;
1
5
Clegg, 2000; Henderson, Venkatraman, & Oldach, 1993). Housing associations are more concerned about learning to work with the new information system and caching up left work than evaluating an historical event. Nevertheless, Clegg (2000) and Solingen, Berghout, Kusters, and Trienekens (2000) stress that evaluations should take place because it is essential for learning. Although much research is available in the field of overcoming IT project failures (e.g. Boddy, Boonstra, and Kennedy, 2005; Benson, Bugnitz, and Walton, 2004), projects still do not meet up to stated expectations. Therefore, the objective of this research is: Increase understanding of differences between upfront and exposed expectations and define guidelines for housing associations to harvest expected benefits from a new information system and guidelines for CEPO to improve its services.
The main research question derived from the research objective is: Why do information systems in housing associations not satisfy expectations?
Types of sub questions one can expect by this research question are related to whether or not the information system performs as expected and has created the expected organizational impact. All sub questions of this research are defined after the literature review in chapter 2.3: Research question and sub-questions.
6
information systems by housing associations acknowledge that projects are implemented using the waterfall approach. Therefore, we will leave the implementation methodology out of the scope of this research.
7
2
THEORETICAL BACKGROUND
In this chapter, literature concerning information system satisfaction is discussed. First, the in literature mentioned dimensions for evaluating success of an information system are elaborated. In addition, these dimensions are evaluation to provide insight in the relevance of these dimensions for this research. Following, an overview is given how these relevant dimensions can be measured. Finally, based upon the insights from the literature, research sub-questions are outlined and a corresponding conceptual model is depicted.
2.1
Dimensions of information system success
Gable, Sedera, and Taizan (2008) state that among researchers little consensus exists on how best to measure the impact of an new information system in organizations. Another writer on this issue (Sabherwal, Jeyaraj, & Chowa, 2006, p. 1849) claims that “[d]espite considerable empirical research, results on the relationships among constructs related to information systems (IS) success, as well as the determinants of IS Success, are often inconsistent”. On the other hand, searching on EBSCOhost and Google’s aggregated article database (Google scholar), one can see that most of literature for evaluating the success of an information system have referred to, or made use of, the information system success model of Delone and McLean (2003). This observation is reflected by different authors (Alter, 2008; Chi-Yuan, Ray-E, Ming-Chien, & Mei-Hsin, 2009; Chung & Kwon, 2009; Palmius, 2007; Wang & Wang, 2009) who state that this model is one of the most widely used model for evaluating the success of the implementation of an information system. According to Delone and McLean (2003) six dimensions are of influence by evaluating the success of an information system (Table 1).
Table 1: Dimensions of information system success Delone and McLean
(2003)
Boddy, Boonstra, and
Kennedy (2005) Palmius (2007) Project effectiveness X System quality X X X Information quality X X X Service quality X Use X X User satisfaction X X X Net benefits X X X
8
judge the success of an information system (Table 1). In addition, Boddy, Boonstra, and Kennedy mention the dimensions project quality as potential dimension to take into account. According to a literature review on information system evaluation by Palmius (2007), most of the literature is focused on system and information quality, user and / or customer satisfaction, and net benefits (Table 1).
2.1.1 Evaluation of information system success dimensions
In this section, the dimensions of information system success outlined in Table 1 will be explained and discussed whether they are relevant within the context of this research. The DeLone and McLean model will be used as guide in since in the previous section this model is indicated as most widely used model. In addition, different authors have assessed this model for being useful by providing a better understanding in what constitutes effective information systems (Petter & McLean, 2009; Wang & Liao, 2008).
This section is outlined as follows: first project effectiveness will be discussed, which is not a part of the DeLone and McLean model. Second, the three independent quality variables of the DeLone and McLean model (Figure 1) will be discussed. Following, the two dimensions related to users, system use and user satisfaction, will be discussed. Finally, the net benefits will be elaborated and conclude how the relevant dimensions can be used within this study.
Figure 1: Updated information systems success model (Delone & McLean, 2003)
Project effectiveness
9
system is an investment for the long-term, project effectiveness is indicated as an insufficient measure for information system success.
Quality dimensions
Delone and McLean (2003) advocate for the use of three quality dimensions by evaluating information systems, system quality, information quality, and service quality. System quality refers to the desirable characteristics of an informant system and can be measured by the performance of an information system in terms of reliability, ease of use, functionality, and other system metrics. Information quality refers to the desirable characteristics of the output of an information system, such as accuracy, traceability, and timeliness. All referred authors shown in Table 1 mention system and information quality as possible dimensions of information system success, which indicates the importance of these dimensions. In addition, if an information system does not provide desired characteristics and output, it is unlikely that expectations will be satisfied (Saarinen, 1996). In contrast to system and information quality, only Delone and McLean mention service quality as important dimension of information system success. They define service quality as the quality of the support that users receive from the IT department. Delone and McLean introduced service quality in their updated information success model (Figure 1) because frequently “the users are now our customers and poor user support will translate into lost customers and lost sales” (Delone & McLean, 2003, p. 23). In addition, they state that for a single information system without customers who directly interact with the information system, service quality is merely a subset of system quality. Therefore, the context should dictate whether service quality should be taken into account (Delone & McLean, 2003).
In conclusion, of the quality dimensions, information quality and system quality are important within the context of this research. The system quality and the information quality together are sometimes labeled information system product quality (Garvin, 1984; Saarinen, 1996). A specific dimension for service quality is less relevant since system quality will cover the relevant aspects within this context.
Use and user satisfaction
10
Even though system use is not a proper dimension within the context of this research, a quality profound information system that is used may not satisfy users (Boddy, Boonstra, & Kennedy, 2005). DeLone and McLean (1992) define user satisfaction as the recipients approval or likeability of an information system and its output. Au, Ngai, and Cheng (2008) define user information system satisfaction as “the IS end-user’s overall affective and cognitive evaluation of the pleasurable level of consumption-related fulfillment experienced with IS”. This definition reveals that user satisfaction is not limited to stated quality expectations; however, also incorporates implied expectations. In contrast to use, all in Table 1 referred authors mention user satisfaction as possible dimension of information system success. In addition, Au, Ngai, and Cheng (2002) state that user satisfaction is probably the most widely used surrogate of information system success.
In summary, use will not be used as measure for information system success. For measuring user satisfaction stated and implied expectations concerning the quality dimensions are indicated as relevant.
Net benefits
All in Table 1 referred authors mention net benefits as dimension of information system success. Realization of benefits is important since an information system of high quality does not automatically lead to a successful information system (Au, Ngai, & Cheng, 2002; Duggan, 2005). As Duggan and Reichgelt (2006, p. 16) state: “the garbage heap of failed information systems contains many brilliantly conceived and technically sound systems”. Delone and McLean (2003) define net benefits as the effect an information system has on an individual, group, organization, industry, etc. They stated that the purpose of the information system must be taken into account to determine on which entity the impacts should be measured. Seddon (1997) supplements to the previous definition by arguing to makes a distinction between past and expected future benefits. Finally, Byrd, Lewis, and Bryan (2006, p. 452) define net benefits as: “the benefits achieved by the organization through using the information system and its impact on the work processes of the organization”. Evaluating these definitions, one can see that all definitions stress the impact the information system has on a particular entity.
To sum up, net benefits is an important dimension of information system success. The vital aspect for realization of net benefits is whether the information system has an impact on a particular entity.
2.1.2 Conclusion concerning relevant dimensions of information system success
11
cases of voluntary system use. Since the information systems subject to research are obligatory to use, use is indicated as irrelevant for the context of this research.
From the quality dimensions, system quality and information quality are indicated as important dimensions since an information system should provide desired characteristics and output to the user. These two quality dimensions together are indicated to represent product quality. A satisfied use will indicate that an information system fulfils the user’s needs and is therefore indicated as relevant dimension. For measuring user satisfaction stated and implied expectations should be taken into account. Finally, net benefits are the goal to reach and are related to whether or not the information system has an impact on a particular entity.
Not all dimensions of the DeLone and McLean model for information system success are indicated as suitable for the context of this research. However, this does not mean that the DeLone and McLean model is invalid and that we cannot explain information system success with these dimensions. As stated by the authors of the model, in order to develop a comprehensive measurement model and instrument for a particular context, the constructs and measures should be systematically selected considering contextual contingencies (DeLone & McLean, 1992; 2003). Most resent research that uses the DeLone and McLean model do indeed select only the relevant dimensions of the model (e.g. Byrd, Thrasher, Lang, and Davidson, 2006; Chi-Yuan, Ray-E, Ming-Chien, and Mei-Hsin, 2009; Chung and Kwon, 2009; Wang and Wang, 2009).
In the next section, measures for the relevant dimensions will be discussed.
2.2
Measures of information system success dimensions
In the previous section, relevant dimensions for the evaluation of information system success are indicated and evaluated. To measure these dimensions, the following three questions need to be answered:
• How to measure user satisfaction by evaluating stated and implied needs?
• How to measure system quality and information quality by evaluating product quality? • How to measure net benefits by evaluating whether the information system has an
impact on a particular entity?
12 2.2.1 Measuring stated and implied expectations
The first logical step for evaluating whether or not an information system matches expectations of its stakeholders is by comparing the implemented information system to the agreed upon requirements of the housing association. In other words, has the software vendor satisfied its customer by realizing agreed requirements? Therefore, a mismatch between agreed requirements and the product quality of an information system in production stage will be used to measure whether or not stated expectations are realized.
Lyytinen and Hirschheim (1987) have labeled the mismatch with agreed requirements as “correspondence failure”. Besides correspondence failure, they defined “expectation failure” as a category of information system failure. Expectation failure reflects to the inability of an information system to meet its stakeholders’ expectations, independent of its specifications (Lyytinen & Hirschheim, 1987). These implied expectations will be labeled “latent needs”; defined as implicit requirements of an information system that the stakeholders expected the information system would possess in the production stage. Therefore, a mismatch between latent needs and the product quality of an information system in production stage will be used to measure whether or not implied expectations are realized. Measures for product quality will be discussed in the next section.
2.2.2 Measuring information system product quality
Behkamal, Kahani, and Akbari (2009) and Hyatt and Rosenberg (1996) have both reviewed models for evaluating the product quality of an information system. Both concluded that the ISO 9126 model is more complete than other models (e.g. the McCall model and the Boehm model) and is free of shortcomings of others. The model is assumed to be built on best practices and is frequently used (Duggan & Reichgelt, 2006; Palmius, 2007). In addition, Chua and Dyson (2004) conclude that the ISO 9126 quality model “…is useful in moving beyond superficial evaluation to achieve a more thorough view of the system’s strengths and weaknesses than can be provided by less systematic approaches”. The ISO 9126 model makes a distinction between internal quality and external quality (ISO/IEC, 2001). The internal quality is not of interest for this research because the internal quality attributes do not rely on execution of the information system. The internal quality can only be measured during the development process (Botella, et al., 2004). In contrast, the external quality refers to the behavior of the running information system and is therefore relevant for this research. For the external quality, the ISO 9126 model specifies six characteristic including functionally, reliability, usability, efficiency, maintainability, and portability (ISO/IEC, 2001).
13
Traceability is categorized under functionality, because this research is focused at transaction processing systems that rely on the traceability of data. Reliability is extended with availability, because housing associations are information insensitive organizations that need the information system for most of the daily activities. By changing from discrete systems to one integrated information system, users are changing from activity-oriented applications to information systems that support the user’s activities as well as a broad, common feature set that can be utilized for any type of organization. In order to allow the users to adjust the information system to specific needs, customizability is considered relevant for usability. Finally, maintainability is extended with manageability and scalability. Manageability is considered relevant because keeping the information system operational is an important daily activity. If changes in the environment occur, it is important that an information system can be adapted (Solingen & Berghout, 1999; Duggan, 2005). Considering the number of mergers of the last years, scalability is regarded relevant for housing associations. In Table 2, these five added sub-characteristics are marked italic.
Table 2: ISO 9126 characteristics and sub-characteristics (adapted from: ISO/IEC, 2001) Characteristic Explanation
Functionality The capability of an information system to provide functions that meet stated and implied needs.
Suitability Availability of adequate functions for the required tasks.
Accuracy Accuracy of the output of the system, in accordance with input and specified transformation.
Interoperability The system's ability to exchange data with other systems. Standards Adhere to standards, conventions, or regulations.
Security The system’s ability to maintain data confidentiality, integrity, and availability.
Traceability The system’s capability for exploring the correctness of the information processing in different stages of a process.
Reliability The capability of an information system to maintain its level of performance.
Fault tolerance The ability to maintain a specified level of performance in case of software fault or of infringement of its specified interface.
Maturity The capability of the system to avoid failures and is over its teething troubles. Recoverability The easiness to recover the system after a breakdown, without losing data.
Availability Extent to which the system is available for users whenever the system is required.
Usability The capability of an information system to be understood, learned, used, and attractive to the user, when used.
Understandability The capability of the system to enable the user to understand the possibilities, and how it can be used for particular tasks.
Learnability The capability of the system to enable the user to learn its application. Operability The capability of the system to enable the user to operate and control it.
14
Characteristic Explanation
Efficiency The capability of an information system to provide appropriate performance, relative to the amount of resources used.
Time behavior Response time, processing time, and batch processing time. Resource
utilization
Amount of required resources (network capacity, disk space, memory, etc.)
Maintainability
The capability of an information system to be maintained and modified. Modifications may include corrections, improvements or adaptations of the software to changes in the environment and in the requirements.
Analyzability The capability of the system to be diagnosed for deficiencies or causes of failures in the software, or for the parts to be modified, to be identified Changeability Easiness to modify, correct, and improved the system.
Stability The capability of the software product to avoid unexpected effects from modifications of the software.
Testability Easiness to test and validate the correct working of the system.
Manageability Easiness to keep the system operational.
Scalability The capability of the system to scale with an increase in users, faster transaction times, storage capacity, etc.
Portability
The capability of an information system to be transferred from one environment to another. The environment may include organizational, hardware, or software environment.
Adaptability The capability of the system to be adapted to different environments or a new version.
Installability Easiness and speed required to install and uninstall the system. Conformance The system’s conformance to technical standards.
Replaceability Easiness to replace other systems, and ability of the system to connect to organizational processes and (manual) procedures.
In conclusion to measuring information system product quality, the ISO 9126 model is indicated as most appropriate because it provides a more thorough view of the information system’s strengths and weaknesses compared to other models. As explained in the previous section, product quality is composed of system quality and information quality. Reflection on the ISO 9126 characteristics, one can see that most characteristics are related to system quality. For the reason that this research is focused at transaction processing systems, most functionalities of these information systems is related to processing the data and not presenting the data. These functions rely on system quality. In cases of transforming the data in information, the information must obvious be accurate, reliable, and available when needed. Since this aspect is also covered by the ISO 9126 characteristics, this model fulfils the requirements for measuring product quality.
2.2.3 Measuring realization of net benefits
15
on financial performance. However, Au, Ngai, and Cheng (2002, p. 452) indicate that “it is difficult to show causality, i.e., to prove that a particular benefit is directly or solely due to the new information system”. In addition, Hirschheim and Smithson (1988) argue that costs and benefits of an information system are largely intangible or qualitative in nature. Finally, Willcocks (1992) state that increasingly answers concerning the financial consequences of an information system produces unsatisfying answers. Therefore, financial performance is unusable in this context to measure net benefits.
Strassman (1999) goes further by arguing that benefits come from reshaping the organizational factors, not from spending on IT. An information system of good quality must be able to support the requirements of the organization; however, only within the organization benefits can be realized (DeLone & McLean, 1992; Seddon, 1997; Willcocks, 1992). Therefore, to translate information system quality into information system success, there must be alignment between the changed IT structure and the organizational structure (Au, Ngai, & Cheng, 2008; Duggan & Reichgelt, 2006). Benson, Bugnitz, and Walton (2004, p. 150) state, “Getting business and IT aligned is a prerequisite for delivering IT value and is on the critical path for using IT to improve business performance”. In addition, Henderson and Venkatraman (1993) state that misalignment between the business domain and the IT domain is a possible cause for not realizing expectations.
As stated in the previous section, the vital aspect for realization of net benefits is whether the information system has an impact on a particular entity. Reflecting on this description, taking into account the above-mentioned possibilities for measuring net benefits, the entity under research will be the organizational processes. In addition, the impact can be measured by alignment between the organizational processes and the information system. Therefore, below theory will be discussed to create and measure alignment and impact in the organization.
Creating alignment
16 Int er na l do m ai n E xt er na l dom ai n
Figure 2: Strategic alignment model (adapted from Henderson and Venkatraman, 1993)
Placing the alignment theory in the context of this research, changes in the external domain will result in a new strategy. To be able to execute the new strategy, the internal domain should be altered. In the case of a new information system, this implicates that the infrastructure and processes of the IT domain will change. To maintain alignment after one domain is changed, the other domains should be updated in a synchronized fashion (Sabherwal, Hirschheim, & Goles, 2001). However, Henderson and Venkatraman do not describe when optimum alignment is created (Sabherwal, Hirschheim, & Goles, 2001). Shang and Seddon (2003) researched this theory in actual practice by evaluating four cases about information systems in utility companies that where in use for at least four years. The characteristics of these cases bear resemblance to information systems at housing associations; not only the line of business is similar to housing associations, also the software vendor of three of the cases, SAP, offers information systems for housing associations. According to Shang and Seddon (2003), four different approaches result in alignment: change organizational processes, change the information system, change both, or change none of them. Changing the information system relates to configuring the information system, designing new reports, and changing actual software code. Shang and Seddon conclude that after three years, all four organizations had changed both, the information system and organizational processes to create alignment. In addition, perceived benefits rose after alignment was created. Therefore, the approach of a housing association for creating alignment is considered relevant for realizing expected benefits.
Creating impact in the organization
17
Boonstra (2009) state that this requires a different approach. They argue that this should be handled as technology-driven organizational change, labeled “technochange”. Harison and Boonstra (2009, p. 283) stress that technochange “has been broadly been used in the technology and project management literature, as well as in other organizational contexts”. Markus (2004) was the first to coin the term technochange. She distinguishes three approaches for organizing a project knowing; IT project, technochange, and organizational change. IT projects focus on improving technical performance, leaving out the impact on the organization. Although it seems logical to involve the organization, projects are still started as initiative of the IT department with limited involvement of the organization (Au, Ngai, & Cheng, 2002; Markus, 2004). A different type of project compared to an IT project is an organizational change approach, focused at improvement in organizational culture and performance. Sometimes, the IT project approach is combined with an organizational change approach to get the best of both worlds. According to Markus (2004), this approach can produce better results than an isolated IT project. However, it suffers from lack of integration between IT and organization. Markus (2004) indicates that “The real payoffs of technochange come from creating new products and processes that exploit the potential of IT, rather than just fitting IT to the way processes are currently done”. Therefore, a technochange approach is preferable where tasks, jobs, and organizational processes all change along with IT. Table 3 summarizes the different characteristics of the three approaches.
Table 3: Technochange versus IT projects and organizational change programs (based on: Markus 2004)
IT project Technochange Organizational change
Target outcomes
Technology performance, reliability, cost of operation and maintenance, within project schedule and budget.
Improvement in organizational performance.
Improvement in
organizational culture and performance.
Solution New IT. New IT, in conjunction with
other changes.
Interventions focused on people, structure, culture and HRM.
Example Replace old system by a new one.
Replacing old procurement into e-procurement.
Transform or reorganize mature organization that is underperforming by making people more result oriented.
Approach IT project structure, led by IT project manager, who is expected to deliver on time and within budget.
Program of change efforts, including IT change. Also process restructuring, reward systems, training etc.
Organizational
18
IT project Technochange Organizational change
Role of managers
Oversight, to approve, provide, initiate, providing input.
Leadership, to initiate the project, act as sponsor, explore process options enabled by the new technology, etc.
Leadership, to initiate and change.
Role of IT experts
Central, to perform the project management role and most of the project labor. Coordinate with business managers, vendors and external consultants.
Central, to work together with organizational managers and other specialists to design a technochange in which the IT part meshes with other changes.
Negligible.
Role of other experts
Technology vendors and consultants may perform various tasks.
Internal staff specialists, external management, technology consultants and technology vendors, may all play key roles.
Internal HRM
management, external development consultants often perform key roles.
The major risk in technochange that Markus (2004) describes is the risk of users that will either not use the technology or use the technology without capturing the expected benefits. Therefore, it is relevant to reveal whether or not the users use all functionalities as expected and to ensure the functionalities of the information system and the activities of the users are aligned.
Conclusion regarding measuring realization of net benefits
19
2.3
Research question and sub-questions
As stated in the introduction, the main research question is: “Why do information systems in housing associations not satisfy expectations?”
Sub-questions deriving from the main research question and the theoretical background are: 1. Is perceived quality of an information system in production stage conform
expectations?
o Did stakeholders perceive issues with realization of agreed requirements? o Did stakeholders perceive issues with realization of latent needs?
2. Do housing associations realize expected benefits of an information system? o Do housing associations focus on aligning the information system with
organizational processes?
o Do housing associations manage the implementation as technochange project? Based upon the literature review concerning dimensions of information system success, three dimensions were indicated as relevant within the context of this research: product quality, user satisfaction, and net benefits. The first sub-question is derived from the product quality dimension. The two related sub-questions of the sub-question are related to the perceived usefulness. First, according to literature stated expectations can be measured by the gap between agreed requirements and product quality. Second, the implied expectations can be measured by the gap between latent needs. As depicts the conceptual model in Figure 3, realization of agreed requirements and realization of latent needs influence product quality. This is in contrast to the DeLone and McLean model (Figure 1) where perceived usefulness (measured by agreed requirements and latent needs) influence the quality dimensions.
20
However, the aim of this research is to elaborate issues with the quality; in contrast to DeLone and McLean, who use the quality as given fact and aim to predict the user satisfaction.
21
3
CASE RESEARCH
In the previous chapter, we described literature concerning information system satisfaction and how this can be measured. Based upon the literature, research sub-questions where developed and depict in a conceptual model. In this chapter, the theory is examined using two cases focused on answering the research sub-questions. Therefore, first the research approach is explained. Next, for both cases a description of the case is given, followed by a discussion of the findings related to the two research sub-questions. In the first case, we provide an analysis on a large housing association named HAX. This company has initiated a sophisticated selection phase in 2005. In the second case, we discuss a small housing association named HAY. This organization has started with a small selection phase in 2003.
3.1
Research approach
The initial research objective of this paper is to provide insight into how an information system can match expectations of its stakeholders. To explore the supposition that information systems in housing associations do not match expectations of its stakeholders, a literature review is conducted to identify theories earlier authors considered relevant for evaluation of information systems and realizing expected benefits. Sub questions are formulated derived from the main research question and literature. Since this research is exploratory, intend to indentify why information systems not satisfy expectations of housing organizations, a qualitative case study approach is used (Walsham, 1995).
22
Figure 4: Overview of research approach
Several authors (Boonstra, 2003; Boonstra & Vries, 2005; Garvin, 1984; Palmius, 2007) indicate the importance of taking into account the diverse and conflicting interests of different stakeholders by evaluating an information system. For example, a financial manager will likely judge on direct cost savings and payback period, a user will focus on support of his work, and a business manager will be more concerned about quality improvements and cost reductions (Boddy, Boonstra, & Kennedy, 2005). To cover all important interests, the alignment theory (Henderson & Venkatraman, 1993) is used. The alignment theory advocates that all four domains should be aligned; therefore, a minimum of one representative for every domain was interviewed. For the business domain, a business manager and end-user were interviewed. To cover all important operational activities supported by the information system, one can interview an end-user of every important business process or interview an representative with a diverse knowledge of the important business processes and day-to-day issues. Due to resource and costs issues by the case organizations, we have chosen for the latter option. For the IT domain, an ICT manager and an ICT administrator were interviewed. Since the organization of the second case is a small organization with only one employee responsible for ICT, this ICT coordinator represented the position of both, ICT manager and ICT administrator.
23
Akbari (2009), the interviewee was required to assign a total of 100 percentage points to the six ISO 9126 characteristics and divide the points among the corresponding sub-characteristics (Table 2) in advance of the interview. For this task, a higher percentage on the characteristic would indicate a higher relative importance. Based on the theoretical background and the knowledge of the case, interview questions where developed. The first part of the interview was focused at discussing issues the interviewees experienced with the information system and determining the origins of issues. The characteristics and sub-characteristics of the ISO 9126 model were used since in the theoretical background this model was identified to be useful for assessing the quality of an information system. Because the quality of the current information system is evaluated, the interviewee was asked to indicate his current perception regarding relative importance of a characteristic. During the interview, the interviewee was asked, for every sub-characteristic he assigned points to, whether or not the characteristic was conform expectation. To determine the cause of revealed issues, the interviewee was encouraged to indicate whether the issue was related to not realizing agreed requirements, latent needs, expected benefits, or another cause. The second part of the interview was focused at the approach of the organization with respect to aligning the information system and the organization. Questions were asked regarding the preparedness of the organization to customize the information system and the preparedness to change the organizational processes. Furthermore, alignment between the functionalities of the information system and the activities of the users was discussed. For example, was the information system altered, the organizational processes changed, users trained, etc? Finally, the characteristics of the implementation project were discussed using the project characteristics as described by Markus (2004) (Table 3). Depending on the role of the interviewee in the organization and in the implementation project, not all questions concerning alignment were asked. Experiences of the interviews of the first case are used to improve the interviews of the second case. The interviews lasted between one and one and a half hours each. Appendix A lists the main interview questions.
24
3.2
Case 1: HAX
The housing association researched for this case will be referred to as “HAX” (Housing Association X). HAX is a housing association in the Netherlands with more than 24.000 tenantable units and over 300 employees. These numbers indicate that HAX ranks in the top 20 of housing associations. HAX has multiple divisions serving their own region. In the vision and mission 2006 – 2010, HAX states to sharpen some of its previous chooses. However, the organization will not make radical reforms. With respect to ICT, HAX states that the potentials of ICT will be leading in improving organizational processes. The organization used about 30 independent applications, including one primary system. The organization and the ICT architecture were organized to efficiently accomplish independent activities. However, efficient activities do not guarantee an efficient process. Therefore, redundant work and a lack of knowledge sharing between related activities are some of the problems the organization wanted to tackle. To be able to realize process improvements, HAX has implemented a new integrated information system that supports all of its main processes.
For selecting the most appropriate information system, HAX has roughly described its requirements and distinguished its most distinctive requirements. First, the information system should support all specific functionalities of the housing association sector and common functionalities like purchasing and human resource management. Second, the information system has to support the following requirements:
• Transparency by storing all information in one information system;
• Uniformity by means of integrating all processes in one information system; • Flexibility by means of an information system that can be used in different
departments without the need to adjust the code for every department;
• (Management) information provision by saving all data in one information system; • Integration by incorporating all processes and information flows in one information
system.
Furthermore, the technology of the information system should be state of the art and support the following technologies:
• The information system uses a n-tier architecture, which assures that the user will always see the same user interface;
• The information system uses the service-oriented architecture concept. This makes it simple to integrate with external systems;
• The information system is web-based to provide access to customers, suppliers, stakeholders, and employees.
25
cases and the total cost of ownership of the information systems, HAX selected its new information system. Following the selection phase, power users (users with above average knowledge of business processes) and consultants of the implementation partner created blueprints of the new system. These revealed the to-be organizational processes in conjunction with the possibilities of the new information system.
The implementation of the functionalities of the information system was divided into two phases. In the first phase, all functionalities offered by the old systems were replaced by the new information system plus some quick wins. The first phase, including the selection phase, was completed in approximately three years, including a delay of five months and some small limitations. For example, the option of annual rent increase was not yet implemented and information provision concerning maintenance projects was incomplete. HAX was able to resolve these limitations in the next two months. The delay resulted from misunderstandings concerning what the software vendor offered and what the implementation partner implemented. At the end of this year, HAX will start with the second phase of the project. In the second phase, HAX intends to implement new functionalities and change organizational processes. This phase is estimated to take another three years. Because the second phase is in the future, this phase is not subject of research. However, the fact that in this phase the organization focuses on optimizing the information system to be able to realize expected benefits, will be kept in mind.
Within the project organization (Figure 5), all organizational processes supported by the new information system were assigned to teams. Every team consisted of power users and a
26
chairman. The chairman of the team communicated with the lead consultant of the implementation partner. The lead consultant managed his team of consultants and was responsible for changing the information system, trainings, supporting tests, etc. The organization and the implementation partner both had their own project leader. The steering committee consisted of a chairman of HAX and representatives of the most important activities of HAX. Both the implementation partner and the software vendor were represented in the steering committee.
3.2.1 Interpretation of findings
In this paragraph, the results of the interviews with the four employees of HAX are discussed by way of our two research sub-questions, knowing:
• Is perceived quality of an information system in production stage conform expectations?
• Do housing associations realize expected benefits of an information system?
Perceived quality of the information system conform expectations of HAX?
For the exploration of this question, the ISO 9126 characteristics are used. As described in the research design, the interviewee was asked to indicate the relevance of all ISO 9126 sub-characteristics by dividing 100 percent points. Figure 6 depicts and overview of the indicated relevance of the main ISO 9126 characteristics. For each characteristic the percentage points of the three stakeholders are summed up. The graph depicts a total of 400 percentage points (100 percentage points from every stakeholder) divided over the six ISO 9126 characteristics. Therefore, one characteristic can receive more than 100 percentage points. When the interviewees of HAX would have assigned all characteristics an equal number of points, every characteristic would have received 67 percentage points. Hence, the indicated relevance of
0% 20% 40% 60% 80% 100% 120% 140% S u m o f in d ic a te d r e le v a n ce
Hax: ISO 9126 characteristics
ICT manager ICT administrator Business manager End-user
27
functionality and maintainability is relative high since they are above average. Consequently, the relevance of reliability, usability, efficiency, and portability was indicated below average. These six characteristics and their percentages will be discussed in the next section along with the given values for the sub-characteristics. In addition, issues with realization of agreed requirements and latent needs regarding characteristics with an indicated relevance are discussed.
Perceived functionality
All stakeholders have indicated the functionality relatively important (Figure 7). For the end-user suitability and accuracy are most important. With respect to suitability, the end-end-user is satisfied; however, the new information system requires entering more data. The user understands this is caused by the extra need of the data in other processes and that the second phase of the project will decrease the extra needed time. The information system’s accuracy delivers a satisfying expectation now; however, according to the end-user there have been issues after going live. The reasons varied from a not described requirement to a completely overlook function. In all cases, it could be solved by reconfiguring the information system. Overall, the end-user is satisfied with the functionalities of the information system.
Functionalities of the system are one of the most important aspects the business manager has focused on during selection. The business manager has judged all sub-characteristics as relevant. In the current situation, the business manager is very satisfied with the quality of the functionalities of the information system. With respect to accuracy, the business manager remarks that most issues are not related to the information system but to the users themselves. Users still need to habituate the new procedures embedded in the information system. To resolve this issue, users are still supported and trained.
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% End-user Business manager ICT administrator ICT manager In d ic a te d r e la ti v e r e le v a n ce o f ch a ra ct e ri st ic
Hax: Functionality
Traceability Security Standards Interoperability Accuracy Suitability28
The ICT administrator perceives functionality as being the least important compared to the other stakeholders. Furthermore, the ICT administrator does not perceive any issues with respect to the sub-characteristics of functionality. The ICT manager perceives the suitability as very important because the ICT department must offer the business the functions that it needs to perform its activities. All issues concerning the suitability are resolved by changing the information system or the organizational processes. The interoperability is above expectations because the software vendor is well known. According to the ICT manager, other software vendors are therefore more willing to change their system, which omitted the need of custom-made software. In addition to interoperability, the security of the information system is above expectations. Therefore, the ICT manager states to be satisfied with functionality. In summary, we can conclude that the stakeholders are satisfied. There have been some issues with suitability and accuracy related to latent needs. Because these issues were relatively easily corrected by reconfiguring the information system, the issues with latent needs is assumed not to influence satisfaction. Despite the extensive training during and after implementation, users still need extra time to become accustomed to the new information system. This is not an issue with the perceived quality of the information system; although it is a limitation for realizing expected benefits.
Perceived reliability
The indicated relative importance of reliability is rather low compared to functionality. Although it seems important to have a reliable information system, the interviewees assumed that any information system would always be reliable. The business manager explained that the perception of functionality differs between organizations; however, every organization will expect a decent level of reliability. Consequently, insufficient attention is given to reliability at the start of the project. Because the ICT administrator is responsible for maintaining the reliability, he only perceives reliability as relatively important (Figure 8). The end-user and ICT administrator do not mention any issue. In accordance with the issues concerning accuracy of the functionality, the business manager stated that issues related to fault tolerance, are mainly related to users and not the information system. With respect to maturity, the business manager states that a number of concepts were perceived as common; therefore, they were not mentioned at the start of the project. Because HAX is the first housing association implementing this system, the expectation was that these issues could occur.
29 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% End-user Business manager ICT administrator ICT manager In d ic a te d r e la ti v e r e le v a n ce o f ch a ra ct e ri st ic
Hax: Reliability
Availability Recoverability Maturity Fault tolerance Figure 8: Indicated relevance of the sub-characteristics of reliability, per stakeholderTo conclude, the reliability is conform agreed requirements because the organization did not write down many requirements. In accordance with the conclusion regarding perceived functionality, the issues are related to users and not the information system. This is not related to latent needs, but limits realization of expected benefits.
Perceived usability
Except the ICT manager, all stakeholders indicated usability relatively important (Figure 9). The end-users indicated relevance of learnability is rather high considering that this should be only an issue at the first months of usage. However, the end-user stated that he indicated learnability relative important because of the perceived difficulties with learning to operate the information system. The business manager stated that an easy to learn information system increase the ability to flexible assign tasks of an ill employees to someone else, which explains the relative high percentage. The end-user is very satisfied with the customizability, which is above expectations. However, the end-user is less satisfied with the other characteristics. In general, this group of users has difficulties with functions of the information system that are not regularly needed. The learnability of the most used functions of the information system is conform latent need.
30
Benson, Bugnitz, and Walton (2004, p. 151) who state, “One of the hardest things to do in any business is stop doing things that are currently in place”.
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% End-user Business manager ICT administrator ICT manager In d ic a te d r e la ti v e r e le v a n ce o f ch a ra ct e ri st ic
Hax: Usability
Customizability Operability Learnability UnderstandabilityFigure 9: Indicated relevance of the sub-characteristics of usability, per stakeholder
The ICT manager perceives all three characteristics conform agreed requirements. To realize latent needs, the understandability should be improved. This topic will be covered in the second phase of the project.
To conclude, although realization of the end-user’s perceived operability, learnability, understandability was not conform latent needs, these are not presumed to be related to the information system. As the business manager outlines, new employees are able to operate the information system without many difficulties. However, issues that have arisen concerning the end-users are related to old work habits. At the moment, this is limiting realization of expected benefits; however, improvements will be implemented in the second phase of the project.
Perceived efficiency
31 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% End-user Business manager ICT administrator ICT manager In d ic a te d r e la ti v e r e le v a n ce o f ch a ra ct e ri st ic
Hax: Efficiency
Resource utilization Time behaviorFigure 10: Indicated relevance of the sub-characteristics of efficiency, per stakeholder
In summary, the perceived efficiency is conform agreed requirements and latent needs. Perceived maintainability
All stakeholders perceive some of the aspects as important (Figure 11). The end-user stated that maintainability was slightly relevant since the characteristic of the information system to avoid unexpected effects and, in case of issues, resolve the issues effectively influences his
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% End-user Business manager ICT administrator ICT manager In d ic a te d r e la ti v e r e le v a n ce o f ch a ra ct e ri st ic
Hax: Maintainability
Scalability Manageability Testability Stability Changeability Analyzability32
experience with the information system. The end-user did not pay attention to the maintainability at the start of the project. Still, this characteristic is conform latent needs. The business manager is not completely satisfied with the changeability of the information system. The ICT manager states this is related to the fact that changing ‘his’ part of the information system influences other processes. Therefore, this is not an issue of the information system, but of business managers who do not have an overview of all processes.
The ICT administrator is responsible for maintenance; therefore, he rates maintainability higher compared to the other stakeholders. The ICT administrator does not have issues regarding maintainability. In addition, the ICT manager is in general satisfied with maintainability. The ICT manager only misjudged the complexity of the new information system. Therefore, all ICT administrators have attended extra training. This was not conform latent needs; however, the ICT manager states that the software vendor has delivered excellent support.
To conclude, the information system was slightly more difficult to manage than expected. The software vendor supported HAX to solve this issue. Other aspects are conform expectation. Perceived portability
Most of the portability characteristics are not perceived relevant, except for replaceability (Figure 12). Three of the stakeholders considers replaceability relevant since the information system was selected to replace a previous information system. However, none of the interviewees indicated an issue regarding any characteristic of portability.
5% 10% 15% 20% 25% 30% 35% 40% 45% 50% End-user Business manager ICT administrator ICT manager In d ic a te d r e la ti v e r e le v a n ce o f ch a ra ct e ri st ic
Hax: Portability
Replaceability Conformance Installability Adaptability33
Conclusion regarding perceived quality at HAX
HAX did not write down requirements concerning reliability, efficiency, and maintainability. For the remaining ISO 9126 characteristics (functionality, usability, and portability), HAX did define requirements. The perception of all these characteristics matches the agreed requirements. The end-users perceived some small issues with respect to realization of latent needs by the characteristics functionality, reliability, and usability. According to the business manager, these issues are related to users who have difficulties of letting go old work habits and changing to a process based work method. Therefore, this is not an issue with realizing of latent needs; however, a limitation for realizing expected benefits. In addition, the stakeholders frequently assume that these issues will be solved in the second phase of the project since this phase is focused on improving the information system. Furthermore, to realize latent needs regarding perceived maintainability, the software vendor had to provide extra trainings. Finally, perceived efficiency and portability are both conform expectation.
Has HAX realized expected benefits of the information system?
To be able to implement the information system with only a limited delay, HAX has chosen to focus only at implementing a system that has the same functions compared to their former information system. Therefore, expected benefits of HAX are not yet realized and HAX has planned for an additional project to realize benefits.
To be able to realize expected benefits, it was presumed necessary to align the new information system with the organizational structure. Therefore, HAX had selected an information system based upon the idea that it should fit in the organization. HAX did not want to alter the information system to any specific needs. During the implementation, it turned out there was a need for small customized software. However, generally speaking, HAX configured a standard version of the information system. Besides tailoring to HAX specific requirements, during implementation the software vendor has tailored the information system to comply with agreed requirements. These where solidly needed to solve software errors. Although the information system was flexible to configure to be implemented without changing the organizational processes, HAX has chosen to use the capabilities of the information system to improve its organizational processes.
34
information system. Therefore, the second phase is important since this phase should enable HAX to realize expected benefits.
The theory of Markus (2004) describes that a technochange approach could be used to implement an information system and being able to directly increase the performance of the organization. Reflecting on the characteristics of an IT project, technochange situation, and organizational change program, HAX wanted the outcomes of a technochange solution and handled the project primarily as an IT project. Only the role of other experts besides IT experts was handled following the principles of technochange management. However, as long as the approach is an IT project, led by an IT project manager, who is expected to deliver on time and within budget, one cannot expect to see immediately improvements in the organization (Harison & Boonstra, 2009; Markus, 2004).
Finally, the major risk of a technology driven organizational change that Markus (2004) describes, is the risk of users that will not use the technology. For the majority of the activities of the users, there is no alternative system. However, the managers of HAX presume that some users do not use the information system for generating letters. Instead, they copy an old letter and change, for example, the name and address manually. Nevertheless, this does not have a large impact and is not presumed to affect expected benefits significantly.
Conclusion regarding realization of expected benefits at HAX
35
3.3
Case 2: HAY
The housing association researched for this case will be referred to as “HAY” (Housing Association Y). HAY is one of the smaller housing associations in the Netherlands with approximately 3.500 tenantable units and under 100 employees. HAY has one headquarters and offers housing in two larger villages (around 25.000 inhabitants) and its surrounding areas. Next to the size of the organization, the case of HAY differs from the previous case as the start date of the project lies ahead of HAX: HAX started in 2005 and HAY already in 2003. The immediate cause for selecting a new information system for HAY was a letter from their software vendor stating that the current information system would be supported for only one more year. Although their information system was already 15 years in use, HAY expected it could be used for approximately another five years. The requirements of HAY that were not supported by its old information system where not urgent to justify an immediate implementation for a new information system.
A team of power users from all departments, managers, and an external consultant were assigned to perform the selection of a new information system. However, there was no representative from the management team because there were issues that resulted in an appointed interim manager. According to the business manager who took up office after selection of the information system, the project team existed for two-thirds of front office uses who focused mainly on functionality and usability. There was no manager with a strategic perspective on the organization and a focus on operational reliability. The most important phase in selecting the information system was a demonstration of the future software vendor in a soccer stadium. Although all representatives were impressed and had chosen for this information system, some users who looked at the capabilities of the information system with more detail, became skeptical. They did not think the information system was mature and capable of delivering the presented benefits. Still, the organization decided to implement this information system.
36
Figure 13 depicts the project organization HAY used during implementation of the implementation system. The three application managers were responsible for a distinct part of the information system, knowing: related to customer, financial, and real estate. The application managers took care of configuring and testing the information system. Consultants of the software vendor supported the application managers and provided trainings. The project lead of HAY was responsible for coordinating the project, including tests and trainings. After the information system was selected, a new management team of HAY settled. Together with a representative of the software vendor, the management team acted as steering committee.
Steering committee
Project leaders
Teams
(customer, financial, real estate)
Hay Software vendor
Chairman and management team Project leader Application managers Representative Consultants
Figure 13: Project organization during implementation at HAY.
3.3.1 Interpretation of findings
In line with the discussion of the first case, the results of the interviews with the four employees of HAY are discussed by way of our two research sub-questions, knowing:
• Is perceived quality of an information system in production stage conform expectations?
• Do housing associations realize expected benefits of an information system?
Perceived quality of the information system conform expectations of HAY?
37
Hence, the indicated relevance of functionality and reliability is relative high since they are above average. Consequently, the relevance of usability, efficiency, maintainability, and portability was indicated below average. These six characteristics and their percentages will be discussed in the next section along with the given values for the sub-characteristics. In addition, issues with realization of agreed requirements and latent needs regarding characteristics with an indicated relevance are discussed.
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% S u m o f in d ic a te d r e le v a n ce
Hay: ISO 9126 characteristics
ICT coordinator Business manager End-user
Figure 14: Indicated relevance of the ISO 9126 characteristics, per stakeholder
To compare the relevance of the ISO 9126 characteristics as indicated by HAX to HAY, Table 4 lists the average relevance of the stakeholders. The average is used because four interviewees represented HAX and three interviewees represented HAY. What strikes one most is the difference in indicated relevance of maintainability, 22% for HAX and 11% for HAY. This can be related to two causes; first, two interviewees from the IT domain represented HAX and one representative represented HAY. Second, HAY has out-sourced a
