Master thesis:
How Data Governance Maturity Influences the Success of
Self-service Business Intelligence
MSc. Business Administration – Digital Business
Author: Nando Bouwhuis Student number: 11206039
Institution: University of Amsterdam / Amsterdam Business School
Program: Executive Program in Management Studies – Digital Business track Date of submission: June 15, 2018
Version: 1.0
First supervisor: Ruben de Bliek Second supervisor: T.b.d.
2
Statement of Originality
This document is written by Student Nando Bouwhuis who declares to take full responsibility for the contents of this document. I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it. The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.
3
Acknowledgment
This study gave me the opportunity to combine my work experience from the last ten years with the theoretical knowledge I gathered in the Business Administration Master programme of the University of Amsterdam. It has been a big challenge to study and to write a master thesis next to a fulltime job. However, studying and writing this thesis broadened my academic knowledge and provided me with insights and experiences I can use during the rest of my future career.
Without the cooperation and support of several persons, this thesis could not have been completed. First of all I want to thank my supervisor Dr. de Bliek for the pleasant cooperation, his constructive feedback, and good advice. Secondly, I want to thank all the interviewees who participated in this research for their openness and cooperation. Finally, I want to thank my girlfriend Annemarie for supporting, advising, and challenging me during the last 2.5 years.
Nando Bouwhuis
4
Abstract
Companies increasingly value their data as an asset and expect important decisions to be based on data. More often ‘data driven’ decisions are made and supported by the use of self-service business intelligence tools and datasets. Therefore, it is important that users of self-service business intelligence are able to use the tooling and data in an appropriate way and are able to draw correct conclusions from the derived insights. Prior research on the success of self-service BI mainly focused on presentation, technology, social features, data, and overall requirements. This thesis analysis the relationship between organisational data governance maturity and the success of self-service business intelligence implementation. Data governance maturity is a way to measure how well an organization is able to manage its data as an asset and to transform this data into reliable and meaningful information. The maturity level gives insight in the current data governance maturity state and the maturity model provides processes and tools to improve the maturity. Adopting a multiple case study approach, ten semi-structured interviews with business intelligence consultants were conducted. This study reveals that there are three conditions that currently hinder the success of self-service business intelligence and where increased data governance maturity can positively influence the degree of success when implementing self-service business intelligence. These three conditions are: (1) end-users and/or self-self-service BI developer being unfamiliar with the data that is presented via self-service BI, (2) self-service business intelligence users lacking data literacy and (3) poor data quality of the source systems. If one or more of these conditions apply in an organization that wants to implement self-service business intelligence, investing in data governance in order to increase data governance maturity can improve the success of the self-service business intelligence implementation.
5
Table of content
Statement of Originality ... 2 Acknowledgment ... 3 Abstract ... 4 1 Introduction ... 6 2. Literature review ... 92.1 Self-service business intelligence... 9
2.2 Prior studies on the success of self-service BI ... 14
2.3 Data governance maturity ... 16
2.4 Relationship between data governance maturity and self-service BI success ... 21
3. Method... 25
3.1 Epistemological assumptions ... 25
3.2 Methodological assumptions ... 26
4. Research results ... 29
4.1 Involved companies and interviewees ... 29
4.2 Self-service BI success ... 30
4.3 Limiting and supporting factors ... 33
4.4 Data governance maturity ... 37
5. Discussion and conclusion ... 41
5.1 Practical implications ... 41
5.1 Theoretical implications ... 44
5.2 Other findings ... 45
5.2 Limitations and suggestions for future research ... 45
REFERENCES ... 47
Appendix A: the DAMA Data Governance Maturity Model ... 53
Appendix B: Interview guideline ... 57
6
1 Introduction
More and more companies consider their data as an ‘asset’ and value data driven decision-making. Data driven decision-making means basing decisions on analysis of data instead of making decisions on intuition (Provost & Fawcett, 2013). The process of acquiring data, transforming data into useful information, and disseminating information throughout the company in order to facilitate data driven decision-making is called ‘Business Intelligence’ (BI) (Olszak & Ziemba, 2007). The way information is delivered to decision-makers in organizations is changing from static reporting, often used at a strategic level, to so called ‘self-service business intelligence’ (self-service BI), often used at an operational level. Self-service BI is a variation on BI in which “search, extraction, and integration of situational data should be accomplished by users through a continuous interaction with the application, without any mediation or intervention by analysts, designers, or programmers” (Abelló et al., 2013, p. 67). Self-service BI implementations are not always successful. As self-service BI users are usually business users with limited knowledge about data, problems with data interpretation and data quality are more likely to occur in self-service BI than in serviced BI. Since self-service BI is increasingly used for decision making, it is important for companies to understand what factors influence the success of a self-service BI solution. Self-self-service BI has been a topic of academic research since 2008 (Spahn, Kleb, Grimm, & Scheidl, 2008). Prior research on the success of self-service BI mainly focused on five dimensions: presentation, technology, social features, data, and overall requirements (Yoko Ogushi & Schulz, Michael, 2015).
7 In this thesis a sixth dimension that could influence the success of a self-service BI implementation (and not has been studied before) will be explored: data governance maturity. Thomas (2013) defines data governance as “a system of decision rights and accountabilities for information-related processes, executed according to agreed-upon models which describe who can take what actions with what information, and when, under what circumstances, using what methods”. Data governance maturity is a way to measure how well an organization is able to manage its data as an asset and to transform this data into reliable and meaningful information. It gives insights in the current data governance maturity state and provides processes and tools to improve the maturity. Companies can improve their data governance maturity with the help of a data governance maturity model. Data governance can be subdivided into five decision domains: data principles, data quality, metadata, data access and data lifecycle.
It is reasonable to assume that a low data governance maturity level decreases the chance of a successful self-service BI solution. On the other side, a high data governance maturity level should increase the chance of a successful self-service BI implementation. For example: if a company scores low (= immature) on the ‘data quality’ decision domain, this means that it is likely there is a lack of procedures to improve the data quality and limited insight in the quality of data. If department x within this company uses a self-service BI solution for their decision making, there is a chance that these decisions are based on bad quality data without the users knowing this. This could lead to unintentionally wrong data driven decisions. A higher score would for example lead to better insights in the quality of data so that department x could make a conscious choice to use the current low quality data or go for an alternative data source that has data of a better quality. Therefore, it could be hypothesized that there is an association between the data
8 governance maturity level and the success or failure of a self-service BI implementation. However, it is still not clear what this association looks like. This thesis analyses this relation by answering the following research question: “How does data governance maturity influence the
success of self-service business intelligence?”
In this study ten self-service BI consultants are interviewed. These self-service BI consultants are involved in self-service BI projects, either in the company they work for (internal self-service BI consultants) or at customers (externally hired self-service BI consultants). To answer the research question, a qualitative approach is used. The research method used in this thesis is a multiple case study using personal face-to-face interviews. This method is chosen because of its interactive character and because it is a good way to get to know more about how people use self-service BI and the processes behind it. In total, ten interviewees were selected. Six interviewees were selected within the personal network. Additionally, four interviewees were selected through snowball sampling. The interviews conducted have been semi-structured, as it provides the opportunity to ask more in depth questions.
It was expected that a higher data governance maturity level leads to a higher self-service BI success rate and a lower data governance maturity level leads to a lower self-service BI success rate. When this is the case this implies that companies can improve the quality of their data driven decisions via self-service BI by initiating a data governance improvement program that focusses on the right decision domains.
9
2. Literature review
In this chapter a review of relevant literature about self-service BI success and data governance maturity is presented. In the final paragraph the relationship between data governance
maturity and self-service BI success is explained.
2.1 Self-service business intelligence
This paragraph explains self-service BI success in five sections. The first two sections describe what business intelligence (non self-service) is and how it evolved over time. This is important to understand as it provides insight in the need for self-service BI. Section three and four describe self-service BI success and the last section explains prior research on self-service BI success.
Business intelligence
Many definitions of BI can be found in academic and practitioner books. BI can be seen both as a product and a process (Jourdan, Rainer, & Marshall, 2008). BI as a process is a chain of systems and methods that “acquire, analyse, and disseminate information from both internal and external information sources significant for business activities and for decision making” (Lönnqvist & Pirttimäki, 2006, p. 32). An example of an internal information source is the customer order database of a company. External information sources are not generated within the company, but are imported from an external source. An example of an external information source is a data feed from the Chamber of Commerce with information about all companies in a country. BI as a product is the “information that will allow organizations to predict the behaviour of their competitors, suppliers, customers, technologies, acquisitions, markets, products and services, and the general business environment with a degree of certainty” (Jourdan et al., 2008,
10 p. 121). Trieu (2017) wrote a literature review and research agenda about business intelligence and concluded that “BI is typically used as an ‘umbrella’ term to describe a process, or concepts and methods, that improve decision making by using fact-based support systems. Many terms (such as ‘business intelligence’, ‘business analytics’, ‘big data’, ‘data mining’, and ‘data warehousing’) are often used interchangeably in the literature, with authors variously describing BI as a ‘process and a product’, ‘a process’, ‘a product’, and ‘a set of technologies’, or a combination of these” (Trieu, 2017, p. 111).
The definition of BI that will be used in this thesis is the definition of Burstein & Holsapple (2008): “Systems that combine data gathering, data storage and knowledge management with analysis to evaluate complex corporate and competitive information for presentation to planners and decision makers, with the objective of improving the timeliness and the quality of the input to the decision process” (Burstein & W. Holsapple, 2008, p. 176).
Evolvement of business intelligence
Academic research to (a precursor of) BI goes back to the early 70s when Scott Morten wrote about how computers could help organizations in making better decisions, later known as Decision Support Systems (Morton, 1971). Sprague (1980) wrote an article about the evolvement of Decision Support Systems. There were two reasons for Sprague to write this article. The first reason was to give insight in the different views on Decision Support Systems. The second reason was to propose a framework in which he addressed major concerns that different stakeholders in the development process of Decision Support Systems experienced. Sprague brought different existing disciplines and techniques needed for a well-functioning Decision Support Systems together in the Decision Support Systems framework and identified issues that could be potential
11 roadblocks in the further development of Decision Support Systems. Finally Sprague concluded that Decision Support Systems was not just a ‘buzz word’, but a movement in information systems because “user organizations, information systems vendors and researchers become aware of the field, its potential, and the many unanswered questions“ (Sprague, 1980, p. 23).
From the early 90’s until about 2005 BI, as the way it is described in chapter 2.1.1, evolved. The development of new data modelling methods, data processing techniques and data analysis tools transformed the concept of BI into a well-developed approach to information extraction, processing and reporting. An important turning point in the history of BI is the point that BI was not just a technology to extract data and transform information anymore, but also started to bridge the gap between strategic management and technology (Golfarelli, Rizzi, & Cella, 2004). Companies were starting to see the value of BI and invested in BI infrastructures and knowledge. The number of BI developers and BI users1 was growing.
BI also had some shortfalls (Berthold et al., 2010). In the first place, it didn’t meet the needs of individual business users. Generated reports were mainly static, and the downside of static reporting is that these reports are not suitable for further analysis of the presented information and they have to be created and maintained by reporting experts. Secondly, BI solutions often lack to provide business context (for example: the meaning of data for a certain business goal was not always clear which could lead to wrong interpretations of this data). Thirdly, the possibilities for business users to collaborate in order to make decisions was limited. Fourthly,
1 BI developers are computer programmers specialized in programming BI solutions. BI users are the consumers of
12 setup and configuration of BI solutions required insights in the data and the business information requirements, which meant a solution had to be tailored to every specific target group (which is time consuming). Finally, BI focused on structured internal data which is hard to integrate with unstructured external data, where there is a business need to combine structured internal with unstructured external data (Berthold et al., 2010) . An example of unstructured data is the body of an e-mail. Where the sender, recipient, date and time sent are structured data, the text in de body of the email is unstructured because it doesn’t have a predicable format.
Self-service BI success
To overcome the shortfalls described in the previous paragraph as from circa ten years ago, more and more so called self-service BI tools became available on the market. In one of the first practitioner articles, Imhoff and White (2011) define self-service BI as “The facilities within the BI environment that enable BI users to become more self-reliant and less dependent on the Information Technology (IT) organization. These facilities focus on four main objectives: easier access to source data for reporting and analysis, easier and improved support for data analysis features, faster deployment options such as appliances and cloud computing, and simpler, customizable, and collaborative end-user interfaces” (Imhoff & White, 2011, p. 4). In this thesis, the term self-service BI is used “to emphasize that the search, extraction, and integration of situational data should be accomplished by users through a continuous interaction with the application, without any mediation or intervention by analysts, designers, or programmers” (Abelló et al., 2013, p. 67) . Since 2005, self-service BI evolved from basic tools that were used especially by so called “power users with a strategic orientation”, to tools used by a much broader
13 group of users. This group is as big as needed in order to perform the needed analyses and can involve employees at all levels in an organization (Yoko Ogushi & Schulz, Michael, 2015).
An example of a self-service BI use case can be found in the marketing department of a telecom provider. To measure the success of a marketing campaign, marketing managers used to rely on a business intelligence department that delivered periodic static reports about campaign effectiveness to them. These reports were standardized and the main purpose was to show (afterwards) if a campaign has been successful or not. Since the marketing managers wanted to know more about the reasons why campaigns are successful or less successful and be able to get more realtime insights, they implemented a self-service BI environment together with the business intelligence department. In this self-service BI environment the marketing managers are able to combine data about campaign effectiveness with sales data, customer satisfaction data and call centre data, depending on the needs of their analysis. The result is that the marketing managers are able to get (near) realtime insights in campaign effectiveness and the reasons behind it, which enables them to adjust a campaign while it is still running in order to get better results. The marketing managers experienced this as a major improvement compared to the situation before they had self-service BI tooling.
Some self-service BI solutions are more successful than others. Since users of self-service BI are often business users, their knowledge about analysing and processing data is often limited. This can lead to wrong interpretation of data and incorrect made decisions. But what exactly defines success in case of self-service BI? Most organizations choose to implement self-service BI in order
14 to generate organizational benefits. These benefits differ per organization and project and they are not generalizable. However, in most situations the expected benefits of self-service BI are (like non-self-service BI) related to an increase of profitability, decrease of costs, efficiency improvement or more advanced insights (Işık, Jones, & Sidorova, 2013). In this thesis the success or failure of a self-service BI solution is defined as “to what extend the anticipated organizational benefits are realized after implementation”. If all anticipated benefits are realized, the self-service BI solution will be considered ‘successful’. If none of the anticipated benefits are realized after implementation, the self-service BI solution will be seen as unsuccessful. If some, but not all of the anticipated benefits are realized, or unexpected benefits are realized, the self-service BI solution will be seen as ‘partly successful’.
2.2 Prior studies on the success of self-service BI
Prior research on the success of self-service BI from 2008 mainly focused on five dimensions: presentation, technology, social features, data, and overall requirements (Yoko Ogushi & Schulz, Michael, 2015). First, research on self-service BI presentation is about creating end user tools that are simple, straightforward, and intuitive, so that business users without any technical background are able to create their own reports, dashboards and data analysis (Imhoff & White, 2011; Spahn, Kleb, et al., 2008). Research by Spahn et al. (2008) for example focused on how data should be presented to business users in a way that these users are able to build meaningful artefacts. They found that the complexity of self-service BI environments should be simplified, even if this results in decreased flexibility for the end user. Complexity reduction can be achieved by focusing on three issues: data abstraction, searching and browsing data, and orchestrating data.
15 Secondly, research on self-service BI technology focuses on increasing the technical performance of self-service BI tooling using a technology called ‘in memory processing’ (Acker, Gröne, Blockus, & Bange, 2011). This is an important technology, because it enables users to process analyses of large amounts of data ‘on the fly’ on real time operational data without causing performance issues on the operational systems. Another important technological development is cloud-based self-service BI. This technology lowers the barrier for organizations to start with (SS)BI. Without having to make large investments in infrastructure, architecture and security, organizations just rent what they need and only pay for the time they use the self-service BI environment (Naish, 2013).
Thirdly, research on social features regarding self-service BI focuses on the integration of social media within self-service BI in order to facilitate collaborative decision making (Liu, Kim, & Sun, 2013). Berthold et al. (2010) describe one of the shortcomings of traditional BI as lacking a collaboration aspect. Collaborative decision making can improve the quality of decision making. If collaborative decision making is used, it is being done outside of the BI system. It would improve the adoption and usage of self-service BI tools, if collaboration were an integrated part of the tool. An architecture to achieve collaboration within self-service BI is proposed by the authors.
Fourthly, research on self-service BI data is about optimizing storage, data integration, and data aggregation in a self-service BI environment (MZaghloul, Ali-Eldin, & Salem, 2013). Savinov (2014) for example proposes a new vision on data integration that simplifies the process especially for
16 non-IT users. Another example is the research of Niemi et al. (2014). This research shows that one of the challenges experienced by self-service BI users is that it is difficult to find out what data can be summarized and, if so, in what way. A new categorization of dimension types and new methods are proposed to data modelers, in order to make summarizing easier for self-service BI users.
Finally, research on overall self-service BI requirements resulted in general design principles for self-service BI. An important challenge that is specifically important for the design of a self-service BI environment is that the complexity (i.e. number of available features and amount of data available to the end user) correlates negatively with its usability. This requires a constant balance of on the one hand giving end-users all the features and data they want for their analysis and on the other hand giving the end-users a seamless and intuitive self-service BI experience (Burnay, Gillain, Jureta, & Faulkner, 2014). Stone et al. (2014) describe the rise of interactive marketing with the use of self-service BI. It is acknowledged that there is a risk that marketing employees using a self-service BI environment draw wrong conclusions based on data. Therefore, a need for a “mature, well-governed relationship between BI experts and marketing users” exists.
2.3 Data governance maturity
In this paragraph data governance maturity is explained. The first section explains what ‘data governance’ is. Next, it is explained what ‘data governance maturity’ is. In the last section prior studies on the effect of data governance are discussed.
17 Data governance
“Data Governance is a system of decision rights and accountabilities for information-related processes, executed according to agreed-upon models which describe who can take what actions with what information, and when, under what circumstances, using what methods” (Thomas, 2013, p. 3). Where Information Technology Governance (IT Governance) refers to decision rights and accountabilities for decision making about IT assets, data governance refers to decision rights and accountabilities about data assets. However, sometimes data governance comprises parts of IT governance (Weber, Otto, & Österle, 2009). Data Governance can be subdivided into five different decision domains: data principles, data quality, metadata, data access and data lifecycle. The Data principles decision domain describes the desired behaviour around data in an organization. Topics that are typically covered in data principles are: data ownership (a data principle on this topic could be ‘every dataset must have a business owner’), principles about sharing and reusing data, the use of external data (for example customer data of third-parties), general rules and regulations around data. The data quality decision domain defines standards and mechanisms to control and improve data quality. Data quality gives users insights in information about the correctness, completeness, timeliness and trustworthiness of this data in relation to its intended use (Khatri & Brown, 2010). The importance of the data quality decision domain becomes clear from recent research. This research indicates that the average financial impact of poor data quality on organizations is $9.7 million per year (“How to Create a Business Case for Data Quality Improvement,” 2017). The metadata decision domain helps to define the meaning of data. The most simple definition of metadata is “data about data” (de Henares, 2006, p. 83). Metadata for example describes where data is stored (physical metadata), who created
18 or modified the data (domain independent metadata), or what the meaning of data is for a business (domain specific metadata) (Khatri & Brown, 2010). The data access decision domain describes rules about who has access to which categories of data under what conditions. Themes like privacy, confidentiality, auditability (tracking who accessed or modified data on what times) and data integrity are applied here. The data lifecycle decision domain generates insights into the movement of data through different stages of its lifecycle. Information about how data is being used, how long it should be retained, and how fast data grows is collected to determine the optimum data storage media. Some authors use more domains. IBM (2007) for example defines ‘Data Architecture’ as a data governance domain, others place this domain under ‘IT Governance’. Weber et al. (2009) suggests that a data governance configuration should be specific to a given company.
Data governance maturity
The state of data governance in an organization can be assessed using a maturity model. Maturity models describe the evolvement of an organizational function over time (in this case, the function is ‘data governance’). The goal of a maturity model is to give an organization insight into the current level of organizational maturity on a certain topic and to provide a roadmap on how to reach a higher level. The concept of measuring maturity of a certain entity emerged in the software development industry. One of the first maturity models was the ‘Capability Maturity Model for Software’ (CMM) (Curtis, Chrissis, & Weber, 1993). In the literature and practitioner books, several maturity models focussing on data governance are described. Sweden (2009) created an overview of seven data governance maturity models developed by thought leaders in the field of data governance (e.g. Gartner, IBM, the MDM Institute, Oracle). All models use
19 different levels of maturity. The number of maturity levels centres between four and six and the naming of levels slightly differs. However, the essence of all models is the same: the lowest level is the most immature state and the highest level is the most mature state. The difference between the models lies in the level of detail used to distinguish data governance maturity from different data governance domains. For this thesis it is important to know which data governance domain relates to self-service BI success. This enhances the value of the results, as companies that want to improve self-service BI success get insight into what data governance domain has the most effect on self-service BI success. Therefore, in this thesis the data governance maturity model of DAMA (the Data Management Association International) is used. Where some models only describe general characteristics per maturity level, other models (especially the models of Garter, IBM, Stanford University and DAMA) classify data governance maturity per data governance domain. The model of DAMA contains the most detailed domain classification, namely ten different domains. For each domain the maturity is assessed for four different dimensions: people, processes, technology and goals & principles. A detailed overview of the DAMA data governance maturity model can be found in appendix A.
Prior studies on the effect of data governance maturity
Prior research on data governance focused mainly on the definition of data governance and different ways to implement data governance in organizations. Research on the effect of data governance (maturity) is still limited. Kamioka et al. (2016) did research the effect of data governance accountabilities on perceived performance in marketing activities. The definition of ‘accountability’ in his research is “the assignment of decision rights and responsibilities” (Kamioka et al., 2016). A positive relationship was found between accountabilities in data
20 governance and perceived marketing performance. An intermediate variable called ‘data utilization’ was used. Data utilization increased when people in (especially smaller) companies were appointed and empowered in data-management roles. The increased data utilization itself led to an increase in the number of sales and customer spending.
Looking a bit broader, IT Governance can be seen as a similar - but more established - concept as Data Governance. Both concepts are about managing assets. In case of IT governance, IT assets are managed. Some examples of IT assets are: servers, software, computer devices and backups. In case of Data Governance, the ‘data asset’ is managed. In IT governance, maturity models are used as well. Simonsson & Johnson (2008) developed a model to predict the effect of IT (e.g. perceived IT quality) based on the maturity level of IT processes. In four case studies, a correlation was found: the higher the IT maturity, the more successful the IT solution. However, in these studies it has not been described how these variables are exactly linked. Therefore, more data was needed.
Simonsson et al. (2010) researched the effect of IT Governance maturity on IT Governance performance. The reason for this research was that there were many best practices on ways to organize the internal structure of an IT organization, but it was never validated in academic research whether there was an actual positive correlation between IT governance maturity and IT governance performance. 34 different IT Governance processes in four categories were assessed in a qualitative research. This resulted in the conclusion that most correlations between IT governance maturity and IT governance performance were medium positive and small positive.
21 Lunardi et al. (2014) drew similar conclusions in their research on the effect of adopting IT governance on financial performance. Companies with a higher IT governance maturity showed higher levels of financial performance, especially a year after the implementation of an IT governance concept.
2.4 Relationship between data governance maturity and self-service BI success
Based on prior research and own experience it is expected that in general the data governance maturity level influences the success or failure of a self-service BI solution. To be more specific, it is expected that a higher data governance maturity level leads to a higher self-service BI success rate and a lower data governance maturity level leads to a lower self-service BI success rate. Different data governance decision domains can influence self-service BI success in different ways. As shown in figure 1, this study is limited to two data governance decision domains: metadata and data quality.
22
Figure 1: Scope of this research
There are different types of metadata: physical metadata, domain independent metadata, domain specific metadata and user specific metadata (Khatri & Brown, 2010). This study will focus on domain specific metadata. Domain specific metadata describes the meaning of data for a specific domain (for example a business unit, department) or for the company as a whole. It creates a connection between the data and the ‘real world’. Better governed domain specific metadata can improve self-service BI success because it helps users to interpret2 the data in the datasets they’re working with. Enhanced data interpretation leads to increased self-service BI success. For example: a dataset containing customer information is presented to a business user
2 Data interpretation is defined as ‘to what extend a SSBI users are able to interpret the meaning of data in a
23 via a self-service BI solution. The set contains customer names, home addresses, and e-mail accounts. Without any context, the business user could interpret this data in many ways. It could be interpreted as the complete customer base of the company (all segments: residential and business-to business customers), but it could also be interpreted as a subset of customers, for example only the residential customers. If a business user wants to generate a self-service BI report about the number of customers in a specific segment, it is very important that more context is provided about the data. Domain specific metadata can give this context.
Improved governed data quality can increase self-service BI success in two ways. First, it gives self-service BI users insight in the quality of the dataset they want to use. The better the information regarding data quality, the better the user can assess whether this dataset is suitable for the intended analysis. If a data quality report shows that the data quality of a dataset is poor, the user can either decide to use the dataset and take the poor data quality for granted or to look for another dataset that suits the analysis better. Secondly, data quality management also facilitates data quality improvement. Facilities like data quality profiling tools3, data quality assurance tools4 and (automated) processes to fix these issues improve data quality in an organization. Improved data quality increases the success of self-service BI.
To conclude, it is expected that an increased data governance maturity – specifically metadata maturity and data quality maturity - leads to improved data interpretation, better insights in data quality and increased data quality. Improved data interpretation, insights in data quality and
3 Tools that automatically detect data quality problems in source systems or data warehouses 4 A tool or website where employees can report and monitor data quality issues
24 increased data quality lead to a higher self-service BI success rate. This could be an interesting finding for organizations that are implementing self-service BI. When improving data governance (especially on the metadata and data quality domain), using a data governance maturity model to guide the improvement process could improve the success rate of self-service BI solutions, which will result in higher quality data driven decisions.
25
3. Method
In this chapter the methodological and epistemological assumptions adopted in this thesis are explained. Additionally, sampling is discussed.
3.1 Epistemological assumptions
The four main epistemological research philosophies within science are ‘Positivism’, ‘Realism’, ‘Interpretivism’, and ‘Pragmatism’. In the literature review, no prior research has been found that links data governance maturity to self-service BI success or failure. Since both data governance and self-service BI are complex processes, this study first tries to clarify the underlying mechanisms of these processes. Next, an answer to the research question is given. Since investigating underlying mechanisms and looking for more advanced insights typically refers to an interpretivist approach, the interpretivist philosophy is adopted. Since this philosophy prefers qualitative research over quantitative research, qualitative research is conducted (Saunders & Lewis, 2011).
A research approach can either be deductive or inductive. A deductive approach is using a research strategy that is designed to test a theory. In an inductive approach theory, development is part of the research strategy. Theory is developed based on the data collection within the research. Since no former research is conducted, an inductive approach is chosen here (Saunders & Lewis, 2011).
There are three different types of study: exploratory, descriptive, and explanatory. Since this study is trying to explain the relationship between two variables, the research will have an
26 explanatory character. Explanatory research is about explaining a certain occurrence by looking for causal relationships between variables (Saunders, Thornhill, & Lewis, 2009).
There are many different research strategies (e.g. experiment, survey, case study, action research, grounded theory). In this study, a multiple case study is used, because it is the most appropriate strategy if a ‘How’ question is to be answered. However, there are also some elements of the ‘grounded theory’, since this thesis tries to develop a theory based on a series of interviews (Saunders & Lewis, 2011).
3.2 Methodological assumptions
The used research methodology has to fit the described epistemological assumptions. The research method used in this study is the personal, face-to-face interview. This method is chosen because of its interactive character and because it is a good way to get to know more about how people use self-service BI and the processes behind it.
Interviews can be unstructured, structured, and semi-structured. The interview type used for this study is the semi-structured interview. This type of interview is a combination of a structured and unstructured interview. A predetermined list of questions is used, but to get a full understanding of the topic the interviewer can decide to switch to an open conversation as well. This interview type is chosen because it provides the opportunity to ask more in-depth questions as a response to answers on the pre-determined questions. That can add value to the outcome of the interview, as additional insights can be gained. All interviewees are asked the same set of questions to get an understanding of why self-service BI solutions are successful or unsuccessful. The interviews
27 will cover two main subjects: the degree of self-service BI success / failure and data governance maturity.
In the first part of the interview, the interviewees are asked to come up with a self-service BI project in which he/she was involved from the start to the end and to grade the project based on their first intuition. Next, questions about how the interviewee defines the degree of success of this project are asked. After that, the anticipated organizational benefit of the projects are discussed and to what extend these benefits were realized. This information is used to define the definition of success used in this research.
In the second part of the interview, interviewees are asked about possible reasons why some self-service BI solutions are successful and others are unsuccessful. After the interviewees come up with possible reasons, questions about data governance maturity are asked in order to define the maturity level of the two different data governance decision domains. The used data governance maturity model can be found in appendix A. Finally, interviewees are asked if they think there is a relationship between the data governance maturity levels and the success/failure of the discussed self-service BI solutions and – if so - what this relationship looks like. Since it cannot be assumed that all interviewees know the concept of data governance, questions about this subject will be asked indirectly using questions that are also being used during a data governance maturity assessment.
To get a deep understanding of factors that lead to successful or unsuccessful self-service BI solutions, ten respondents are interviewed. Eight of them were selected within the personal network. The rest of the respondents were selected through snowball sampling. Snowball
28 sampling means asking respondents if they have other suitable participants in their network that can be contacted for one of the next interviews (Saunders et al., 2009).
All respondents either work in the field of self-service BI delivery, e.g. BI consultants or managers of BI departments that realized self-service BI solutions for businesses. When a respondent agrees to participate, an e-mail is sent with additional information such as the research objectives, an explanation of how the data is used and how the respondents identity is protected.
The interviews all take place at the workplace of the interviewee, because it lowers barriers for the interviewee to participate. The interviews are recorded upon approval by the interviewee. This helps the interviewer to actively engage and listen to the interviewee. The interview guideline can be found in appendix B.
29
4. Research results
In this chapter the results from the semi-structured interviews are presented. First, the background about the involved companies, interviewees and cases are described. Next, the interview results related to the success of the self-service BI cases are presented. Then, the limiting and supporting factors that influenced the success of self-service BI are discussed. Finally, the results of governance maturity assessments are presented.
4.1 Involved companies and interviewees
All interviews focused on self-service BI projects that were undertaken by companies that are situated in the Netherlands. The company sizes varied from circa 1.000 to 40.000 employees. The discussed self-service BI projects (cases) took place in various industries: Telecom (5), Government (3), and Fashion (1). In total ten people were interviewed which resulted in discussing nine different self-service BI projects (cases). One interviewee did not refer to specific self-service BI project/ case, but talked about his general experience with self-service BI.
The experience level of the interviewees with (self-service) BI varied between 2 to 20 years. Most of the interviewees are currently working for a BI consulting firm. This group referred to projects they performed for one of their clients. The other part of the interviewees are working as internal BI consultants in a BI Competence Centre. This group brought in cases of projects they participated in within the organization they work for.
30 4.2 Self-service BI success
The success of the self-service BI cases is measured in two ways. First, interviewees were asked to indicate the success of the self-service BI solution they worked on, by grading their case on a scale from one to ten; where ‘one’ indicates that the case was unsuccessful and ‘ten’ indicates that the case was considered to be very successful. Secondly, interviewees were asked to describe the predefined organizational benefits of the cases and to what degree these benefits were realized after the project completion.
Of the ten interviewees, seven scored their case a seven or higher when indicating the degree of success. The lowest grade given was a 6.5, however this is an average grade. ‘I would give an
eight for the technical solution, but a five for the usability and business value’ (Interviewee 4,
personal communication, April 12, 2018). The highest grade given was a nine. Two of the interviewees were not able to give a grade because the project was not finished yet (interviewee 7) or they did not have a specific case in mind during the interview (interviewee 6).
To the question related to defining the predefined organizational benefits, several answers were given which can be categorised into three categories. The first category is the creation of flexible insights into operational performance of a department in order to improve its performance. ‘The
goal was to create insights for call center agents and their team leaders. The agents wanted insight in their own performance, the team leaders wanted insight in the performance of their teams and employees. Next to insight, one wanted to be able to ‘drill down’ on the numbers in order to find out more about the reasons behind a certain score and make changes to improve performance’ (Interviewee 5, personal communication, April 13, 2018).
31 The second category is the replacement of manual (legacy) reporting activities. ‘The most
important goal was to replace the current manual Excel reports. Every month it took a lot of time to create these reports. By using self-service BI, the reporting process would be more efficient and less dependent on one or two persons’ (Interviewee 8, personal communication, April 18, 2018).
The final category is creating a company broad analysis platform to generate a wide range of insights. ‘The goal was to generate a general self-service BI analysis environment for the whole
organization, which contains company broad information’ (Interviewee 4, personal
communication, April 12, 2018).
All of these categories are in line with prior research on the success of business intelligence, such as the research of Işık et al. (2013) where it is found that in most situations the expected benefits of business intelligence are related to an increase of profitability, decrease of costs, efficiency improvement or more advanced insights.
Table 1 shows the categories and the frequencies that indicate the number of times they were mentioned.
# Anticipated organizational benefits # of interviewees that mentioned benefit
1 Create flexible insights in operational performance of a department in order to improve its performance
5x
2 Replace recurring (legacy) manual reporting activities in order to save time, report more reliable data, or be less dependent on one person to generate reports
4x
3 Creating a company broad analysis platform in order to generate a wide range of insights
32
Table 1: Anticipated organizational benefits
In six cases the anticipated organizational benefits were completely realized after implementation. “Yes, the anticipated benefits were met, and even more. From the moment the
users started using the self-service BI solution they realized they could use the tool and the data for additional benefits as well. For example, to create insights in the order processing time in order to improve it” (Interviewee 1, personal communication, April 5, 2018). In two cases the
anticipated benefits were partly realized. “A self-service BI analysis environment was made
available to the end users, however we did not manage to replace all legacy reports. The current legacy reports contain more information than which is made currently available in the self-service BI environment” (Interviewee 4, personal communication, April 12, 2018). In two cases, the
interviewees answered that although the anticipated organizational benefits were completely realized, the organization still could get more out of the self-service BI solution if the business users would have a better understanding of the data, the data model, and on how to use the self-service BI tool. “Technically, the targeted benefits were met. It is possible to acquire the insights
that the company aimed to make available through the tool. However, the self-service BI solution is not used very often, because the end users (marketing employees) do not know how to use the tool in combination with the underlying data model. For example, employees do not understand well how to relate different data entities to each other and they are unable to assess how certain actions affect the result” (Interviewee 2, personal communication, April 6, 2018).
The results of the questions related to self-service BI success of the cases described by the interviewees are summarized in table 2.
33
Case #
Grade Organizational benefits realized?
Remarks
1 8 Yes, completely
2 7 Partly
3 7 Yes, completely
4 6.5 Partly Interviewee gave an 8 for the technical solution but a 5 for usability/business value. The rate depicted in the table is the average rate
5 8 Yes, completely Although all planned benefits are realized, the organization could get more out of it
6 n/a n/a Interviewee did not have a specific case in mind
7 n/a n/a The project was not finished yet
8 7 Yes, completely Although all planned benefits are realized, the organization could get more out of it
9 9 Yes, completely
10 7 Yes, completely
Table 2: self-service BI success
4.3 Limiting and supporting factors
During the interviews, several factors were mentioned that influenced the success of self-service BI in a positive or negative way. These factors are categorised and shown in table 3. Additionally, for each factor it is indicated whether it was seen by the interviewees as a limiting or a supporting factor. Also, it is shown how often interviewees mentioned each factor. A factor is indicated to be ‘supporting’ when this factor actually helped the project to be more successful or it could have helped the project to be more successful. A factor is marked to be ‘limiting’ when it made the project less successful. Some factors were mentioned both in a positive and a negative way.
34
# SSBI success factor Limiting / supporting # of interviewees that mentioned influencer
1 Having domain specific metadata Supporting 8 2 Business involvement during the
design and development of a self-service BI solution
Supporting 8
3 Prior knowledge of the presented data
Supporting 7
4 Bad data quality Limiting 6
5 (Lack of) domain specific metadata Limiting 4 6 Creating tailored self-service BI
solutions for different target groups
Limiting 3
7 Having data quality management Supporting 2
8 (Lack of) data literacy Limiting 2
9 Limiting the data that is presented in a self-service BI solution
Supporting 1
Table 3: Stimulating and withholding success factors
Eight interviewees indicated that having some form of domain specific metadata has been a supporting factor. “Many problems were prevented because we created good documentation,
containing metadata. This documentation contained lists with facts and dimensions and their functional meanings. It also contained information about which data entities one could combine and which not” (Interviewee 5, personal communication, April 13, 2018). Four Interviewees
mentioned that lack of domain specific metadata made the project less successful. “The
datamodel was complex, among other things because it contained much more data entities than most end users needed. Lack of metadata made it unclear for end users how to deal with this data” (Interviewee 4, personal communication, April 12, 2018).
35 Another supporting factor that was mentioned eight times is the involvement of business users during the design and development phase of a self-service BI solution. “For self-service BI, the
requirements phase is completely different compared to developing a fixed report or dashboard. One should understand the question behind the question in order to design an optimal data model. To obtain this insight, close corporation with the business is necessary” (Interviewee 4,
personal communication, April 12, 2018).
Seven interviewees mentioned that prior knowledge of the presented data among self-service BI developers and self-service BI users positively influenced the success of self-service BI. In cases where users or developers already worked with the dataset before starting the project, having metadata and data quality management in place seems less important. “The end users already
knew the data very well. That’s why data quality issues were quickly recognized and fixed”
(Interviewee 8, personal communication, April 18, 2018). “End users already had a lot of
experience with this specific dataset because they already used it in their Excel analyses. That is why it was less important to have good metadata. However, it would still be important to have metadata for future colleagues who don’t have this knowledge” (Interviewee 1, personal
communication, April 5, 2018).
The limiting factor that was most often mentioned by the interviewees (six times) is having to deal with data of bad quality. “Data quality certainly was an issue. In the source systems, bookings
were registered on wrong codes. This had to be fixed and cost a lot of extra time” (Interviewee 3,
personal communication, April 6, 2018). “The data quality was bad and it was a long process to
36 mentioned that having more mature data quality management would have helped their project to be more successful. “Better data quality would have improved the project. Next to that, a good
channel to report data quality issues would help to address the issues at the right place in the business. This would have saved a lot of time for the BI Competence Center team” (Interviewee
4, personal communication, April 12, 2018). Two interviewees mentioned that having more mature data quality management would not necessarily make the project more successful, but it would speed up the process of delivering the project.
A supporting factor that was mentioned 3 times is the use of different target user groups. Interviewees thought that defining different target user groups and customizing the self-service BI solution to a specific target group increased the success of self-service BI. For example, one interviewee said that they created a different version of the same self-service BI solution for board members, managers/ team leaders, and analysts. By doing so they assured that all target groups used the same data for analyzing and reporting, but the level of detail, analysis options, and possibilities to publish the results differed per target group.
Another similar supporting factor was mentioned by one interviewee: making self-service BI less complex by limiting the data that is presented in the self-service BI solution to data that is well known by the users. This prevents users from exploring new data and can decrease the chance of incorrect data interpretations. A similar result was found by Burnay et al. (2014). They found that complexity in self-service BI correlates negatively with its usability.
37 Lack of data literacy was mentioned as another limiting factor. Data literacy was explained by the interviewee as “the ability to transform data into information”. Lack of data literacy with self-service BI users can result in wrong data interpretation.
Overall, the described findings in this paragraph are partly in line with the findings of Spahn et al. (2008) who found that end-users experience problems in understanding, finding, and interpreting data in datasets that are made available to them via self-service BI tools. Spahn et al. (2008) did not find that lack of metadata, or low data quality could be a possible cause of this problem. However they did find that there is a need for system knowledge among end-users in order to be able to know what data exists, how to find it and how to use it.
4.4 Data governance maturity
In order to obtain insight in the data governance maturity of the companies where the cases took place, a data governance maturity assessment for the decision domains ‘metadata’ and ‘data quality’ was part of the interview. Maturity was measured over four dimensions: Goals and Principles, People, Process, and Technology. The interviewee rated each dimension on a scale from one to five. The most immature state is one and the most mature state is five.
38 Metadata
The results of the metadata maturity assessment are shown in table 4.
Case# Metadata maturity
Goals & principles
People Process Technology
1 2 2 2 2
2 4 2 3 2
3 1 1 1 1
4 2 3 2 2
5 4 3 3 4
6 n/a n/a n/a n/a
7 3 2 3 2
8 2 2 2 2
9 3 3 3 4
10 2 2 2 2
Table 4: Metadata maturity
To the question whether the project would have been more successful if it had a higher metadata maturity level, four out of ten interviewees responded that they thought this would be the case. Three interviewees stated that the project would not have been more successful in the sense of meeting business goals, but it would have helped them to deliver the project in a more efficient and faster way. Two interviewees thought the metadata maturity was good enough to support their project and thought a higher maturity would not increase the success of the self-service BI project. One interviewee indicated that because of the immature state of metadata
39 management, the project manager decided to limit the dataset that was presented via self-service BI in order to prevent problems with data interpretation. A higher metadata maturity would in this case have led to a broader dataset and additional analysis possibilities, because the project managers would not have had to limit the dataset.
Data quality
The maturity of data quality is measured in a similar way as metadata is measured. The results are shown in table 5.
Case# Data quality maturity
Goals & principles
People Process Technology
1 1 1 1 1
2 4 3 unkown unkown
3 1 1 1 1
4 2 2 2 2
5 5 4 4 4
6 n/a n/a n/a n/a
7 2 2 2 2
8 2 2 2 2
9 4 3 4 3
10 2 2 2 2
Table 5: Data quality maturity
To the question whether the project would have been more successful if it would have a higher data quality maturity level, three of the interviewees responded that they thought this would be
40 the case. Two interviewees indicated that because there were no data quality issues in their specific projects it would not have made the project more successful. However, in general they agreed that it would improve the success of self-service BI. Two interviewees indicated that the project would not have been more successful in the sense of meeting business goals, but it would have helped to deliver the project in a more efficient and fast way. Two interviewees indicated that it would not have made any difference.
Overall, for metadata, eight out of the ten interviewees and for data quality seven out of the ten interviewees indicated that the project would in some way have been more successful if the maturity of the organization would have been higher. Similar results have been found by Kamioka et al. (2016), Simonson & Johnson (2008) and Lunardi et al. (2014). They all found that a higher maturity (of data governance or IT governance) resulted in a better performance or more success.
41
5. Discussion and conclusion
Recognizing the importance of self-service BI, the purpose of this study is to examine the effect of data governance maturity on the success of self-service BI solutions. In this chapter the practical and theoretical implications are described, followed by other findings, limitations and suggestions for future research.
5.1 Practical implications
Overall, the results of this study show that especially under certain conditions, investing in data governance maturity before implementing self-service BI can contribute positively to the success of self-service BI. These conditions can be seen as moderating variables and can influence the relationship between data governance maturity and self-service BI success independently from each other. The conditions are further explained in this section. In figure 2 a schematic overview of the relationship between data governance maturity and the success of self-service BI together with its moderating variables is shown.
42 The first condition is when the end-users and/or self-service BI developers are unfamiliar with the data that is presented via self-service BI. This study shows, that when this is the case, end-users might incorrectly interpret the data or draw incorrect conclusions. In the cases that are studied as part of this research, three situations are identified that trigger this condition. The first situation that might lead to end-users/ self-service BI developers being unfamiliar with the presented data is the implementation of a self-service BI solution that uses a company broad dataset. The dataset is then presented via one self-service BI solution; hence, no customized dataset is used for different target groups. The second situation triggering unfamiliarity with the data is the unavailability of data to end-users or developers in previous reporting solutions. Lastly, when the data has been transformed in a structure that is unknown by the end-users (e.g. because it was transferred from the source model to a dimensional model5) this might trigger unfamiliarity. When the first condition applies, a higher data governance metadata maturity can help end-users and self-service BI developers to gain increased insight into how to interpret the data, which data elements can be combined with each other, and in what way they can be combined.
The second condition under which data governance maturity influences self-service BI is when end-users lack data literacy. This research has shown that lack of data literacy is an important factor influencing the success of self-service BI. Training end-users in data literacy or hiring employees that have better data literacy might solve this problem. However, as long as data literacy of end-users does not improve, a higher level of governance maturity of metadata can
43 support end-users by providing explanations about the meaning of – and relationships between – the used data. A similar conclusion has been the reason for the research of Niemi et al. (2014). They found in earlier research that one of the challenges that self-service BI users face is to find out what data in their self-service report can be summarized and how this should be done. Lack of data literacy can be a reason for this challenge. To overcome this challenge a new way of categorizing dimension types is suggested, which should make it easier for self-service BI users to understand what data can be summarized and how to do this. Their solution can be seen as an improvement of metadata.
The third and last condition under which data governance maturity positively can impact the success of self-service BI is when the likelihood of using data of poor quality is high. For example, if it is already known that one of the source systems of a self-service BI solution contains data quality problems, a higher data governance maturity data quality level can help to specify these problems and speed up the process of solving them.
Based on this research it can also be concluded, that if the three conditions identified do not apply, the need to invest in data governance maturity before initiating a self-service BI initiative is less urgent. However, even when this is the case, one should realize that the absence or presence of the identified conditions can change over time. For example, users being unfamiliar with the data and users lacking data literacy, are both conditions that can change when for example new employees are hired that will work with the self-service BI solution. Additionally, the data quality of a source system can be good at a certain moment of time, but data quality issues can still occur in the future. As a consequence, even if the explained conditions do not
44 apply when starting the process of developing a self-service BI solution, analyzing and improving data governance maturity before, during and after the implementation of a self-service BI solution can still contribute to its success.
5.1 Theoretical implications
As described in the literature review, Spahn et al. (2008) found that end-users have problems in understanding, finding, and interpreting data in technically oriented databases that are made available to them via self-service BI tools. Similar results were found by Burnay et al. (2014).They found that complexity in self-service BI correlates negatively with its usability. This research found similar problems with self-service BI, but especially under the earlier described conditions. Spahn et al. (2008) suggests to reduce complexity of the self-service BI solution to solve this problem, even if this leads to a limitation of the flexibility and analysis possibilities. Burnay et al. (2014) states that a constant balance is needed, of on the one hand giving end-users all the features and data they want for their analysis and on the other hand giving the end-users a seamless and intuitive self-service BI experience. In some of the studied cases in this research a similar solution was mentioned: by limiting the presented dataset to a set that only contains data that end-users need, problems with data interpretation and data quality were minimized. However, it should be discussed on a case by case basis whether this is a desired solution. As described before, one of the main advantages of self-service BI compared to static reporting is that it provides end-users with the ability to perform further analyses on the presented data; for example by drilling down on results. When limiting datasets, this advantage might be decreased as well. This research found another way to improve the ability of end-users to understand, find, and interpret data. By improving data governance maturity (especially metadata maturity and
