Validating the design principles of an Information System Blueprint to attain cycle time measurability in Head and Neck Oncology pathways MSc Thesis DDM TOM

(1)

Validating the design principles of an Information

System Blueprint to attain cycle time measurability in

Head and Neck Oncology pathways

MSc Thesis DDM TOM

R.W.M. Oldenburger (s2229986 / B140614114)

DD MSc Technology & Operations Management, Faculty of Economics and Business – University of Groningen

DD Operations & Supply Chain Management, Newcastle University Business School – Newcastle University

Supervised by: Dr. H. Balsters and Dr. C. Hicks Word Count: 15.029

December 7th, 2015

Abstract

Currently, the majority of information systems implemented in the healthcare sector are not equipped to accurately measure cycle times. Especially in Oncology pathways, hospitals are not able to show their adherence to the new SONCOS quality standards imposed throughout the Netherlands. Moreover, no validated standardized method existed in literature to attain this cycle measurability in a manner least affecting the workflow of the domain experts (e.g. doctors). Therefore, this overall study aimed to design and validate an Information System Blueprint (ISB), composed of methods to create process- and data models, to monitor the cycle time of the diagnostic phase within a Head and Neck Oncology (HNO) pathway. In order to assess if the final design of the ISB adhered to all relevant stakeholder requirements, this specific study focused on the principles of design science to validate the process- and database models. Since no method existed for this purpose, this involved the initial design of a validation test method by means of User Interface Mock-ups. The study was conducted at a Large Teaching Hospital in the Netherlands (LTHN), and the results were found to be generalizable to other diagnostic phases of HNO pathways in LTHN subject to the SONCOS requirements.

Key words: Design Science, Validation, User Interface Mock-ups, BPMN, ORM, Detailed

(2)

Preface

(3)

Glossary

BPMN = Business Process Model and Notation

CSF = Critical Success Factors

DCM = Detailed Clinical Models

EHR system = Electronic Health Record system

FBM = Fact-Based Modelling

HL7 = Health Level 7

HNO = Head and Neck Oncology

HQMF = Health Quality Measure Format

ISB = Information System Blueprint

LTHN = Large Teaching Hospital in the Netherlands MDO = Multidisciplinary consult (without patient) MDS = Multidisciplinary office hour (with patient)

MDWE = Model-Driven Web Engineering

ORM = Object Role Modelling

PCP = Patient Care Pathways

SONCOS = Foundation Oncology Cooperation (Stichting Oncologische Samenwerking)

UI = User Interfaces

UIM = User Interfaces Mock-ups

UML = Unified Modelling Language

(6)

1 Introduction

Healthcare expenditure in the Netherlands is soaring (van Rooijen et al. 2013). An increase in chronic diseases, diseases related to a first world lifestyle, and an ageing population put pressure on the supply of healthcare. Efforts to curb costs risk a negative effect on the quality of patient care (Aiken et al. 2012). Simultaneously, healthcare is becoming more and more complex, involving various professionals in the treatment of one patient (Igz 2011). Despite this complexity, retaining a high quality of care can be crucial to ensure that treatment is provided on time. Correspondingly, foundations concerned with the quality of care of patients in critical conditions are introducing more countrywide guidelines (e.g. SONCOS, 2015). Hospitals have to show that they adhere to these guidelines via accurate measurement of patient information. Consequently, while there exists an increasing pressure to reduce costs, the significance of measuring and demonstrating the adherence to specific quality requirements in an increasingly complex healthcare environment becomes more significant.

(7)

(8)

One of the departments within hospitals that can benefit significantly from the ability to measure quality indicators is the Head and Neck Oncology (HNO) department. All Dutch HNO departments recently became subject to the requirements of SONCOS (2015). SONCOS, a Dutch foundation for oncology collaboration, prescribes quality guidelines for the HNO departments, which all hospitals should adhere to by the first of January 2016. One of these requirements refers to the cycle time of the diagnostic phase of the HNO pathway, which should be no more than three weeks.

Currently, the design and validation of a structured method to detail the workflow processes in HNO pathways complemented by the ability to enable cycle time measurability is an unexplored issue in literature. Lichtenberg (2015), Meerman (2015) and Schriever (2015) initiated the development of the design principles for a monitoring system for HNO pathway at the Oncology Department of a Large Teaching Hospital in The Netherlands (LTHN). This method to create a monitoring system for care pathways is referred to in this study as the Information System Blueprint (ISB). This ISB consisted of a combination of a method to create process models, called Business Process Model and Notation (BPMN), and a method to create data models, called Object Role Modelling (ORM). However, validation of their design, especially with respect to validation with end users, has never been fully executed. Hence, this study together with Alders (2016) and ten Holt (2016) will extend their work to add to the theory of a validated ISB in the context of the diagnostic phase of an HNO pathway, leading to the following overall research objective:

"In an Head and Neck Oncology (HNO) care pathway, to design and validate an Information System Blueprint (ISB) to attain cycle time measurability adhering to the SONCOS requirements"

(9)

for measurement. Therefore, in this individual study a third method to create a validation test for the correctness of the process- and data models, referred to as User Interface Mock-ups (UIM) was created as a supplement to this ISB. The three methods will be further detailed in the literature section. Correspondingly, the main research objective of this individual thesis is:

"In an Head and Neck Oncology (HNO) care pathway, to design a validation test method to attain accurate and complete feedback on the ability of an Information System Blueprint (ISB) to attain cycle time measurability adhering to the SONCOS requirements"

Since the “as-is” situation of the ability of HNO departments to measure cycle times is unsatisfactory, this problem can be characterized as a design science problem (van Strien 1997). Thereby, a practical-knowledge problem is proposed which needs to be solved (Wieringa 2009). In order to analyse the practical-knowledge problem, the structure of a regulative cycle is followed, entailing an iterative flow of problem investigation, solution design, and design validation focused on the requirements of end-users (van Strien 1997; Wieringa 2009). This ensures that the final design will take into account the needs of all stakeholders, by taking a sociotechnical rather than a pure IT perspective on development. Based on the individual research objective to design a validation test method for the evaluation of an ISB for HNO pathways via the regulative cycle, the following sub-questions were analysed:

1. Which quality attributes are available in literature to determine the utility of a validation method to assess information products?

2. What are existing methods for the validation of healthcare information product designs able to assess these quality attributes?

(10)

4. How should a validation test method be designed to attain accurate validation of information product designs for the diagnostic phase of HNO pathways in general?

Based on the main overarching objective to design an ISB to attain cycle time measurability adhering to SONCOS requirements via the regulative cycle, the following additional sub-questions related to validation were analysed in this study:

5. Which quality attributes are available in literature to determine the utility of an information product?

6. Who are the stakeholders affected by an Information System Blueprint (ISB) for the diagnostic phase of a Head and Neck Oncology (HNO) pathway and what are their goals and Critical Success Factors?

7. How should an ISB be designed to attain efficient and effective cycle time measurability in the diagnostic phase of HNO pathways in general?

(11)

2 Theoretical background

This section will discuss the theory relevant to a design science study in the context of Head and Neck Oncology (HNO) departments. Figure 1 shows the conceptual framework, based on the IS research framework as proposed by Hevner, Match and Park (2004). The right hand side of Figure 2.1 outlines which foundations and methodologies are relevant to the validation of an Information System Blueprint (ISB) for Head and Neck Oncology (HNO) pathways, carried out in this thesis. Both the applied foundations and the methodologies used for the provision of guidelines in the justify/evaluation phase will be discussed. Moreover, theoretical and practical considerations associated with the measurement of cycle time will be outlined. The aspects of the overall design science study, represented in the middle box in Figure 2.1, that were conducted in this individual thesis are shown in red.

(12)

2.1 Concurrent Engineering

In the design of complex systems, individual parts of the system can be designed separately but they have a high influence on each other (Loch et al. 2003). Within complex systems, the optimal system cannot be identified from the outset. Therefore, in order to reduce uncertainty in the design of complex systems, the development process can be transformed from a sequential to a concurrent process, (Koufteros et al. 2001). Thereby, at each decision moment, the system blueprint developer considers the other components in the system, which affect each other’s performance. In the context of software development, complexity in the current environment of many projects has been acknowledged, resulting in the emergence of methods for building software in an incremental way. One of these is the Agile/SCRUM software development process (Rising & Janoff 2000). The Agile/SCRUM approach is a manner of software development via iterative development cycles, ensuring that progress is made even when requirements are unstable. Agile/SCRUM-type development processes focus on short feedback loops, testing early and frequently whether the system being developed delivers the highest value as defined by the stakeholders (Schwaber 2004).

In their current situation, many Dutch care pathway include various stakeholders and databases, which face continuous problem choices. Stakeholders are widely defined by Freeman (2010, p.53) as “[…] any group or individual who can affect or is affected by the achievement of the organization’s objectives”. Correspondingly, a high variety of stakeholders who are affected by the implementation of a monitoring system can make the requirements for such a system unclear or uncertain. Hence, the design of an ISB for a HNO pathway can be characterized as a complex system which should be organized via a similar concurrent engineering approach where the three individual roles of process model development, data model development, and validation will be executed in an iterative manner.

2.2 Business Process Model and Notation

(13)

representation, designed to simplify the understanding of processes by all business users, including non-technical stakeholders (OMG 2011). Therefore, BPMN is currently considered a standard for business process modelling. However, the application of BPMN in a medical context is fairly new (Scheuerlein et al. 2012). Rojo et al. (2008) analysed the use of BPMN for the modelling of processes of Anatomic Pathology, and reported positive findings on the its applicability to model the medical sub processes in an understandable way. Scheuerlein et al. (2012) showed in their results from a pilot project on the feasibility of the construction of clinical pathways by means of BPMN that the standard is sufficiently suitable for the modelling of clinical pathways. It allows medical and IT knowledge to be integrated, and be adjusted to individual requirements. Barros and Quezada (2013) took a first attempt at developing an integrated design approach for the overall process architecture of hospital processes expressed in BPMN. Notwithstanding, a framework of how exactly findings of a stakeholder analysis can be translated into BPMN models in order to assure that the workflow “as-is” is fully correct according to their requirements has not been developed yet. Hence, this will be the focus of the process modelling aspect of the Information System Blueprint, designed by Alders (2016). Moreover, a structured method for the design of BPMN models in a healthcare context has never been used in combination with a data modelling method to allow for the identification of standardized information objects corresponding to the “as-is” workflow. This combination could lead to the ability of Head and Neck Oncology pathways to measure cycle times of their current processes. The following section will analyse the data modelling method pursued in this study for this purpose.

2.3 Object Role Modelling

(14)

in terms of entities having specific attributes, ORM simply describes objects participating in relationships. Relationships in ORM are not restricted to binaries but ORM allows relationships of any arity (number of roles), enhancing its clarity as well. Being attribute-free assures that if later in the system development process an information requirement is found which was initially not clear, this can easily be included as an additional object. Thereby, the attribute-free ORM models and the corresponding queries are more stable, since they are free of changes caused by attributes evolving into other constructs, such as changes in the business domain.

This advantage of ORM has also been stressed by Halpin and Bloesch (1999) who provided an extensive comparison between Unified Modelling Language (UML) and ORM. The authors stressed that ORM allows the data-model developer to overcome the shortcomings of UML by including the ability to validate data concepts and business rules with domain experts. This ensures that ORM can serve as a database modelling approach of which the correctness can be evaluated with non-technical end-users. Notwithstanding, to the best of our knowledge it has never been applied in a healthcare context, apart from within theses preceding the current study. However, considering its possibilities for accurate validation due to clarity and semantic stability, ORM is expected to be an appropriate data modelling method for the purposes of designing an Information System Blueprint (ISB). Correspondingly, its appropriateness will be assessed in this design science study.

2.4 Design Validation

This section will first explain the requirements of validation within design science. Subsequently, a review will be provided on the available methods for validation or evaluation of information systems in a hospital setting.

(15)

2.4.1 Validation Within Design Science

Design validation in the context of the regulative cycle entails the assessment of whether a specified design, when implemented correctly, would adequately fulfil all stakeholder objectives (Wieringa 2009). Boonstra (2006) explained this type of validation in the general context of ERP-systems. These systems are constructed according to certain business models, based on their own assumption of integration, control, generic processes and centralisation. According to Boonstra (2006), this paradigm can fit or misfit with an organisation, which may lead to either organisational validity, in the case of a fit, or organisational invalidity, in the case of a misfit. In the context of design science research, Hevner et al. (2004, p. 85) stated, “the business environment establishes the requirements upon which the evaluation of the artefact is based”.

As shown in the ‘knowledge base’ box in Figure 2.1, three types of knowledge questions should be discussed in order to assess validity (Wieringa 2009). Firstly, the extent of internal validity of the design should be assessed. This entails testing whether the design satisfies the criteria identified in the problem investigation phase in the specific context being analysed. Secondly, trade-offs between marginally different designs should be analysed. Thirdly, external validity should be confirmed, meaning that the design also adheres to the criteria when implemented in a comparable context (ibid.). This latter validity test is also referred to as a sensitivity analysis of the designed method and provides an indication on its generalizability.

2.4.2 Usability Engineering

(16)

acceptance measures. The main goal of UT is twofold: improving the usability of the product which is being tested, and to improve the actual process by which the products are designed and developed (Dumas & Redish 1999). To this purpose, the ISO/IEC 25010:2011 (section 4.2.4) describes usability, referring to a subset of quality characteristics of a product of a system, as: “the degree to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”.

Since Hevner et al. (2004) stressed that IT artefacts developed via designs science should be rigorously evaluated to assess the utility, quality and efficacy of the design solution, techniques of UT appear suitable for this purpose. Moreover, Rolón et al. (2015) expressed the need for the development of a framework which allows the evaluation of the usability of process models designed in the healthcare context. Hence, this suggests that a method for the analysis of the validity of the Information System Blueprint (ISB) designed in this overall study should be able to apply principles of UT to assess its effectiveness, efficiency and satisfaction. A method for the assessment of these quality attributes will be proposed in the methodology section.

2.5 The measure of cycle time and Detailed Clinical Models

(17)

eMeasure Cycle Time = final activity diagnostic phase – first policlinic visit. An unambiguous eMeasure of cycle time can only be extracted if agreements can be made on the exact information retrieval points which represent the start and the end of the diagnostic phase of the HNO pathway. Correspondingly, a specific requirement for Object Role Modelling (ORM) in the context of clinical pathways stipulates the need to incorporate Detailed Clinical Models (DCM) in the construction of objects. This assures that the data elements as specified in the diagrams are in accordance with the clinical object standards as developed for the healthcare sector (Goossen et al. 2010). For Dutch healthcare, DCM have been specifically designed within the programme “Registration at the source” (“Registratie aan de bron”), a cooperate programme between eight University Medical Centres in the Netherlands and Nictiz, to depict clinical concepts in a standardized manner based on the HL7 standard (ISO/TS 2015). An example of the DCM of the concept “Concern” is shown in Figure 2.3 (NFU 2015b). This DCM includes for instance the data element “ConcernType” (“ProbleemType”) which represents certain

types of patient problems. The document “ConcernTypeCodelist”

(18)

(19)

3 Methodology

This section will discuss the methods that have been applied in order to answer the research question according to the research onion framework presented by Saunders et al. (2009). First the research philosophy and approach advocated by this thesis will be discussed. Subsequently, the methodological choice and pursued strategy of design science will be elaborated on. Finally, the time horizon will be discussed and the procedures for data collection and analysis will be outlined.

3.1 Research Philosophy and Approach

Saunders, Lewis and Thornhill (2009) outlined several broad research philosophies related to the development of knowledge and the nature of that knowledge. The perspective taken on the effective development of an Information System Blueprint (ISB) and a validation test method in this thesis was one which advocates a business rather than an IT point of view. Hence, the aspect of ontology proposed by the current research was one of subjectivism, in which the ‘social actors’ can place a variety of different interpretations on the problem situation, which need to be incorporated. Moreover, since the nature of this study assumed that acceptable knowledge could be based on subjective meanings of stakeholders, in terms of epistemology it can be referred to as more interpretivist research. This necessity to incorporate subjectivity in the construction of information systems in emphasized by Pinch and Bijker (1984, p.414) who noted that for a problem to be determined relevant “[…] a problem is only defined as such, when there is a social group for which it constitutes a problem’’.

Furthermore, since this study entailed the creation of an Information System Blueprint (ISB) and a validation test method by means of input from both the existing knowledge base as well as the problem context (see Figure 2.1), the research could be characterized as abductive, in which data and theory were applied in an iterative manner throughout the research (Suddaby 2006).

3.2 Methods and Research Strategy

(20)

constructive meanings were interpreted, the research method associated with this philosophy was a qualitative research design (Denzin and Lincoln 2005).

Moroever, the corresponding research type applied in this thesis to develop the Information System Blueprint (ISB) was the method of design science. In terms of the methods outlined in Saunders, Lewis and Thornhill (2009), design science is comparable to the process of Action Research. Similarly, design science requires the participation of both the researcher and members of the organization through collaboration in iterative design cycles. The following sections will discuss in further detail why the overarching research question analysed in this thesis could best be answered by means of a design science approach. Subsequently, the specific method, which was used for validation of the ISB will be outlined.

3.2.1 Design Science

(21)

3.2.2 The Regulative Cycle

Hevner et al. (2004) emphasized that in the context of information systems’ design, incorporating a behavioural perspective can be highly effective in assuring that the system will actually offer opportunities for improving the efficiency and effectiveness of organizational processes. The authors stressed that design science research is naturally iterative where “effective design requires knowledge of both the application domain (e.g. requirements and constraints) and the solution domain (e.g. technical and organizational). Ultimately, this iterative structure should result in a “satisfying” solution (Simon, 1996). One of the methods which can be used for this purpose, and has been applied as the research strategy in this study, is referred to as the regulative cycle (Wieringa 2009; van Strien 1997). Van Strien (1997) detailed the phases of the regulative cycle, including the following stages:

1. Design Problem: Mapping and understanding the application and solution context of the problem through an analysis of the stakeholders and their corresponding requirements in the form of goals and Critical Success Factors (CSF);

2. Diagnosis/Analysis: Further analysis of the application and solution context of the CSF. This phase involves identifying the considerations involved with resolving the CSF, by outlining the corresponding quality attributes and identifying possible interdependencies;

3. Design Solution: Designing the actual artefact either from scratch, or as assembled from “old solutions”;

4. Implementation: Executing “the goal” of the solution design. Wieringa (2009) emphasized that the meaning of implementing the (initial) design solution depends on the solution that has been designed;

5. Validation: The development of a test method in order to validate the fulfilment of all end user requirements by the artefact.

(22)

(2009) where design validation takes place before the actual implementation to maximize the chance of acceptance by the end users before resources are devoted to implementation.

In this individual study, a validation test method was created by means of the regulative cycle, henceforth referred to as the “individual regulative cycle”. Subsequently, this regulative cycle was part of a regulative cycle in which the Information System Blueprint (ISB) was designed, henceforth the “overall regulative cycle”. Here, the validation test method formed the framework for validating the ISB (in the validation cycle), developed in the process- and data-model cycles by Alders (2016) and ten Holt (2016) respectively. An overview of this structure is shown in Figure 3.1. The red lines in Figure 3.1 detail how each individual phase provides input to the subsequent one. The blue lines show how the principles of Concurrent Engineering, described in section 2.1, were added to this feedback process such that findings of the validation phases of an individual cycle were directly incorporated in the other individual cycles.

(23)

3.2.3 Design Validation Method

As identified in the literature section on Usability Engineering (section 3.2.3), an appropriate method for the analysis of the validity of the Information System Blueprint (ISB) designed in the overall regulative cycle should be able to assess its effectiveness, efficiency and satisfaction.This section will analyse existing methods for the evaluation of clinical information systems, and identify a possible appropriate method for the purposes of this study.

For internal validation, Staron, Kuzniarz and Wohlin (2006) assessed the effectiveness of stereotypes in UML class diagrams to comprehend Object-Oriented applications in the telecommunication domain. They reported positive findings on the use of stereotypes to help students and professionals improve their comprehension of the diagrams. Conversely, Ricca et al. (2007) found that stereotypes offered little help in a context where users have different levels of ability and experience. Hence, this method would not be suitable in the current context where many different types of stakeholders with varying technical capabilities exist.

(24)

A more suitable method can be found in design engineering research. According to Rivero et al. (2014), Model-Driven Web Engineering (MDWE) methods, which apply a process through which a set of models is created that can generate a Web Application automatically, often ignore incorporating customer feedback or early design of user interfaces (UI). With Web Engineering, changes to the UI might represent entire changes in the business logic, and when this is left to the end of the development cycle, this can result in the introduction of uncovered requirements (Rivero et al. 2014). Hence, various authors have proposed the use of UI prototypes, known as mock-ups, to aid the early evaluation of the quality of a solution design (Baumer et al. 1996). UI prototyping is a development method advocating the development of executable software systems, prototypes, for experimental purposes. Ricca et al. (2010) confirmed the effectiveness of screen mock-ups by showing that it enhances the understanding of system requirements by end users. This opportunity for increasing the understanding of all requirements by end users is crucial in the validation of a social technical system. For this reason were UI prototypes of actual systems such as Electronic Health Record (EHR) systems also introduced as a tool to evaluate the usability of prototypes of clinical information systems (Kushniruk & Patel 2004; Jaspers et al. 2004; Kushniruk et al. 1997). Since UI mock-ups (UIM) optimize the understanding of systems’ specifications by non-technical end users, UI mock-ups (UIM) could also be effective in assessing the effectiveness, efficiency and satisfaction with which stakeholders can achieve their specified goals for an ISB. Here, the UIM would not represent prototypes of an actual clinical information system, but reflect the process- and data-models which form the ISB. Therefore, UIM were applied in this study to validate the ISB.

(25)

appropriate tool for the purposes of this study needed to enable the most simple and efficient manner of interface design. If the most simplest manner of interface building could allow for accurate validation of an information system design, this confirmed the practical value of the development of a general method for building UIM as a validation test method. Therefore, the form building software Google Forms (Google 2015) was pursued in this study. Google Forms (Google 2015) is widely available to all with a Google account, and allows for simple form creation.

Finally, external validity, or sensitivity, of the ISB to attain cycle time measurability in Head and Neck Oncology (HNO) pathways in general can be assessed corresponding expert opinions on the generalisation capabilities of the ISB.

3.3 Input from Business Process Model and Notation (BPMN)

In order to be able to understand the elements of BPMN which are validated in this study, a description of the structure of BPMN as modelled in the software tool (Bizagi 2015), applied by Meerman (2015) and Alders (2016) and outlined by Chinosi & Trombetta (2012) follows.

(26)

description of the elements of BPMN used as input for the User Interface Mock-ups, the reader is referred to Appendix A.

3.4 Input from Object Role Modelling (ORM)

This section will explain in more detail the structure of ORM models in the ORM 2 notation, applied by Schriever (2015) and ten Holt (2016) supported by the NORMA (Neumont ORM Architect) tool (Halphin & Morgan 2010).

The notation of Object Role Modelling (ORM) consist of two main type of objects, ‘entity types’ and ‘value types’. Entity types are named, soft rectangles with a reference scheme. This scheme is included to refer to specific instances of that entity type, such as “(.nr)” for a RoomNr., depicted in Figure 3.3 (Halphin & Morgan 2010). Value types do depicted as named, dashed soft rectangles. Furthermore, relationships between these objects are shown as named sequences of one or multiple role boxes (the rectangular boxes between object types), known as fact types. The lines above the role boxes indicate uniqueness constraints indicating unique relationships. Finally, the solid dots represent a mandatory role constraint, entailing that each instance in the population of that object must play that role (e.g. each Activity must have an ActivityName). Some unary fact types in ORM can have an outcome of either true or false. The outcome “true” is called an equality constraint and is represented by a circle with an ”=” sign included. The outcome “false” is referred to as an exclusion constraint and is depicted by a circle with

(27)

an “x” included. For a more extensive description of the elements of ORM used as input for the User Interface Mock-ups, the reader is referred to Appendix B.

3.5 Time Horizon

The research time frame mostly corresponding to a Design Science or Action Research type of study is a longitudinal study. The final solution design was developed over a period of three months in which the requirements of the design could alter.

3.6 Data Collection and Analysis Procedures

Data collection for the initial design problem phase and diagnosis/analysis phase of the individual- and overall regulative cycle was done by means of an unstructured, one-to-one interview with the gatekeeper of the study. Unstructured interviews are particularly useful in order to be able to explore in depth a general area of interest (Saunders et al. 2009). Notwithstanding, there should be a clear understanding about the aspect which is being explored (ibid.) Since this study continued on the research of a previous group of students, these interviews discussed the stakeholder analysis concerning the Information System Blueprint (ISB) as created by Meerman (2015) and Schriever (2015), and the stakeholder analysis for the validation test method as created by (Lichtenberg 2015).

Data collection for the subsequent iterations of the individual regulative cycle was done via presenting the User Interface Mock-ups (UIM) to the end-users, guided by a semi-structured interview. As explained in section 2.4.2, the validation test method developed in this study was linked to the principles of Usability Testing (UT), in order to be able to assess the usability of the Information System Blueprint (ISB), as well as to Figure 3.3: Example ORM diagram for room scheduling. Source: Halphin

(28)

improve the actual process by which the products are designed (Dumas & Redish 1999). In this individual study this process entailed the method of validation. A prerequisite for UT was the use of subjects who are representative of the target user population, and who perform representative tasks (Kushniruk & Patel, 2004). Hence, the UIM have been discussed with the end-users of both the Information System Blueprint (ISB) and the validation test method. The use of semi-structured interviews is in line with the type of interview proposed by Saunders, Lewis and Thornhill (2009) for inductive research. Semi-structured interviews allow for flexibility in interviews, which can provide a broad understanding of the subject (Verhoeven 2011). Moreover, some group interviews were held for the benefits of allowing the interviewees to challenge one another’s view (Saunders et al. 2009). A principle of UT which has been applied during these validation interviews is the technique of “think aloud” reports (Kushniruk & Patel 2004). In “think aloud” reports, the subjects verbalize their thoughts while using the product in its particular context of use. UT can be, amongst other stages, useful during both the stage of analysis, involving gathering of the system requirements, and during the design of the system (Kushniruk & Patel 2004; Jaspers et al. 2004). Hence, the data collection via interviews focused both on verifying the stakeholder requirements identified in the unstructured interviews, as well as on validation of the ISB and the design of the UIM themselves.

(29)

(30)

Table 3.1: Category definitions – Individual regulative cycle

* Main categories follow from the goals specified by the stakeholders and are shown in bold and italics

** Sub-categories follow from the quality attributes corresponding to the Critical Success Factors specified by the stakeholders and are shown in bold

Table 3.2: Category definitions - Overall regulative cycle

* Main categories follow from the goals specified by the stakeholders and are shown in bold and italics

** Sub-categories follow from the quality attributes corresponding to the Critical Success Factors specified by the stakeholders and are shown in bold

Main categories* and Sub-categories** Explanation

Stakeholder Analysis All statements defining the goals and Critical Success Factors (CSF) for the individual regulative cycle

Efficiency and Effectiveness - Easiness to interpret

(Analysability)

The clearness and comfort with which the elements shown by the validation test method can be understood.

- Accuracy in reflecting elements of the ISB

The extent to which all elements of the BPMN models and the ORM models are correctly represented

Availability to apply in further projects

- Flexibility The degree to which the validation test method is usable in contexts beyond those initially specified in the requirements

Appropriateness recognisability of the ISB - Clarity on the start and end of the

diagnostic phase

The extent to which the validation test method can identify the start and end of the diagnostic phase in an unambiguous manner

- Unambiguity The extent to which the validation test method is precise or clear in describing the workflow and the corresponding information objects

- Completeness recognisability of the DCM

The extent to which the validation test method can identify gaps in the existing set of DCM

Main categories* and Sub-categories** Explanation

Stakeholder Analysis All statements defining the goals and Critical Success Factors (CSF) for the overall regulative cycle

Minimization affection current workflow

- Completeness The extent to which all relevant working processes are correctly represented

- Minimization of the necessity for additional actions

The degree of additional checks or proceedings necessary to allow for cycle time measurability

Availability to apply in further projects

- Flexibility The degree to which the ISB is usable in contexts beyond those initially specified in the requirements

Ability to enable measurability of cycle time - Unambiguity on the start and end

of the diagnostic phase

The degree to which the ISB allows the Head and Neck Oncology (HNO) pathway to indicate their adherence to the SONCOS (2015) requirements for cycle time

- Relevance recognisability of DCM

(31)

(32)

4 Results

The discussion of the results will be divided in a section presenting the results of all phases of the individual regulative cycle of this thesis, and a section outlining the results of the design problem, diagnosis/analysis and design validation phase for the overall regulative cycle.

4.1 Results Individual Regulative Cycle

Since the aim of this individual study was to develop a validation test method via the regulative cycle in order to be able to assess the ability of the Information System Blueprint (ISB) to attain cycle time measurability, the results of all four phases of this cycle will be discussed.

4.1.1 Design Problem: Stakeholder Analysis

(33)

Table 4.1: Stakeholder analysis - Validation test method

Stakeholder Goal CSF

Di

re

ct

Healthcare provider A validation test method which allows efficient and effective validation of the ISB

The validation test method must be easy to interpret and accurately reflect all elements of the ISB Case manager A validation test method which

allows efficient and effective validation of the ISB

The validation test method must be easy to interpret and accurately reflect all elements of the ISB

In

d

ir

ec

t

HNO process manager An appropriate validation test method which is available to apply in further projects

The validation test method must be flexible and accurately reflect all elements of the ISB

Principle specialist oncology department

A validation test method to assess the ability of the ISB to enable accurate measurement of cycle times

The validation test method must be clear on the start and end of the diagnostic process, unambiguous, and accurately reflect all elements of the ISB

Information manager A validation method to assesses the ability of the ISB to provide an accurate eMeasure of cycle time.

The validation test method must be clear on the start and end of the diagnostic process and the relevant DCM must be complete

4.1.2 Diagnosis / Analysis

(34)

4.1.3 Solution Design

In order to determine if the User Interface Mock-ups (UIM) could serve as a an accurate test for the evaluation of the correctness of the Information System Blueprint (ISB), the extent to which the method was able to fulfil all its stakeholder was assessed. Lichtenberg (2015) proposed an initial general method for creating interfaces by means of Google Forms derived from Business Process Model and Notation (BPMN) models and data models in the Object Role Modelling (ORM) notation. However, Lichtenberg (2015) was not able to subject this method to a validation according to the principles of design science.

The feedback provided by the stakeholders during the interviews enabled the iterative improvement of a general method to create the UIM. This general method for creating UIM is described below. A discussion of how the final design of the method was created based on end-user feedback will be provided in the section presenting the results of the validation phases of the individual regulative cycle (section 4.1.4). Correspondingly, section 4.1.4 will also discuss the extent to which this final design of a general method to create UIM as a validation test is able to fulfil all quality attributes described above.

Method for creating UI mock-ups as validation test

(35)

(36)

Step 1: Define the banner information

According to the design guidance document for clinical noting in forms (Microsoft 2009), some information should always be present in the system’s interface to ensure the safety of the patient. Hence, a banner can be constructed in the Google, which includes this required information (see Figure 4.2).

Step 2: Define the responsible stakeholder and the title of the activity

In order to validate whether the stakeholder responsible for each activity, as designed in the BPMN and ORM models, are correct, they should be incorporated in the forms. As shown in Figure 4.3, the direct stakeholders represented in the swimlanes of the BPMN models have been directly transferred to the ORM model, where the stakeholder is indicated by the relationship “performed by”. Correspondingly, these stakeholder references should be translated to the page titles of the forms that specify the activities for which they are responsible. Subsequently, the title of each activity should be indicated after the stakeholder. The denotation of each activity should correspond to the activities in the BPMN models. Thereby, the active description of each activity in BPMN is transformed into a corresponding passive description of this activity in the UIM, as shown in Figure 4.3. Moreover, the denotation of each activity should also correspond to the entity types with the reference scheme “.nr”, representing the same activities in the ORM notation. Hence, the description of the object, or entity type, is transformed into the same corresponding passive description of the role played by this entity type. For example, the first visit of the patient at the policlinic leads to the activity “ConductAnamnesis” in BPMN, to the object “Anamnesis” being performed by the healthcare provider in ORM, and to the activity “Anamnesis Conduction” in the UIM. Consequently, the structure of a “page title” becomes: “Stakeholder – Activity”.

(37)

Step 3: Define the context of the activity and the corresponding input, output, pre-conditions and post-pre-conditions.

Subsequently, a context description should be provided with each activity. This context description is twofold. Firstly, a visual representation of the location of each activity in the larger process should be provided by inserting an image as shown in the upper half of Figure 4.4. Secondly, a textual context description phrased in language used by the end-users should detail who are concerned with the activity and the specific actions performed during the activity (see bottom half Figure 4.4).

Figure 4.4: Visual and textual description of context

(38)

Moreover, pre-conditions, post-conditions, input and output for each activity should be defined. Pre-conditions refer to specific statements that have to be “true” before a certain activity can start (Kaiser 1989). In more general terms is the pre-condition of each activity that follows a decision moment, linked to an output edge of that gateway in the BPMN model and the corresponding constraint of a unary fact-type in the ORM model, as shown in Figure 4.5. An example of a pre-condition for the activity “Information Check” is “There is an observation of insufficient patient examination(s)”. In contrast, the input of an activity refers to the information that has to be processed by the user of the system at the beginning of an activity to be able to execute the activity. This input corresponds to the input of each activity in the BPMN model (OMG 2011). An example would be “An indication of the necessary additional patient examination(s)” as input in to the same activity “Request additional patient examination(s)”.

In a similar manner do post-conditions refer to characteristics of the results of an activity which have to be “true” (Kaiser 1989). This can be a required status change in the system at the end of an activity, e.g. “A request has been filed for additional patient examination(s)”. These status changes could serve as validation of at what moment in the HNO process changes in the patient file are made indicating the end of the diagnostic phase. Contrastingly, the output of an activity refers to what the activity actually makes available for the progress of the process. This output corresponds to the output of each activity in the BPMN model (OMG 2011). This would hence be “A request for additional patient examination(s)”. An example of the combined description of the

(39)

context, pre-conditions, post-conditions, input and output of the activity “Anamnesis” is shown in Figure 4.6

Step 3: Define the question fields of the detailed interfaces

Forms which include question fields are referred to as detailed interfaces rather than direction interfaces. Direction interfaces serve merely as a guidance through the process in order to validate its correctness. Detailed interfaces allow the validation of gateways represented in the BPMN models, and specific unary fact-types coupled to the process in the ORM models (see Figure 4.7).

(40)

Moreover, detailed interfaces allow for identification of a link between the moments where modifications in the description of a patient concern are being made in the current “as-is” system and the corresponding concrete changes in an integrated “to-be” system. Validation of these data elements identified in the ORM models is crucial for enablement of the extraction of the eMeasure of cycle time from a “to-be” information system. An example is the conclusion of the letters drawn up after each anamnesis or multidisciplinary consult in the current process, which can be represented in the UI as open paragraph questions where the healthcare provider responsible for the tracking of the diagnosis status of the patient can indicate the conclusion on the status of the patient’s tumor (see Figure 4.8). Hence, a representation of such a field could aid the indication of where the diagnostic phase ends. Subsequently, the conclusions on the patients’ status filled in in these open fields could be possibly linked to the DCM “PatientConcern”, which has been identified during the validation of the ISB as the DCM allowing for the capture of the various patient concern types (“ConcernType”) (NFU 2015b). The example of this link is shown in Figure 4.8.

(41)

Step 4: Define the form flow logic

The form flow logic is based on the sequence flows in the BPMN models and associations in the ORM models. This can be defined in Google Forms in a similar manner as explained by Lichtenberg (2015). The form flow can result from regular sequence flows and associations between two activities or objects, or from the fact that certain paths in the process are only pursued when a specific condition is true as in Figure 4.5 and Figure 4.9, In other words, if a certain answer, or output edge, is chosen as “true”, the processes will continue to the activity corresponding to this specific answer.

Figure 4.8: Description of the validation question of patient’s status modifications

(42)

4.1.4 Design Validation

This section will present the main findings of the validation of the User Interface Mock-ups (UIM) as validation test method for an Information System Blueprint (ISB) consisting of process- and data models. Internal validation and trade-offs will detail the analysis of the extent to which the UIM were able to fulfil all stakeholder requirements presented in the stakeholder analysis in section 4.1.1. In terms of Usability Testing, this entails that the objective of the validations had been formulated as the assessment of the UI functionality and usability, and as the main input into the refinement of emerging prototypes of the UI (Dumas and Redish, 1999). Here, UI functionality and usability is defined by the extent to which the UIM as a validation test method fulfil all stakeholder requirements. A discussion on how the general method was developed for the creation of UIM as a validation test, based on these requirements, will be presented. Moreover, the considerations for external validation, i.e. the extent to which this design solution can serve as a general method for the validation of information system designs, will be outlined.

Internal Validation and Trade-offs

For the validation of the UIM, as outlined in the methodology section, semi structured interview via the principles of “thinking aloud” reporting were conducted. These interviews together with findings during the design solution phases have exposed several considerations for internal validation as well as for trade-offs.

Easiness to interpret

(43)

information systems’ end-users, a description of the context of each activity was mentioned as a practice to make to validation test method more tangible for the end-users. It was stated by a Healthcare Provider during one of the earlier interviews that “What actually happens during at the end of an MDO is that the healthcare provider writes down in a letter what the current diagnosis is. This is not clear from the interfaces” (Translation). Moreover, sometimes it was found to be difficult to comprehend for end users when two activities would happen in parallel. Overall, after the inclusion of a textual and visual context description and the denotation of al textual parts in language understood by the end-users, it became apparent that the validation interviews progressed fluently without large misconceptions on the elements shown by the UIM.

Accuracy in reflecting elements of the ISB

The multiple design solution phases showed that the method for creating UIM based on BPMN and ORM models was able to translate all objects, relationships and constraints modelled in the ORM models to Google Forms in a standardized manner. Moreover, all flow objects, out-put edges and events shown in the BPMN could be included in a standardized way. Notwithstanding, not all gateways of the BPMN models could be translated to the UIM one-to-one. If a gateway resulted from an activity which is performed by multiple stakeholders, but only one of these stakeholders was responsible for the decision represented by this gateway, an additional form had to be created. This form would then represent an activity which is not a flow object in BPMN, but should be incorporated to ensure that the correct responsible stakeholder is attached to each activity or gateway.

Flexibility

(44)

Clarity on the start and end of the diagnostic phase

With respect to this quality attribute is was noted that the most important way of ensuring that the scope of the diagnostic phase is clear is the creation of a link to the “letter structure” used in the “as-is” HNO pathway to keep track of the status of knowledge on the patient’s tumor. It was stated by a Case Manager that “During the MDO, the final status of the tumor and the final care advise are discussed. These are discussed with the patient that afternoon and then they are “made definitive” in the letter, which ends up in the system” (Translation). In addition, a Healthcare Provider stated “During each consult with or about the patient, a letter is drawn up with at least the reason of visit, the anamnesis, physical examinations, possible necessary additional examinations, a conclusion on the status of the tumor, and the treatment policy”. The inclusion of a data entry field in the UIM for the validation of the content of this letter enabled various stakeholders to identify what they consider to be the start and end of the diagnostic phase.

Unambiguity

(45)

Completeness recognisability of the DCM

The ability to assess the completeness of the DCM was found to be possible by relating specific elements of the letter structure, referred to above, to question fields on possible corresponding DCM attributes. Validation of these DCM attributes led to the acknowledgement that some DCM within the current set of DCM were insufficiently able to make a clear link to the “as-is” pathway terminology of domain experts. For example, a Case Manager indicated that “The patient concern type corresponding to the end of the MDO would be a final diagnosis” (Translation). However, the current content of the DCM “Concern” does not yet include the “PatientConcernType” final diagnosis. Correspondingly, it was confirmed by an Information Manager that indeed the UIM identified this correctly, and that it should be considered whether this element should be included in the concerned DCM.

External Validation

During the consecutive cycles of validation of the ISB and the validation test method, several specificities of the context of this study’s scope became apparent. External validity of the validation test method was assessed via expert opinions on the extent to which this these specifics are similar for other pathways and hospitals, and hence on the suitability of the UIM to assess the appropriateness of the ISB in other (HNO) pathways and hospitals as well.

It was discovered that the diagnostic phase of the Head and Neck Oncology (HNO) pathway in the focal hospital is characterized by a relatively simple workflow, with a relatively clear overall sequence of activities. A few activities can happen in parallel and there are some specific tasks for which the responsibility is not predetermined. The different roles (core disciplines and supporting disciplines) involved in the HNO pathway are clearly determined.

(46)

need to adhere to the SONCOS requirements. Therefore, they would benefit from the same identification of the start and end point of the diagnostic phase. Correspondingly, it was emphasized by Information Managers that “Important is that the “as-is” situation needs to be translated to the “to-be” situation. The interfaces should make a link to the DCM to validate with, for example, a case manager which information is stored” (Translation). As discussed above, the UIM were able to create a link between the “letter structure” with open text field, currently applied in many pathways and hospitals to capture status changes (“as-is”), and the Detailed Clinical Models (DCM) structure (“to-be”). Moreover, these DCM have been developed by the eight UMC in the Netherlands and Nictiz based on the same HL7 standard. Hence, this indicates that the validation test method could assess the ability of the ISB to attain cycle time measurability by means of DCM identification for other diagnostic phase of HNO pathways as well.

(47)

4.2 Results Overall Regulative Cycle

This section will discuss the results of the design problem phase, the diagnosis/analysis phase and the validation phase of the overall regulative cycle, covered in this thesis. For an overview of the other phases, the reader is referred to Alders (2016) and ten Holt (2016).

4.2.1 Design Problem: Stakeholder Analysis

(48)

Table 4.2: Stakeholder analysis – Information System Blueprint

4.3 Diagnosis / Analysis

The diagnosis phase of the overall regulative cycle entailed evaluating what the general quality attributes are corresponding to the Critical Success Factors (CSF) for the Information System Blueprint (ISB) identified above, and whether there exist any possible order dependencies or trade-offs between these attributes. This section will elaborate on these context considerations by means of describing the evaluation of the stakeholder analysis presented above.

From an evaluation of the CSF of the ISB identified in the stakeholder analysis, several corresponding quality attributes became clear. These quality attributes could be summarized as: completeness, minimization of the necessity for additional actions, flexibility, unambiguity on the start and end of the diagnostic phase, and relevance recognisability of the DCM. No order dependency or trade-offs were expected for these quality attributes.

Stakeholder Goal CSF

Di

re

ct

Healthcare provider An ISB which allows cycle time measurability in a manner least affecting the current workflow

The ISB must correctly represent all relevant working processes and minimize the necessity for additional actions

Case manager An ISB which allows cycle time measurability in a manner least affecting the current workflow

The ISB must correctly represent all relevant working processes and minimize the necessity for additional actions In d ir ec t

HNO process manager An ISB which can be applied in further projects for the creation of a monitoring system

The ISB must correctly represent all relevant working processes and be flexible

Principle specialist oncology department

An ISB which can be used to enable the measurability of cycle times

The ISB must be unambiguous on the start and end of the diagnostic phase, and correctly represent all relevant working processes Information manager An ISB which allows the

identification of necessary improvements of the quality of- and validation of the country wide system of DCM and eMeasures

(49)

4.3.1 Design Validation

This section will present the main findings of the validation of the ISB of the combination of BPMN-ORM and UIM as a method to attain cycle time measurability adhering to the SONCOS requirements. This entailed the analysis of the extent to which the ISB was able to fulfil all stakeholder requirements presented in the stakeholder analysis in section 4.2.1.

Internal Validation and Trade-offs

Internal validation of the ISB as a whole was assessed via the same semi-structured validation interviews with the various end-users of the ISB, as referred to in section 4.1.4. This sub-section will present the results of these validation cycles and findings during consecutive design solution phases, describing the extent to which the identified quality attributes could be fulfilled.

Completeness

During the validation interviews it became apparent that all relevant working processes of the HNO diagnostic phase could be represented by the ISB. Overall, the workflow as translated from the BPMN and ORM models to the final design of the UIM was acknowledged to be correct by all end-users. However, validation pointed out that the ORM models were not able to represent all complex working processes correctly. Due to the requirement for databases to be deterministic, the ORM models were designed based on an assumption of sequentiality in activities. However, as noted by an HNO Process Manager: “Not everything within care processes is that explicit. Sometimes information can be requested by a healthcare provider, and another time the doctor decides to ask care administration to do this for him. However, this is not predetermined” (Translation). Therefore, by assuming a sequence in determination of who does what with this kind of activities, a part of the fit with reality is lost.

Minimization of the necessity for additional actions

(50)

they correspond to the terminology and actions of this workflow in its “as-is” state. Thereby, the necessity for additional actions to be able to integrate cycle time measurability in the current HNO information system was minimized.

Flexibility

The multiplicity of iterative validation cycles, which could be conducted in this individual study over a relatively short time span, indicated that both ORM and BPMN were capable of incorporating additional elements in a considerably fast manner. The technical flexibility of the two methods could not be assessed by means of UIM, but should be assessed based on the BPMN and ORM models themselves by the developers.

Unambiguity on the start and end of the diagnostic phase

Unambiguity on the start and end of the diagnostic phase of the HNO pathway could only be assured by the ability to identify one moment in the “as-is” HNO process which indicates the unanimous start of this process, and one moment in the “as-is” process to indicate the end. These moments should then be allowed to be linked to DCM to assure that an unambiguous information retrieval point is decided on.

(51)

referred to by the healthcare providers and the case managers in the “as-is” process as a transition form a “preliminary diagnosis” to a “final diagnosis”. However, the DCM “PatientConcern” currently only includes the “ConcernType” with a value “Diagnosis”. Hence, with the current detailing of this DCM it is not possible to indicate the moment where a diagnosis becomes final, and hence where the end of the diagnostic phase is reached. Hence, a second possible moment for the end of the diagnostic phase was the construction of a final care plan or advice. Notwithstanding, in the current set of DCM, a DCM for care plan or care advise does not yet exist. Hence, currently the possibilities for the ISB to identify an unambiguous end of the diagnostic phase were found to be limited. However, on the ability of the final design of the ISB to identify necessary information to enable measurability of cycle time, it was commented by the Principle Specialist that “The information which you collected is very important for current control information, meaning: am I going through the system within the time limits” (Translation).

Relevance recognisability of DCM

Following from the findings outlined above on the ability on the ISB to be unambiguous on the start and end of the diagnostic phase, it was clear that the ISB was able to identify DCM relevant for the measurement of cycle time. This was confirmed by an Information Manager who noted “The method was able to identify DCM which can now be placed over the current primary production” (Translation).

External Validation

Based on the specifics of the context of the diagnostics phase of the HNO pathway in the focal hospital, as outlined in section 4.1.4, external validity of the ISB was assessed via expert opinions on the generalisation capabilities of the ISB.

(52)

validation results show that the ISB is able to identify relevant DCM. Therefore, the ISB could generate cycle time measurability by means of identification these DCM for other diagnostic phases of HNO pathways as well.

Validating the design principles of an Information System Blueprint to attain cycle time measurability in Head and Neck Oncology pathways MSc Thesis DDM TOM