UNIVERSITY OF GRONINGEN July 6, 2015
Exploring the contribution of process-mining
for performance management in complex
healthcare delivery: a case study.
MSc. Technology & Operations Management
MSc. Supply Chain Management
Author
Floris R. Willenborg BSc Student no. 1812890 fwillenborg@gmail.com
Supervisors University of Groningen Dr. J.T. van der Vaart Dr. ing. J. Drupsteen
Exploring the contribution of process-mining for performance management in complex healthcare delivery: a case study.
Abstract—With the cost of healthcare rising to unsustainable levels, looking for new ways to gain
operational efficiency is necessary. Healthcare has long been an industry in which measurement is of vital importance for improvement. Performance management is in theory a simple loop of deciding on
performance direction, measuring KPIs, reviewing performance, and restarting the cycle. In healthcare, especially multidisciplinary care, this is often not so straightforward. New technological advancements have allowed greater storage of measurement in datawarehouses, but more measurement has not solved all questions regarding healthcare delivery processes. Professionals have voiced their inability to gain insight into the patient's care pathway in complex multidisciplinary healthcare delivery. A data analysis
technique originally for production environments—process-mining—holds great promise of utilising available data to provide insight into these complex healthcare processes. This research explores how process-mining can contribute to the performance management in multidisciplinary healthcare provision. A single in-depth case study in Mamma (breast cancer) care is performed by both archival data analysis and semi-structured interviews that include different professional perspectives. This research finds that process-mining input requires logistic measurement which is not readily available in this
multidisciplinary setting with fragmented data sources, making input creation a laborious and time-consuming affair. Process-mining does expose much more pathway variation than was anticipated by stakeholders, a valuable insight, but at the same time this makes interpretability of output difficult. At times the process-mining software seems underdeveloped for easy use with such high-variation and unstructured processes. Process-mining’s biggest value to performance management could be its speed of filtering and usability in multidisciplinary meetings. Findings indicate how process-mining is able to expose processes of complex healthcare delivery but its contribution to performance management relies strongly on both input and output challenges.
Keywords: healthcare; multidisciplinary; data; performance management; performance measurement;
PREFACE………..4
INTRODUCTION ... 5
THEORETICAL BACKGROUND ... 7
Performance management ... 7
Performance management in services... 8
Performance management in Healthcare ... 9
Process-mining... 10
RESEARCH DESIGN AND METHODOLOGY ... 10
Research design... 10 Data collection... 12 Data Analysis ... 14 FINDINGS ... 14 Process-mining input ... 14 Source ... 14
Setting the scene... 14
Primary data source ... 15
Data preparation ... 18
Key performance indicators dictate ... 19
Isolating patient group ... 19
Aggregation of activities ... 20
Increasing timestamp accuracy... 21
Filtering out duplications and other disturbances... 22
Process-mining output ... 23
Variation ... 24
Slider bar deception ... 27
Activity repetition and parallelism ... 29
Algorithm... 30
Paradox of information... 32
Perspectives ... 33
Mamma department ... 34
Logistics and innovation department... 37
Healthcare consultant (expert Disco user) ... 40
DISCUSSION... 41 Limitations ... 44 CONCLUSION ... 44 REFERENCES... 46 APPENDIX A: Input ... 53 APPENDIX B: Output ... 55
PREFACE
I would like to express my appreciation to Peer Goudswaard, Tjibbe Hoogstins, Igor van der Weide, Gerben Brandsema, and Paul Schüren for being ever so helpful in answering any of my questions regarding this project, or any of life’s oddities I found interesting at the time. It was very helpful to have these bright gentlemen think with me on any obstacle or fork in the road I encountered during this project. They have made my research period at the UMCG a highly informative experience, moreover, a very pleasant one.
Also, I would like to thank my supervisor at the University, Taco van der Vaart, for
INTRODUCTION
The healthcare sector has been experiencing increasing pressure to become more
efficient. Rising costs in healthcare have been putting enormous pressure on the GDP of many countries in the developed world. In the Netherlands, healthcare expenses have increased their stake from 9% of GDP in 2000 to 13% in 2012, expected to reach 31% in 2040 when expenses keep growing at the same pace. This growth can be attributed to an aging population, but even more to new diagnosis and treatment options and demands for higher quality (Centraal Plan Bureau, 2013). Consequently, care has become less affordable to those whom it is intended for, the population of the Netherlands, and at the same time more of their tax money is going towards healthcare. In response to these developments, healthcare organisations are trying to make a significant effort to increase their efficiency by looking at other fields of knowledge such as operations management (Crema & Verbano, 2013). However, the understanding of many hospital processes is still relatively little because of the inevitable heterogeneity of patients through diversity of treatments and clinical outcomes (Smith, 2015). Traditionally, healthcare providers were held responsible for their own professional domain and contribution. Now a shift—especially in complex care such as oncology—is seeing multidisciplinary teams taking responsibility over the entire course of care provision. New challenges of
bringing effective improvement in multidisciplinary care arise because now professionals from different backgrounds and cultures are crossing traditional
professional boundaries (Ferlie et al., 2005; Meltzer et al., 2010). Central to improvement of healthcare delivery, hospitals have been measuring quality indicators for a long time, but considerable variation exists in implementation methods and the level of
improvement (De Vos et al., 2009). Despite the large amount of data and quality indicators being recorded, professionals in healthcare are still having trouble getting grip on a patient’s actual chain of care processes within the hospital. It seems that what Behn (2003, p. 586) stated holds: “neither the act of measuring performance nor the resulting data accomplishes anything itself”. Process-mining is a technique that can automatically turn data into visual process diagrams, a promising tool for insights into complex care processes. This study will explore the applicability of process-mining as contributor to performance management in healthcare. Rather than trying to analyse and improve the process of healthcare delivery, the subject of study is the practical use of process-mining software itself.
organisations focuses on performance measurements (Sillanpää, 2011; Gort et al., 2013) or the use of quantitative modelling to enhance performance management in one department (Santos et al., 2007). Although there is PM literature with a supply chain perspective (Gunasekaran et al., 2001, Gunasekaran et al., 2007; Cuthbertson & Piotrowicz, 2013) there is not much that focuses on complex supply chain processes within a service organisation such as a hospital.
Performance management is preceded by performance measurement, thus data gathering. Throughout the years, as a result of technological developments, large amounts of data have been gathered for a wide variety of purposes. Even though many healthcare providers regard operational improvements as valuable to their organisation and the quality of their care, there is not much measurement performed from an
operational or supply chain perspective. Most measurement in hospitals is related to clinical indicators for external auditing purposes (for example mortality or readmission rates) or financial reimbursement from insurance companies, but not tailored to
decision-making on operational process improvement. As a result, professionals in healthcare organisations often have a hard time figuring out how healthcare delivery processes are behaving, and consequently, how to improve them. This struggle is more evident in complex and multidisciplinary healthcare delivery where the patient’s illness requires customised treatment by multiple departments. Interestingly, more data is available than ever before but performance management in hospitals is struggling to benefit, because little informational value is extracted from it. Process-mining is a technique developed to analyse processes from event logs, a type of data that should be increasingly available because of new technologies (e.g. RFID, cloud computing, sensor networks) (Van der Aalst, 2012). To see if this technique can be of value in hospitals the research question is formulated as: How can process-mining contribute in the process of performance management in healthcare delivery?
The practical contribution of this research lies in seeing whether primary care can benefit from a data analysis technique such as process-mining in their process of performance management. At the same time it provides insights on practices and challenges regarding performance measurement and management in contemporary hospital setting. The theoretical contribution lies in exploring uncharted territory in the field of performance management in services, by specifically looking into complex care processes with strong supply chain characteristics. This study also adds to the
Define objectives Performance measurement Performance reporting Review and adjust
THEORETICAL BACKGROUND
Performance ManagementThis study will speak of performance management in the following sense of the concept:
[...] the process where steering of the organization takes place through the systematic definition of mission, strategy and objectives of the organization, making these measurable through critical success factors and key performance indicators, in order to be able to take corrective actions to keep the organization on track (De Waal, 2007, p. 19).
In plain terms, performance management can be considered a combination of performance measurement and the initiated action in response to those measurements (De Waal et al., 2011). Performance management is a process of managing performance according to organisational and functional strategies and objectives (figure 1). This entails deploying a closed-loop control system that embeds these strategies and
objectives across all business processes, activities, and personnel, while at the same time receiving feedback through the performance measurement system to aid management decisions (Bititci et al., 1997) (figure 2). This performance measurement system (PMS) can be described as: “the information system which enables the performance
management process to function effectively and efficiently” (Bititci et al., 1997, p. 524).
Performance management and performance measurement are two distinctively different—yet inseparable—concepts. In an iterative loop, performance management precedes, follows, and creates a context for performance measurement (Lebas, 1995). Performance management—more of a philosophy or style—is supported by
performance measurement as a tool or source of data to substantiate management decision-making. Figure 2 depicts the central role of performance measurement in performance management. The effectiveness of the PMS to honour its objectives is dependent on how well it takes into account the organisation’s strategy and environment, the organisational structure, its processes, functions, and their relationships (Bititci et al., 1997).
Many authors have listed the positive effects of PM on both financial and non-financial outcomes (Jowett & Rothwell, 1988; Kaplan & Norton, 1996; Epstein et al., 2000; Ahn, 2001; Ittner et al., 2003; Davis & Albright, 2004; Neely et al., 2004; Koufteros, 2014), although this has arguably been mostly qualitative, sometimes ambiguous, and has neglected implications for non-profit organisations (De Waal et al., 2011; De Leeuw & Van den Berg, 2011). Proving the impact of performance management on non-profit organisations is even “more difficult due to the relative lack of clarity in the purpose and direction of performance management in non-profit organisations” (De Waal et al., 2011, p.780).
Performance Management in Services
What seems to be a trend within large organisations—both product and service operations—is their need for more sophisticated measurement systems, of which output is more difficult to interpret and analyse (Jääskeläinen et al., 2012). Moreover, there can be multiple internal parties interested in varying measurements (Jääskeläinen, 2010). The hospital is a shining example of such an organisation. Performance measurement of service operations is known to be a daunting challenge compared to measuring
Performance Management in Healthcare
In accordance with the inherent assumption that PMS should be tailored to the
context of strategic objectives (Bourne et al., 2005; Kaplan & Norton, 1996), hospitals too require tailored PMS. Especially since they face their own management difficulties (Elg et al., 2013) because of specific challenges like an ageing population (Nolte & McKee, 2003), advances in medicine and technology, and patients demanding to be a well-informed co-actor of their own treatment (Elg et al., 2011). Performance measurement has captivated different groups in healthcare such as elected officials, administrative leaders, and professionals alike (Radnor & Lovell, 2003; De Vos et al., 2009; Klassen et al., 2010). Its popularity in healthcare can be ascribed to its versatility (Behn, 2003), as it is often being used for resource funding, development (Radnor & Lovell, 2003),
administrative control, improvement of clinical practice (Nelson et al., 1996), public transparent reporting (Schneider & Epstein, 1998), and research. As Bititci et al. (1997) describe, the PMS’s responsibility within performance management is to provide a structured framework in order to feed back relevant information to appropriate points and thereby facilitate decision and control processes. Nevertheless, healthcare
organisations are struggling to manage performance of their complex processes based on their measurements. The field of performance management in healthcare is still exploratory, and empirical research into how PMS can be used to improve clinical practice is rare (Elg et al., 2013). Gort et al. (2013) explored the use of performance indicators in oncological care and their subsequent use by multidisciplinary teams for performance gain. Moreover, De Vos et al. (2009) reviewed 21 studies on the use of quality indicators for performance improvement, and concluded that effective strategies of performance measurement are unusual. Moreover, the operations management perspective in healthcare is found to be lacking whilst significant research into
performance measurement for management accounting purposes exists (Elg et al., 2013). The lack of understanding about organising performance measurement to support operational performance is a result of practitioners being unable to make good use of the data, despite good intentions behind measuring (Neely & Al Najjar, 2006). Research on quality indicators and performance measurement in healthcare keeps reflecting the inability to act upon the information that is available. In practice this resounds in healthcare professionals expressing inability of understanding the behaviour of their processes and consequently feeling less able to manage their performance. When assuming that indeed a missing part of the puzzle is operational understanding of the healthcare processes, a logical step is to explore means for improving this
Process-mining
Process-mining is a data-mining technique that uses large amounts of event data with the aim to objectively discover, monitor, and also improve processes by extracting
knowledge from event logs available in information systems (Van der Aalst, 2011). The main distinction from classic data-mining techniques (classification, clustering,
regression, etc.) is that it is process-centric, bridging the gap between data-mining and business process modelling. It provides an objective representation of the process under examination purely based on event logs containing a case ID, an activity, and a
timestamp. It has been used in multiple environments such as municipalities, banks, insurance agencies, multinationals, high-tech manufacturing, and media (Van der Aalst, 2012). Nevertheless, services and less-structured processes pose specific challenges (Rozinat et al., 2009; Van der Aalst, 2012). This study will elaborate on the ability of process-mining to contribute to performance management in healthcare.
RESEARCH DESIGN AND METHODOLOGY
Research Design
This research is performed as a single in-depth case study. With regard to the
purpose of a case study, Voss et al. (2002) distinguish four variants: exploration, theory building, theory testing, and theory extension. The aim of this study is to explore the phenomenon of performance management in healthcare and specifically the
The specific case under study is that of Mamma oncological primary care (breast cancer) at the University Medical Centre Groningen. In Cohen & Bailey’s (1997) extended model, Lemieux-Charles & McGuire (2006) try to depict the complex
interactions and variables that influence the effectiveness (e.g. clinical quality, patient satisfaction, cost effectiveness) of healthcare delivery by teams. It states how task design (task type, task features, and team composition) influences team effectiveness both directly and indirectly. The task design of oncological care is much more complex than (e.g.) orthopaedic care on many of the model’s dimensions of task design. Professionals are more interested in the use of process-mining for such complex oncological care as it is more multidisciplinary, is subject to stricter performance norms, and has more hidden structure to expose. From preceding dialogue with both medical and logistical
professionals in the hospital, a consensus was reached that Mamma care would be the complex care case. This decision was based on reasons such as a presumed
unambiguous care process and small difference in diagnosis and treatment, relative to other types of oncological care. Within the limited timeframe of the research the choice for Mamma care increased its viability.
Case
characteristics
Hospital type: University Medical Centre
Diagnosis: Malignant mamma tumour (breast cancer)
Period: 01-01-2013 to 01-01-2015 No. of patients >500
Table 1. Case characteristics.
The first part of the research is about learning which data is available and which source is most useful to serve as input for process-mining. The next step concerned adjusting the data, for it to become adequate input for the process-mining software. At the same time—but also the penultimate step—was to analyse what process-mining is offering as a tool in performance management of Mamma care provision. The last step of analysis concerned differentiating between perspectives of both the Mamma group and the department of Logistics & Innovation on performance management and
Mamma group and Logistics & Innovation about the best ways to adjust input data and to interpret output happened on a continuous basis.
Note that this study is not about analysing the healthcare delivery process of primary Mamma care by the use of process-mining. It is about exploring the
applicability and ease by which process-mining could be a contributor to performance management of such complex care.
Data Collection
Two ways of data collection were used for internal triangulation: information systems (archival) data analysis, and semi-structured interviews. Questions in the interviews were inspired by elements of performance management in Bititci et al.’s (1997) model (figure 2), of which protocols can be found in Appendix C. Data collection started by studying the sources of data available from information systems and the actual data sets that could be retrieved from the data warehouse.
Next, a period of familiarising with the archival data, interpreting, and adjusting it ensued. Subsequently, the required adjustments were being documented as
data/findings. Following this, semi-structured interviews of approximately 60 minutes were conducted with two staff-advisors to the department of oncology of which one responsible for the Mamma patient data. Another interview was conducted with a senior laboratory technician of radiotherapy who was, at that time, involved in process analysis and improvement within the Mamma care process. This provided data on performance measurement and performance management practices at the Mamma group. Additionally this provided data on their view of process-mining as a tool within their performance management practice.
Tag: Interviewee:
R1 Staff advisor to department of oncology R2 Staff advisor to department of oncology R3 Senior laboratory technician of radiotherapy
R4 Chief officer health operations management & innovation R5 Healthcare logistics advisor
R7 Healthcare logistics advisor
R8 Consultant in healthcare operations
Table 2. Interviewees and corresponding tags.
Thereafter, semi-structured interviews of roughly 90 minutes were conducted with three advisors in healthcare logistics and one senior manager of the Logistics & Innovation department. This provided data on their position within and different perspective on care, as well as, their view of process-mining as a tool within
performance management in oncological care. The Logistics & Innovation department was interviewed because they are the authority within the hospital when it comes to operational advice on processes, throughput times, data analyses, and other logistical improvement and innovation questions coming from other departments.
In order to gain more specific knowledge about the applicability of process-mining in healthcare an additional 30 min interview with an expert was conducted. A healthcare consultant, experienced in using process-mining for data analyses, is able to provide valuable extra insights from another (external) perspective. In total 8 people were interviewed (Table 2).
Find appropriate data source
Adjust data to create process-mining input
Report findings
Discuss input with stakeholders in process Research structure
Analyse the output generated with process-mining
Discuss output with stakeholders in process
Data Analysis
The data analysis started by evaluating the best possible primary source for process-mining input. This was decided upon together with the staff of the Logistics &
Innovation department since they have a clear view on what data is available hospital-wide and work with data every day. Also the Mamma group was involved as they keep their own specific dataset on Mamma care patients. Data sources were compared on dimensions of applicability, completeness and measurement rigour. After this initial phase the research remained to be highly iterative through close contact with
stakeholders in the Mamma process. The following phase of the analysis meant keeping record of the required adjustments to the input file for process-mining, with the goal of creating output that would help gain insight into the Mamma care process. At the same time adjustment to the input data was minimised because process-mining promises an objective representation of the process have the software do most of the work for you. The chosen approach for adjusting the input data was to cluster activities up to a high level of abstraction and then pursue adding more detail if applicable. Iteratively during, but also after the adjustment of input, the dataset was used as input for process-mining software called Disco by the company Fluxicon. This process-mining tool was chosen because it is most accessible for academia and one of the easiest to use out of the limited range of alternatives. It is intuitive and no knowledge of Petrinet modelling language is required or any other informatics behind it. This allowed the analysis to focus more on the required input and the gained output, instead of learning to operate the tool.
The following interviews consisted of showing the functionality of Disco on basis of the prepared input data and asking questions on performance measurement, performance management and the role and worth of process-mining. Important insights from multiple perspectives were obtained and included.
FINDINGS
Process-mining Input
Source
Setting the scene
A large healthcare institution such as the UMCG most often has a vast amount of data
it became easier for hospitals to store data thanks to technological advancements in computer technology and storage capacity. This has led to a measurement urge and expanding lists of performance indicators. Recently, people have started to recognise that measuring more is not per se adding value to operations, it can cause confusion about what is important, and it adds pressure on professionals because of manual input. Not only a lot of measuring takes place, but most of it is taking place decentralised in the departments which are running their own semi-autonomous operations. This means that departments measure (1) what they are obliged to do by national rules and
guidelines, and (2) things they find interesting for the functioning of their department. Responsibility and accountability for matters happening within the boundaries of the department is still defining most measurement. With little operational measurement transcending these departmental preferences, the result is a great bulk of data made up of small heaps of data. Even though originated from different perspectives or goals, in combined state the data could hold valuable information.
Primary data source
Process-mining is a data-analysis technique which only requires input consisting of three elements: a unique ‘case’ id for each patient, an ‘activity’ that was performed in the healthcare delivery process, and a ‘timestamp’ connected to the activity.
In order to satisfy the above requirements for process-mining input, a choice was made to take Diagnose Behandel Combinatie (from now on referred to as DBC) data as the primary source. This DBC system for financial reimbursement is comparable to the more internationally accepted diagnosis-related groups (DRG) method. By Dutch law— in order to receive reimbursement—all hospitals have to record diagnosis and treatment activities at the individual patient level. The choice to go with the DBC data was made in consideration and concurrence with the people at the Logistics & Innovation
department (who have access to hospital-wide data and work with on a regular basis). The reasons for choosing this primary source of data for process-mining are the
following:
It is central in the sense that it revolves around a patient and therefore does not know the boundaries of departments and their measurements. Therefore it has less chance of issues related to stitching together peripheral data.
It is considered to be very complete. Because it serves financial reimbursement purpose, it holds record of literally every activity in the healthcare delivery process.
It is regarded to be semi accurate regarding the activity timestamp; only the day is registered (no exact time).
It is rigorous because activities are generally registered the same by all departments and the same activity is labelled the same for every patient that undergoes it.
The DBC system is a national and univocal system which will make any findings in this study more generalizable for the entire Dutch healthcare sector.
Mans et al. (2012) write in their article specifically about challenges of process-mining in healthcare, and refer to the data available within hospitals in the categories as seen in figure 3. The dimension “level of abstraction” on the y-axis refers to the breakdown of the activities from high being an entire activity, medium level as individual tasks, and low level as movements within tasks. The dimension of “accuracy” on the x-axis refers to the accuracy of the timestamp attached to each activity. This is low for DBC data because only the day of the activity is recorded. The distinction between central and peripheral/departmental data sources is also depicted in figure 3. The healthcare logistics
systems are mainly planning tools for departments, whereas clinical support systems and medical devices are department and specialisation specific support tools.
The departmental data set of the Mamma group was inappropriate for different reasons. This data was recorded manually, which proved very laborious. And when this one
Peripheral/ Departmental Central (DBC data)
person responsible was sick, gaps in data started occurring. Moreover, it included many KPIs for medical reasons and hardly any logistically oriented ones. Except for times related to very important norms, from this however, a proper care pathway could not have been constructed. What should also be taken into account is that access to data in hospitals is heavily restricted and often there are only a few people that have ‘hospital wide’ authority, such as the people in the Logistics & Innovation sector at the case hospital.
The choice of data source depends on what process is subject of the analysis. This study is looking at the process of healthcare provision which moves a patient through a hospital on the course of many treatment activities, and at throughput times in between critical activities. Therefore, it is not relevant to have a very low level of abstraction which would refer to steps within an activity (e.g. setting-up an MRI machine). Rather a high level of abstraction is preferred in this study as it indicates an activity relevant to the patient (e.g. MRI scan). Figure 3 shows how there are no data sources with a high level of abstraction and a high level of accuracy in the hospital, which would be positioned in the top-right corner. The chosen DBC data source has many of the
characteristics making it appropriate for process-mining, but it is not suitable yet. It still needs significant adjustment to serve its purpose as process-mining input.
The ‘raw’ data set as retrieved from the data warehouse consisted of the following (a fragment is shown in Appendix A, figure 2A):
612 unique patients/cases
24.057 recorded health care activities (rows) 656 unique activity descriptions
20 columns with information concerning the activity Properties of the data: numerical and text
Data preparation
As mentioned earlier, the DBC data is measured for the purpose of financial
reimbursement from insurers. Consequently, it covers all activities that have happened for a patient within the hospital, even if the patient need not be at the hospital
physically. A lab test is an example of such an activity.
Key performance indicators dictate
When applying process-mining the input is heavily influencing the output, as would
be the case with any other type of data analysis. The rule of garbage-in-garbage-out holds, so it is important to realise beforehand what process you want to unearth. The question coming from the hospital was “can we get more insight into our
oncological process?” and whether process-mining would be able to offer this insight. One level deeper you trace that the reason they want to have this insight is a SONCOS (Stichting Oncologische Samenwerking) 1-3-6 norm requiring breast cancer patients to have their first outpatient clinic consult within a week of breast cancer suspicion, after which diagnosis should be completed in 3 weeks, and treatment needs to commence within 6 weeks. Not adhering to this norm can have profound consequences for the hospital’s national image, ability to attract funds, and sell care to insurers. Having knowledge of such KPIs is a prerequisite for input preparation.
There are two reasons the raw dataset is unusable as process-mining input in this case. Because (1) the output will be an enormous ‘spaghetti’-diagram as can be seen in Appendix A figure 1A, and (2) it would not hold value to understanding the process with regard to the KPIs as expressed by the Mamma Oncology. As they are not
interested in knowing the times between two billable activities within the care pathway, but rather require insight into aggregated core activities for which the patient needs to come to the hospital. As stated by Mamma care personnel:
[…] performance indicators that are now very important to the Mamma Group are access and throughput times, simply because we are too often below the norms set. [R1]
This means that not all of the activities within the healthcare delivery process that are in the data set are relevant to the insights you are trying to gain about your
shows activities which are highly relevant to the patient (e.g. physical presence) and which are extremely critical for the continuing course of the process. Meaning input data of even higher abstraction is needed indicated by the red-striped square in figure 3.
Isolating patient group
One of the first and most important requirements to the dataset is to clean it up by isolating the patients with the same diagnosis—in this case malignant breast cancer— and filtering out those patients that do not belong to this group. In this part of the process the shortcomings of the data already manifest themselves.
The DBC data does have a labelling of patients by diagnosis under
“DBC_DIAGNOSE_CODE” and a description matching that code. So for the actual illness of malignant Mamma tumours there are three codes: “811”, “318”, and “105” corresponding to “maligniteit mamma”, “maligne neoplasma mamma”, and “Mamma tumoren”, respectively. This is because different departments have created various diagnosis descriptions, making awareness of all possibilities important. Another
potential problem is that these codes can correspond to different diagnoses, as they are not unique. Other departments can put the same code on diagnoses descriptions
completely unrelated to malignant Mamma tumours. The current system is a free-for-all of assigning diagnosis codes and descriptions, which makes it almost impossible to select the right group of patients automatically and is prone to making errors when trying to isolate a patient group.
Next, the search of a process-view requires patients in the dataset to have completely traversed their process of treatments. No patients should be halfway into their treatment at the start of the dataset, and none should be halfway through their treatment at the end of the dataset. This can be managed with the DBC data because there is a starting (“BEGIN_DBC_DATUM”) and end (“EIND_DBC_DATUM”) date for every patient. In this case only patients that have a starting date after 01-01-2013 and an ending date before 01-01-15 are allowed in the dataset.
Having an expert from the Mamma Group look at the dataset after the above stated process, it became clear that there still was ‘noise’ in the set because of many activities unrelated to breast cancer treatment. In a last attempt to clean up the set, only patients were selected on basis of specific surgical activities with either the words “ablatio”, “mastectomy”, “lumpectomy”, or “amputation”. These activities indicated that this patient underwent a surgical procedure to remove a malignant Mamma tumour.
undergone the same treatment. Moreover, coupling medical attributes from the Mamma department’s own dataset was not easy because of access and the fact this dataset was not maintained rigorously. Because of this, isolating a patient group again requires a lot of interpretation and manual work instead of it being made easy by relevant attributes.
Aggregation of activities
The dataset is extremely large as every billable activity is reported, therefore
aggregation of activities is required. Aggregation or clustering of activities is a necessary but very tricky manipulation because it has a strong influence on the actual
(visualisation of) process-mining output. When for example all outpatient clinic visits and other consults are clustered into one activity—because they are comparable—a loop from almost every activity in the process to this cluster is generated.
Clustering activities to an umbrella definition such as “operation” for all surgical procedures will essentially not change the process, it will only leave one datapoint for an entire strip of billable activities during the operation. Also there may be no need to differentiate between an MRI scan of the left or right side, you can cluster it to MRI or even Imaging Diagnostics (figure 4). A part of the input file containing this clustering is shown in Appendix A figure 3A.
A national DBC-database is kept by the Nederlandse Zorg Autoriteit (Dutch
Healthcare Authority) and publicly accessible on the Internet which explains how DBC care products are build up from single activities. This provides some guidance for clustering but the most of it comes down to interpretation and discussion with medical professionals. In the end, the highest abstraction level considered to be informative
MRI WERVELKOLOM – LUMBAAL (Raw DBC data activity)
MRI (Detail activity cluster)
Imaging diagnostics (Core activity cluster)
consisted of 27 core activity clusters (Appendix A, Table 1A) down from 656 unique activities in the raw DBC data.
Clustering activities is a necessary but very laborious activity requiring both medical knowledge and assistance from practitioners. Caution is required because some
activities from different departments are labelled exactly the same by code and description for (e.g.) a first outpatient clinic consult. This process is trial and error, because often you may find clustered activities to distort the process-mining output significantly and require iterative adjustments.
Increasing timestamp accuracy
Low accuracy of DBC data is a big issue when using it for process-mining. As per activity it is only required to register the date of execution—and not the exact time of execution on that date—a number of issues arise when preparing a ‘correct’ input file for process-mining. This correctness is explained in the following problems. The principle of process-mining is to have an objective image of the process as data projects this image, without too much interference or adjustment of the data. However, if the input data is not necessarily measured to serve as input for process-mining, than problems may occur.
One large problem caused by the dataset was that of activity sequence due to timestamp inaccuracy. All activities are at exactly the same time (00:00:00) on any particular date, which results in a random sequence of activities purely based on how they are organised within the input file. This is not desirable when an objective image of a process is the aim.
Moreover, this takes away any indication of waiting time on the same day which a patient has to endure between (e.g.) an outpatient clinic consult and an actual
examination or operation. But this is of lower importance than the issue of sequence. Because process-mining ought to be an objective view on the process, one should not manipulate input data too much. Certainly not rearrange activities according to how people in the hospital feel the process should look like. Nevertheless, it is those people with substantive knowledge of the healthcare delivery process who should be consulted for their opinion about the correctness of what is seen in the process map. This can lead to issues with input getting resolved but can also raise interesting questions as to why the process is not behaving as they would have expected it to.
Such sources are the peripheral planning systems of patient visits at outpatient clinics or the planned time of surgery.
This needs to be done for most of the departments (specialisms) involved in the process, which can be selected from a list of all departments. Unfortunately, the
information system did not allow a retrieval of visit information for a selected group of patients. Instead it provided all outpatient clinic information about all patients that had paid a visit in the timeframe of interest. This resulted in data lists too large for Excel to handle—or being more specific—to filter through. The filter option was needed to filter out the right patients, but Excel was unable to do this because of the size. Another program such as Microsoft Access needed to be used for reduction of the set to contain the right patients. Executing these queries in Microsoft Access required expertise not by default familiar to someone who is highly skilled in Excel. Caution is required as not all planned outpatient clinic visits are actually taking place at that moment but might be rescheduled. Often though, there will be an indication for this and the right times can be filtered.
It is vital for the additional time-specific information to be added in order to increase the accuracy of the timestamp. This part of the process is critical in order to have the patient’s activities in the right order for further analysis of the care process. The manual process of coupling the times to the activities, by matching the patient number,
department involved (producing specialism), and the date of activity, is both laborious and not ironclad. This makes a great example of the little attention to logistical
measurement, because adding time should be an easy job when working with IT systems that can clock automatically when something
is being registered.
Filtering out duplications and other disturbances
After clustering activities there are a lot of
duplicate activities left, because many were given the same aggregated description (figure 5). Those
activities with an accurate timestamp should be preserved and ‘duplicates’ on the same date removed. This step does result in losing information about the producing specialism that was involved in the
removed duplicate row. Duplicates can also be caused
Some activities are registered for administration or reimbursement reasons, (e.g.) “DOT-15A026” or “EERSTE ADMINISTRATIEVE CONSULT”, and thus irrelevant to the care process.
Or, as seen in table 3 below, the same activity description of the hospital with the same code can have a different description in the national database. This also happens the other way round.
Table 3. Part of dataset containing hospital description, code, and national description of the activity.
In the light of performance management, preferably this input adjustment is to be
automated, assuming that new dedicated measurement will not be implemented any time soon. Automation possibilities would mostly depend on a multitude of factors related to the used primary source (here DBC), the IT infrastructure and access, skill level of personnel adjusting the data, and undoubtedly many more. The biggest pitfalls of the DBC as primary source is that the first time aggregation will require heavy interpretation and manual work. Moreover, ambiguity and multiplicity of activity description and code, and diagnosis codes referring to unrelated diagnoses, will make automating more difficult. The next difficulty for automation would be the filtering of duplicates that needs to be done on superfluous clustered activities. Multiple rounds of filtering were required in order to adjust sequence of activities per patient and their timestamp. In between these rounds, the order of several columns needed to be adjusted manually.
Process-mining Output
This chapter analyses the output from the process-mining software as derived from carefully prepared input data. Key insights are elaborated upon regarding the value of process-mining within the context of applying it to oncological healthcare delivery processes. This should also provide more clarity on the key topics of the previous chapter on process-mining input.
182 patients which are diagnosed with malignant Mamma tumours and have had surgery (either ablatio, mastectomy, lumpectomy, or amputation) as part of their treatment. (134 after steps below);
and 27 aggregated activity clusters (Appendix A, Table 1A).
Initial steps have been applied within Disco in order to cut back variation and create a more ‘homogenous’ patient group. The creators of the software actually advise to apply these steps when dealing with complex processes. These steps include:
1. Utilising Disco’s variation filter. However, this showed only two options: (1) select three cases that share an exact sequence of activities or (2) select the rest of the patients, whom all have a unique care pathway.
2. Select first consult – surgery as obligatory starting activity and select intermediate
consult – surgery, intermediate consult – nurse, and teleconsult as obligatory ending
activities. This is another extra method to ensure that only patients having traversed their entire treatment are in the process map you will be analysing. Patients with half treatment in the analysis would distort the overall process. The following subchapters elaborate on the main findings of the process in which the author interprets the output of the process-mining tool Disco by Fluxicon in the context of the value it offers for gaining insight into multidisciplinary oncological healthcare processes.
Variation
Figure 6. Process-mining diagram at 100% activities and 100% paths.
However it does show that still a large amount of care pathway variation remains within this homogenously diagnosed patient group. Even after variation reduction, as the number of unique activities (and thus combinations) has dropped from 656 to 27
Figure 7. Variation of care pathways.
A filter option allows for automatic filtering of variations, but there is simply so much variation—even at a high level of abstraction—that only three patients show exactly the same care pathway and the rest are all different (figure 7). This shared pathway is depicted in figures 8 & 9 below at 100% activities and paths. “Instant” should be interpreted as “n/a” as this info was out of scope.
2%
98%
Figure 8. Output diagram showing maximum repetitions for activities and routes.
Figure 9. Output diagram showing mean duration.
Even though, it is not a straight process because of the looping back to the intermediate
consult – nurse, diagnostic activity – radiology twice, and Imaging diagnostics, it does present
Moreover, there is a clear indication of routes where most of the waiting takes place in figure 9, where the red arrows are thickest in between activities.
The statistics section of Disco (figure 10) shows these three patients are among the top ten in the dataset when it comes to total length of treatment, with total durations of 28.5, 30.7, and 40.5 days. The same file shows they are ranked 18th, 19th, and 20th respectively
in terms of number of activities (events) during their treatment. The number of
events/activities spanning the treatment is somewhat correlated to the total duration of care, but many very long durations also occur in care episodes containing a relatively small amount of events (shown in Appendix B figure 1B). This and the total number of activities per patients are indicators of variation within the care process.
Figure 10. Top ten lowest total duration/throughput time statistics.
Process-mining was able to show quite some insight into the care processes of these 3 patients with identical care pathways. However, this technique should be handling big amounts of data and show its user the law of large numbers, in order to get to grips with dynamics and peculiarities of a larger group of patients.
Slider bar deception
Figure 2. 100% Activities, 50% paths.
Such a slide bar also exists for adjusting the number of activities that are shown in the process- diagram. When the activities slider bar is lowered to 60% and the slider bar of paths is lowered to only 1%, it appears as if finally a ‘standard’ pathway is materialising, as shown in figure 12. It seems like process-mining has uncovered the care pathway that most patients in this group have traversed. This is a false observation though, caused by a heavily distorted model of the truth because the entire case does not disappear but only its unusual activity or connection. Thereby, the two slider bars (activities and paths) are able to minimise the ‘spaghetti’-ness of the diagram, but the high variation of pathways leaves the now interpretable diagram as highly deceptive. Activity positions and connections have become unreliable because many important connections are missing and apparent vertical chronological order is inaccurate. Result is, that what is being shown as the ‘standard’ pathway can in no sense be interpreted as such.
Figure 3. 60% activities, 1% paths.
Activity repetition and parallelism
Figure 13. Core activities; numbers showing maximum repetitions (100% activities, 100% paths)
Parallelism is also causing some variation in the setting of oncological care because some activities are running parallel to each other as different departments are working at the same time for one patient. For example, lab tests can be run simultaneously while a patient is getting a scan at radiology. This parallelism is very dependent on planning within different departments. Moreover, the input data can also create parallelism because of low time accuracy, causing some activities to happen simultaneously. This can cause both variation and deceptive activity sequence.
The low interpretability of Disco output is mainly caused by the process’s inherent variation resulting in most activities being connected to multiple other activities, both in- and outgoing connections. These so called ‘spider’ activities (best seen in figures 6 & 11), are creating a web of activities instead of a pathway. In all fairness, this is a good depiction of reality, but it leaves little informational value when trying to find a pathway, except for a reality check on variation.
Algorithm
The process-mining algorithm behind the software tool that has been used in this study is apparently making decisions that are not always in line with expectations of diagram visibility. In some instances the algorithm is deciding what important
Because when trying to analyse a healthcare delivery process with so much variation, you are orienting towards finding a ‘standard’, and thus expect to see those connections which most patients have traversed (unless configured/filtered otherwise yourself). A great example can be derived from figure 12 where 129 patients have had an Operation within their treatment. The process-mining algorithm shows how 38 of those patients moved to having a Teleconsult and 18 received Revalidation therapy afterwards. This leaves the paths of 73 patients unspecified, more than 56% of all outgoing connections. For the outgoing connections of Imaging diagnostics this is only ~22%.
To try and uncover these lost paths the slider for paths is cranked up to 11% to show the process-diagram in figure 14. Another 32 outgoing connections now become visible from Operation to Intermediate consult – surgery which—on basis of frequency—can be considered to be more important than the connections to Revalidation therapy.
Additionally, suddenly 36 patients are shown to have moved from Imaging diagnostics to
Preop. Screening, which is both a significant number and in line with logical clinical
procedure. Moreover, this can be considered a very important connection with regard to the 1-3-6 SONCOS-norm as it symbolises a move from diagnostics towards treatment (in this case an operation).
Also, the extra 36 patients add up to 136 unique outgoing connections from Imaging
diagnostics, even though only 129 unique cases have had imaging diagnostics as part of
their care pathway. This is because some patients have had up to 10 Imaging diagnostics repetitions (Appendix B, figure 3B) and could therefore have traversed many of the outgoing connections after Imaging diagnostics. Nevertheless, one would expect that
Imaging diagnostics would be placed above Preop. Screening within the process-diagram
for reasons of interpretability, because only 16 patients have never received Preop.
Screening after Imaging diagnostics (found by use of the subsequence filter option). Some
high-frequency connections even require the paths slider to be at 100% before being shown.
Figure 4. 60% activities, 11% paths
Paradox of information
In the case of extracting useful information from process-mining in healthcare
processes, it really is a very difficult balance to strike between adding detail and leaving out detail concerning the activities of treatment. When interested in deviating behaviour of the process, the instinctive response is to add detail to the input in order to give the output more explanatory information. In this case, however, adding more detail to the input (through lower level of abstraction or aggregation) will cause the output to be less interpretable, thus less informative. I call this the paradox of information.
logically this should allow you to get a better explanation of which treatment might be causing longer waiting times within the category of imaging diagnostics. However, by doing this, you also increase the amount of variation within the process-diagram. The same logic holds for the amount of activity repetitions allowed. Looking at Mamma healthcare provision, allowing replicable activities as input instead of ‘one-time’ non-replicable Gate moments will see variation increase drastically. The arrow in figure 15 depicts the envisioned way of analysing Disco’s value with Only core clustered
process activities as a starting point. However, moving forward did not increase the value
of output.
Figure 5. Visualisation of the paradox of information
Perspectives
Because process-mining is a technique that originated in production and with operations-minded people, it is especially interesting to have different perspectives in the hospital reflect on this technique’s role in the process of performance management in complex healthcare.
Gate
moments
Only core
clustered
process
activities
All
clustered
activities
Raw dbc
data
A
m
b
ig
u
it
y
o
f
o
u
tp
u
t
D
e
ta
il
a
d
d
e
d
t
o
in
p
u
t
(w
e
a
lt
h
)
H ig hVariation in care pathways
HighMamma department
The Mamma group has responsibility for the treatment of all the patients with a diagnosed Mamma tumour or suspicion. It is a multidisciplinary group responsible for coordination and performance of the overall process of healthcare delivery for Mamma patients.
Performance measurement
Measurement at the Mamma group is a manual and thus laborious affair. There is a list of over 200 (performance) indicators that should be registered for every patient. One person is responsible for this and when illness strikes there is no replacement, therefore the measurement does have some gaps. This is not very rigorous.
Furthermore, almost all indicators that are to be registered at this department have been determined by external parties. The Mamma group indicates that many of the things they are measuring are the right things to oversee their performance:
[…] we can see how much of our patients have been performing according to the norms and how much have not. Moreover, our KPIs are those that are being dictated by external parties as these are the national norms and guidelines which indicate excellence in healthcare delivery. [R1]
Even though the department feels it is measuring the right KPIs they think the list is a bit extensive and high maintenance:
[…] the entire list of indicators was originally based on those KPIs that are put forward by external parties, which we find most important, making the list
highly accurate. But we do see the list extending and it is very laborious and time consuming to keep up-to-date. [R1]
Furthermore, not everything being measured is actually serving any purpose at the department itself, let alone be actively used to monitor or improve performance:
[…] most indicators are being registered without specified purpose. We assume DICA (Dutch Institute for Clinical Auditing) or other authorities might use them for their own research. [R1]
When trying to gather information about specific passage times of Mamma patients in the sub-process of radiotherapy, a senior laboratory technician was very surprised at how difficult it was to gather the right data for her project:
information I required was stored in different sources and I needed to fabricate my own new file in order to be able to analyse it. [R3]
From the interviews with professionals it seems as if people think performance measurement is accurately executed at Mamma as they are able to keep track of their most important KPIs. But, when data is needed in order to analyse the throughput times of only a sub-process of Mamma care (radiotherapy), it is not so straightforward. This implies that establishing the entire course of care from current measurement is even harder.
When discussing the fragmentation of data and the status of the current information systems in the hospital, they do recognise that they are looking for easier ways to get all data available in the hospital on their specific patients:
[…] currently we have just started working on an IT project called BROC which should in a smart way be able to connect all data of oncological patients that is now scattered throughout the hospital in multiple information systems. [R2] This is another indication of how IT in the hospital is lagging with regard to data management, and a consequence of that is a lot of manual registration, double registration, and difficulty of retrieving complete data.
Performance management
Managing your performance as a response to what has been measured is very
difficult to do in a hospital. Processes are not linear; variation is inherently high because every patient is different in some respect or many; human planning and limited capacity are involved; and many more variables can be involved.
Managing the performance of the Mamma oncology department is based on what is measured, but it does not appear to be very systematic nor very efficient because there is little attention for the large population, only for the individual.
[…] if a patient shows a long throughput time or waiting time in between one of the main activities which are important for us (first consult, diagnostic, start treatment), we will investigate this case. This means we look into the details of this patient’s care pathway in our own data to see if there were any special circumstances in the hospital or with the patient”. [R1]
[…] the search for answers can lead you to speak with many different
professionals…sometimes it was just a small miscommunication that led to the exceptional behaviour and it answers the question but took a lot of time and does not really hold value for the future. [R1]
The way to make sure that something is learned from looking at specific cases is to address it to stakeholders.
[…] if we see from our detailed data that it took a large amount of time before a patient could be seen for the purpose of for example imaging diagnostics we will start a dialogue with the department involved, and discuss how this can be prevented in the future. [R1]
The Mamma group is in a very responsible position as it needs to oversee all the breast cancer patients within the hospital and account for their performance conform to the many external norms and guidelines for Mamma care delivery. This seems to be somewhat misaligned with their actual authority to manage performance:
[…] we have little power to make departments do as we please, we cannot point our finger to a department that seems to be the origin of bad performance and demand them to improve…We have to rely on a charismatic Mamma group head to start a dialogue and get things done with the departments. [R1]
When dialogue is pursued there can be other roadblocks:
[…] doctors sometimes do not believe that waiting times were actually as long for their department as measured…and they are often more sensitive to larger
populations and larger deviations from the norm. [R1]
Furthermore, it seems as if performance is driven by a constant pressure of adhering to the external authorities’ expectations and the fear of consequences if not adhering to them.
[…] it is not a bad thing these external norms have such a big influence on measurement of performance of Mamma Oncology care, because the pressure keeps everybody sharp and we feel, eventually, this leads to increased
performance. [R2]
managing future performance is hard because of a lack of authority and solid evidence to show practitioners.
Process-mining
As mentioned before, the Mamma group is focussing on looking at details when looking into their performance and looking at ways to improve it. When it comes to process-mining having a place to fulfil in that process, the response is quite unanimous:
[…] at this stage it is not showing enough detail in order for us to make any assessment of the performance whatsoever, we cannot see the underlying reasons for performance. [R1]
This was a response to process-mining output on one of the lowest abstraction levels, as shown in figure 8. It clearly indicates that this department is not so much interested in looking at general trends within their process but are more interested in small details that explain why single cases perform out of line.
Logistics and Innovation Department
Performance measurement
Regarding measurement, the advisors are clear in saying that measurement at the Mamma department is accurate because it is exactly in tune with what external parties want to see. However, they are doubting their contribution to quality of healthcare: […] the most important norms for Mamma are the exact same norms for all other oncology groups. These ‘quality indicators’ can be very arbitrary and even make little sense to us and medical professionals. Having to adhere to these norms results in less flexibility and I doubt that it is increasing quality of care. [R6] Concerning underlying reasons for performance measurement at oncology groups, the advisors and manager feel as if the logistics of care is not taken into account very often:
[…] registration happens either for adding to the patient file or for external organisations such as DICA and insurance companies, very rarely is something measured because it is important from a logistical perspective. [R4]
This results in one of the biggest drawbacks in available data for the advisors at the logistics department to work with, time accuracy:
process in order to analyse it. A rare example of systematically measuring time for operational purpose is the OR, where a button is pushed after or before a process step to log the time. [R5]
Another big drawback to the available data is its fragmented nature:
[…] measurement is scattered throughout the hospital and therefore also within the information system. Additionally, departments decide on their own measures and thus it is not always easily comparable or compatible. [R7]
This issue of data management within the hospital becomes even more clear when talking about a large project led by a world-renowned IT company with strong ties to healthcare:
[…] this project was supposed to create centrality, get rid of all the scattered data issues, and even give us more timestamp data automatically. However, after a few years the company realised it was unable to deliver what it had promised and the project was abandoned. [R6]
Performance management
It seems as if the Logistics & Innovation department often acts as a facilitator for other departments’ performance management:
[…] we try to help departments the best way we can, for example by structurally delivering performance reports. But only when they are genuinely interested in it, the motivation needs to come from them. [R4]
Often no immediate answer exists to someone’s question or enquiry, rather it needs to be retrieved from an analysis on manually restructured data:
[...] many times we are able to conduct good analyses with the data that we have, but we need to be creative in putting together data from multiple sources and it is time consuming. [R7]
Process-mining
The advisors and manager at the Logistics & Innovation department were rather surprised by what they got to see when being introduced to process-mining output of Mamma care:
