Patient Scheduling For Multi-Priority Patients With Inﬂammatory Bowel Disease: A Markov Decision Process Approach

(1)

Patient Scheduling For Multi-Priority Patients With

Inflammatory Bowel Disease: A Markov Decision Process

(2)

Master’s Thesis Econometrics, Operations Research and Actuarial Studies Supervisor: Dr. N.D. van Foreest

(3)

Patient Scheduling For Multi-Priority Patients With

Inflammatory Bowel Disease: A Markov Decision Process

Approach

T.L. de Haan

Abstract

(4)

3.6 Conclusion . . . 20 4 Analysis by MDP 20 4.1 State space . . . 21 4.2 Action space . . . 22 4.3 Cost structure . . . 22 4.4 Transition probabilities . . . 23 4.5 DP algorithm . . . 24 4.6 Determining parameters . . . 25 4.7 Results of the MDP . . . 27 4.8 Conclusion of MDP . . . 28 5 Conclusion 29 Appendices 33 A Results of the analysis by simulation 33 A.1 Scenario I . . . 33

A.2 Scenario II . . . 35

(5)

1 Introduction

Inflammatory bowel disease (IBD) entails the chronic conditions whereby patients suffer from inflammation of the gastrointestinal tract. Although IBD usually is not fatal, it could aggravate unexpectedly, causing life-threatening complications. In addition, IBD, even when in remission, could lead to severe psychological distress (Vidal et al., 2008). Hence, ideally patients regularly visit an outpatient clinic to check the state of their disease and have quick access to the outpatient clinic when their disease aggravates.

To provide both types of services, the outpatient clinic has to make two decisions. First, the outpatient clinic has to decide how much time of it’s physicians it will devote to the services of IBD patients each day, i.e. capacity decision. Second, a decision is made with respect to the allocation of time between regular check-ups and quick access services. The latter lies within the scheduling of patients as it is the planning of patients that decides at what times a regular check-up can take place and when to reserve time for potential future urgent arrivals. Therefore, this second decision can be referred to as a scheduling decision. Clearly, the outpatient clinic then experiences a trade-off between the utilization of physi-cians, i.e. occupancy rate, and the expected time between the request for an appointment and the actual moment of being served, i.e. expected admission time (AT), of more urgent patients.

This paper will focus on both decisions made by the outpatient clinic which is often referred to as the patient scheduling problem in the literature. While doing so, we will consider three types of patients that arrive according to a Poisson Process and vary in priority. These types from low to high priority are follow-up, new, and urgent patients. We will then design a rule that determines the capacity an outpatient clinic should offer given the arrival rates of patients. For the scheduling decision we will develop five policies that vary in the reservation of time slots for urgent patients and the time slots proposed to each patient. The goal is to achieve high occupancy rates for the outpatient clinic and low ATs for patients.

Klassen and Rohleder (1996) investigated the patient scheduling problem with the inclu-sion of multi-priority patients. To accommodate for these multi-priority patients, priority is given to more urgent groups of patients by reserving time slots that only they can be assigned to. Their goals was to then find the optimal number of reserved time slots and the positioning of these slots throughout a day such that patients experience a minimal waiting time at the clinic while maintaining low levels of idle time of the service provider. Relying on a similar approach as Klassen and Rohleder (1996), Broekhuis et al. (2008) did address the patient scheduling problem specifically for IBD with the distinction of three types of patients based on priority. To adapt for IBD patients, Broekhuis et al. (2008) put emphasis on AT instead of waiting time. Klassen and Rohleder (1996), Klassen and Rohleder (2004), and Broekhuis et al. (2008) all relied on simulation-based models in which they evaluated multiple setups which varied in the number of time slots reserved.

Patrick and Puterman (2007) and Patrick et al. (2008) also investigated the patient scheduling problem with the inclusion of multi-priority patients. However, they did not consider a number of time slots reserved for more urgent patients. Instead, they formulated the patient scheduling problem as a Markov Decision (MDP) process. In their MDP, all patients are assigned to time slots at the beginning of the day. Given these assignments, costs are incurred that increase with the length of ATs and the priority of the patients. The MDP formulation offers the advantage of not having to manually reserve time slots a priori as each decision includes the current state of the schedule and, therefore, allow for a more dynamic approach of assigning patients. However, due to the large state space that arises from including different priority patients, they could not use traditional MDP solving algorithms that rely on Dynamic Programming (DP).

(6)

be extended and different patterns of time slot reservations will be evaluated. Thereafter, we try to improve the occupancy rate and levels of AT with the insights of the simple policies even further by formulating the patient scheduling problem as a MDP. In contrast to previous work based on MDPs, it is considered that patients are scheduled directly when they request an appointment rather than being assigned all at once at the beginning or end of the day. To derive a policy from the MDP we apply the Value Iteration Algorithm (VIA) with which we try to minimize expected costs that are coupled to the AT and priority of patients. Yet, the MDP quickly suffers from the curse of dimensionality such that only a short planning horizon can be considered.

This paper is structured by the following sections. First, Section 2 will present relevant literature in the field of patient scheduling. Second, in Section 3 we analyze the problem by means of simulation. Third, in Section 4 we will formulate the patient scheduling problem as a MDP and evaluate the policy derived from this MDP. Fourth and last, Section 5 will conclude this paper and provide recommendations for further research.

2 Literature

This section will briefly review the literature related to the patient scheduling problem. The purpose of this review is to grasp what previously has been studied within the field of patient scheduling and to get inspiration for the design of the models used within this paper. For a more detailed overview of the existing literature on patient scheduling I would recommend Cayirli and Veral (2003), Gupta and Denton (2008), and Lakshmi and Iyer (2013).

Within the literature on the patient scheduling problem with the inclusion of multi-priority patients we observe two classes of researches, namely intra-day scheduling and inter-day scheduling. The first class focuses on the ordering of appointments throughout a inter-day with the goal to optimize the balance between the waiting time of patients at the clinic and the utilization of resources provided by the clinic. The second class focuses on the day a patient should be assigned to with the goal to optimize the balance between the time a patient waits for an appointment after the patient makes a request , i.e. AT, and the utilization of resources provided by the clinic. Since a large part of the literature is concerned with intra-day scheduling and only very few investigate the inter-day scheduling (see Gupta and Denton, 2008), we will, even though our research will focus on inter-day scheduling, briefly discuss literature on intra-day scheduling. Also, because most studies on inter-day scheduling are build on fundamentals obtained from studies on intra-day scheduling.

Initially, studies on intra-day scheduling analyzed the structure and positioning of ap-pointments within a schedule, later the focus shifted to involve more features such as patients with priority or the probability of no-shows and walk-ins. Klassen and Rohleder (1996) in-vestigated intra-day scheduling with the inclusion of multi-priority patients by proposing a daily scheme in which appointments are reserved for higher priority patients with the goal to lower wait time of higher priority patients. Kaandorp and Koole (2007) designed a local search procedure in which the expected waiting times of patients, idle time of physicians and overtime of physicians is optimized under the occurrence of no-shows. This local search procedure is then generalized by Koeleman and Koole (2012) to include emergence arrivals and general service time distributions.

Based on Klassen and Rohleder (1996) and their extension study Klassen and Rohleder (2004), Broekhuis et al. (2008) also investigated the scheduling of multi-priority patients but instead focused on inter-day scheduling. Since their focus is on inter-day scheduling, they did not propose the exact timing of reserved appointments but only the number of reserved appointments per day that optimizes the occupancy rate of physicians and the AT of patients. In addition, they also included the preference of patients to visit the same doctor each time a patient visits the outpatient clinic.

(7)

optimal policy dynamically allocate available capacity to match the request for appointments by patients. Alternatively,Liu et al. (2010) propose heuristic dynamic policies where the cur-rent state of the schedule is considered each time a request for an appointment occurs while Feldman et al. (2014) developed an appointment scheduling model in which the outpatient clinic dynamically makes days with appointments available from which patients can select. Liu et al. (2010) and Feldman et al. (2014) both include the possibility of cancellations and no-shows by patients.

Over time, many different methodologies have been applied in the field of the patient scheduling problem. Cayirli and Veral (2003) differentiate three type of methodologies; Analytical Studies, Simulation Studies, and Case Studies. While analytical studies and simulation-based studies will be most relevant for this paper, case studies do provide useful information with respect to the applicability of appointment systems. (O’Keefe, 1985) per-formed a case study on three clinics in the U.K. and proposed an appointment system that distinguishes between new and returning patients. Outpatient clinics were not fond of this proposal as physicians had difficulty adapting habits to the proposed appointment system. Hence, complicated assignment rules may not always be preferred over simplistic assignment rules.

The most interesting analytical studies for our work are Patrick and Puterman (2007) and Patrick et al. (2008). Patrick and Puterman (2007) and Patrick et al. (2008) describe the patient scheduling problem as a MDP with multi-priority patients. The benefit of a MDP is that it provides a mathematical framework which can be solved via a variety of mathematical programming algorithms. In both studies, each day is divided up in time slots in which a single patient can be served. In addition, they define the priority of patients by a limit on AT that the patient may experience. Then, at the beginning of each day all patients are assigned at once where the assignment of each patient is coupled to cost per day of AT a patient experiences. The optimal policy is then obtained by minimizing the expected costs. The issue that arises in their formulation of the MDP is twofold. First, from an outpatient clinic perspective it could be beneficial to schedule all patients at the beginning of a day but contemporary patient wish to be assigned directly when they request an appointment and do not want to wait till the next morning. Second, the state space within their MDP is too large because it includes all possible combination of types of patients that can arrive in a single day such that their MDP suffers from the curse of dimensionality.

While assignment rules tested in simulation-based models are less dynamic than the assignment rules obtained from a MDP formulation, simulation-based models are easier to set up even in complicated cases and most likely easier to implement in real life (see O’Keefe, 1985; Cayirli and Veral, 2003). Therefore, we also consider the two most interesting studies to our work that rely on simulation models, namely Broekhuis et al. (2008) and Van Buizen (2014). They specifically address the inter-day scheduling problem for an outpatient clinic serving patients that suffer from IBD by evaluating the AT of patients and the occupancy rate of physicians via the reservation of time slots specifically for higher priority patients. While their studies provide many insights into the multi-priority patient scheduling problem, a downside in their research is that they assume an old fashioned scheduling system in which follow-up patients are scheduled directly at the end of their appointment for their next appointment. This initially seems beneficial to the outpatient clinic as it gives the clinic more control over their future arrivals. However, much information is lost on the actual need for these appointments. Palvannan and Teow (2012) confirm that the use of such scheduling method neglects the value encapsulated in, for example, a Poisson arrival process that is observable when patients themselves contact the outpatient clinic for an appointment.

3 Analysis by simulation

(8)

what their important characteristics are that the outpatient clinic needs to cope with when scheduling patients. Therefore, this section will start with a case description in which we will elaborate on the classification of patients and their characteristics. Thereafter, we will respectively elaborate on the capacity decision by implementing a capacity threshold and the scheduling decision by means of four policies. After that, this section will present the results of these four policies with a given capacity threshold. At the end of this section we will conclude with a summary of the results and aspects that could be used in the development of the MDP.

3.1 Case description

When approaching this problem, a specific scenario is considered in which patients request an appointment via an online application and are directly offered an appointment. To accomplish this, it is assumed that patient types can be distinguished through a small ques-tionnaire filled in upon requesting an appointment. We will now classify the IBD patients and specify their main differences. Thereafter, we will specify the arrival rates and service times for which we rely on data presented in Broekhuis et al. (2008).

3.1.1 Patient classification and characterization

In the case of IBD, patients can be classified into three types, namely follow-up, urgent, and new. Let i indicate the type of patient such that i = 1 indicates follow-up patients, i = 2 urgent patients, and i = 3 new patients. In our research we consider these three types of patients to have three distinctive characteristics.

First, the number of request per time period differs between each patient type. Swartz-man (1970) evaluated the Poisson Process as arrival process for hospitals. He finds that a Poisson Process is a good approximation for the arrival process, given that the hospital is sufficiently large. He also states that accurate arrival rates can be obtained relatively quickly. We consider the group of patients at the outpatient clinic to be sufficiently large and we assume that patients arrive independent of each other as their state of disease does not depend on other patients. Therefore, in line with Swartzman (1970), we assume that each type of patient arrives according to a Poisson Process with rate λi, i ∈ {1, 2, 3} per

day. For convenience, the arrival rates are assumed to be constant over time which is in line with the number of appointments per year as reported in Broekhuis et al. (2008).

Second, types also differ in their required service time. Based on other researches (see Broekhuis et al., 2008; Van Buizen, 2014) it is assumed that service times required by IBD patients are relatively constant for each type, but differ between types. We define the service time required by the patient type(s) that requires the least amount of time for service as a single block. Then other types will require an integer multiple of these blocks. We can then define βi as the number of blocks required by patient type i, i ∈ {1, 2, 3}. This definition of

service time is useful in the development of the schedule as it allows us to split a physician’s day into blocks equivalent to those required by patients.

Third, each patient type is characterized by a time window in which they must be served. In other words, the AT of each type of patient may not exceed a specified threshold, i.e. a limit on AT. We define this limit with the use of the ‘Treeknorm’ 1_{. The ‘Treeknom’ is a}

norm on AT set by the Dutch government in cooperation with hospitals and describes the time within which each type of patient must be served by a clinic. Since the subject for this thesis originates from an outpatient clinic in the Netherlands we consider the ‘Treeknorm’ to be appropriate to use. Hence, the limits on AT for urgent and new patients are equal to the time set by this norm which are the following; urgent patients need to be strictly served within 8 calender days and new patients strictly within 28 calender days. The norm also provides a restriction for follow-up patients which is that they need to be served within 11 to 13 weeks since their last appointment.

(9)

Recall that AT is defined as the time between the requests for an appointment and the moment the patient is actually served. Hence, as we assume that follow-up patients do not request a new appointment right at the end of their last appointment, we cannot use the ‘Treeknorm’ for follow-up patients as an indicator for the limit on their AT. The ‘Treeknorm’, however, does specify that the norm within which follow-up patients need to be served is not strict due to the variety in needs of follow-up patients. Therefore, if follow-up patients are served outside the time window proposed by the norm it can still be assumed appropriate unless the outpatient clinic deviates too far from this norm. Hence, we assume that as long as follow-up patients can be scheduled within the planning horizon, which we will define in Section 3.2, we can ignore the limit on AT for follow-up patients. In addition, as outpatient clinics are usually closed during the weekend we assume for convenience a limit on AT for urgent and new patients of respectively 6 and 20 working days. For the remainder of this paper we will only consider working days.

3.1.2 Arrival rates

The data used by Broekhuis et al. (2008) contains the number of appointments realized by each patient type in the year 2005 at a Dutch outpatient clinic that serves IBD patients. In this data a total of 847, 88, and 38 appointments were recorded for respectively follow-up, urgent, and new patients. Since we assume arrival rates per day to be constant through time, we determine the arrival rate per day by dividing the number of appointments by the number of working days in a year. For the latter we consider a five-day work week, in line with the AT limit, such that a year contains 260 days (=52 · 5). These arrivals rates are presented in Table 1 as setup 1.

In addition to setup 1, we will also consider two different setups of arrival rates. As patients themselves request appointments, we could potentially see a shift in the number of requests of follow-up patients to urgent patients. Follow-up patients might request fewer appointments as they do not feel the need to make an appointment. However, as their state of the disease is checked less often, we assume that the risk of them becoming urgent increases. Setup 2 is resembling this potential effect by a 10% reduction in the arrival rate of follow-up patients which seems reasonable as this would imply that on average the time between two appointments for follow-up patients increases by roughly 1 week. For urgent patients a 10% increase is assumed. While we do think the arrival rate of urgent patients increases, we do not think this is in a one-to-one relation with a reduction to follow-up patients in absolute value. However, in relative terms we think this is an acceptable assumption as it implies that one out of ten follow-up appointments that are no longer present is replaced by an urgent appointment. Setup 3 extends the idea of setup 2 by adjusting both rates even further, namely 20% reduction and increase in the arrival rates of follow-up patients and urgent patients respectively. The arrival rates in setups 2 and 3 are presented in Table 1. Note that we have no reason to assume the arrival rate of new patients to change.

Table 1: Overview of arrival rates per day for each patient type in setups 1, 2 and 3. Arrival rates

Patient type Setup 1 Setup 2 Setup 3 Follow-up 3.26 2.93 2.61 Urgent 0.34 0.37 0.41

New 0.15 0.15 0.15

3.1.3 Service times

(10)

that one service block can represent 15 minutes in which one physician can serve a follow-up or urgent patient while new patients will require two service blocks. Table 2 represents the implications of the service times with βi, i ∈ {1, 2, 3}, as defined in Section 3.1.1.

Table 2: Required service blocks per patient type. Patient type β

Follow-up 1 Urgent 1

New 2

3.2 Capacity decision

With the capacity decision we decide on the shape and the size of the schedule to which patients can be assigned to. Hence, let the schedule contain a set of N days starting at the next day up to N days in the future with a rolling horizon. The latter implies that at the end of each day the first day of the schedule is removed and a new day is added at the back of the schedule. Then, the length of the schedule, i.e. planning horizon, will then remain constant over time. As the scale of AT is in terms of days, the exact time on a day is not relevant when a patient is requesting an appointment nor when the patient is served. Therefore, within the schedule we only consider the number of available service blocks, bn ∈ {1, ..., Bn}, at n, n ∈ N , days in the future with Bn as the maximum number

of service blocks at that day. In addition, we assume that patients will accept the proposed service blocks and are present at the appointments. Given this structure, the AT of patients is equal to the number of days in the future the patient is assigned to, i.e. AT = n. This section will continue by specifying N and Bn, n ∈ N .

3.2.1 Planning horizon (N )

For the length of the planning horizon a period of 20 days, N = 20, is considered. We choose this length because the highest limit on AT is 20 days for new patients. In addition, in practice physicians often know their schedule four weeks in advance which implies that a shorter planning horizon is not desired as it will require physicians to change the way they work (see O’Keefe, 1985).

3.2.2 Daily capacity (Bn)

To determine the daily capacity, we rely on a rule derived from queuing theory. To avoid long-term congestion, we need a service rate that is higher than the relative arrival rate (Van Foreest, 2019). The service rate is equivalent to the number of service blocks per time period and the relative arrival rate is the arrival rate multiplied by the number of service blocks required by each type. However, setting the service rate only minimally higher than the relative arrival rate might still cause short-term congestion when unexpectedly many patients arrive. If either long-term of short-term congestion occurs, patients have to be rejected which implies that the patient must be served in overtime with additional costs or has to be referred to a different health care provider resulting in revenue loss and image damage. On the other hand, setting the service rate too high implies lots of overcapacity which should also be avoided as it brings about unnecessary costs, e.g. salary of an idle physician or unused consulting rooms.

Let Bxbe the total number of service blocks within a period of x days, let Φ be the sum of daily relative arrival rates such that Φ =P3

i=1βiλi, and let M be a Poisson distributed

(11)

horizon with length N . Then, to avoid congestion we require BN _{to be such that P {M >}

BN_{} ≤ δ for a chosen threshold δ. Thus,} ∞ X M =BN₊₁ (N ∗ Φ)M M ! · e −N ∗Φ_{≤ δ.} ₍₁₎

In our model, we assume that a probability of less than 10% is sufficient to avoid a high number of rejections, i.e. δ = 0.1. Hence, for N = 20 we obtain B20 _{for each setup as}

shown in Table 3. Yet, with B20 _{determined we still have to decide how to distribute these}

service blocks over the planning horizon. Firstly, we will distribute the service blocks equally between weeks in the planning horizon by dividing B20_{by four and then rounded up to the}

nearest integer to obtain B5 _{as given in Table 3. The reason for rounding up is that we}

still want to comply with the rule as in Equation 1 which rounding to the nearest integer does not necessarily provide. Secondly, also within each week we will distribute the service blocks as equally as possible. This is achieved in two steps. In step one we allocate service blocks to each day equal to the integer quotient of the number of service blocks per week divided by the number of days in a week. In step two, the remaining service blocks are one by one allocated to each day of the week starting at the last day of the week until no more service blocks remain to be allocated. We have chosen to start at the last day of the week to enhance the probability of patients arriving before the service blocks are elapsed.

Table 3 also reports ρ which we define as the ratio of the weekly relative arrival rate to the weekly service rate, i.e. ρ = 5 ∗ Φ/B20_{. If no patients are rejected, then ρ indicates the}

expected occupancy rate of our model.

Table 3: Overview of the sum of relative arrival rates per day, Φ, the total number of service blocks in a period of 5 and 20 days, B5 and B20, and the ratio of the weekly relative arrival rate to the weekly service rate, ρ, for each setup with a capacity threshold δ = 0.1.

Setup Φ B20 B5 ρ Setup 1 3.90 89 23 0.85 Setup 2 3.60 83 21 0.86 Setup 3 3.32 77 20 0.83

As example of the distribution of service blocks, Figure 1 presents the case of setup 1 with B20 _{= 89 and B}5 _{= 23. First, we allocate four service blocks (=b}23

5c) to each day as

depicted in Figure 1a. Second, we allocate the three remaining service blocks one by one where Friday receives the first, then Thursday the second, etc. Eventually, we end up in a schedule as in Figure 1b. A full planning horizon with N = 20 consists of four of these weekly blocks. The behavior of allocating capacity is cyclical in the sense that when we move to the next day, the capacity of the day removed at the beginning of the schedule is equivalent to the capacity of the day added at the back of the schedule. Note, the service blocks on each day do not indicate an order in which service blocks are handled by the outpatient clinic. They only indicate on which day the service block is handled. For example, it is possible for these service blocks to be handled simultaneously if there are multiple physicians at work.

3.3 Scheduling decision

(12)

Mon Tue Wed Thu Fri Block 1: Block 2: Block 3: Block 4: Block 5:

(a) Allocating 4 service blocks to each day of the week.

Mon Tue Wed Thu Fri Block 1:

Block 2: Block 3: Block 4: Block 5:

(b) Allocating the 3 remaining service blocks.

Figure 1: Example of the distribution of service blocks per week for the case of setup 1. In (a) the integer quotient of weekly service blocks is equally distributed over the days of the week. In (b) the remaining 3 service blocks are allocated to the last three days of the week such that no service block remain to be allocated.

that patients who request an appointment get the first available service block, coupled to their type, proposed. The fourth policy does not reserve service blocks nor assumes the FCFS principle. Instead, the fourth policy lets patients select any slot available within the planning horizon. We will now give a more detailed description of each policy.

3.3.1 No reservation

In this policy all service blocks can be used by all types of patients. With the FCFS principle this implies that patients are served in the order of arrival independent of type. When a patient arrives and all service blocks within the planning horizon are full, the patient is automatically rejected, i.e. served in overtime or referred to a different health care provider. This policy is expected to achieve high occupancy rates as it will try to fill as many service blocks as possible. However, follow-up patients and new patients might use up all blocks in the first few days such that urgent patients might not be served within their limit on AT.

3.3.2 Constant reservation

With this policy a constant number of service blocks is reserved throughout the planning horizon that can solely be used by urgent patients. This implies that each day the same number of service blocks are reserved equal to the arrival rate of urgent patients per day rounded up to the nearest integer, dλ2e.

The implication of this policy is that whenever a follow-up or new patient arrives they get the first open, i.e. non-reserved, service block available proposed. However, when an urgent patient arrives (s)he will get a service block proposed on the first day with at least one service block available. If both an open and reserved service block are available, the urgent patient will get the reserved block proposed as otherwise we potentially have too many reserved service blocks that will remain unused. Figure 2 depicts an example of the application of this policy.

(13)

Block 4: Block 3: Block 2: Block 1: R R Mon R R Tue R R Wed

(a) Before a patient arrivals.

Block 4: Block 3: Block 2: Block 1: R R N N Mon R R Tue R R Wed

(b) After assigning new pa-tient. Block 4: Block 3: Block 2: Block 1: R U N N Mon R R Tue R R Wed

(c) After assigning urgent pa-tient. Block 4: Block 3: Block 2: Block 1: R U N N Mon R R N N Tue R R Wed

(d) After assigning new pa-tient. Block 4: Block 3: Block 2: Block 1: R U N N Mon R R N N Tue R R F Wed

(e) After assigning follow-up patient. Block 4: Block 3: Block 2: Block 1: U U N N Mon R R N N Tue R R F Wed

(f) After assigning urgent pa-tient.

Figure 2: Example of the constant reservation policy with a planning horizon of three days and four blocks per day. Empty blocks are available and blocks with a R are reserved for urgent patients. Blocks with a F , U , or N indicate a follow-up, urgent, or new patient assigned to that service block respectively. The sequence of patients arriving on Friday: New, Urgent, New, Follow-up, Urgent.

3.3.3 Increasing reservation

This policy is similar to the constant reservation policy with the exception of the first few days in the planning horizon. Initially, the number of reserved service blocks is zero for the first day of the schedule and increases by increments of one per day until the same number of service blocks per day are reserved as with the constant reservation policy. However, due to low the arrival rates of urgent patients (see Table 1), we adjust the policy slightly by placing the first reserved service block at d1/λ2e days in the future. Figure 3 provides an

example of this policy where for convenience of the example we do start reserving at the second day of the planning horizon. With this policy, patients are rejected under the same condition as with the constant reservation policy.

Note that when we move to the next day, the policy will evaluate the remaining reserved service blocks in the first few days of the planning horizon. If we consider the example of Figure 3, then Figure 3f depicts the state of the schedule at the end of Friday. By applying the rolling horizon, Wednesday will become the second day of the planning horizon and, according to the this policy, should have at most one reserved service blocks. Hence, the policy will enforce one of the remaining reserved service blocks on Wednesday to be no longer reserved for urgent patients. The new day added to the schedule, Thursday, will contain two reserved service blocks. Figure 4 depicts the application of the rolling horizon with Figure 3f as the state of the schedule at the end of Friday.

(14)

Block 4: Block 3: Block 2: Block 1: Mon R Tue R R Wed

(a) Before a patient arrivals.

Block 4: Block 3: Block 2: Block 1: N N Mon R Tue R R Wed

(b) After assigning new pa-tient. Block 4: Block 3: Block 2: Block 1: U N N Mon R Tue R R Wed

(c) After assigning urgent pa-tient. Block 4: Block 3: Block 2: Block 1: U N N Mon R N N Tue R R Wed

(d) After assigning new pa-tient. Block 4: Block 3: Block 2: Block 1: F U N N Mon R N N Tue R R Wed

(e) After assigning follow-up patient. Block 4: Block 3: Block 2: Block 1: F U N N Mon U N N Tue R R Wed

(f) After assigning urgent pa-tient.

Figure 3: Example of the increasing reservation policy with a planning horizon of three days and four blocks per day. Empty blocks are available and blocks with a R are reserved for urgent patients. Blocks with a F , U , or N indicate a follow-up, urgent, or new patient assigned to that service block respectively. The sequence of patients arriving on Friday: New, Urgent, New, Follow-up, Urgent.

Block 4: Block 3: Block 2: Block 1: F U N N Mon U N N Tue R R Wed

(a) Schedule at the end of Friday.

Block 4: Block 3: Block 2: Block 1: U N N Tue R Wed R R Thu

(b) Schedule at the beginning of Monday.

Figure 4: Example of the application of the rolling horizon with the increasing reservation policy.

3.3.4 Free selection

(15)

urgent patients.

To resemble the selection behavior, each type of patient will choose service blocks based on a probability distribution defined below. While we could not find any evidence for these probability distribution, we do think that they reflect real life behavior to a large extent. Follow-up patients

These patients will not feel much urgency to quickly access the outpatient clinic. Therefore, follow-up patients will select service blocks based on a uniform distribution such that each service block has an equal probability to be selected. If all service blocks within the planning horizon are full when a follow-up patient arrives, then this patient is automatically rejected. Urgent patients

These patients will feel high urgency and want to access the outpatient clinic quickly. There-fore, urgent patients will select service blocks within the first three days of the planning horizon given a uniform distribution such that each service block within those three days has an equal probability to be selected. However, when no service block is available within those three days, the urgent patient will select the first service block available within the remaining portion of planning horizon. Similar to follow-up patients, when no service blocks are available throughout the planning horizon, the patient is automatically rejected. New patients

These patient are not urgent but feel insecure about their health condition. Therefore, they will feel mediocre urgency, i.e. they want quick access to the outpatient clinic but not as quick as urgent patients. To accommodate for this behavior, new patients select service blocks within the first two weeks of the planning horizon given a uniform distribution such that each service block within those two weeks has equal probability to be selected. Recall that new patients will need two service blocks to be served. Therefore, service blocks that are the last one available on a day are not proposed to the patient and, thus, are not in-cluded in the uniform distribution. When no service blocks are available within those first two weeks, the new patient will select the first two consecutive service blocks available on a single day within the remaining portion of the planning horizon. Similar to follow-up and urgent patients, when no service blocks are available throughout the planning horizon, the patient is automatically rejected.

The expectations of this policy is that the ATs of patients might go up but that the ATs for urgent patients remain within acceptable levels while, also, follow-up and new patients do not experience dissatisfaction as they potentially choose longer ATs themselves. However, the occupancy rate with this policy might be lower if patients select too many service blocks other than the first service blocks available in the schedule such that the early service blocks eventually remain unused.

3.4 Results of simulation

The simulation model and the policies are coded in the programming language Python (version 3.7)2_{. Within each simulation a stream of arrivals is generated according to the}

arrivals rates as in Table 1. This is done 10 times for each policy over a period of 50 years. The 50 years start with a warm-up period of 1 year to avoid measurement errors deriving from an empty schedule at the start. For convenience, the tables in this section will only highlight the most important results. For the complete results, we refer the reader to Appendix A.

(16)

3.4.1 Setup 1

Table 4 presents the occupancy rates of each policy with the arrival rates of setup 1 and the capacity threshold δ = 0.1. These occupancy rates can be considered low but are in line with ρ = 0.85 with the exception of the constant reservation policy. This implies that we use the capacity as expected and there should be close to zero rejections. This is indeed the case, if we inspect Table 5 with the results of ATs and rejections. In the last column of Table 5 we only observe low rejection rates with the constant reservation policy. In addition, we see that the mean AT of both urgent and new patients is much lower than their limit and only a small proportion of urgent patients is served beyond their limit on AT.

From the results, we suspect that we are able to use less capacity to enhance the occu-pancy rate while keeping the mean ATs, the proportions of urgent patients served beyond their limit on AT, and the rejection rates sufficiently low. Especially, because we observe no significant differences in the results of the no reservation policy and the increasing reserva-tion policy which implies that with the current capacity, the reserved service blocks within the increasing reservation policy are barely used. However, the constant reservation policy is an exception to this. If we take the ratio of the weekly relative arrival rates of follow-up and new patients together and the number of service blocks available to these patients, we see that this ratio is 0.99 (= 5 · (3.26 + 2 · 0.15)/(23 − 5)). Hence, due to the constant number of reserved service blocks each day, only a minimal number of service blocks remain available for follow-up and new patients. Reducing the capacity potentially leads to ratio above 1 causing a permanent shortage in capacity for these type of patients.

Table 4: Results w.r.t. occupancy rate of each policy with setup 1. The values are the weighted averages of 10 runs over a period of 50 years.

No Constant Increasing Free reservation reservation reservation selection Mean Var Mean Var Mean Var Mean Var Occupancy rate 0.850 0.006 0.844 0.002 0.850 0.006 0.848 0.004

Table 5: Results w.r.t. AT (in days) of each policy with setup 1. The values are the weighted averages of 10 runs over a period of 50 years. For each patient type the mean AT, variance of AT, and the rejection rate are given. In addition, the proportion of urgent patients not served within their AT limit is given.

AT Proportion of patients Rejection Policy Patient type Mean Var with AT > limit rate

(17)

3.4.2 Setups 2 & 3

In line with the results of setup 1, the results of setups 2 and 3 also indicate that we could potentially use less capacity. Therefore, we will not go in detail on the results and refer the reader to Appendix A for the results of these two setups.

3.5 Alternative capacity scenarios

From our results, we observed that three out of four policies can potentially still provide sufficient results with less capacity. Therefore, we decided to evaluate these three policies in two scenarios with less capacity. Recall that in our capacity decision we set the threshold on the probability that the relative arrival rate over a period of N days exceed the number of service blocks in a period of N days equal to 10%, δ = 0.1. Setting this threshold δ higher would imply that the model will allocate less capacity for the service of patients. The two alternative scenarios will be referred to as scenario II and scenario III and will have a threshold δ = 0.25 and δ = 0.5 respectively. Note that scenario III with δ = 0.50 implies that the relative arrival rate over a period of N days is equal or minimally higher due to rounding up to integers than the number of service blocks in a period of N days. While this level of capacity usually causes problems, we suspect that the application of the rolling horizon can potentially offer a sufficient number of service blocks as a buffer in case unexpectedly many arrivals occur.

Table 6 provides an overview of the number of service blocks in the planning horizon with a length of 20 days, B20_{, the number of weekly service blocks, B}5_{, and the ratio of}

the weekly relative arrival rate to the weekly number of service blocks, ρ, in each of the two alternative scenarios.

As mentioned in Section 3.4.1, we expect that there will be a shortage in capacity for follow-up and new patients when the constant reservation policy would be applied in sce-narios II and III. This is true as the ratio between the weekly relative arrival rate of these patients together and the weekly number of service blocks in scenario II would be 1.22, 1.20, and 1.28 for setups 1 to 3 respectively. These values will even be higher in scenario III. Hence, the constant reservation policy is excluded in the analysis of these two alternative scenarios.

Table 6: Overview of the effects of the threshold δ in scenarios II and III on the number of service blocks in the planning horizon with a length of 20 days, B20_{, the}

number of weekly service blocks, B5_{, and the ratio of the weekly relative arrival}

rate to the weekly number of service blocks, ρ.

Scenario II Scenario III

Setup Φ δ B20 _B5 _ρ _δ _B20 _B5 _ρ

Setup 1 3.90 0.25 84 21 0.93 0.50 78 20 0.98 Setup 2 3.60 0.25 78 20 0.90 0.50 72 18 1.00 Setup 3 3.32 0.25 72 18 0.92 0.50 66 17 0.98

3.5.1 Results scenario II

(18)

3.5.1.1 Setup 1

Table 7 presents the occupancy rates of each policy with the arrival rates of setup 1. While these rates are higher than in scenario I with δ = 0.1, they are again in line with ρ = 0.93 which again implies that we use the capacity as expected and there should be close to zero rejections. Table 8, which presents the results with respect to ATs and rejections, confirms this. In the last column of this table we only see a very small proportion of new patients being rejected with the free selection policy.

In Table 8 we also observe that the no reservation policy and the increasing reservation policy still provide equivalent results with a low proportion of urgent patients served beyond their limit on AT, zero rejections, and a mean AT far below the limit on AT for urgent and new patients. Therefore, we again suspect that we are able to use less capacity to enhance the occupancy rates further without harming the ATs and rejection rates of patients given that either of these two policies are chosen.

For the free selection policy it might be less desirable to lower the capacity as the proportion of urgent patients served beyond their limit on AT is 16% with a threshold of 25%, i.e. δ = 0.25. This proportion is likely to increase further as we lower the capacity. However, the mean AT of urgent and new patients is still almost 2 and 12 days below their respective limit on AT. Also, the rejection rates with the free selection policy are currently neglectable small.

Table 7: Results w.r.t. occupancy rate of each policy with setup 1 in scenario II. The values are the weighted averages of 10 runs over a period of 50 years.

No reservation Increasing reservation Free selection

Mean Var Mean Var Mean Var

Occupancy rate 0.929 0.003 0.929 0.003 0.928 0.003

Table 8: Results w.r.t. AT (in days) of each policy with setup 1 in scenario II. The values are the weighted averages of 10 runs over a period of 50 years. For each patient type the mean AT, variance of AT, and the rejection rate are given. In addition, the proportion of urgent patients not served within their AT limit is given.

No Follow-up 2.52 3.23 - 0.000 Reservation Urgent 2.52 3.15 0.037 0.000 New 2.74 3.15 - 0.000 Increasing Follow-up 2.53 3.48 - 0.000 Reservation Urgent 2.36 1.79 0.001 0.000 New 2.81 4.10 - 0.000 Free Follow-up 14.14 24.27 - 0.000 Selection Urgent 4.04 9.39 0.160 0.000 New 7.72 14.08 - 0.001 3.5.1.2 Setups 2 & 3

(19)

3.5.2 Results scenario III

Here we will present the results of scenario III with the capacity threshold δ = 0.5. Note that this implies that we have the lowest possible capacity given our capacity decision rule without the weekly relative arrival rate exceeding the number of weekly service blocks. For the simulations we used the same procedure as in scenarios I and II where each setup is evaluated 10 times for each of the three policies considered over a period of 50 years with a warm-up period of 1 year. Also here, for convenience, the tables in this section will only highlight the most important results. For the complete results, we refer the reader to Appendix A.

3.5.2.1 Setup 1

In Table 9 we see that the occupancy rates of the three policies considered in this scenario. Clearly, these rates are higher than in the previous two scenarios. Interesting is that the occupancy rates are only slightly lower than the expected occupancy rate ρ (=98%). This is especially interesting for the free selection policy as we expected a lower occupancy rate for this policy. Apparently, the time preference of urgent and new patients to be served quickly almost offsets the probability of service blocks remaining unused due to patients not being forced to select the first service block available.

Similar to previous scenarios, an occupancy rate close to ρ implies that we should have rejections but not many. Once more, Table 10 with the results of ATs and rejections confirms this. In the last column we see rejection rates between 0% and 1%. Hence, we almost have sufficient capacity to serve all patients within the planning horizon even with this minimum level of capacity. However, we do observe large proportions of urgent patients being served beyond their limit on AT with the exception of the increasing reservation policy.

Table 9: Results w.r.t. occupancy rate of each policy with setup 1 in scenario III. The values are the weighted averages of 10 runs over a period of 50 years.

Occupancy rate 0.974 0.001 0.973 0.001 0.970 0.001

Table 10: Results w.r.t. AT (in days) of each policy with setup 1 in scenario III. The values are the weighted averages of 10 runs over a period of 50 years. For each patient type the mean AT, variance of AT, and the rejection rate are given. In addition, the proportion of urgent patients not served within their AT limit is given.

(20)

As expected, the proportion of urgent patients served beyond their limit has increased to 45.9% with the free selection policy. However, now also for the no reservation this proportion is large with 31.2% of urgent patients served beyond the limit. For the free selection policy this even implies that the mean AT of all urgent patients is larger than the limit. In contrast, with the increasing reservation policy applied, this proportion is only 0.9%. In addition, the increasing reservation policy also provides the lowest mean AT for urgent patients. Hence, the benefits of reserved service blocks solely for urgent patients in the increasing reservation policy can now be utilized more often than in the previous scenarios.

Interestingly, the increasing reservation policy also offers the lowest mean AT for follow-up patients. Although this advantage seems to arise fully at the expense of new patients as their mean AT and rejection rates are slightly higher than with the no reservation policy. These additional rejections of new patients cause more service blocks to remain available for follow-up patients.

3.5.2.2 Setup 2

Recall that in this setup we have ρ = 1 such that the weekly service rate exactly matches the weekly relative arrival rate. Therefore, the occupancy rates presented in Table 11 are the highest in this scenario and setup. Since these rates are close to ρ for all policies, we should again observe only a few rejections. Also here, Table 12 with the results of ATs and rejections confirms this with low rejection rates in the last column of this table. Although the rejection rates are slightly higher than with the arrival rates of setup 1 in scenario III, the highest rejection rate is still only 3.3% for new patients with the free selection policy. However, the no reservation and free selection policies cause a large proportion of urgent patients to be served beyond their limit on AT. Remarkably, this proportion of urgent patients is still only 2.1% with the increasing reservation policy.

Occupancy rate 0.993 0.000 0.991 0.001 0.985 0.001

(21)

In Table 12 we also observe for both the no reservation policy and the free selection policy that the mean AT of all urgent patients is larger than their limit. Therefore, the differences in the mean AT of urgent patients between the two policies and the increasing reservation policy has only become larger. Also, the difference in the mean AT for follow-up patients has become larger in favor of the increasing reservation policy. In addition, the increasing reservation policy now also provides the lowest mean AT for new patients. However, the increasing reservation policy in turn rejects more follow-up and new patients than with the no reservation policy.

3.5.2.3 Setup 3

Table 13 presents the results with respect to the occupancy rate of each policy in scenario III with the arrival rates of setup 3. Since in this setup the ratio ρ is equivalent to that of setup 1 in this scenario, we also observe occupancy rates close to those of setup 1. Yet, we do not observe this equivalence in the mean ATs, proportions of urgent patients served beyond their limit on AT, and the rejections rates. Table 14 presents these results for setup 3 in this scenario. While the difference between the policies are the same as in setup 1 of this scenario, overall we observe higher mean ATs, larger proportion of urgent patients served beyond their limit on AT, and higher rejection rates. The mean AT of urgent patients with the no reservation policy is even larger than their limit on AT. We think that this is caused by the fact that in setup 3 there are three days per week where the number of service blocks is lower than the daily relative arrival rate whereas in setup 1 each day has atleast a number of service blocks that exceeds the daily relative arrival rate. Therefore, in setup 3 patients more often have to be allocated one additional day further in the planning horizon and, thus, experience a higher AT. Logically, this also implies a higher variance in AT and urgent patients more often being served beyond their limit on AT.

Occupancy rate 0.977 0.001 0.975 0.001 0.971 0.002

(22)

3.6 Conclusion

From the analysis by simulation we gained many insights. We will briefly summarize the most interesting observations.

First of all, to determine our capacity we set a threshold δ on the probability that the relative arrival rate over a period of 20 days exceed the number of service blocks in a period of 20 days. When setting this threshold we underestimated the buffer of service blocks the rolling horizon provides as a new day is added at the back of the planning horizon each time a day has elapsed. In addition, the arrival rates per day in each setup are relative low such that, independent of the value δ, the rounding up of weekly capacity also provides a relative big buffer of service blocks. Hence, we found that a threshold of 50%, δ = 0.5, can provide almost sufficient capacity to serve all patients within 20 days.

Second, the constant reservation policy should not be adopted by an outpatient clinic. Due to a constant number of service blocks reserved solely for urgent patients each day, there remain too few service blocks open for follow-up and new patients even with δ = 0.1. Third, the free selection policy reports large proportion of urgent patients not served within their limit on AT if δ is set to 0.25 or 0.5. While the policy does offer freedom to patients which could offer benefits, e.g. less cancellations, we do not expect these benefits to offset this high proportion. Especially, because urgent and new patients might not experience freedom of selection as with these levels of capacity more often days early in the planning horizon are already fully occupied such that these patients will feel the need to only select the first service block available. However, the inclusion of these benefits is out of the scope of this paper such that we cannot know this for certain.

Fourth, only when a minimal capacity is adopted, i.e. δ = 0.5, we find differences between the performance of the no reservation policy and the increasing reservation. The increasing reservation policy provides proportions of urgent patients served beyond their limit on AT between 1% and 2% while these proportions are between 31% and 70% with the no reservation policy. In addition, the increasing reservation provides a lower AT to follow-up and urgent patients in all three setups while slightly higher for new patients in two of the three setups than the no reservation. Hence, the increasing reservation policy overall performs better.

Fifth, the capacity decision and the scheduling decision perform relative stable such that if a policy performs well under the current conditions of arrival rates, then most likely in the future the policy will perform equally well given our capacity decision.

Sixth, overall the increasing reservation policy provides the best results with respect to mean AT and low proportion of patients served beyond their limit on AT even with an absolute minimal weekly capacity. However, some of these benefits come at the expense of new patients being rejected.

Even though the rejection rates with the increasing reservation policy are low, we still want to see if these rejections could be completely avoided by deriving an optimal policy from a MDP formulation of the patient scheduling problem. The formulation of the MDP and the analysis of its optimal policy is given in the next section.

4 Analysis by MDP

(23)

of the MDP policy. Fourth, this section ends with a conclusion on the analysis by MDP.

4.1 State space

The state space entails all states that are observed upon each decision epoch. For this we have to make a capacity decision that entails the length of the planning horizon and the daily capacity. In addition, we need to define the decision epochs at which the the schedule is observed.

4.1.1 Capacity decision

The capacity decision will be similar to the capacity decision made in the model for the analysis by simulation. However, for a rolling horizon in the MDP we need each day to have the same capacity. Hence, let B be the maximum number of service blocks available per day and let N be the length of the planning horizon. In addition, let bn ∈ {0, 1, ..., B}

indicate how many service blocks are already occupied on the nth, n ∈ {1, ..., N }, day of the planning horizon. Note that also here the first day in the schedule indicates tomorrow. 4.1.2 Decision epochs

To mimic reality of scheduling a patient directly and to avoid exploiting knowledge of future arrivals we must have at most one request at each decision epoch. For this we assume the arrival rates to remain constant throughout a day.

Let K be the total number of intervals, i.e. decision epochs, per day with the length of an interval equal to 1/K and let Λ be the sum of daily arrival rates. Then, similar to the approach taken in Section 3.2.2 for the capacity, we have to set K such that P {M > 1|_K1} ≤ ζ is satisfied where M is Poisson distributed variable with rate Λ/K which indicates the number of arrivals in a single interval and ζ a chosen threshold on the probability of more than one arrival in an interval. Thus, K given Λ and ζ needs to satisfy,

P {M > 1|1

K} = 1 − (1 + Λ K) · e

−Λ/K_{≤ ζ.} ₍₂₎

At each decision epoch the schedule will be observed and also a patient of type i, i ∈ {0, 1, 2, 3}, where i is defined as in Section 3.1.1 with the addition that i = 0 indicates that no patient has arrived.

4.1.3 Defining the state space

Combing the capacity decision and the decision epochs we have that the state space takes the form:

S = (~b; k, i) = (b1, ..., bN; k, i), (3)

where k indicates the interval on the current day with k ∈ {1, ..., K}.

Table 15 depicts a small example of some states that can be observed throughout a day. In this example a day is cut into three intervals, K = 3, and entails a planning horizon of three days, N = 3. At the end of each interval k, k ∈ {1, 2, 3}, a decision takes place in which the current state of the schedule is represented by b1, b2, and b3, and the type of

(24)

Table 15: Example of the state space for a day with three decision epochs, K = 3, a planning horizon of three days, N = 3. bn indicates the number of service blocks

already used on the nth, n ∈ {1, 2, N }, day of the planning horizon. The patient type that has arrived at each k, k ∈ {1, 2, K} is indicated by i ∈ {0, 1, 2, 3}.

Decision epoch b1 b2 b3 Arrival i

k = 1 3 1 2 1

k = 2 3 2 2 2

k = 3 = K 3 3 2 3

4.2 Action space

The action set will incorporate a set of all possible actions, i.e. scheduling decisions, that can be taken given the state s, s ∈ S, of the system, i.e. As. This set of actions will entail

the assignment of a patient or the rejection of a patient when there are not sufficient service blocks available. Also here, the rejection of patients imply that they have to be served in overtime or are redirected to another health care provider.

Let an{0, 1}, n ∈ {1, ..., N } be the action of assigning a patient to the nth day of the

planning horizon and let ar ∈ {0, 1} be the action of rejecting a patient. Then, to make

sure that a patient is either assigned or rejected only once we must havePN

n=1an+ ar= 1.

In addition, a patient cannot be assigned to a day with insufficient service blocks available. Hence, the assignment of a patient of type i, i ∈ {0, 1, 2, 3}, must satisfy the condition bn+ βian≤ B for all n ∈ N and where βiis as defined in Section 3.1.3. The values of βiare

corresponding to those in Table 2 with the addition of β0= 1 for ease the modeling. Then,

we define the action space as follows,

As= (~a|s) = a1, ..., aN, ar| N X n=1 an+ ar= 1, bn+ βian≤ B ! . (4)

Table 16 provides an example of the action space that continues on the example of the state space (see Table 15). Hence, K = 3 and N = 3. In addition, a maximum of three service blocks is available per day, B = 3. At k = 1, first the schedule and a follow-up patient is observed. Thereafter, the follow-up patient is assigned to the second day of the planning horizon indicated by a2= 1 such that both conditions are satisfied. Similarly, the

urgent patient at k = 2 is also assigned to the second day. To comply with both conditions, the new patient at k = 3 must be rejected as β3= 2.

4.3 Cost structure

Each decision taken is valued by costs. These costs need to incorporate time preferences by urgent and new patients to be served rather earlier than later, a penalty if urgent and new patients exceed their limits on AT as given in Section 3.1.1, and a penalty if a patient is rejected. Let us define the following variables,

αi costs per day of AT for a patient of type i, i ∈ {2, 3}.

li, defines the limit on AT for a patient of type i, i ∈ {2, 3}.

hi, defines the additional costs per day of AT beyond lifor a patient of type i, i ∈ {2, 3}.

(25)

Table 16: Example of the action space for a day with three decision epochs, K = 3, a planning horizon of three days, N = 3, and a maximum of three service blocks per day, B = 3. bn indicates the number of service blocks already used on the

nth, n ∈ {1, 2, N }, day of the planning horizon, an ∈ {0, 1} indicates whether the

patient is assigned to the nth, n ∈ {1, 2, N }, day of the planning horizon, and ar ∈ {0, 1} indicates whether the patient is rejected. The patient type that has

arrived at each k, k ∈ {1, 2, K} is indicated by i ∈ {0, 1, 2, 3}. Decision epoch Action day 1 day 2 day 3

k = 1 Observing b1 b2 b3 arrival i 3 1 2 1 Assigning a1 a2 a3 ar 0 1 0 0 k = 2 Observing b1 b2 b3 arrival i 3 2 2 2 Assigning a1 a2 a3 ar 0 1 0 0 k = 3 = K Observing b1 b2 b3 arrival i 3 3 2 3 Assigning a1 a2 a3 ar 0 0 0 1

We assume that with the costs per day of AT that there is sufficient pressure to not let capacity be unused. In addition, we assume that the rejection of any type of patients is equally undesirable. However, to make sure that this also holds in relative terms, we have to define a rejection cost per type of patient. Also, recall that follow-up patients do not have a limit such that we cannot apply a penalty for exceeding a limit on AT and neither do we assume time preference for this type.

With these variables the cost structures will be as follows,

ci=      P n∈NZan, for i = 0 r1ar, for i = 1 P n∈N[(n − 1)αi+ [n − li] +_h i] an+ riar, for i = 2, 3 (5)

with Z sufficiently large such that the action to assign is never chosen when no patient has arrived (i = 0) and [n − li]+= max{n − li, 0}.

These costs structures imply that when no patients arrive, then choosing the action of assigning incurs Z costs while rejecting is free. If a follow-up patient arrives, then there are only costs incurred when this patient is rejected equal to r1. If an urgent or new patient

arrives, then assigning the patient incurs costs that increase linearly with the days of AT for this patient,P

n∈N(n − 1)αian with i ∈ {2, 3}. However, if the limit on AT for the urgent

or new patient is exceeded, then the costs are increased with an additional cost per day of AT beyond their limit, P

n∈N[n − li]+hian with i ∈ {2, 3}, such that the linear increase

becomes steeper. Note, since we do not see a reason to incur costs for assigning a patient to the first day of the planning horizon, we decide that the costs per day of AT are only incurred from assigning the patient to the second day of the planning horizon and onward. In addition, rejecting an urgent or new patient incurs a cost of ri, i ∈ {2, 3}.

4.4 Transition probabilities

(26)

potential transitions and thereafter assign their probabilities of occurring.

The state before and after a transition can differ in any of the three components of the state spaces, namely ~b, k, and i. In other words, during a transition we can observe changes in any these components. The changes to the components ~b and k are deterministic.

If a patient is rejected and k < K, then ~b will remain unchanged and k changes to k + 1. However, if k = K, then k changes to 1 instead and also ~b has to be adjusted for rolling horizon by removing the first day of the planning horizon and adding a new empty day at the back of the planning horizon.

If a patient is assigned and k < K, then ~b will be adjusted for the number of occupied service blocks and k changes to k + 1. However, if k = K, then also in this case we have that k changes to 1 instead and, after the adjustment of ~b for the number of occupied service blocks, ~b has to be adjusted for the rolling horizon by removing the first day of the planning horizon and adding a new day at the back of the planning horizon. With k = K, note that the order of adjustments to ~b is important because otherwise assignment of a patient is recorded on the wrong day of the planning horizon.

In contrast to ~b and k, a change in i is stochastic as independent of the chosen action we do not know the type of the next arrival.

To summarize these changes as transitions, we let s be the state we transition from and let s0 be the state we transition to. In addition, let i, i ∈ {0, 1, 2, 3}, indicate the type of the patient in state s and let j, j ∈ {0, 1, 2, 3}, indicate the type of patient in state s0. Then, for each state s we have the following two cases with each two sets of possible transitions,

s = (b1, ..., bN; k, i) Rejecting =====⇒ s0 = ( (b1, ..., bN; k + 1, j) , for k < K, (b2, ..., bN; 0; 1, j) , for k = K, (6) s = (b1, ..., bN; k, i) Assigning ======⇒ s0 = ( (b1+ βia1, ..., bN+ βiaN; k + 1, j) , for k < K, (b2+ βia2, ..., bN+ βiaN, 0; 1, j) , for k = K. (7) Figure 5 presents a general overview of each of the four transitions from state s to s0 with an arrival of type j ∈ {0, 1, 2, 3} that can occur within each transition set. Along the arcs of this figure, we then find the transition probabilities. More formally, define pj, j ∈ {0, 1, 2, 3},

as the transition probabilities corresponding to a transition from state s, s ∈ S, to any state s0, s ∈ S, with an arrival of type j, j ∈ {0, 1, 2, 3} and let X be the number of arrivals per day. Then, for the transition probabilities to represent the arrival rates per day, we need

E[X] = Λ =⇒ 3 X i=1 pj· K = Λ =⇒ 3 X i=1 pj= Λ/K, (8)

If we want to keep the proportion between the arrivals rates of each type in combination with Equation 8, we have that the probability of each type of arrival is,

pj= λj Λ 3 X i=1 pj= λj/K, for j ∈ {1, 2, 3}, (9)

with the probability of no patient arriving equal to p0= 1 −P 3 i=1pj.

4.5 DP algorithm

(27)

s s0 with j = 0 s0 with j = 1 s0 with j = 2 s0 with j = 3 p0 p1 p2 p3

Figure 5: Overview of the possible transitions from state s to state s0 with their corresponding transition probabilities.

4.6 Determining parameters

This section will discuss the values of the parameters that are used when we apply the VIA to our MDP and when we perform a simulation with the policy obtained through the VIA. 4.6.1 Arrival rates and service times

For the arrival rates and service times of patients we use the same setups as in the analysis by simulation. Therefore, the arrival rates and service times can be found in Tables 1 and 2 respectively.

4.6.2 State space variables

The state space is determined by the length of the planning horizon, the daily capacity, and the number of decision epochs per day.

4.6.2.1 Planning horizon (N )

For the planning horizon we assume a length of 20 days, N = 20, similar to the length of the planning horizon used in the analysis by simulation.

4.6.2.2 Daily capacity

As mentioned in Section 4.1.1, to adopt the rolling horizon we need every day in the planning horizon to have the same capacity. Therefore, we set the number of service blocks per day equal to the sum of relative arrival rates per day rounded up to the nearest integer such that B = dΦe. With the definitions of Φ, B5_{, and ρ as in Section 3.2.2 we have respectively}

Φ =P3

i=1βiλi, B5= 5B, and ρ = Φ/B5. Table 17 reports the values of B5 in each setup

and their corresponding ρ.

Table 17: Overview of the number of weekly service blocks, B5_{, and the ratio of}

the weekly relative arrival rate to the weekly service rate, ρ, in each setup. Setup B5 _ρ

1 20 0.98 2 20 0.90 3 20 0.85

(28)

respective ρ is equal to that of setup 2 in scenario II and setup 3 in scenario I respectively. Since we had too much capacity in scenarios I and II, we decided to no longer consider the two potential future setups 2 and 3 in our MDP.

4.6.2.3 Decision epochs

We assume that the probability of less than 5% for observing more than one arrival between two decision epochs, ζ = 0.05, is sufficient. Solving Equation 2 numerically gives 11 decision epochs per day, K = 11, for setup 1.

4.6.2.4 Curse of dimensionality

With the chosen parameters for the length of the planning horizon, daily capacity, and decision epochs we found that our MDP suffers from the curse of dimensionality as the size of our state space becomes enormous with a total of 4 · (B + 1)N_{·K states, where 4 originates}

from the different types of patients including no arrivals. The curse of dimensionality refers to the case where a problem is defined in a high dimensional space such that the problem becomes computational intractable. For our MDP this implies that the VIA has a run-time exponentially in the size of the state space and requires too many values to be stored.

To avoid the curse of dimensionality, B, K, and N could be adjusted. However, as the arrival rates are low, so is the value of B and we cannot set the value of B below the arrival rates. In addition, the adjustment of the threshold ζ to achieve a low value for K is not desired as K can then easily become a limit on the maximum number of arrivals that can occur on a day. Hence, reducing N is the only reasonable option to reduce the size of the state space as we expect it to have the least influence on the performance of our MDP policy and N also contributes the most to the size of the state space as it is in the exponent. After trial and error, we found that a planning horizon of five days, N = 5, is the maximum length in which the VIA will still execute within a reasonable time.

4.6.3 Cost parameters

Table 18 presents the values of the cost parameters. For li, i ∈ {2, 3}, we use the limits

given in Section 3.1.1. For αi, i ∈ {2, 3}, we assume that the proportions between the limits

on AT can also be used as the proportion between the time preferences such that for urgent and new patients we have α2= 4α3. Then, for convenience of modeling we let α3= 1 from

which we then have α2 = 4. For hi, i ∈ {2, 3}, we have no values because the planning

horizon of five days is not sufficiently long for patients to exceed their limit on AT. For r1

we choose a cost of 21 and for ri, i{2, 3} we choose a cost of 21 + (N − 1)αi. These rejection

costs make sure that it is not beneficial to reject a patient of one type over the other if they would require the same number of service blocks for service.

Table 18: Overview of the cost parameters for each type of patients with a cost per day of AT, α, a limit on their AT, l, additional costs per day of AT beyond their limit on AT, h, and a cost of rejecting the patient, r.

Patient type α l (in days) h r

Follow-up - - - 21

Urgent 4 6 - 37

New 1 20 - 25

4.6.4 Transition probabilities

Patient Scheduling For Multi-Priority Patients With Inﬂammatory Bowel Disease: A Markov Decision Process Approach

Patient Scheduling For Multi-Priority Patients With

Inflammatory Bowel Disease: A Markov Decision Process

Patient Scheduling For Multi-Priority Patients With

Inflammatory Bowel Disease: A Markov Decision Process

Approach

T.L. de Haan

Contents

1

Introduction

2

Literature

3

Analysis by simulation

3.1

Case description

3.2

Capacity decision

3.3

Scheduling decision

3.4

Results of simulation

3.5

Alternative capacity scenarios

3.6

Conclusion

4

Analysis by MDP

4.1

State space

4.2

Action space

4.3

Cost structure

4.4

Transition probabilities

4.5

DP algorithm

4.6

Determining parameters