Condition-based maintenance optimization with
multiple types of imperfect inspections
Master’s Thesis Econometrics, Operations Research & Actuarial Studies University of Groningen
Contents
1 Introduction 2 2 Problem Description 3 3 Discretisation 4 3.1 Deterioration process . . . 5 3.2 Imperfect inspections . . . 54 Markov decision process formulation 6 4.1 Knowledge state space . . . 6
4.2 Possible actions . . . 6
4.3 Transition probabilities . . . 7
4.3.1 Do nothing . . . 7
4.3.2 Preventive maintenance . . . 7
4.3.3 Inspection . . . 7
4.4 Reduction to finite knowledge state space . . . 8
5 Solution method 9 5.1 Costs-to-go . . . 9
5.1.1 Value iteration . . . 9
6 Numerical Experiments 10 6.1 Mixing perfect and imperfect inspections . . . 11
6.2 Two imperfect inspections of different qualities . . . 13
6.3 Degree of approximation to reality . . . 16
7 Conclusion 17
8 Acknowledgements 19
Condition-based maintenance optimization with multiple
types of imperfect inspections
Sagarika Patra (S2318970)
Abstract
1 INTRODUCTION
1
Introduction
Condition-based maintenance (CBM) has been an increasing area of interest in the maintenance optimisation literature. The benefits when this type of maintenance is effective compared to others depends on the circumstances. Furthermore, condition-based maintenance is increasingly common due to improvements in technology and feasibility of such methods. A full overview of different types of maintenance policies can be found in Wang (2002). CBM aims to monitor the condition of the system by means of variables indicating wear of units in order to predict failures and perform preventive maintenance efficiently to reduce costs. A comparison of the strengths and weaknesses of CBM when compared to time-based maintenance can be found in De Jonge et al., (2017a). In many cases, CBM offers a distinct advantage over time-based maintenance as it is able to take advantage of useful remaining lifetimes by opportunistically postponing a preventive maintenance. Within the systems studied under CBM policies many variations are possible with respect to the number of units in a system, failure modes and dependencies, deterioration processes, and inspection and maintenance types. Various models have been applied for maintenance optimisation. An overview of these can be found in Dekker (1996) and include linear and nonlinear programming, renewal theory, heuristic approaches and, particularly for CBM, various types of Markov decision processes.
Two main types of maintenance can be identified, namely preventive and corrective. Cor-rective maintenance generally occurs when a unit has failed, and is therefore generally more costly than preventive maintenance. An additional distinction can be made between perfect and imperfect maintenance, where perfect maintenance returns the unit to an as-good-as-new state, whereas imperfect maintenance brings the unit back to a state somewhere between as-good-as-new and as-bad-as-old. In most of the literature, both corrective and preventive maintenance are considered to be perfect, however an example of imperfect maintenance can be seen in Do et al., (2015), in which the amount of repair performed is stochastic.
Next, we focus on the literature concerning single unit systems with multiple inspection types. For CBM, there are many methods with which to obtain information about the condition of a system. A brief survey of such methods can be found in Tsang (1995). The existing methods can be grouped into two main categories for determining the deterioration of a unit, namely systems with continuous monitoring and systems which require inspections. For the the first, the degradation level of the unit is known at all times, and the cost of obtaining this information is constant for each unit of time, or can be obtained for free. However, continuous monitoring can often be infeasible due to technical limitations. In contrast, for systems with inspections, the deterioration information is unknown until a costly inspection is performed. A further distinction can be made between perfect and imperfect inspections. Perfect inspection results in an accurate measurement of the deterioration level, whereas for imperfect inspection there exists uncertainty about the observation. Furthermore, Tsang (1995) confirms that perfect inspections tend to be more costly than imperfect inspections. Often this also means that the time required to perform the perfect inspection is longer, making them unsuitable in certain cases. Hence the scheduling of inspections may become an important factor in the cost minimisation.
2 PROBLEM DESCRIPTION
On the other hand, Maillart (2006) develops two separate discrete-time models for aperiodic inspection and repair scheduling with CBM for a single unit. The first is for perfect inspection whereas the second uses partially observable Markov decision processes for imperfect inspection. No new information can be obtained about the state of the system unless it is inspected. Optimal policies as well as heuristic solution methods are provided for each. Wang et al.(2014) consider a model with three deterioration states and two inspection types. Whereas the first inspection is imperfect in determining certain deterioration states, the second is perfect for identifying all states. Furthermore, instead of dynamically scheduling inspections depending on the deteriora-tion, the inspection interval is simply halved when the unit is found in the deteriorated state. Zequeira and B´erenguer (2006) consider three types of inspections of which one is perfect, as well as three competing types of failures. An age-based maintenance policy is employed to pre-ventively maintain the system, and optimal intervals for periodic inspections and maintenance are determined. Lastly, Makis and Jiang (2003) consider a system in continuous time, with un-observable discrete deterioration states, except for failure. Inspections are imperfect and are set to occur at periodic time intervals. Thus only preventive maintenance is adaptively scheduled.
In this paper we consider CBM for a single unit with a continuous deterioration process. Inspections are required to obtain degradation information about the unit, and we assume that two types of inspections that differ in their quality can be carried out at different costs. Both in-spection types can be imperfect, and the special case of a system with both imperfect and perfect inspections will be considered. As opposed to previous papers with multiple imperfect inspection types, no additional deterioration information is obtained when the system is not inspected. Con-trary to previous research with multiple inspection types, we aim to determine aperiodic optimal policies for both inspection and preventive maintenance by modeling the problem as a partially observable Markov decision process. We approximate the continuous deterioration process by a discrete-time Markov chain (DTMC) using the discretisation approach developed by N. Kremers (2017) to enable the use of Markov decision processes (MDPs) to determine optimal policies rather than apply heuristics. Furthermore, the observations obtained by imperfect inspections are modeled continuously and will be discretised accordingly. This paper investigates how two distinct qualities of imperfect inspection can be combined and in what ways they can interact in optimal policies to minimise the long term average cost per time unit.
The paper is structured as follows. In Section 2, a problem description is given and subse-quently formulated as a mathematical model with continuous deterioration in continuous time. This model is then approximated by a discretised model in Section 3. The solution method used to solve the discretised problem is a partially observable Markov decision process (POMDP), which is constructed in Section 4 and a solution method is outlined in Section 5, followed by a numerical investigation in Section 6. Lastly, our conclusions are summarised in Section 7, and limitations of this study and recommendations for future research are discussed.
2
Problem Description
We consider condition-based maintenance optimisation for a single-unit system with a continuous non-decreasing deterioration process, where the deterioration level of the unit is not observable unless it is inspected. We consider an infinite time horizon and the aim is to minimise the long-run average maintenance and inspection cost per time unit.
3 DISCRETISATION
taking negligible time. The first inspection method is more precise than the second. As a result, the second inspection method is less costly, and we have c1 > c2. We assume that failures of
the system are self-announcing and that immediate corrective maintenance is executed when a failure occurs. We also assume that both preventive maintenance at cost cpm and corrective
maintenance at a cost ccm take negligible time and restore the deterioration level to X(t) = 0,
where the system is as-good-as-new. It follows that the probability that the system is in the failed state is zero at all times.
Subsequently, the continuous deterioration process allows the observed deterioration level from imperfect inspections to be modeled by a continuous probability distribution. Any suitable continuous distribution may be chosen. Going forward, we assume that observations of the deterioration level are normally distributed with mean equal to the true deterioration level, and standard deviation σi ≥ 0 which depends on inspection type i = 1, 2. It is evident that σi
is a measure of the relative imprecision of each inspection method i. Because we assume that inspection type 1 is more precise than type 2, we must have we must have σ1 < σ2. Then,
let the density function of the probability of observing a deterioration level of x when the true deterioration is X(t) be denoted by fi(x|µ = X(t)). This density function should be centered
at µ = X(t) given that the true deterioration level is indeed X(t). As this density function is dependent on the true deterioration level, we henceforth call it the conditional observation density. Lastly, note that a perfect inspection in this model is simply a special case of an imperfect inspection with observations originating from a degenerate distribution with σ = 0. Therefore σ = 0 indicates a perfect inspection.
Additionally, we assume that the inspection methods will never report observed deterioration levels that are deemed impossible. Recall that, as a result of self-announcing failures and imme-diate corrective maintenance, the probability of the system in a failed state is zero. It follows that observations indicating a failed state must be prohibited. Therefore, the probability that an inspection yields an observation of a failed state is also zero at any point in time. Similarly, the probability of observing negative deterioration levels should also be zero. This is achieved by truncating the normal distribution at zero to the left and at L to the right. The probability density function of the resulting truncated distribution is given by:
fi(x|µ = X(t)) = 1 σi φ x − X(t) σi Φ L − X(t) σi − Φ 0 − X(t) σi I[0,L](x), i = 1, 2, (1)
where φ denotes the density function of the standard normal distribution, Φ its cumulative distribution function and
I[0,L](x) =
1, if 0 ≤ x ≤ L,
0, otherwise, . (2)
Note that if the truncation is asymmetric around the mean, the mean of the truncated normal distribution is different from the original normal distribution. Therefore µ = X(t) is no longer the mean of the truncated normal distribution, and becomes merely the mode, that is, the value at which the probability density function attains its maximum. The derivation of a truncated normal distribution and its moments and properties can be found in Greene (2003).
3
Discretisation
3 DISCRETISATION
continuous deterioration process is discretised into a discrete-time Markov chain (DTMC) in Subsection 3.1. Following suit, in 3.2 the probability distribution for inspection observations is modified and then converted into discrete probabilities.
3.1
Deterioration process
In order to formulate the problem that we consider as a Markov decision process (MDP), the con-tinuous deterioration process X(t) must be discretised. This is done using the method described by Kremers (2017) for discretising stationary continuous-time continuous-state non-decreasing deterioration processes. We approximate the continuous deterioration process by a DTMC by di-viding the continuous time scale into steps of length ∆t. Let the cumulative distribution function of the additional amount of deterioration during such a time period with length ∆t be denoted by F∆t(x).
To discretise the deterioration levels up to L into m discrete states for the DTMC, we get intervals with length ∆x = mL, and the kth interval is denoted by xk = [(k − 1)∆x, k∆x],
k = 1, 2, ..., m. We let the failed state in the continuous model correspond to the failed state m + 1 in the DTMC. The approach used by Kremers (2017) assumes that when the deterioration is in state k in the DTMC, the continuous deterioration level X(t) is uniformly distributed over the interval xk, that is, {X(t)|X(t) ∈ xk} ∼ Unif(xk). Let qi, i = 0, 1, 2, ..., denote the
probability that the deterioration does not exceed the interval xk+i at time t + ∆t, given the
assumption above, for arbitrary t and k. Then we have:
qi= P (X(t + ∆t) ≤ (k + i)∆x|X(t) ∼ Unif(xk)) = P (X(t + ∆t) ≤ (1 + i)∆x|X(t) ∼ Unif(x1)) = 1 ∆x ∆x Z 0 F∆t((1 + i)∆x − x)dx. (3)
Hence for the DTMC, qi is the probability that the unit deteriorates at most i states within one
time period. It follows that the probability of the unit deteriorating exactly i states within one time period, denoted by pi, i = 0, 1, 2, ..., is given by:
p0= q0 (4)
pi= qi− qi−1, i = 1, 2, ... (5)
With the latter probabilities we may construct an (m + 1) × (m + 1) matrix P describing the transition probabilities of the DTMC. As listed in the lecture notes by de Jonge (2017b), this matrix has several properties. It is upper diagonal, such that all entries Pij, j < i are equal to
zero. Each entry Pij with 1 ≤ i ≤ m, i ≤ j ≤ m equals pj−i. The last column, containing the
probabilities to deteriorate into the failed state, has entries Pi,m+1= 1 −P m+1−i
k=1 pk equal to one
minus the sum of the other probabilities in the row, namely the remaining probability. Lastly, we have that the failed state is an absorbing state, and thus Pm+1,m+1= 1.
3.2
Imperfect inspections
Next, we require observation probabilities for each discrete state as defined in the DTMC. Recall that the discretisation of the deterioration process is constructed under the assumption that if X(t) ∈ xk, the deterioration level is uniformly distributed within xk. With this in mind, it seems
4 MARKOV DECISION PROCESS FORMULATION
to the midpoint of the interval xk when X(t) ∈ xk, instead of the exact deterioration X(t) itself,
as in the continuous model. To simplify notation, let
fi,k(x) = fi(x|µ = (k −12)∆x), i = 1, 2, k = 1, 2, ..., m, (6)
denote the shifted probability density function for inspection method i, given deterioration level k. Subsequently, to obtain discrete conditional observation probabilities, we calculate the prob-ability mass of fi,k that falls within each interval xj, j = 1, 2, ..., m, and set this to be the
probability that an observation is made indicating that the deterioration level is j. Let ri(j, k)
denote the probability of observing the unit in state j using inspection type i, given that the true state of the unit is k. We have
ri(j, k) = j∆x Z (j−1)∆x fi,k(x) dx = Fi,k(j∆x) − Fi,k((j − 1)∆x), (7)
where Fi,k denotes the cumulative observation distribution function corresponding to fi,k.
4
Markov decision process formulation
In this Section we will formulate the problem as a Markov decision process (MDP). We refer to Puterman (1994) for more information on Markov decision processes. However, we note that there exists uncertainty about the deterioration state of the unit, requiring a special case of an MDP, namely a partially observable Markov decision process (POMDP). As the problem involves an infinite horizon, we define the set of decision epochs as T = {1, 2, ..., ∞}, and aim to determine the inspection and maintenance policy that minimises the long-run mean cost per unit time. We describe and define the necessary components to formulate the POMDP.
4.1
Knowledge state space
To define a POMDP, we require a set of states. However, we do not know the true state of the unit without performing an inspection, and even after an inspection, uncertainty about the current state remains. Hence we must define a set of knowledge states, or belief states, based on the underlying deterioration states and transition probabilities. We let π = [π1, π2, ..., πm, πm+1]
denote a knowledge state in which πj ≥ 0 is the probability that the unit is currently in
deteri-oration state j. The set of all knowledge states is given by Ω =nπ :Pm+1
j=1 πj = 1, πj ≥ 0
o and the number of such knowledge states is infinite as the probabilities are continuous. Note that because immediate corrective maintenance is carried out when failure occurs, we have πm+1= 0
with certainty.
4.2
Possible actions
Secondly, we define the possible actions that can be taken each decision epoch. There are four possible actions, namely do nothing (DN ), preventive maintenance (P M ), a precise inspection (I1), and an imprecise inspection (I2). Recall that we have assumed that inspection and
4 MARKOV DECISION PROCESS FORMULATION
inspections in a decision epoch to one. After the inspection, it is possible to either perform preventive maintenance or to do nothing in the same decision epoch. For ease of notation, this second stage of decision after an inspection is considered to be a part of the actions I1 and I2
themselves, and the set of all possible first stage actions is denoted by A(π) = {DN, P M, I1, I2}.
4.3
Transition probabilities
Next, a method is required to to find the next knowledge state, π0, depending on the current knowledge state and chosen action.
4.3.1 Do nothing
When the action DN is chosen, the current knowledge state is updated according to the transition matrix P . This is done by multiplying the current knowledge state π with the transition matrix as follows:
˜
π0 = πP. (8)
However, these probabilities must be rescaled because a transition to the failed state is observed immediately and followed by corrective maintenance. The computation attributes a positive probability to the unit being in a failed state. The probabilities of the non-failed states π0j, j = 1, ..., m must therefore be divided by 1 − ˜πm+10 to obtain
π0j=
˜ πj
1 − ˜πm+10 , i = 1, ..., m, (9)
and π0m+1= 0. Then π0(π) = [π10, π02, ..., π0m, 0] indicates the knowledge state for the next time period after the action DN is chosen.
4.3.2 Preventive maintenance
Conversely, when preventive maintenance is carried out, the state of the unit is known with certainty immediately afterwards, as the unit is restored to state 1 where it is as-good-as-new. However, note that since the maintenance takes negligible time, the unit may continue to dete-riorate afterwards during the same time period.
4.3.3 Inspection
In the case that an inspection is performed, we must construct an update function from the finite set of possible deterioration levels that can be observed and the probabilities of encoun-tering them. From the modeling in Section 3.2, the set of possible observations is the set of all deterioration states except the failed state. First, we define the probability ui(j, π) of observing
deterioration level j, j = 1, 2, ..., m, when the knowledge state is π and inspection type i is used. Let this be denoted by
ui(j, π) = m
X
k=1
πkri(j, k). (10)
Note that these probabilities are no longer dependent on the true state of the unit, but only on the knowledge state π. Hence ui(j, π) gives the unconditional observation probabilities.
4 MARKOV DECISION PROCESS FORMULATION
deterioration level j using inspection method i given knowledge state π is constructed using the following probabilities (Maillart, 2006):
π00l(i, j, π) = mπlri(j, l) P k=1 πkri(j, k) = πlri(j, l) ui(j, π) for l = 1, 2, ..., m. (11)
We multiply the probability that the unit was indeed in state l before inspection, by the probability of observing j given this state. It is then divided by the overall probability of observing a deterioration level j from any state given knowledge state π. Hence, we obtain the probability of being in state l given that the inspection yields an observation of deteri-oration level j. Again, it must hold that πm+100 (i, j, π) = 0 for all j = 1, 2, ..., m, as we al-ways have πm+1 = 0. The updated knowledge state after and inspection is thus given by
π00(i, j, π) = [π001(i, j, π), π200(i, j, π), ..., π00m(i, j, π), 0].
4.4
Reduction to finite knowledge state space
A continuous, and therefore infinite, state space leads to difficulties in finding appropriate so-lution algorithms. One method to reduce the state space to a finite one is to consider the fixed series or paths of consecutive knowledge states that result from the fixed knowledge state transition probabilities π(π) when choosing the action DN repeatedly in succession from the as-good-as-new state. As a result, not every knowledge state in between needs to be considered. Furthermore, when inspections are perfect, the knowledge state resulting from inspection does not depend on the preceding knowledge state and will reset to one of the m − 1 fixed knowledge states where the state is known with certainty. Thus we only need consider the fixed paths of knowledge states when starting from these points and again choosing DN repeatedly. It is additionally possible to provide an upper bound on the number of time periods that the action DN is chosen consecutively between two inspections, due to which the state space is restricted to a finite set.
However, when working with imperfect inspections, this approach becomes infeasible. Firstly the set of knowledge states that can result after an inspection is much larger and remains de-pendent on the preceding knowledge state. Due to this, even if it is possible to provide an upper bound on the number of time periods between consecutive inspections, the paths of the knowl-edge states are not fixed as before. Therefore, even if it is possible to restrict Ω to solely the knowledge states that are actually visited with the method above, it may still become too large to compute solutions effectively.
We instead choose to approximate Ω by discretising it. In the fashion of Maillart (2006), we discretise the continuous state space in order to obtain a finite state space Ω0 by restricting the
probabilities πi, i = 1, 2, ..., m, which can take any value on the continuous interval [0, 1], to a
grid such that they may only take values from the discrete set {0, 1 M,
2
M, ..., 1}, where M > 0
is an integer denoting the number of partitions and is sufficiently large. Then the discretised knowledge state space is defined as follows:
Ω0= (" j1 1 M, j2 1 M, ..., (1 − m−1 X i=1 ji) 1 M, 0 # : ji∈ N0, i = 1, 2, ..., m − 1 and m−1 X i=1 ji≤ M ) . (12)
5 SOLUTION METHOD
Furthermore, it must be ensured that the approximation does not become infeasible as a prob-ability, so points with
m
P
i=1
ji > M are not considered. Lastly, in case of multiple points with
minimum distance, the closest approximating point on the grid is selected by giving priority to points minimising the values πi in order of descending i.
5
Solution method
In this Section we describe how we obtain optimal policies for the POMDP formulated in Section 4.
5.1
Costs-to-go
Let the probability of failure when the knowledge state is π and the chosen action is DN be denoted by R(π). We calculate R(π) by multiplying the probability of being in a state k by the probability of failure when in state k, as follows:
R(π) =
m
X
k=1
πkPk,m+1. (13)
Next, let the value function vn(π) denote the minimum total expected cost given that there are n decision epochs left. We define the costs-to-go as the total expected cost that will be incurred over all subsequent time periods when a specific action is chosen in the current decision epoch. Then the costs-to-go for the actions DN , P M , and inspections I1 and I2respectively are given
by DN (n, π) = R(π)(ccm+ vn−1(e1)) + (1 − R(π))vn−1(π0), (14) P M (n, π) = cpm+ DN (n, e1), (15) I1(n, π) = c1+ m X k=1 m X j=1 πk· r1(j, k) · min{DN (n, π00(j, π)), P M (n, π00(j, π))}, (16) I2(n, π) = c2+ m X k=1 m X j=1 πk· r2(j, k) · min{DN (n, π00(j, π)), P M (n, π00(j, π))}, (17)
where e1 denotes a vector of length m + 1 with 1 as its first element and all others zero. Note
that since P M depends on DN in the same period, and I1, I2depend on both DN and P M , we
must compute these costs-to-go in the order shown. Using the costs-to-go, we define the value function as the minimum cost resulting from the minimising action for every knowledge state:
vn(π) = min{DN (n, π), P M (n, π), I1(n, π), I2(n, π)}. (18)
5.1.1 Value iteration
The value iteration algorithm finds an ε-optimal policy by minimising the average cost per time period. It generates a sequence of values v0, v1, v2, ... ∈ V where V represents the space of
functions Ω0 → R from the finite knowledge state space to real numbers. Hence every value v ∈ V is such a function, and the value v0 denotes the final costs when no decision epochs are
6 NUMERICAL EXPERIMENTS
action, as shown in equations 14-18. The algorithm prescribes when to terminate depending on the parameter ε > 0, which should be chosen small, and the span of v, sp(v), which is defined as
sp(vn) = max
π∈Ω0v
n(π) − min π∈Ω0v
n(π). (19)
The span is the difference between the maximum and minimum value given all possible knowledge states, and when this difference becomes small enough the algorithm terminates. Hence when sp(v) is sufficiently small, we obtain a cost v0(π) that is independent of the final state, in other words approximately equal for all π. Lastly, note that for the stopping criterion to be fulfilled at finite n, we require that for any optimal stationary policy, the transition matrix is aperiodic. By the nature of our problem, this is the case. The value iteration algorithm proceeds as follows:
1. Specify ε > 0 and set n = 0. Select a starting value v0∈ V .
2. For each π ∈ Ω0, compute vn+1(π) as described above.
3. If sp(vn+1− vn) < ε, go to step 4. Otherwise increment n by 1 and return to step 2.
4. For each π ∈ Ω0, choose the ε-optimal decision that minimises the costs as follows
dε(π) ∈ arg min a∈A(π)
{DN (n, π), P M (n, π), I1(n, π), I2(n, π)}
and stop.
When the algorithm terminates, the tightest lower bound of the average cost per time period is given by
min
π∈Ω0{v
n+1(π) − vn(π)}, (20)
and the tightest upper bound by max
π∈Ω0{v
n+1(π) − vn(π)}. (21)
The optimal cost per time period g∗ is approximated by the average of the two:
g∗= min π∈Ω0{v n+1(π) − vn(π)} + max π∈Ω0{v n+1(π) − vn(π)} 2 . (22)
6
Numerical Experiments
In this section we will consider various instances of the model that we have formulated. We assume that the unit deteriorates according to a stationary gamma process, which is widely used to model the continuous deterioration level of single units, and according to Van Noortwijk (2009) is most suitable for application to monotonic and gradual deterioration. Therefore, in the remainder of our analysis we work with the stationary gamma process, although other deterio-ration processes with the required properties may be readily substituted. The density function of the gamma distribution is defined as follows, for x > 0:
fα,β(x) =
1 Γ(α)βαx
α−1
e−xβ (23)
with α > 0 and β > 0 the shape and scale parameters respectively, and Γ(α) =
Z ∞
0
6 NUMERICAL EXPERIMENTS
denoting the gamma function. The stationary gamma process has shape function α(t) = at with shape parameter a > 0, and a scale parameter b > 0 with the following properties (Van Noortwijk, 2009):
1. X(0) = 0 with probability 1,
2. X(τ ) − X(t) ∼ fa(τ −t),b for τ > t ≥ 0,
3. X(t) has independent increments,
4. X(t) is a jump process with infinitely many jumps in any time interval.
We select a gamma process with a = 15 and b = 0.3. The gamma process is discretised using time steps of ∆t = 0.05. We also set m = 3, L = 3. This yields a discrete-time Markov chain with the following transition probability matrix:
P = 0.7808 0.2136 0.0054 0.0002 0 0.7808 0.2136 0.0056 0 0 0.7808 0.2192 0 0 0 1 . (25)
We start by setting M = 50 as the grid size for the knowledge states. With this level of discretisation, we obtain a knowledge state space with size |ΩM =50| = 1326. The discretised
knowledge state space grows relatively fast in M and causes steep increases in computation time. In the remainder of the analysis, we choose a value of ε = 0.00001 as span threshold for the VIA algorithm, as this is the value at which decreasing ε further does change the optimal average cost per time period nor the optimal policy. Firstly, we explore the effect of changing the comparative costs of the two inspections, c1 and c2 while keeping the quality of the inspections
fixed. Secondly, we will observe the effect of the levels of inspection imprecision σi on the
minimum average cost per time unit and on the optimal policy itself, while keeping the costs fixed. Lastly, we wish to investigate the effect of changing M , which is a measure of how finely the knowledge state space is discretised, and m, the amount of deterioration levels before failure, on the quality of the resulting policy. Both of these variables are measures that determine how well the model approximates reality.
Note that for many combinations of σ1, σ2, c1and c2, only one of the two types of inspection
is chosen to be optimal at all knowledge states rather than a mix of both. As Maillart (2006) states, for an (imperfect) inspection to occur, the cost of inspection must be smaller than that for preventive maintenance, ci< cpm, and the signal-to-noise ratio must be sufficiently large. In
our case, this ratio is measured by the quantity L/m σi
.
6.1
Mixing perfect and imperfect inspections
We begin by considering the case where σ1 = 0, that is, when it is possible to carry out both
perfect and imperfect inspection at different costs. For interesting results, we first find parameter values for which both inspection types are interspersed in the optimal policy. This policy will be of the type with aperiodic inspections, and immediate preventive maintenance when required. We choose ccm = 120 and cpm = 30. By setting σ1 = 0 and σ2 = 1 we observe the problem
with a choice between an imperfect and a perfect inspection. The costs of inspection are c1= 5
and c2= 1 respectively. This yields an optimal average cost per time period of 6.2635 and the
optimal policy in figure 1. Note that if c1 is decreased further, imperfect inspection does not
6 NUMERICAL EXPERIMENTS 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM
Figure 1: Optimal policy for perfect and imperfect inspection with σ2 = 1 and costs c1 = 5,
c2= 1.
We observe when π1 is approaching 1, we have the knowledge state of near certainty that a
unit is good-as-new. Conversely, when π2is approaching values near 1, we are confident that the
unit is not yet in deterioration level 3. For values between these, the same holds and we are again certain that the unit is not in deterioration level 3. Hence, the optimal policy is to do nothing, and wait and see. When π1and π2take small values, it is probable that the unit is about to break
6 NUMERICAL EXPERIMENTS
discernible, it does not affect the minimum cost per time period or our analysis significantly. Next, we try to obtain more information about when imperfect inspection is optimal by increasing the cost of the perfect inspection, c1. In figure 2, the optimal policies are displayed
for c1 = 6, 7, 8, 9. As the cost for perfect inspection increases, we see that perfect inspections
are gradually replaced by imperfect inspections which are cheaper. Again, this replacement converges from relatively informative knowledge states towards the most uncertain knowledge states. That is, the policy region where imperfect inspection is optimal becomes wider and replaces part of the region of perfect inspections. In particular, the knowledge states having almost equal probability of the unit being in deterioration level 2 and 3 are last ones to have perfect inspection replaced by imperfect inspection, as is evident in sub-figure (c). This gives us valuable information the types of situations in which perfect inspections are the most appropriate and may warrant performing even at higher cost. Secondly, we notice that the optimal policy is quite sensitive even to relatively small changes in the relative inspection cost, that is even a small change in c1immediately translates to a change in the optimal policy. The optimal cost per
time period can be seen as a function of c1 in figure 3. Between c1= 7 and c1= 9, the changes
in optimal cost are barely discernible. Although the optimal policy still shows some states for which perfect inspection is optimal, these states are too few or not visited often enough to make a significant reduction in costs. We conclude that for our chosen cost parameters, when c1< 5,
perfect inspection is always preferable over an imperfect inspection with precision σ2 = 1 and
cost c2 = 1. On the other hand, when c1 ≥ 9, it has become too expensive and is no longer
optimal for any knowledge state.
6.2
Two imperfect inspections of different qualities
Now we proceed by gradually decreasing the precision of the first inspection type, by increasing σ1 from zero, such that it is no longer a perfect inspection. By doing this we obtain an idea
of the value of the precision of imperfect inspections without implicitly changing the costs of inspections c1 and c2 relative to the corrective and preventive maintenance costs, ccm and cpm
respectively. The cost parameters remain the same as in Subsection 6.1. First, we wish to determine the level of imprecision at which inspections are no longer desirable. We do this by setting σ1 = σ2 such that the first inspection type is never used in the optimal policy due to
c1> c2, and any inspection that occurs is done with cost c2. Then we increase σ2up to the value
where inspections are foregone altogether. Doing this, we find that around σ2= 1.75, inspections
no longer yield cost savings, as can be seen from figure 4. Increasing σ2beyond this point results
in the optimal policy slowly phasing out the last few points where inspections were optimal, but the optimal cost per time period no longer changes. Note that since inspection type 1 is more costly, the value of σ1at which inspections are no longer beneficial is even lower.
Considering this analysis, we will keep σ2 = 1, and increase σ1 from 0 to σ2 in steps of
0.05, until inspection type 1 no longer occurs in the optimal policy. The corresponding graph of optimal costs per time period can be found in figure 6. From this graph, we see that in the beginning, up to σ1 = 0.2, the increased effective cost of information is small and does
not affect the optimal cost per time period much. This is further reflected in the fact that the optimal policy does not change at all within these values, which can be seen from figure 5. It can be argued that truly perfect inspection is not necessary, provided that the precision remains sufficiently small. However, after this point, the optimal cost per time period increases rapidly until again, inspection type 2 becomes preferred over type 1 for almost every knowledge state and the costs per time period flatten off. By observing changes in the optimal policy, it is evident that a threshold is reached at σ1 = 0.2, where directly performing preventive maintenance as
6 NUMERICAL EXPERIMENTS 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (a) c1= 6, c2= 1 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (b) c1= 7, c2= 1 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (c) c1= 8, c2= 1 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 2 PM (d) c1= 9, c2= 1
Figure 2: Optimal policies for perfect and imperfect inspections with σ1= 0 (perfect inspection)
and σ2= 1 (imperfect inspection).
Using the cost graph we can draw a parallel with the earlier cost graph in figure 3 and its corresponding optimal policies. We note that in both cases, inspection type 1 is phased out gradually. An important difference however is that the knowledge state regions where inspection type 2 is inserted differs between the two. Comparing figures 2 and 5, it appears that when the cost c1 is increased, inspection type 2 becomes optimal uniformly from both directions of
the policy region. That is, inspection type 1 is replaced both on the side of the policy region bordering the region where P M is optimal as well as the one bordering the region where DN is optimal. On the other hand, the change when σ1 is increased is much more one-sided. P M
6 NUMERICAL EXPERIMENTS 6.30 6.35 6.40 6.45 6.50 5 6 7 8 9 10 c1 A v er
age Cost per Time Unit
Figure 3: Average cost per time unit as a function of the perfect inspection cost.
6.5 6.7 6.9 7.1 1.00 1.25 1.50 1.75 2.00 σ1 A v er
age Cost per Time Unit
6 NUMERICAL EXPERIMENTS 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (a) σ1= 0.15 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (b) σ1= 0.35 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (c) σ1 = 0.45 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 π1 π2 Optimal Action DN Inspection 1 Inspection 2 PM (d) σ1= 0.55
Figure 5: Optimal policies for two types imperfect inspections with σ2= 1 and various values of
σ1.
Even when increasing σ1further to the point that inspection type 2 begins replacing inspection
type 1, this starts occurring on the side bordering the P M policy region.
6.3
Degree of approximation to reality
7 CONCLUSION 6.30 6.35 6.40 6.45 6.50 0.00 0.25 0.50 0.75 1.00 σ1 A v er
age Cost per Time Unit
Figure 6: Average cost per time unit as a function of σ1, for values of σ1 ranging from perfect
inspection (0) to 1.
in subsection 6.1, with a perfect inspection type and an imperfect inspection type. We consider values of M from 40 to 70 in increments of 10. All other parameters remain equal. The results are mostly as expected, slowly decreasing as the knowledge state discretisation happens on more fine-mazed grids, although an unexpected slight increase in the optimal cost occurs for M = 70.
7
Conclusion
In this thesis, we present a continuous-time model with continuous deterioration of a single unitin for which two distinct inspection types are present at different costs. A special case of this model is one where both imperfect and perfect inspections can be performed. We proceed to discretise this model and present the problem as a partially observable Markov decision process. We then obtain optimal policies for selected instances of the problem. Although different inspection qualities have been combined previously in the literature, this thesis contributes to the literature by presenting a generalised model. Additionally, preceding papers assume that some form of continuous monitoring or other deterioration information is present even when the system is not inspected, which we do not. Lastly, this thesis is unique from the existing literature in that we are able to explore the effects of the extent of discretisation of the model.
7 CONCLUSION 6.20 6.25 40 50 60 70 M A v er
age Cost per Time Unit
Figure 7: Average cost per time unit as a function of M , for values of σ1ranging from 40 to 70
in increments of 10.
are available, part of the inspections of one type are replaced by the other type depending on the particular balance of parameters. Although decreasing the precision of inspection type 1 leads to an increase in the effective cost of information, a notable difference appears in the be-haviour of the optimal policy compared to when the cost of the inspection itself is increased. We conclude that the policy regions are complex functions of the precision and costs of inspection, and further investigation is required to deduce this correctly. As Maillart (2006) has already derived heuristic rules that define policy regions that partition the knowledge state space into sets where DN , P M or a single type of imperfect inspection is optimal, we do not reiterate these results. However, it might be interesting to provide similar heuristic rules that define when one inspection type replaces another. Due to time restrictions, further investigation into the specifics of such heuristic rules could not be carried out. This is good opportunity for future research.
Secondly, we investigate the changes in the optimal cost per time period when the model approximates reality better by increasing the M parameter, which makes the grid to which the knowledge states are discretised more fine-mazed. The results of this analysis are mostly as expected, with a slight increase in cost for M=70. However, computation times become prohibitive quickly for larger M (number of intervals for knowledge state space discretisation), but especially more so for m (number deterioration levels before failure). This prohibited us from running analyses on values of m larger than 3. Due to this, the model may not be applicable to larger instances of the problem, or attempts at less reductive discretisations. Nevertheless, it is a useful model for deducing properties and general behaviours of a system, which is what this thesis aims for. Similarly, due to restrictions in the maximum size of the problem and time restrictions, it is not possible to fully explore interaction effects between each of the variables.
9 REFERENCES
the numbers are low enough to not affect optimal costs significantly, these atypical and possibly incorrect ‘optimal’ actions are likely caused by the way the knowledge state space is discretised in combination with the choice of solution algorithm. For instance, Maillart (2006) discretises the knowledge state space in the same way as in this study, but applies policy iteration to find optimal policies without running into the aforementioned problem. This is because for policy iteration, the costs-to-go themselves need not be approximated to correspond to the grid points of the discretised state space. Another option is to find other ways to obtain a finite knowledge state space that do not involve rounding points in the knowledge state space to their nearest point on the grid.
Some assumptions made in this model may not apply to reality. For instance, it is quite difficult to obtain values for the standard deviations or other measures of precision of imperfect inspections, and these may not be measurable. Interesting avenues for future research include extending the model by adding dependencies or variation in these standard deviations, as well as adding imperfect maintenance, or more than two inspection types. Furthermore, it may also be interesting to explore the original continuous model using other solution methods rather than discretising it.
8
Acknowledgements
I would like to thank my supervisor, dr. Bram de Jonge, for his patience, valuable suggestions and advice.
9
References
De Jonge, B., Teunter, R. and Tinga, T. (2017). The influence of practical factors on the benefits of condition-based maintenance over time-based maintenance. Reliability Engineering & System Safety, 158, 21-30.
De Jonge, B. (2017b). Maintenance Planning and Optimization 2017-2018. Lecture Notes. Dekker, R. (1996). Applications of maintenance optimization models: a review and analysis.
Reliability Engineering & System Safety 133, 22-32.
Greene, William H. (2003). Econometric Analysis (5th ed.). Prentice Hall. ISBN 0-13-066189-9. Kremers, N. (2017). Optimal dynamic aperiodic inspection and maintenance scheduling for single-unit systems with continuous deterioration processes. MSc Thesis, University of Groningen.
Maillart, L. (2006). Maintenance policies for systems with condition monitoring and obvious failures. IIE Transactions, 38, 463-475.
Ohnishi, M., Kawai, H. and Mine, H. (1986). An optimal inspection and replacement policy under incomplete state information. European Journal of Operational Research, 27, 117-128. Puterman, M. L. and Feinberg, E. A. (1996). Markov Decision Processes: Discrete Stochastic
Dynamic Programming. SIAM review, 38(4), 689.
9 REFERENCES
Van Noortwijk, J. M. (2009). A survey of the application of gamma processes in maintenance. Reliability Engineering and System Safety, 94(1), 2-21.
Wang, H. (2002). A survey of maintenance policies of deteriorating systems. European Journal of Operational Research, 139(3), 469-489.
Wang, W., Zhao, F. and Peng, R. (2014). A preventive maintenance model with a two-level inspection policy based on a three-stage failure process. Reliability Engineering and System Safety, 121, 207-220.
White, C. (1977). A Markov quality control process subject to partial observation. Management Science, 23(8), 843-852.
