Experimental Design for Decision Support Systems for Shunting Planners

(1)

Closing the gap between theory and practice

Experimental Design for Decision Support Systems for Shunting Planners

Sijbrand Jakob Hofstra Groningen, 29^th of August 2005

Section Operations and Supply Chains

(2)

Closing the gap between theory and practice

Experimental Design for Decision Support Systems for Shunting Planners

Author: Sijbrand Jakob Hofstra University: University of Groningen Faculty: Management and Organization

Specialisation: MSc. Business Administration – Operations and Supply Chains Student number: 1380494

Date: Groningen, 29^th of August 2005 Supervisor: dr. J. Riezebos

Co-assessor: dr. W.M.C. van Wezel Principal: RuGRintel Research Group

Faculty of Management and Organization

University of Groningen

Section Operations and Supply Chains

(3)

Preface

“Until there are experiments, I take no notice of anything that’s done”

(William Stephenson, 1902-1989) The specialisation of the Master of Science in Business Administration study, Operations and Supply Chains aims to let students perform their thesis projects at companies. Although I did not choose for a research at a company, this thesis is to no extent a mere theoretical paper.

Some years ago the largest railway operator in the Netherlands approached several universities to participate in a research. The relationship that followed eventually gave me – and other students before me – the opportunity to test and to elaborate on scientific knowledge in a real world situation.

Some experts on the field of reporting state that it is inappropriate to thank those in a preface from whom you could have expected assistance in the first place. Nevertheless, I would like to thank both my supervisor and co-assessor – also in their role as principal – Jan Riezebos and Wout van Wezel for their enthusiasm, inspiration and sharp remarks throughout this thesis project and for letting me explore the more theoretical side of scientific research.

Sijbrand Hofstra

Groningen, August 2005

(4)

Abstract

The Dutch railway company ‘Nederlandse Spoorwegen’ (NS) has asked several universities to participate in a research on the field of planning support systems for so called shunting planners. In total, some 300 people are involved in daily planning activities of the NS, of which around 130 are assigned with shunting planning. These people plan, using mainly pen and paper, the allocation of trains on shunting yards, making sure that the trains are parked (in such a way that they can leave/arrive at a predetermined time at a predetermine place), that trains are recombined into new combinations and that related activities are carried out. It is remarkable that in time where the role of the computer in the daily life plays such a large role, these planning activities are mainly carried out using pen and paper

Both the Rijksuniversiteit Groningen (RuG) and the Erasmus Universiteit Rotterdam (EUR) have created prototypes of planning or decision support systems (DSS). The mainspring of this thesis was the need to determine which form of planning support is preferable for shunting planners, in order to increase performance for finding a feasible and efficient solution for a planning problem.

This graduation thesis for the study of Management and Organization addresses the issue of a lack of rigorous testing of systems that can support human planners during their job.

Although there is literature on differences between DSSs, many authors only present illustrative examples or tests are carried out with non-experts such as students.

Based on an analysis of the current situation of the shunting planners, combined with theory on human factors of planning, a model of essential elements in shunting planning is been created. Using both an Ishikawa diagram and a Slack polygon the essential elements are separated from the less important ones. The result is a group of elements which can be used to determine the difference in performance between the traditional working order, and the use of either prototype.

It is shown that the main difference between the two developed DSSs is that whereas the EUR solution functions as a black box and takes over a number of tasks from the planner, the RuG solution follows the path of a human-computer interactive system.

One of the most important conclusions is that the best way to give a decisive answer regarding the question on which system is the best in which situation, would be by conducting a field experiment. Unfortunately, the DSSs are not ready to be used in a production environment at this stage. It would also cost considerable resources to implement these systems just to carry out an experiment. The next best alternative would be to conduct a laboratory experiment where all the essential planning characteristics – where possible – are recreated. Important elements in this conclusion are that only expert planners should participate in the experiment and that the domain should reflect the planners’ regular domain.

The result is a 2x3x2 full factorial laboratory experiment. To guide the principals further towards the actual experiment, a basic framework for the experiments is presented and strategic choices concerning the experimental design are discussed.

(5)

1 – Introduction

As evening falls in the Netherlands and all the passengers are at home, the train stations are filled with trains that have been in service during the day. These trains will have to be cleaned and washed before the start of the following day. The order and combination in which the trains arrive at the stations during the evening is not necessarily the order and combination in which they have to start service the following day. Since space at the several strategically located shunting yards is scarce and the schedule of departing trains in the morning is tight, a tough job is presented to the people who plan and execute the shunting of trains.

The Dutch railway company ‘Nederlandse Spoorwegen’ (NS) has asked several universities to participate in a research on the field of planning support systems for so called shunting planners. Shunting is the process of parking trains, recombining them into new combinations and related activities. The goal of these support systems is to enable the shunting planners to spend more time on finding adequate solutions for complex situations by reducing the time needed for normal situations. In the current situation a lot of the planning is done by hand, even partially with pen and paper. Instead of complete automation, the focus is on supporting employees with making the right decision by means of Decision Supporting Systems (DSS). This focus can also be found in the name that is given to the research; Rintel, an acronym that stands for ‘Rangeer intelligent’ (Shunt intelligent).

The NS uses several levels of planning. From the head office in Utrecht, the local planning offices receive a general planning which is valid for a period of around 8 weeks.

This plan consists of arriving and departing trains per station (Barten et al, 2002). The local offices make changes in these plans when needed (for example if a track section is out of order because of maintenance or if an extra train is temporarily needed).

The Rintel research focuses on two levels of planning for shunting activities; ‘year planning local’ and ‘day planning local’. The other existing types of planning; ‘planning central’ and ‘Traffic control / Platform service control (PerronDienstLeiding; PDL)’ are not taken into account. In total some 300 people are involved with the daily planning activities of which 130~140 are assigned to shunting planning.

It is remarkable that in a time where a lot of processes are either automated or where humans use computers during their work, the majority of the planning activities involved at the NS are carried out manually. The local offices receive a generic planning electronically, but planners use paper and pen to draw revised versions. After these changes have been made, the paper planning is then entered in a computer system. Literature explains (Higgens 2001, Jackson and Browne 1989, Hoch and Schkade 1996) that it is not desirable to fully automate these types of planning processes, but it is feasible to strive for a form of digital decision or planning support.

Both the Rijksuniversiteit Groningen (RuG) and the Erasmus Universiteit Rotterdam (EUR) have created prototypes of planning support systems, a specific type of DSSs (from this point onwards DSS and planning support systems are seen as one). Currently two types of supporting systems have been developed:

1. Black box approach (EUR; EURRintel) 2. Electronic Planning board (RuG; RuGRintel)

Both systems are based on a form of task algorithmic, but whereas the RuGRintel approach focuses on interaction with the planner, the EURRintel is less of a DSS and functions as a fully automated black box. The EURRintel Black Box approach tries to automatically find a full-integrated solution. The RuGRintel Electronic Planning Board is a smart digital version of the traditional pen and paper work style.

(7)

1.1 Research question

The mainspring for this thesis project was the need to determine which planning support system/technique is preferable. The initial question of the Rintel research group was to set-up field annex laboratory experiments in order to decide which out of two techniques of planning support is preferable for the specific situation of train shunt planning. The actual underlying question was: which form of planning support for shunting planners of the ‘Nederlandse Spoorwegen’ is preferable? This underlying question is broader in the sense that it also takes into consideration that these systems are not multiple exclusive. The reason to use field annex laboratory experiments instead of simulation is threefold; firstly, one of the two systems is a pure decision support system, meaning that human input is necessary. Secondly, the current systems have already been tested using simulation during the construction phase. Finally, as Wiers (1997) puts it, the share of theoretical work dominates by far the total research. In other words: it is time to really test these systems.

There were some constraints that had to be respected during the research. For example, it was important that these experiments would be set-up in such a way that, with a minimal required effort from a select group of planners, a general conclusion could be drawn for the total population of shunting planners regarding the most preferable form of planning support.

A method is a way to produce an answer. If one asks with which method this answer can be found, you’re dealing with the field of methodology (Leeuw, de, 2001). The following question reflects this methodological point of view.

How can one determine which form of planning support – given the developed systems – is preferable?

The main focus of this thesis is to investigate this methodological question in order to set-up the actual experiments. The actual conduct of these experiments and the data processing were – given the available time frame of four months – not carried out as part of the thesis project. However extensive attention has been paid to these phases, since these are vital steps for further research.

The posed methodological question addresses the question of ‘which system is preferable’. In this thesis this question is looked at from a different angle, therefore it’s main research question focus more on the question of ‘how can one determine which system is preferable’. The main research question reflects the methodological character of the main issue of setting up the experiments:

How can one determine which form of the current developed planning support systems is preferable for shunting planners, in order to increase performance for finding a feasible and efficient solution for a planning problem.

This thesis will form a protocol for the RuGRintel research group for conducting experiments in order to research which form of planning support for shunting planners of the ‘Nederlandse Spoorwegen’ is preferable. This thesis forms a bridge between the theory based systems and the real life planning task, hence the title of this thesis: ‘Closing the gap between theory and practice – Experimental Design for Decision Support Systems for Shunting Planners’.

SUB QUESTIONS TO THE MAIN RESEARCH QUESTION

To give further structure to both the research and this report, the following sub questions have been formulated. The four sections also indicate a rough structure for the rest of this thesis, a structure which will be elaborated further on.

(8)

A. Current situation

1. What are the characteristics of the planning process?

2. What are the characteristics of the domain of the planners?

B. Role of the human planner

1. What is the role of the human planner in the planning process?

C. New developed systems

1. What are the characteristics of each system?

2. Which system is likely to be the best/worst suited for which kind of problem?

D. Experimental design

1. Which type of experimental design is best suited to research which form of developed planning support system is preferable?

Question D1 may appear premature at this stage, yet during preliminary research it became evident that a controlled (laboratory) experiment would be the most logical approach. For example Aytug et. al. (2005) point out that many researchers only present an illustrative example instead of actually experimenting with scheduling support systems. In other cases (see for example Kuo and Hwang, 1991) experiments are conducted, yet they are performed with non-experts (students) in a simplified setting. As will be shown later the difference between using experts and non-experts such as students is quite important. Aytug et al therefore stress the need for conducting ‘well-designed computational experiments and [to]

analyze their results in an appropriate manner’.

BUILD UP OF THE THESIS

In the next chapter, the current situation planning situation at the NS analysed. Attention will be paid to the characteristics of the planning process and the broader domain characteristics which influence the planning task. In chapter 3 the focus is on the human factors of planning in order to get a better understanding on why certain elements of planning are important. The results of chapter 2 and 3 are combined in chapter 4 into a model of those elements which influence the planning performance. In chapter 5 the developed Decision Support Systems are reviewed based on their distinctive characteristics. After these first five chapters, the results are combined to create an experimental design in chapter 6. On the next page these chapters are presented as a flow chart.

The reader is correct when he concludes that attention is paid to the human factors, but not to for example mathematical aspects of planning. The reason to include the human factors is that the term ‘decision support system’ indicates a form of cooperation between a human planner and a system. In order to get a good understanding on the role of the human in the planning process an entire chapter is dedicated to this topic.

(9)

Figure 1.1 Visual representation of the structure of the thesis

The model depicted above may suggest that the process which lead to this thesis was pure sequential. In fact from day one attention has been paid to diverse elements of the research.

As soon as the context of the research project became clear, attention was paid to the experimental design. This gave focus to essential elements of the total research while stressing the need to keep a clear eye on the bigger picture.

(10)

2 Current situation

In this chapter the current planning situation at the NS is depicted. The goal is to get a clear understanding of the essential characteristics of the planning of shunting activities at the NS.

Based upon the sub questions of the main research question, this chapter will deal with both the planning characteristics of the planning process itself and the characteristics of the domain of the planners.

2.1 Grasping the planning process

A very important step in this thesis is to get a good understanding of the planning process.

Therefore in this paragraph the characteristics of the planning process will be analysed. As already depicted in the introduction, the focus of this research is on both the local year and day planning of the NS. Barten et al (2002) have performed research on the planning of shunting planners of the NS. The characteristics they have found appear interesting for this thesis.

LEVEL OF PLANNING

The NS uses several levels of planning. From the head office in Utrecht, the local planning offices receive a general planning which is valid for a period of about 8 weeks. This plan consists of arriving and departing trains per station (Barten et al, 2002). The year planners get one week to analyse this concept. It’s the planners’ job to turn this concept into a feasible plan; to convert the list of trains with arrival and departure times into a full plan. The plan is always based on altering an existing plan. This information is then returned to the head office and later on the local office receives a new plan which may or may not include all the suggested changes by the year planners.

As the planned week approaches, it becomes the task of the day planners to make changes in these plans where needed (for example if a track section is out of order because of maintenance or if an extra train is temporarily needed).

COMMUNICATION AND AUTOMATION

The year planners receive the plan electronically. In the system the year planner gets a visual representation of the tracks and a list of train movements. Since this is all the system can do, he plots the graph and other information onto a piece of paper. This shadow version of the planning is then send to the day planner who makes a copy for specific weeks. After making the necessary changes, the plan is send back to the PDL (the management of the stations) (Barten, et al).

This also illustrates why research on DSS is useful; in this era of computer support it would be foolish not to make use of technology to improve processes.

GENERAL CHARACTERISTICS

Planning involves certain subtasks, depending on which author you follow you can get various sets of standard subtasks. To give an idea, two sets will be mentioned: getting the right information, communicating about the plan, negotiating with shareholder, making calculations and problem solving (Van Wezel and Jorna, 2005a). The second set is:

information gathering, negotiating, problem solving and evaluating (Sjarbaini and Jorna, 2005).

Crawford and Wiers (2001) show that in general, planners spend 80 to 90% of their time on finding the constraints which influence their planning task. Although Crawford and Wiers call the cited research recent, these findings date from halfway the 1980’s. It is possible

(11)

that these figures have shifted over time, perhaps even as a result of the introduction of computers to take over some work of planners. Later research (Van Wezel and Jorna, 2001) come up with more recent research. Although the variations are quite large, the works cited by Van Wezel and Jorna do show that finding constraints does take a large part of the planner’s time.

In some cases the planner cannot grasp the entire planning problem at once. To cope with this situation, the planner chops the tasks into smaller subtasks.

What also is important is that people generally do not look for a perfect solution, but for a satisficing solution. Their decision making capabilities are influenced to a strong degree by a so called bounded rationality. Simon (1982) originally described this as: ‘that property of an agent that behaves in a manner that is nearly optimal with respect to its goals as its resources will allow’. This is something to keep in mind when one tries to increase planning efficiency.

Barten et al point out that there are several generic characteristics which play a role for the shunting planners.

• Both day and year planners use an existing plan as the basis for their activities. The strategy is to respect that plan as much as possible. Since alterations in one section of a plan may lead to new conflicts at other points in the plan, it is desirable to keep changes to a bare minimum.

• With every alteration in the planning, the consequences for those people who have to carry out the actual shunting will have to be determined.

• As mentioned in the introduction chapter, the shunting planning activity is something which is mainly carried out by hand.

Furthermore there are some constraints that play a role on a more detailed level. To a large extent, the quality of a plan is determined by the degree of constraints respected. It should be noted that not every constraint is as hard as another.

- The length of a train should not exceed the length of the track it’s parked on.

- Trains should be positioned in the right order when they are coupled.

- Trains should not be enclosed by other trains.

- There should be no more than 1 train arriving or departing on a specific track within a 3 minute interval.

- Transit tracks (for trains passing by) should be kept free as much as possible.

- The combination of equipment should be legitimate.

- Trains moving to or from a non-secured area are allowed some delay.

- Internal cleansing should be carried out on a daily basis.

- External cleansing should be carried out on a daily basis; exceptions can be made as a final solution.

- Keep the walking time of drivers and shunters to a minimum.

- Make as less alterations as possible when it involves passenger trains.

- Changes in platform tracks are to be considered as the final option YEAR PLANNING

Based upon the general planning, the planners involved with the year planning have the task to convert the list of incoming and outgoing trains to a complete local plan. The planners have to check for the following conditions during the planning process:

• Check whether the length of the trains matches the capabilities of the train stations

• Maximize efficiency for plans during the day and night

(12)

It may appear a bit vague what these planners actually do. Since they base their plans on previous used plans their activities show a resemblance with the activities of the day planners;

altering existing plans to meet new constraints.

DAY PLANNING

The activities of the planners entrusted with the day planning are triggered by events. Typical events include:

• Changes in arrival time of trains

• Changes in departure time of trains

• Changes in arrival track/platform

• Changes in departure track/platform

• Changes in the combination of trains (arriving/departure)

• Unavailability of a track

• Extra arriving train

• Extra departing train

• Cancelled arriving train

• Cancelled departing train FREQUENT TASKS

During the research Barten et al came across a number of tasks which frequently occurred.

They give a random selection of these tasks and events:

• A train can no longer stay on a certain track as a result of the track becoming unavailable (find an alternative track).

• A (unit of a) train is defective. At later point in time a replacement will arrive (make sure that the defective unit is replaced by the replacement on the same track).

• The departure time is altered (analyse the consequences and resolve any disturbances in the plan caused by this alteration).

• Link a group of shunters to a shunting activity.

• Choose a route for a shunting activity.

• Evaluate the plan.

The authors point out that a complicating factor is formed by the fact that these tasks may trigger other tasks. If not dealt with carefully a snowball effect – events triggering new events which in turn trigger new events exponentially increasing the number of events – may arise, disrupting the entire plan. Barten et al have furthermore researched the order in which planners deal with these kinds of problems. A typical work order depicting how planners deal with tracks which are out of service can be found in appendix I.

According to the principals of this thesis, research from R.J. Jorna shows that although year planners and day planners carry out the same basic sub tasks differences exist in the speed and the outcome of the planning process. Year planners are slower, but since their plans are used for several weeks they try to find more robust solutions for events than day planners.

To capture the tasks in a more general framework, the following typology of typical planning activities related to domain and horizon responsibility can be used (Lastivkova et al, 2005).

The info in the table depicts the situation for local year and day planners at the NS (the horizon is automatically determined by the type of planner).

(13)

Activity Description

Forecasting Not applicable => Forecasting is mainly performed by the head office, although some form of forecasting is used when trying to resolve conflicts.

Order acceptation Not applicable => Only to a small extent the planners can suggest changes in the in- and outbound traffic

Material planning* This plays an important role (interwoven with capacity planning) In this context

Capacity planning* This plays an important role (interwoven with material planning) Work force planning The workforce is an important constraint during the planning

process, therefore the work force planning is important just as well

Aggregate planning The term aggregate planning refers to a higher level planning with less detail than ‘normal’ planning. At the NS this higher level planning is done by the head office.

There is however a difference between local year and day planners. In both plans there is the same detail in time and incoming and outgoing trains, but year planners do not care about incidental changes in the plan

Scheduling This plays a very important role Table 2.1 Typical activities for shunting planners

The list of activities is based on regular industries. Material planning and capacity planning at shunting yards do differ from regular industries. Material planning regularly deals with the flow of products or parts through a factory, whereas at shunting yards, material planning deals with the flow of trains (the product being a form of service). Capacity planning is more than just the work force (as a logical equivalent to capacity in regular industries); it also includes the availability of tracks.

2.2 Planning domain characteristics

The complexity of the planning process can only be understood by a thorough understanding of the context (domain) of the planning process. The analysis of the domain plays an important role in this research since it is likely to have consequences for the way the experiments should be setup. It is not only interesting to see how certain elements influence the planning process and the planning performance, but it will also give insights (later on) on how the different DSSs are likely to perform depending on the experimental design.

In contrast to for example planners in industrial environments, the shunting planners have some tougher constraints. Whereas in most situations the planner can to some extend delay the product in the process, shunting planners are bound by pre-determined time limits. Of course all planners are bound by some form of time limits, but shunting planners have limits set in minutes, not in hours or days. The planning of trains in the entire country can be influenced if trains get delayed at a single station. In other words: delay is not an option.

Furthermore, the planners have to deal with a large number of interacting decisions and they have to deal with the inherent unpredictability of the rail network. One of the ways planners deal with this is to split up a task into subtasks and to stick to the original plan as much as possible in order to prevent the snowball effect (see paragraph 2.1).

Another complicating domain factor is the shunting yards itself. In order for planners to handle adequately and quickly, it is required that they are aware of the capabilities and

(14)

limitations of every part of the shunting yard. Three basic elements that determine the capabilities and limitations are:

1. The type of track; the existence of LIFO tracks (Dutch: kopsporen), platform tracks (Dutch: perronsporen), shunting tracks (Dutch: opstelsporen), transit or permanently free tracks (doorrijsporen) (important because of nightly activity on the tracks), and secured and unsecured tracks.

2. Available resources per track; the availability of depot feeds (electricity), water feed, washing internally, washing externally, electrification of the tracks, diesel fuel station.

3. Availability of drivers and shunters to execute the plans.

Perhaps not a true domain characteristic but a direct related variable is the traffic density and nature of the traffic of the shunting yard (activity). Not only do these factors differ from shunting yard to shunting yard, but it also differs from day to day. During weekdays almost every train is used (although this varies from intensive use during rush hours to little use during the night) while in the weekend a large number of trains remains at the shunting yards.

This poses two challenges: during the weekend there is limited excess space for shunting activities, while during weekdays there is a lot of traffic which requires another form of planning strategy.

2.3 Conclusion

In this chapter attention was paid to the characteristics of the planning process and the planning domain. It was shown that year and day planners plan a different time frame, but that their tasks consist of the same basic activities. It was also shown that pen and paper still play a large role in the planning of shunting activities. In the first paragraph the planning tasks were captured in a general framework.

In the second paragraph attention was paid to the domain characteristics which play an important role during the planning process. It was shown that the complexity of a domain varies from shunting yard to shunting yard as well as from day to day and even from day to night.

Now that a clear understanding on the planning of shunting activities is achieved, a closer look can be taken on the human factors of planning in the next chapter.

(15)

3 Theory on human factors of planning

In the previous chapter it was shown that the planners and the tools and procedures they use play a very important role in the planning process. In this chapter the planning is studied from a human perspective. Since DSSs involve humans and their tasks, it is – in order to get a good understanding on DSSs – important to study the human factors of planning. This information enables to predict how the developed DSSs are likely to perform in a real life setting. This will assist us in mimicking the real life setting where needed during the actual experiments.

Crawford and Wiers (2001) have analysed the existing knowledge on human factors of planning and scheduling. Since their work provides a clear and comprehensive basis, the remainder of this chapter will be largely based on their work. Although the paragraph titles partially originate from Crawford and Wiers, the content is based on more than just their work, although it does play an important role.

This chapter will, combined with the previous chapter, contribute to the model of essential elements in shunting planning which will be presented in the next chapter.

3.1 Cognitive issues

A very important element of planning is the way schedulers deal with information. It appears that human planners make use of a so called mental model to grasp the planning domain. It is furthermore interesting to note that human schedulers look no more than three steps in the future when planning or an equivalent of 1.5 hours (Crawford and Wiers, 2001). Crawford and Wiers wonder whether the three-step limitation is a conscious human strategy and or whether the environment in which planners operate determine their success.

3.2 Decision making

The decision making process forms an important part of both the planning tasks and the cognitive issues altogether. Crawford and Wiers hand out various interesting remarks on decision making. In this paragraph the focus will be on those remarks which are most important for this thesis.

There is consensus on the fact that scheduling deals with making some sort of predictive decisions (Crawford and Wiers). The question arises whether the developed DSSs can handle this human strategy.

Crawford and Wiers state that schedulers have an executive routine that directs the general decision making process and that they especially use untaught and non-routine heuristics in exceptional conditions.

It should be mentioned that planners are individuals and that decision behaviour differs among planners because of individual differences. Besides the individual differences the decision making process also influences these individuals (Crawford and Wiers)

The authors also state that the decision making system is far from static. It should be noted that IT designers normally ignore this dynamic element.

Finally, the authors claim that performance is apparently not important to the planners while planning, ‘because the scheduler is not often able to match the feedback to the appropriate decision making behaviour’. For the planners of the NS it is extra difficult, since the planners are physically separated from the workforce which executes the plans.

3.3 Environmental factors

The environment is perhaps one of the most complex and elaborate elements of the domain.

Crawford and Wiers make the important remark that it is likely that a researcher will overlook some of the environmental factors because of the immense size of the domain. This is not to

(16)

be seen as a failure, but it should be taken into account nevertheless. The problem is that a researcher may not be able to trace every environmental factor which a planner takes into account while performing his task. An important characteristic of planning is how the planer interacts with other departments within the organisation. Crawford and Wiers point out that the quality of the plan is determined by the degree of authority and responsibility the planner had within the organisation.

3.4 Domain-free versus context

There are two schools in scheduling research: those who believe that the scheduling problem should be studied within its environment, and those who believe that the parameters influencing the planner are not organisation specific.

The domain-free school has two strands:

1. ‘Where researchers attempt to define the factors that affect scheduling performance as objective domain-free parameters, e.g. the number of jobs, the number of machines that will have to be scheduled by the scheduler. However, this approach is considered ineffective by the majority of researchers because such parameters only indirectly influence the scheduler’s performance.

2. Factors beyond the level of individual scheduling situations that influence the scheduler recognition, decision-making and action are subjective domain-free parameters e.g. time available to examine the system, familiarity of system state, and the number of plausible action alternatives. Certain researchers suggest that there is a need to understand these types of parameters to develop a generic understanding of human scheduling performance (Sanderson and Moray, 1990)

(Crawford and Wiers, 2001)

The context school states that scheduling can only be fully understood in context. The reason for this would be that a planner needs to have ‘a thorough understanding of the process and product that they are being asked to schedule. This is to allow the scheduler to make decisions based on an ability to recognise hidden relationships and identify possible alternative resource assignments’ (Crawford and Wiers).

As showed in the previous chapter, there are a lot of domain factors that have to be kept in mind during the planning process. The logical consequence of this characteristic is that planners at the NS need a thorough and intimate understanding of their domain. Therefore, if one has to choose between the domain-free or context school, in this case the context school would be preferable.

3.5 Instability and complexity of the environment

An unstable environment may lead to a situation where planners need to adjust their plans which were already in a final state. Instability may be caused by internal and external factors.

Internal factors are caused by problems and uncertainties in the production process. For the NS internal factors could be strikes, break down of material and deviations in the time it takes to execute a shunting activity. External factors are perhaps even more important for the NS. If there is a disruption somewhere in the system than it may cause delays and cancellation of trains elsewhere (at stations and shunting yards).

Please note that the term ‘disruption’ should not be merely asscociated with negative elements. Extra trains for special events such as concerts and Koninginnedag (Queens' day; a typically Dutch holiday) also lead to disruptions in the schedule. It should be noted that year planners and day planners only deal with disruptions which are known several days or weeks in advance. Disruptions on a shorter notice are dealt with by the PDL.

(17)

Instability may cause extra complexity for the planners. But the complexity is to a large extent determined by other domain characteristics. It should be noted that if complexity is increased beyond a certain point, the planner may lose track of the constraints he’s working under (Crawford and Wiers; also see the remark on the time planners need to determine constraints in the previous chapter). The authors present a solution in the form of an aid which can determine which constraint to relax in order to stick to the restraints as close as possible.

3.6 Temporal and production constraints

Planning is in essence bound by time restrictions. Crawford and Wiers point out the difference between expirable and non-expirable plans. For the NS applies that the majority of plans are expirable, except for the year planners who use existing plans and then adapt those to changes indicated by the head office to create long term plans. The plans generated by day planners are valid for a shorter period of time and are executed on a shorter term.

3.7 Information systems

Evidently, for planners information plays a very important role in fulfilling the planning task.

A lot of research has been performed on the field of ecological interface design. Interestingly enough it appears that experts, as opposed to novices, also take into account information from upstream stations in the organisation (more information on the differences between novices and experts will be presented later on). Crawford and Wiers furthermore refer to the works of McKay et al. McKay et al. show that to an important extent planners rely on enriched data;

data which includes historical, organisational, cultural, personal and environmental elements.

It may seem logical that it is hard to capture this enriched data into a model for an algorithm.

On the other hand computers may be able to supply (new forms of) enriched data to the planner.

3.8 Conclusion

In this chapter human factors of planning were discussed in order to get a firm understanding of the human elements in shunting planning. It has become clear that the human plays an important role in the planning process. Combined with the findings in chapter 2 the findings of this chapter form a solid basis for the experimental design. However, the collected data extends to a broad area and it may not be all of equal importance. Therefore in the next chapter the relevance of the gathered information will be discussed.

(18)

4 Essential elements in shunting planning

In the previous chapters essential characteristics of shunting planning were presented. In this chapter the information will be consolidated in two ways. First, using an Ishikawa diagram, the essential elements which influence the planning performance are depicted. Second, using Slack and Lewis’s performance criteria model variables which can be used to measure differences during the experiment are determined.

On one hand the Ishikawa diagram assists in determining the most important characteristics that play a role for planners (the focus is on the domain). On the other hand, Slack and Lewis’s performance indicators focus on the performance of the planning system itself.

4.1 Ishikawa diagram

Kaoru Ishikawa developed a so-called Ishikawa diagram to relate quality characteristics to cause factors (Ishikawa, 1985). This tool is also known as the cause-and-effect diagram and as (given its shape) the fishbone diagram. ‘Achieving quality characteristics is the effect and is also the goal of the system. The words appearing on the tips of the branches are causes’

(Ishikawa, 1985). The strength of this tool lies in the fact that it filters out a fast majority of cause factors, by selecting only the important ones, those that will sharply influence the main effect or goal.

In the previous two chapters a number of characteristics have been depicted which describe the planning activity of the shunting yard planners of the NS and general planning characteristics. In this paragraph we’ll use the Ishikawa diagram to create a model of those elements which are likely to influence one of the higher goals of research of the DSSs:

increasing planning performance. In particular, the Ishikawa diagram allows us to identify and control elements that could otherwise lead to potential confounding effects, which in turn could decrease the power of an experimental study to uncover causal relationships among variables.

Although this type of diagram is mainly used in Quality Control settings, there is no reason not to use this concept in related situations. The main difference is that where normally the elements on the branches depict distortions, in this diagram they represent (non-value laden) variables.

(19)

Figure 4.1 Ishikawa diagram

The diagram consists of the information of the previous chapters and that the structure should be a logical consequence of the presented information, nevertheless the branches will be briefly clarified. The selection of the main and sub causes was done by writing down a large variety of causes found in the studied literature on a piece of paper. Next these causes were ranked, separating the main causes from the sub causes and eliminating minor causes. This process could best be described as a brainstorming session.

AD TARGET / GOAL

Analogue to the main research question, the Ishikawa diagram focuses on differences in planning performance based on changes on other fields.

AD PERSONAL SKILLS

As was shown, the role of the human planner is very important. Elements like domain knowledge and finding and processing enriched data are to be considered as tacit knowledge.

Also the effect of bounded rationality should not be underestimated (people are not looking for the best solution, but for a satisfying solution). It seems that the strategies humans use differ from situation to situation, but the most important element is the element of creativity of human beings when trying to solve non-standard problems.

AD COMMUNICATION

Information forms the basis for a planner to conduct his or her planning task. This aspect therefore gets a prominent position in the Ishikawa diagram depicted above. In reality communication also has an overlap with ‘tools and procedures’ and ‘personal skills’.

AD DOMAIN COMPLEXITY (LINKED TO THE TACIT DOMAIN KNOWLEDGE OF THE PLANNERS) A lot of attention has been paid to the domain characteristics of the shunting planners. It was shown that it plays an essential role in executing the task of planning.

- Activity

- Interface with information systems - Layout

- Conditions (incl.

rules and laws)

- Priority of (sub)tasks

- Type of tasks

- DSS used (if any)

- Tacit knowledge - Bounded rationality - Processing capabilities - Strategies

Domain complexity Tools and procedures

Communication Personal skills

- Interaction with other parties

Target / goal Planning performance

(20)

AD TOOLS AND PROCEDURES

Apart from the domain specific characteristics, the tasks and subtasks of the planners play an important role.

Fifty percent of the branches are related to the complexity of the planning tasks (tools and procedures and domain complexity). This shows that complexity plays a very important role in the planning process.

4.2 Important performance parameters

If we want to check for changes in planning performance when using different forms of planning support (represented in figure 4.1 as ‘DSS used’), we’ll first have to determine how performance can be measured. A starting point would be to analyse the planning task and its outcome (the plan) with the use of Slack and Lewis’s (2002) five performance criteria:

Quality, Speed, Dependability, Flexibility and Cost. Below the performance criteria are operationalised and an indication is given how each element can be measured resulting in performance indicators. It should be noted that the choice of these performance indicators is arbitrary. Furthermore, the means of measuring are focussed on a non-longitudinal study type (as will presented in chapter 6, the final result will not be a longitudinal study). The goal is not to provide a full list of every possible performance indicator, but to identify the key performance indicators which should enable to objectively determine which planning DSS is to be preferred.

Please keep in mind that all these performance criteria refer to both the planning process and the generated plan. In between brackets it is shown if an element is planning or process oriented [process] or plan oriented [plan].

QUALITY

• The degree of respected constraints [process]

o Planners have certain constraints which they have to respect as much as possible during their work (see section 2.1).

¾ The quality can to a large extent be determined by looking at the number of not-respected constraints and to what extent these constraints were not respected.

• Robustness of solutions [process]

o Planners generally anticipate on events which are likely to occur in the near future. A robust plan assures the planners that a plan does not have to be designed from scratch (exaggerated) if a new distortion is introduced into the system.

¾ The best way to evaluate this performance indicator is by asking questions to the planner like: ‘if track X would be closed down, what would be the consequences of the choices you’ve just made?’. More objectively this can be measured by looking at the implications if a new standard distortion is introduced. In case of a DSS one can monitor how many changes a planner makes after a solution is offered.

• Robustness of solutions [plan]

o Planners generally anticipate on events which are likely to occur in the near future. A robust plan is not affected if a train arrives for example 2 minutes late.

¾ The best way to evaluate this performance indicator is by analyzing the time frame between activities on a track. If the time between activities is short, it may be an indication of a less robust plan.

(21)

SPEED

• Time needed to generate the plan [process]

o The time it takes a planner to find a solution for a problem.

¾ This can effectively be measured by taking the time from the moment the planner starts with a problem to the moment where he or she submits the generated plan.

• Time needed to execute the plan [plan]

o Apart from the time it takes to generate the plan (performance of the planning process) it is also, if not more interesting to look at the time it will take the shunters and the drivers to execute the plan.

¾ This aspect can be measured by analysing the time needed by the shunters and the drivers to execute the plan.

The reason for including both performance indicators is that if the time it takes to execute the plan decreases but the total time of planning and executing increases then the effective performance is lower. This is a trade-off which is made by the planners.

DEPENDABILITY

• Deviation from deadlines for schedules [process]

o The dependability of the scheduling process is determined by the percentage of plans delivered in time.

¾ This can be measure directly by looking at the percentage of plans delivered in time.

FLEXIBILITY

The element ‘flexibility’ is strange member of the family of performance criteria. The following items cannot be directly measured as with the other elements; their value can only be determined by looking at the other performance criteria.

• Volume flexibility [process]

o To what extent is the system able to cope with variations in the number of shunting activities? This characteristic may influence the overall performance of a system.

¾ The easiest way to test this would be by experimenting with both simple and quiet shunting yards and with complex and busy shunting yards or experimenting with one shunting yard on quiet and busier time intervals. The result can be measured by evaluating the opinions of the test subjects (the planners) and the changes in quality and speed.

• Domain complexity [process]

o Closely related to the previous point, domain complexity refers to variations in complexity. For example during weekdays a lot of material is on the move, resulting in more space to park trains. During weekends however, more material is already parked at the shunting yards making it more difficult to find a spot to park a train. Again, this characteristic may influence the overall performance of a system.

¾ The same tactics as with volume flexibility can be used in this case.

• Task complexity [process]

o Not every task is as difficult as another. It may be that different systems result in different performance outputs. It is therefore interesting to see to what extent the systems differ in performance on this point. Again, this characteristic may influence the overall performance of a system.

(22)

¾ This can be easily measured by introducing two or more levels of complexity in the experiment and then look for changes in quality and speed.

COST

• Productivity per unit of time [process] + [plan]

o If more can be done within the same timeframe, savings can be realized on the field of personnel.

¾ This item can be split up on two levels: the time (ergo costs) needed by planners and the time (ergo costs) needed by the drivers and the shunters. Since there is a learning curve, especially with systems that differ from the current task approach of planners, an experiment may not be the right moment to establish accurate readings on this point.

• Cost / Benefit ratio [process] + [plan]

o Introducing a DSS in the work flow will require money to implement the system and later on money to maintain the system. On the other hand, the system will hopefully contribute to a higher productivity per planner. It may also prove possible to increase the efficiency for shunters and drivers.

¾ This can be measured objectively, yet it remains unclear at this stage what it will cost to implement a particular DSS in a real life setting.

The presented performance objectives can be represented graphically (from now on referred to as the Slack polygon). This way the systems can be easily evaluated combining both subjective and objective data. Figure 4.2 depicts a simple example of how the data can be presented. The elements are valued on a scale of 0 to 5, where 0 is the lowest value and 5 the highest. Figure 4.2 has however one downside; it implies that the separate elements of each of the performance criteria should have a weight. This approach would mean a loss of data.

Alternatively the pentagon depicted below could be changed in a more generic polygon where each element is represented separately.

0 1 2 3 4 5

Quality

Speed

Dependability Flexibility

Cost

max Current DSS1 DSS2

Figure 4.2 Slack pentagon; fictive example

(23)

5 New developed systems

After having discussed the characteristics of the planning process, it is time to take a look at the DSSs which have been developed and which will eventually be tested. This chapter is build up in two phases; first the characteristics of each DSS will be presented after which we’ll discuss which system is (based on previous research) likely to be best suited for which specific situation. The differences can be used to create an experiment (chapter 6) in which the differences of the systems can be teased out effectively.

5.1 General DSS types

Basically three types of computerized solutions for scheduling problems exist; the first is based on Artificial Intelligence (AI), the second is simulation-based and the third is a human- computer interactive system (Kuo and Hwang, 1999). The differences between these approaches will now be further elaborated. It should be noted that this division in three approaches does differ from the division used by Van Wezel and Jorna (2005a). The first and third approaches are practically the same, but whereas Kuo and Hwang discuss the simulation approach, Van Wezel and Jorna discuss the expert or knowledge based system approach.

Firstly, the AI approach is based on human experiences (Kuo and Hwang) and domain analysis, but without analyzing the way human planners solve problems (Van Wezel and Jorna; 2005a). While this system is not constricted by the constraints of the human planner (Van Wezel and Jorna) and it is able to solve general scheduling problems, the AI approach is restricted in problem size (Kuo and Hwang). Another weakness of this approach is the fact that in general the modelling of the domain is done just once, whereas in real life the domain is dynamic (Van Wezel and Jorna).

Secondly, the simulation-based approach is rather straightforward; by simulating the implementation of scheduling rules the effects can be determined. This type of system can function as a DSS. The disadvantage of this system is that it may take too much time to simulate a case (Kuo and Hwang).

Thirdly, the human-computer interactive system (or hybrid system) does not try to incorporate the human decision process (as was tried with the AI approach), instead as a true DSS the system supports the human planner. This system does not take into account the cognitive characteristics of human problem solving; instead the analysis which forms the basis of the system is limited to the domain and task characteristics (Van Wezel and Jorna, 2005a).

Now that there is a basic framework to position the two different developed DSSs, it is time to look at the unique characteristics of both and the underlying paradigms.

5.2 EURRintel

The EURRintel project is based on an Artificial Intelligence approach (AI) and it focuses on a so called black box approach. Instead of following the line of breaking up the main planning problem into smaller, more comprehensible, sub-tasks like humans would do, the black box tries to find a direct answer to a planning problem. The black box fits in with the AI approach (Van Wezel and Jorna, 2005b).

For the EURRintel approach it is not critical that a solution for a problem is found in seconds. If the system takes 10 minutes but then comes up with a suitable complete solution which doesn’t require the human interaction to fine tune the last details, then that’s just as fine. The planners still keep the opportunity to make changes as they find appropriate. The

(24)

EURRintel approach works by abstracting the planning problem, potentially resulting in an incomplete solution.

It should be noted that the EURRintel system is a mere algorithm which still requires a graphical user interface (GUI) to be used by planners. The RuGRintel research group is currently active to fit this algorithm into the already developed GUI for the RuGRintel system.

5.3 RuGRintel

The RuGRintel system is an Operations Research (OR) approach of solving problems.

RuGRintel is a DSS which fits in with the human-computer interactive system; ‘human and computer should do the tasks they are best at’ (Van Wezel and Jorna, 2005b). The main difference is that RuGRintel does not aim to create a complete solution in one run. Instead it offers tools to help planners to carry out subtasks; the RuGRintel system assists on problem solving on a lower aggregate level. The system can suggest alternatives to a planner, but the planner keeps full control of the situation.

It should be noted that the boundaries of RuGRintel are not 100% clear as new functionality is still being added to the program.

5.4 Discussion

Now that the basics are clear one important question arises: which system is the best, or more subtle; which system is likely the best/worst suited for which kind of problem? The term

‘suited’ can be operationalised by looking at two separate questions; ‘to what extent will each system likely contribute to an increased planning performance’ and ‘to what extent are planners likely to accept the combination of problem, DSS and the solution found in this way’. In this paragraph an attempt is made to clarify to which extent the developed systems differ. This information will be used in the experimental design as it points out areas which can help to tease out the difference in performance of RuGRintel and EURRintel.

According to Sanderson (1989) the necessity of hybrid systems is threefold. First of all humans are necessary for monitoring the process and for troubleshooting should the system fail. Second, without the human factor the risk exists that operators become too remote from the process. This may lead to a loss of an up-to-date mental model of the system and on the long term it may lead to a loss of a mental model of system functioning and structure altogether. Third, hybrid systems outperform the other two systems as it combines the positive elements of both. It would go too far to claim that Sanderson states that a system like RuGRintel would be better than the EURRintel approach. It is likely that the domain complexity is too high to fully automate the planning function using the black box approach.

Although one never knows what the future shall bring.

Jackson and Browne (1989) argue that although interactive scheduling systems will seldom give optimal solutions, they can model a given manufacturing system more closely than a schedule produced automatically, since the user – with his strategies, experience, knowledge and intuition – has played an explicit part in its development. On the other hand, the black box approach may lead to a more optimal solution. However since its domain model is static and the black box model doesn’t have the ability to predict the future as humans do the degree of optimization is eroded.

The researchers from the RuGRintel research group expect that – based on literature research – their DSS will prevail over the EURRintel black box approach in complex situations. In straightforward situations they expect that the black box approach will perform better. Moreover, Van Wezel and Jorna expect that the increase in the quality of the plan will be small in the case of EURRintel as it does not support individual subtasks of the planners.

(25)

Their reasoning is as follows: ‘If we analyze the planner’s task without looking analytically at the elements in the planning problem, we will surely inherit planning habits that are based on the planner’s limited information processing capacity’ (p.8). On the other hand the authors seem to base their opinion on theoretical literature studies, not on real case studies. Crawford and Wiers (2001) point out in their literature review that Sanderson claims something similar:

‘Humans and algorithms [appear] to have complementary strengths that could be effectively combined’.

Another problem dooms up;Bainbridge (1983) states that ‘persons are disinclined to trust decisions that vary from their own, where they cannot understand the methods and criteria used’. If this holds true, then both types of DSSs may be affected. The barriers for the shunting planners may be lower with RuGRintel than with EURRintel as planners remain in full control with the first, while offering some autonomy with the latter.

Rasmussen (1999) stresses the need for transparency of the strategy of black boxes for pilots; ‘The issue is not, as often mentioned, to “open the black box” of automation to help the pilot to understand its function, but to help him understand the reasons for its actions – which are not found inside the box.’ The effect will probably be stronger when using the EURRintel approach, since it has less interaction with the planner. This effect can be eliminated by introducing a sort of progress window where users can see what the system is doing and why it’s doing what it’s doing. This is however something to keep in mind when evaluating the experiment.

Interesting is whether a black box approach is viable altogether in this particular situation. Fransoo and Wiers (2004) point out, that planners in general do not create a plan and than leave it for what it is, instead planners make substantial changes in plans over time.

The EURRintel approach on the other hand however does aim at a one time only run, except if the input for the plan or other parameters changes.

Of course it is not only interesting to look at the differences between EURRintel and RuGRintel, but it is also interesting to see how they perform compared to the traditional way of planning. The magic word is performance as was already argued in the previous chapter. In the next chapter the previous research will be consolidated into an experimental design.

(26)

6 Experimental design

“Unfortunately there are no good statistical adjustments for poor design”

Paul Spector Being the most important chapter of this thesis, this chapter combines the data accumulated in the previous parts in order to create the design of the actual experiment. A good starting point is to look again at the main research question: How can one determine which form of the current developed planning support systems is preferable for shunting planners, in order to increase performance for finding a feasible and efficient solution for a planning problem.

The main objective of the experiment is in line with the central research question: to tease out the differences in performance between the developed DSSs and the traditional planning approach. The build up of this chapter is as follows; in paragraph 6.1 attention will be paid to which type of experiment will be suitable in the end. In the following paragraph a closer look is taken at the scope of the experiment. Since there is not a single perfect way to conduct experiments the strategic choices of the design will be discussed in paragraph 6.3.

Finally in paragraph 6.4 these strategic choices are used to create the actual experimental design.

6.1 Experimental type

The goal of the research is straightforward; to test two developed DSSs in a realistic setting (in other words: reflecting a real life situation) to determine which type yields the best performance. The need for a realistic setting became clear in the previous chapters where it was shown that the role of the human planner and the complexity and dynamics of the planning domain play an important role. Now that the goal is clear, the way that this goal can be achieved can be determined. Simulation would not be suitable since there is no known system that can realistically mimic the behaviour of human planners. The only form of research that is available to conduct the research is the experiment.

The term experiment implies some form of putting a (in most cases new or renewed) concept to the test. Since one of the conditions of the goal is a realistic setting, only two basic types of experimental research remain: field experiments and laboratory experiments.

Field experiments are conducted by altering one or more parameters in real life and studying the effects. More specific for this case: offer the planners (one of the) new developed systems to use in their daily routine. The big advantage is that the domain characteristics – which were found to be very important – remain fully intact. Also the enriched data, as was pointed out in paragraph 3.7, is not compromised. The problem on the other hand is that in the current condition, the DSSs are not suitable to be implemented in a production environment.

Laboratory experiments on the other hand, do not require the full integration of the DSSs in the daily work environment. Some authors are pessimistic on the use of laboratory experiments: “The laboratory approach [of conducting experiments] has been used for sequencing research in a very limited problem definition that is sterile and devoid of context and uncertainty. If the goal is more effective planning and scheduling in the workplace, either the laboratory must reflect the situation or the research must be taken into the field. […] It is doubtful that a laboratory setting could capture the problem solving process adequately”

(MacCarthy et al. 2001). First of all, the scepticism of MacCarthy et al is probably due to the fact that, in the past years, they have spent quite some time on conducting case studies. Of course they are right when they claim that ‘the laboratory must reflect the situation’. They

Experimental Design for Decision Support Systems for Shunting Planners

Closing the gap between theory and practice

Experimental Design for Decision Support Systems for Shunting Planners

Closing the gap between theory and practice

Experimental Design for Decision Support Systems for Shunting Planners

Preface

Abstract

Table of contents

1 – Introduction

2 Current situation

3 Theory on human factors of planning

4 Essential elements in shunting planning

4.1 Ishikawa diagram

4.2 Important performance parameters

5 New developed systems

6 Experimental design

6.1 Experimental type