Maximizing the simulation output: A competition

Tilburg University

Maximizing the simulation output

Kleijnen, J.P.C.; Pala, O.

Published in:

Simulation: Technical journal of the Society for Computer Simulation

Publication date:

1999

Document Version

Publisher's PDF, also known as Version of record Link to publication in Tilburg University Research Portal

Citation for published version (APA):

Kleijnen, J. P. C., & Pala, O. (1999). Maximizing the simulation output: A competition. Simulation: Technical journal of the Society for Computer Simulation, 73(3), 168-173.

http://sim.sagepub.com/

SIMULATION

http://sim.sagepub.com/content/73/3/168

The online version of this article can be found at:

DOI: 10.1177/003754979907300304

1999 73: 168

SIMULATION

Jack P. C. Kleijnen and Özge Pala

Maximizing the Simulation Output: A Competition

Published by:

http://www.sagepublications.com

On behalf of:

Society for Modeling and Simulation International (SCS)

can be found at:

SIMULATION

Additional services and information for

168

TECHNICAL ARTICLE

Maximizing

the

Simulation

_Output:

A

_Competition

Jack

P.C.

_Kleijnen

and

_Özge

Pala

Department

of Information

_{Systems (BIK)/Center}

for Economic Research

_(CentER)

School of

_Management

and Economics

_(FEW),

_Tilburg

_{University (KUB)}

Postbox _90153, 5000 LE

_Tilburg,

The Netherlands

E-mail:

_{kleijnen@kub.nl;}

O.Pala@kub.nl

The following

competition

was

_{organized by}

the Business Section

_of

the Netherlands

_{Society for}

Statistics and

_Operations

Research _(VVS): maxi-mize the

_output

_{of a given}

simulation model

_by

selecting

the best combination

_of

six

_inputs;

_only

32 runs are

_permitted.

Twelve teams

_competed;

these teams

_{came from industry}

and academia. This paper is written

_by

the

_winning

team,

ex-plaining

its

_design

and

_analysis.

That

_design

pro-ceeded in

_stages.

First,

a

_{special design}

was used

to estimate all main

_effects

and

_two-factor

inter-actions

_(namely,

_{Rechtschaffner’s}

saturated

de-sign).

Then

_{quadratic effects}

were estimated

_by

changing factors

one at a time.

_Finally,

the

re-sulting

estimated second-order

_polynomial

was

used to

_estimate

the

_{optimal input}

combination.

The paper

presents a combination

of design

of

ex-periment techniques

and

common

sense that may

have

more

_{applications in solving}

real

1. Introduction: The

_{Competition Explained}

The

_{following problem}

was defined in the VVS

Bulle-tin

_(November

_{1997, pages}

_150-151;

December _1997,

pages

162-163).

The translation from the

original

Dutch text into

_English

is ours.

&dquo;Optimize

your own

_output!

You have

devel-oped

an advanced

_computer

model that

com-putes

the

_output

of the

_synthesis

of zeolite on

gauze

pads,

for

given

values of the

following

Rules _{of the game:}

1.

_[Given

is the

_following

_table.]

2. We

_[the

_organizers

of the

_competition]

will

e-mail _you a similar

list,

_including

the

corre-sponding

output.

Note:

Of course, the table above is

_only

an

_example,

in which

_only

the factors

_{A, B,}

and C were

varied. You are

_permitted

to _vary more factors or fewer factors as

_long

as _{you indicate for}

each of the six _{factors how you wish} to set its value. In the

_{example eight}

runs were offered.

So 24 runs remain for new

_experiments.

You

_yourself

_{determine how you will}

_spread

the 32 runs over the

_experiments,

_e.g., one

ex-periment

with 32 runs, two

_experiments

with

16 runs, one

_experiment

of 16 runs and two of

eight

runs, etc.

...You can

_register

no later than 5

_January

1998 ...&dquo;

At the start of our

search,

this was all we knew

about the

_problem!

In other

words,

we had no

informa-tion on the process

itself,

the ranges of its

_inputs

or

factors,

say,

zj

with j

= 1, ..., 6, _etc. We did know one

input

combination and its

_resulting

_output;

we call

this latter run

the free

base run.

(The

initial estimates

will turn out to _{be poor, which} is a realistic

_situation.)

We

_organize

this

_report

on our search as follows:

Section 2. Solution

_Strategy

Selected

Section 3. Rechtschaffner’s Saturated R-5

_Design

Section 4.

_Quadratic

Effects: One-at-a-time

De-sign

Section 5.

_{Re-estimating}

the

_Optimal

Combina-tion

Section 6. Conclusions

Section 7.

_Epilogue

Section 8. Final Comments on the

Competition

Appendix.

All 33 Runs with

_Inputs

and

_Outputs

2. Solution

_Strategy

Selected

Any

simulation model

_implies

an

_input/output

_(I/O)

function or _{response surface. Since the simulation}

model of this

_competition

_represents

a chemical

sys-tem, we assume that interactions _{among the} six

fac-tors are

_important.

_Moreover,

it concerns a maximiza-tion

_problem,

so we assume that

_quadratic

effects are

important.

Therefore we

_approximate

the _I/O

func-tion

_by

a

_{second-degree polynomial}

over the whole area of

_{experimentation.}

This

_polynomial

has 28

param-eters : one overall mean or

_intercept,

_say,

_/30’

six main or first-order effects

_{(3j ,}

15 two-factor interactions

_{3j;j’

(j ’ > j; j ’

= 2,...,

6),

and six

_quadratic

effects

_{/3j; j’}

’

Which

_{experimental design}

should we select to

esti-mate these

_parameters?

We have a

tight

_{&dquo;computer}

budget&dquo; allowing only

32 runs. To estimate all

effects,

we need 27 more runs

besides the free base run. Since we do not wish to

spend

most of our

_computer

_budget

in one

shot,

we

proceed

stage-wise:

computer

runs are executed one

by

one. We further focus on _{interactions,} before qua-dratic effects

_(also

see Section

_8).

Once we have also

estimated the

_quadratic

effects,

we take the six

_partial

derivatives

_{8y / 8z, ,}

_equate

them to zero, and estimate

the

_optimum

factor combination.

3. Rechtschaffner’s Saturated Resolution-5

Design

Our

_strategy

_implies

that we first estimate the overall

mean, the six main

effects,

and the fifteen two-factor

interactions

_(in

total,

22

_effects).

Because of the

_tight

computer

budget,

we select a saturated

_design,

that is, a

design

with a number of _{runs, say,} n

_equal

to the

num-ber of

effects,

q. There are several

_types

of saturated

designs,

satisfying

different criteria.

_By

definition,

resolution-5

_(R-5)

_designs

_give

unbiased estimators of the overall mean, all main

effects,

and all two-factor

interactions. We select a saturated R-5

_design

that is

readily

available,

namely

the

_design

derived in Recht-schaffner

_[1]

and

_replicated

in

_Kleijnen

_[2,

_pp

_310-311]

_]

(see

Table

_1).

This table

_gives

the standardized factor _{values, say,}

x: - stands for -1, and + for 1;

further, -

means that the factor has its lowest

value,

and + means that the

factor has its

_highest

value in the

_experiment.

We let

+

_correspond

to a 10% increase of the factor relative to

the base

value;

for

_example,

factor A or _{z, has} a base

value of 150

_(see

Section

_1),

so its +

_equals

165. Stan-dardization

_implies

that effects can be

directly

com-pared-without thinking

about their different units

(factor

A is in _mM, factor C in

_{C°) :}

it reveals the most

important

factors. In the next

_{stage, however,}

we shall use the

_original

scales. Also see

_Kleijnen

_[3].

3.1 Main

_{Effects Only:}

First

_Eight

Runs

Table 1. Rechtschaffner’s saturated R-5

_design

_[1], in standardized values (- is -1; + is ₁₎

This estimation

_requires

at least seven runs. Run #1 is

the free run. Now we execute runs #2

_through

#8 in

Table 1

_(actually,

runs #2

_through

#7 would have

suf-ficed,

but we were misled

_by

the fact that a 2k-P

_design

would have

_{required eight}

_runs).

The

_resulting

esti-mators _may be biased

_by

two-factor interactions and

quadratic

effects. Hence it is

_dangerous

to declare a

factor

_unimportant

when its estimated main effect is

not

_significant!

To estimate the effects

_(3,

we use

ordinary

least squares

(OLS),

giving,

say,

(3.

The

resulting

first-order

polyno-mial

_gives

excellent fit:

_R-square

is

_0.99999,

and

R-square

adjusted

for the number of effects is 0.99996.

We use

_SPSS,

which assumes

_{normally identically}

and

independently

distributed

_(NIID)

_fitting

errors with

constant variance

_(estimated

to be

_0.015162).

_Further,

SPSS

_applies

Student’s t statistic to estimate 95%

con-fidence

intervals;

their low and upper limits are

dis-played

in the last two columns of Table 2

_(all

effects

have

_roughly

the same standard error,

namely

0.007;

see column

_3).

All main effects are

significant

(last

two

_columns).

Actually,

we use

_only

the

_magnitudes

of the OLS

point

estimates

_{(column 2)}

to sort the factors. This shows

that factor B is the most

_important

_factor;

factor D the least

_important;

factor F the

_{only &dquo;negative&dquo;}

factor.

However,

these are

_only

tentative

conclusions,

be-cause main effect estimators _may be biased

_by

higher-order effects and statistical

_significance

_testing

as-sumes NIID. Our conclusion after the first

_stage

is

that there is not

_enough

information to eliminate a

factor or to _{make any}

_changes

in the factor levels.

3.2 Two-Factor Interactions:

_Remaining

Runs

Next we execute the

_remaining

runs #9

_through

#22

in Table 1. The

_outputs

turn out to _{vary between} 90.369 and 99.204

_(base

_output

_90.900);

see the

_Appendix.

Since the

_design

is

saturated,

_R-square

is 1.0. The factor estimates

_change:

_{(a) (30 =}

_94.3616;

_(b)

_{the (3~ ’s}

become

0.79105,1.79775, 0.5415, 0.41918, 0.61275,

and

_-0.66778;

(c)

_the

_{~3j; ~~’s}

equal

0.00225,

_except

for

_~1;6

=

0.0014,

and

_P4;6

= 0.002225. These estimates

suggest

that all

two-factor interactions are

_unimportant

_(see

Section

_7).

4.

_Quadratic

Effects: One-at-a-Time

_Design

Next we estimate the

_quadratic

effects,

_{by changing}

one factor at a time. Each factor should have at least

three values: we

change

zj

to, say,

_cj

with

_{cj ~}

-1 and

Cj 7=

1.

Moreover,

we execute runs one

_by

one

(chang-ing

the level of

_only

one

factor).

After each run, we

re-estimate the main

effect,

interactions, and

_quadratic

effect of that one

_input.

If the

_resulting

estimated

opti-mum value of that

_input

lies far outside the current

range, we are

_searching

in _{the wrong} area!

The first factor we

_change

is the

_seemingly

most

im-portant

factor,

B

_(see

Section

_3.2).

Furthermore,

we

change

this factor in the combination that

_yielded

the

highest

output

so far

_(run

_#7; see the

_Appendix).

Since

B’s estimated main effect is

_positive,

we increase B’s

value. We do so

_by

another _10%, which

_gives

_z2 = 484

(or

x2 = 3.2: _a 10%

change

in _z is _{not a} 10%

change

in

x).

This increases the

_output

to 102.79.

After

_adding

this run to the

_previous

22 runs, we

estimate the second-order

_{polynomial. Taking}

its par-tial derivatives

_{ay/azj ,}

_equating

them to zero, and

solving

gives

the estimated

_{optimum input}

values.

(The

values for the other factors besides B do not make sense: their

_quadratic

effects are not

_yet

_estimated.)

The

_{&dquo;optimal&dquo;}

B value turns out to be far _{away: x2} =

15.4843 or ₂₂ = 729.68599.

Next we also increase the factors A and C

_through

G

_by

20% in the

_original

scales;

we decrease F

_by

that same

_percentage:

runs #24

_through

#28. We

re-esti-mate the

_polynomial;

the overall mean and main

effects remain close to those in Section 3.2; the interac-tions remain

_unchanged;

the

_quadratic

effects are

-0.011488, -0.041071, -0.016225, -0.004175, -0.023099,

and -0.019517.

5.

_{Re-estimating}

the

_{Optimal Input}

Combination

After run _#28, we re-estimate the second-order

poly-nomial,

which

_gives

the estimated

_{optimum input}

values:

_{530.9438, 955.02,}

_623.2063,

_51.96925,

_647.079, and 74.437. This combination is the

_input

for run #29,

which

_gives

an

_output

of 145.4481

_(a

drastic increase

of

41.04%,

_compared

with the

_highest

_output

so

far).

Again re-estimating

the

_polynomial

_gives

overall

mean, main and

_quadratic

effects that

_hardly

_change,

and interactions that

_change

_quite

a bit. The

re-esti-mated

_{optimal input}

values are shown in the

Appen-dix. These values are the

_input

for run

_#30,

which

yields

a further increase to 159.5943.

Next we re-estimate the

_optimal

_inputs

and find

-7.0495 for factor _F; such a

_negative

_{value, however,}

is

impossible

since F denotes the factor _copper.

There-fore we

_keep

F’s level at zero in the next run

_(run

#31).

This

_yields

an

_output

of

157.5518,

a decrease

_compared

with the

_{immediately preceding}

run.

Next we

_again

re-estimate the effects and find

_they

hardly change.

We re-estimate the

_{optimal inputs:}

some

_inputs

increase, some

_decrease,

factor F becomes

positive again

(58.2545),

which is more

_meaningful.

Run #32

_yields

an

_output

of 151.3

(a decrease).

Finally,

we re-estimate the

_optimal

_input

values for

run #33, which

_yields

an

_output

of 152.6. This is not

the maximum

_output

over all 33 runs; the maximum

in our search is that of run #30

_(namely,

_159.5943).

6. Conclusions

Our

_computer

_budget

was restricted to a total of 33

runs,

_including

the free base run

_(provided

in the

prob-lem

_definition).

We used the first 22 runs to estimate

the six main effects and the 15 two-factor _{interactions,}

besides the overall mean. To

_specify

these runs we

used Rechschaffner’s saturated

_design

_{(Table 1),}

and

we

_changed

the factors

_by

10%

(see

_Appendix).

These runs _gave

_outputs

that increased

_by

no more than 9%

(90.9

in the base run became 99.2 in run

_#7).

Next we estimated the

_quadratic

effects. We used

six runs,

_increasing

each factor one at a _time,

_by

20%

(runs

#23

_through

_#28).

This increased the

_output

to a

maximum of _103.1, a modest increase

_(see

run

_#24).

For the

_remaining

five runs

_(#29

_through

_#33)

we

used the five combinations estimated to be

_optimal,

using

the second-order

_polynomial

re-estimated after

each run. These runs _gave

_{substantially}

_improved

outputs.

The overall maximum

_output

turns out to be the

re-sult of run _#30; this maximum is 159.5943. This is a 76% increase

_compared

with the base

_output,

90.9000.

Ob-viously,

our estimated maximum is not

_necessarily

the

_global

maximum

_(we

_might

have

_gotten

stuck at a

local

_maximum).

_Actually,

the true maximum

_output

turns out to be 160

_(see

Section

_7),

so we have succeeded

in

_{approximating}

the true maximum _very

_closely!

7.

_Epilogue

After we finished the search for the maximum

160. The simulation model that was a black box to us,

turned out to be:

y = 160 +

-

(zl - 420)2/5000 - (z2 -

870)2/10000

-(Z3 - 480)2/10000 -

(z4 -

40)2/70

-(Z5 - 520)2/10000 -

(z6 -

40)

2/1000

+ +

_{30/ {[(zl - 420)(z6 -}

_40)/1000]2

+

_{5} -}

_30/5.

So there are no main effects and no interactions

ex-cept

for that between z, and z2. There is no random

noise. The

_{optimal input}

values are 420,..., 40

(com-pare with the values of run

#30).

The last term

_(30/5)

is

subtracted,

because the value of the interaction term

for the

_optimal

_input

values is

_30/(02

+

_5).

8. Final Comments on the

_Competition

We were

disappointed

to learn that the simulation model was

_only

a mathematical

function,

not a

real-life

_problem

that we were

_helping

to solve. This fact

explains why

the

_participants

did not

_get

_any

informa-tion on the process itself and the ranges of its

_inputs.

Hence,

in our view the

_competition

was unrealistic: in

real life the

_analysts

accumulate much

_knowledge

while

_developing

their simulation model. This

1-nnIAll-edge

concerns both the model and the

_underlying

real

system.

In real

life,

_analysts

and

_{problem &dquo;owners&dquo;}

should

_cooperate!

Notwithstanding

this _criticism, not

_only

we found

this an

_interesting

and

_{challenging problem:}

12 teams

competed, employed by

operations

research and

sta-tistics

_departments

of well-known international

com-panies (Philips,

Unilever),

research institutes

_(TNO,

DLO),

and universities

_(Amsterdam,

_Tilburg).

We won

the

_competition,

but it was a

_&dquo;photo

finish&dquo;: our

maxi-mum

_output

was _159.6, whereas the

_second-place

out-put

was 159.4.

On

_hindsight,

interactions were not so

_important

(see

Section

_7),

so an R-4

_{design might}

have been

bet-ter than Rechtschaffner’s R-5

_design

_{(Table 1).}

How-ever, a

_complication

is that we used

_stage-wise

experi-mentation. In this

_approach,

classical

_designs

_(such

as

2k-p

_designs)

were not suited: we were limited to 32

runs

altogether,

and we also wanted to estimate the

quadratic

effects. This limit also

_implies

that we could

not

_{apply Response}

Surface

_Methodology

_(RSM),

which combines a series of local

designs

with

_steepest

ascent.

_Kleijnen

[3]

gives

details,

including nearly

100

references;

we limit our references to those

publica-that

_really

used.

a

_joint

_paper

_by

the better

_teams).

At the

_meeting

at

which the

_{competitors presented}

their

solutions,

it

turned out that

_typically

our

_strategy

_gave

_relatively

low results

_(compared

with our

_{competitors) during}

the first 28 runs; in runs #29

_through

#33, however,

our

_strategy

accelerated and overtook the

competi-tors’

_outputs.

In

_general,

our

_strategy

seems a

_good

heuristic for

real-life

_{applications. Obviously}

no heuristic is

_always

&dquo;best&dquo;

_(it

would not be a

_heuristic).

_Determining

when a

_particular

heuristic is

_applicable

is rather difficult.

One

_practical

solution

_might

be:

_apply

the heuristic

that is most familiar

_(&dquo;a

_carpenter

can solve any

prob-lem with a

_{hammer&dquo;).}

Our

_strategy

was a combination

of

_design

of

_experiment

_techniques

and common sense.

Moreover, in some other

_respects

this

_competition

was realistic: the number of runs was limited

_{(to 32),}

and there was a deadline

₍₅

_January

_1998).

So the

techniques applied

in this _{paper may} have more

ap-plications

in

_solving

real

_problems.

9.

_Postscript

In

_May

1999 this

_competition

was

repeated

with five

teams of students at the

_University

of

_Canterbury

in

t-11r4S4-c 1, --1- ~T..~.. _{1wl- --} &dquo;- -

autl- or

Christchurch,

New Zealand

(when

the first ~o author

visited the

_University

as a

_Visiting

Erskine

_Fellow).

Each team consisted of two members. These teams

used

_strategies

that differed from the

_strategy

that the authors

_{applied. Actually,}

the students did not

_try

to

estimate

_quadratic

effects and interactions. Instead

they

fitted first-order

_polynomials

to local

input/out-put

data, each time followed

_by

several

_steepest

as-cent trials. In

_hindsight,

interactions _may indeed be

ignored

in this

_competition!

The

_winning

team

suc-ceeded in

_obtaining

an

_output

of

_159.4-very

close to

the authors’

_output

of 159.6 and the true maximum of 160

_(the

&dquo;worst&dquo; team realized an

_output

of

139.2).

So

different

_strategies

_may

_yield

_(roughly)

the same

re-sult :

_&dquo;many

roads lead to Rome!&dquo;

10. References

[1] Rechtschaffner, R.L. "Saturated Fractions of 2n and 3n

Facto-rial _Designs." Technometrics, Vol. 9, pp 569-575, 1967.

[2] Kleijnen, J.P.C. Statistical _{Tools for} Simulation Practitioners,

Marcel Dekker, NY, 1987.

[3] Kleijnen, J.P.C. "Experimental Design for _{Sensitivity Analysis,} Optimization, and Validation of Simulation Models."

Hand-book _of Simulation, _Jerry Banks (ed.), Wiley, NY, 1998.

Acknowledgment

Jack P.C.

_Kleijnen

is a Professor of Simulation and

Informa-tion

_Systems.

His research concerns _simulation,

mathemati-cal statistics, information _systems, and

_logistics,

which have led to six books and

_nearly

160 articles. He has been a

consultant for several

_{organizations}

in the U.S. and

_Europe,

and has served on _many international editorial boards and scientific committees. He _spent several _years in the U.S., at

both universities and

_companies,

and received a number of international

_fellowships

and awards. More information is

provided

at

_{http:llcwis.kub.nll-few5lcenterlstafflkleijnenl.}

6zge

Pala is a PhD student in

_Operations

Research. She earned her BSc in Industrial

_Engineering

from

_Bogazici

University

in _Istanbul,

_Turkey,

and an MSc in

_Management

Science from

_Tilburg

_University

in The Netherlands. Her research interests concern _system

_{dynamics methodology,}

simulation and soft OR

_{methodologies.}

More information is

provided

at

_{http://cwis.kub.nl/ few5/center/phd_stud/pala/.}