Evaluation Data for MoDe4SLA
Lianne Bodenstaff
∗, Andreas Wombacher
University of Twente
{l.bodenstaff,a.wombacher}@utwente.nlManfred Reichert
University of Ulm
manfred.reichert@uni-ulm.de1 Transcript
Friday 9 January 2009 the first group of experts evaluated usefulness of MoDe4SLA. It was a try out with only three experts. The complete session was around one hour and 45 minutes which is divided in a presentation part, an explanation part with two examples, and the actual evaluation part with three test cases. A transcript of the exact times is depicted in Table 1.
Although each participant is employed in an information technology environment, not everyone is familiar with service compositions. Therefore, the presentation comprises an explanation of the problem and what the exact research gap is. The first part of the presentation discusses the necessity of identifying dependencies between different services in a composition, why identifying these dependencies is not straightforward. The second part of the presentation is on the MoDe4SLA approach in which is explained how we identify these dependencies and solve the problem. Since the first group already participated in previous presentations on MoDe4SLA, the time frame of 15 minutes for the presentation should be considered a minimum.
In the second part is through two examples explained how the survey will be conducted. Both examples have the same structure as the three test cases. The goal of introducing these examples is to allow the participants to get familiar with the MoDe4SLA approach. First, the representation of the service composition and its parameters (e.g., average response times) in the bilateral documents for both the estimations and the realized values is discussed. Second, the analysis done with MoDe4SLA on the realized values of the composition is discussed. Together with a legend the participants discuss how to use both the bilateral and the analysis documents.
The last part is done by the participants separately, without interference of the presenter. First the introductional questions are answered after which the participants go through the three test cases. After the test cases the concluding questions are answered. Interested participants receive an evaluation of the three test cases on how to read the analysis done through MoDe4SLA.
2 Hand-out
This Appendix contains the complete hand-out for participants of the evaluation. This starts with a cover sheet and a legend, after which the two examples and three test cases are given. The hand-out concludes with the survey itself and some suggested answers to the presented problems.
Time Subject Minutes
15:09-15:26 Presentation 15
15:27-15:40 Example 1 13
15:40-15:58 Example 2 18
16:00-16:04 Before evaluation: Q1-Q7 4
16:04-16:15 Test Case 1: 5 Services 11
• Q8-Q11 without MoDe4SLA: 4min • Q12-Q18 with MoDe4SLA: 7min
16:15-16:28 Test Case 2: 10 Services 13
• Q19-Q22 without MoDe4SLA: 5min • Q23-Q29 with MoDe4SLA: 8min
16:28-16:43 Test Case 3: 17 Services 15
• Q30-Q33 without MoDe4SLA: 8min • Q34-Q40 with MoDe4SLA: 7min
16:43-16:53 After evaluation: Q41-Q47 10
15:09-16:53 Total time 104
Table 1. Time Transcript
CS
Loop sequence 1, 3x
XOR 2
WS1
WS2
WS3
WS4
WS5
CS
AND 1
WS1
ORDISC 2
WS6
WS9
WS10
Example 1
Example 2
Impact Model of Costs composition AND 1 (Loop) XOR 2 3.0 WS 5 IF: 0.5278 3.0 WS 1 IF: 0.0612 0.6977 WS 2 IF: 0.2461 1.4186 WS 3 IF: 0.0438 0.3023 WS 4 IF: 0.1211 0.5814
Impact Model of Response Time composition AND 1 (Loop) XOR 2 3.0 WS 5 IF: 0.22 3.0 WS 1 IF: 0.3841 0.6977 WS 2 IF: 0.095 1.4186 WS 3 IF: 0.0347 0.3023 WS 4 IF: 0.2228 0.5814
Impact Model of Costs composition AND 1 WS 1 IF: 0.0336 1.0 OR 2 1.0 WS 6 IF: 0.2563 1.0 WS 7 IF: 0.098 1.0 WS 8 IF: 0.2486 1.0 WS 9 IF: 0.0454 1.0 WS 10 IF: 0.1273 1.0 AND 3 (Seq) 0.922 WS 5 IF: 0.0134 0.078 WS 2 IF: 0.0059 0.922 WS 3 IF: 0.1255 0.922 WS 4 IF: 0.0459 0.922
Impact Model of Response Time composition XOR 1 WS 1 IF: 0.3882 0.4255 XOR 2 0.2482 WS 6 IF: 0.0574 0.0567 WS 7 IF: 0.0 0.0 WS 8 IF: 0.0 0.0 WS 9 IF: 0.3265 0.2695 WS 10 IF: 0.0 0.0 AND 3 (Seq) 0.2482 WS 5 IF: 0.0 0.0 WS 2 IF: 0.0626 0.2482 WS 3 IF: 0.1216 0.2482 WS 4 IF: 0.0367 0.2482
Impact Model of Costs composition OR 1 OR 2 0.284 WS 5 IF: 0.1115 0.716 WS 1 IF: 0.21 0.2222 AND 3 (Loop) 0.2593 WS 4 IF: 0.0539 0.0864 WS 2 IF: 0.4298 0.8395 WS 3 IF: 0.1948 0.8395
Impact Model of Response Time composition XOR 1 XOR 2 0.284 WS 5 IF: 0.3205 0.716 WS 1 IF: 0.0 0.0 AND 3 (Loop) 0.2593 WS 4 IF: 0.0174 0.0247 WS 2 IF: 0.2825 0.8395 WS 3 IF: 0.3681 0.8395
Impact Model of Costs composition OR 1 WS 1 IF: 0.0249 0.0962 WS 2 IF: 0.1733 0.3808 WS 3 IF: 0.202 0.7231 WS 4 IF: 0.0545 0.2462 WS 5 IF: 0.0446 0.6192 WS 6 IF: 0.1145 0.7731 WS 7 IF: 0.0396 0.2808 WS 8 IF: 0.0964 0.7346 WS 9 IF: 0.115 0.4192 WS 10 IF: 0.1351 0.7269
Impact Model of Response Time composition XOR 1 WS 1 IF: 0.0 0.0 WS 2 IF: 0.1032 0.1038 WS 3 IF: 0.0 0.0 WS 4 IF: 0.0 0.0 WS 5 IF: 0.0359 0.0269 WS 6 IF: 0.4255 0.3808 WS 7 IF: 0.027 0.0346 WS 8 IF: 0.2773 0.3038 WS 9 IF: 0.1293 0.1423 WS 10 IF: 0.0018 0.0077
Impact Model of Costs composition AND 1 WS 1 IF: 0.0332 1.0 WS 2 IF: 0.1063 1.0 WS 3 IF: 0.0662 1.0 WS 4 IF: 0.0978 1.0 WS 5 IF: 0.0181 1.0 AND 2 1.0 OR 5 1.0 WS 16 IF: 0.033 1.0 WS 17 IF: 0.023 1.0 WS 6 IF: 0.0517 1.0 WS 7 IF: 0.038 1.0 XOR 3 1.0 WS 13 IF: 0.1763 1.0 WS 8 IF: 0.0504 0.7391 AND 4 (Loop) 0.1304 WS 12 IF: 0.0128 0.1304 WS 9 IF: 0.134 1.0 WS 10 IF: 0.0066 1.0 WS 11 IF: 0.0935 1.0 WS 14 IF: 0.0589 0.9565 WS 15 IF: 0.0 0.0435
Impact Model of Response Time composition XOR 1 WS 1 IF: 0.0 0.0 WS 2 IF: 0.0 0.0 WS 3 IF: 0.0 0.0 WS 4 IF: 0.0 0.0 WS 5 IF: 0.0 0.0 XOR 2 0.913 XOR 5 0.0 WS 16 IF: 0.0 0.0 WS 17 IF: 0.0348 0.087 WS 6 IF: 0.0791 0.2174 WS 7 IF: 0.0975 0.2174 XOR 3 0.4783 WS 13 IF: 0.0 0.0 WS 8 IF: 0.1648 0.3478 AND 4 (Loop) 0.1304 WS 12 IF: 0.0 0.0 WS 9 IF: 0.1967 1.0 WS 10 IF: 0.1818 1.0 WS 11 IF: 0.229 1.0 WS 14 IF: 0.0 0.0 WS 15 IF: 0.0 0.0
