Citation
Kolkman, W. (2006, November 14). Laparoscopic surgery in gynecology : studies about implementaion and training. Retrieved from
https://hdl.handle.net/1887/4980
Version: Corrected Publisher’s Version
construct validity
and establishing performance standards
for residency training
Abstract
Background: To test our laparoscopic simulator for construct validity and estab-lishing performance standards for simulator training.
Methods: Medical students with no laparoscopic experience (novices, n=18) and
laparoscopic advanced gynecologists (experts, n=5) participated in our study and were tested on our inanimate simulator by performing 5 tasks (pipe cleaner, rubber band, beads, cutting circle, intra-corporeal knot tying). Sumscore was the sum of scores of all 5 tasks. Scores were calculated by adding completion time with penalty points, thus rewarding speed and precision. After baseline evaluation, novices were assigned to 5 weekly training sessions (n=8, training group) or no training (n=10, control group). Both groups were retested. The training group had a total of 7 trials of each task (1 baseline test, 5 training sessions, 1 final test) and scores were analyzed for the effect of repetition. Experts were tested once and their performance was compared to the baseline scores of all novices to establish construct validity.
Results: The training group improved significantly in all tasks during the study
period. The final scores of the trained group were significantly better than the control group for all tasks and sumscore. The training group reached plateau of the learning curve within 7 trials, except for intra-corporeal knot tying. During final testing, the trained group reached the experts’ level of skills on the simulator.
Conclusions: Our simulation model has construct validity. Novices can reach
Introduction
Laparoscopy has become the method of choice for many open procedures. This minimal invasive approach has considerable benefits for patients, such as reduced morbidity, shorter hospitalization, better cosmetic results, and earlier return to normal activity. [Darzi 2002]
Performing laparoscopic surgery requires psychomotor skills that are different from those needed to perform open surgical procedures and result in long learning curves. [Lehmann 2005] These skills include the shift from three-dimensional operating field to two-dimensional monitor display, judgment of altered depth perception and spatial relationships, distorted eye-hand coordination, adaptation to the fulcrum effect, manipulation of long surgical instruments while adjusting for amplified tremor, diminished tactile feedback and fewer degrees of freedom. [Munz 2004]
In addition to the difficulties of acquiring laparoscopic skills, issues such as quality control, patient safety, financial constraints, efficiency and cost-effective-ness and less exposure to the operating room resulted in the need for skills training outside the operating room. [Fried 2004, Munz 2004]
Simulators are devices that recreate operative conditions as a substitute for real life performance and play an increasingly important role in training of basic laparoscopic skills. Simulator technology is initially proven to be a useful teaching tool in aviation; nowadays it has great potential for training [Derossis 1999, Jordan 2001, Lentz 2001, Rosser 1997, Scott 2001] and objectively assessing laparoscopic skills. [Derossis 1999, Jordan 2001, Reznick 1997, Scott 2001] Simulator training is shown to be effective in providing skills that are transferable to the operating room. [Grantcharov 2004, Hamilton 2002, Hyltander 2002, Schijven 2005]
Methods
This study was performed in the skills laboratory located at the Department of Gynecology of the Leiden University Medical Center (LUMC) in The Netherlands. The simulator was designed (author FWJ) and fabricated at the LUMC. It consisted of an inanimate 5-task box trainer with a non-transparent cover, measuring 45 x 30 x 25 cm using a 0º scope.
Outcome measures
The performance on the box trainer was measured using a scoring system that rewards precision and speed. During each task the time to completion (seconds) and penalty points were measured. Scores were calculated by adding completion time and penalty points, thus rewarding both speed and precision (score = time + penalty points). This scoring system rewards faster and more accurate performance with lower scores. Besides a score for each task, a sumscore was calculated, which was defined as the sum of scores of all 5 tasks. All participants were instructed how to perform the 5 tasks and how penalty points were obtained by watching a 10-minute introduction video. No practice was permitted before testing and the 5 tasks were performed in a set order.
Tasks
The simulator and the tasks in this study, as well as the scoring system were based on the studies of Derossis et al. [Derossis 1998] and are visualized in figure 1 and figure 2.
1. Pipe cleaner
This task involved placing a pipe cleaner though 4 small circles. A penalty was calcu-lated when a circle was missed. Score = time in seconds + (the number of missed circles x 10).
2. Placing rubber band
This task required stretching a rubber band around 16 nails on a wooden board. A penalty was calculated when the rubber band was not stretched around a nail at the end of the task. Score = time in seconds + (the number of missed nails x 10).
3. Placing beads
This task involved placing 13 beads in a letter ‘B’. A penalty was calculated when a bead was dropped next to the pegboard. Score = time in seconds + (the number of dropped beads x 10).
4. Cutting circle
This task required cutting a circle from a rubber glove stretched over 16 nails in a wooden board. Penalty points were calculated when deviated from cutting on the line. Score = time in seconds + surface of glove in milligrams deviated from circle.
5. Intra-corporeal knot tying
This task involved tying an intra-corporeal knot (2 turn, square knots) in a foam uterus. A penalty was calculated to reflect the security (slipping or too loose) of the knot. Score = time in seconds + 10 when knot was slipping or loose.
Figure 2 | Five inanimate tasks on laparoscopic simulator.
1. Pipe cleaner 2. Rubber band 3. Beads
Construct validity
Five gynecologists with extensive experience in advanced laparoscopy (‘experts’ that performed more than 100 advanced laparoscopic procedures) were invited to complete the 5 simulator tasks once. Their scores were compared to the novices’ baseline evaluation in order to establish construct validity.
Effect of training
A total of 18 medical students (novices) volunteered to participate in the study. They were in their 2nd- 5th years as medical students at LUMC and had no experience with simulator training or clinical laparoscopy prior to, or during the study period.
After baseline testing on the simulator, the novices were assigned to either 5 weekly training sessions on a simulator (training group, n=8) or to a control group (n=10). Assignment occurred based on availability of the student during the study period. The control group and training group were then again measured as final testing, as shown in figure 3. The control group received no skills training. During
NFEJDBMTUVEFOUTXJUI OPTVSHJDBMFYQFSJFODF ´OPWJDFTµ #BTFMJOFUFTUJOH POTJNVMBUPS "TTJHOFEUPOPUSBJOJOH O DPOUSPMHSPVQ /PUSBJOJOHPO "TTJHOFEUPUSBJOJOH O USBJOJOHHSPVQ XFFLMZUSBJOJOH FYQFSJFODFE MBQBSPTDPQJTUT ´FYQFSUTµ 5FTUJOH POTJNVMBUPS
baseline testing, all 5 training sessions and final testing the novices completed all 5 tasks once. This means that at the end of the study the training group completed all tasks a total of 7 times (1 baseline test, 5 training sessions, 1 final test) and will be referred to in this manuscript as 7 trials. These results were analyzed for the effect of repetition (learning curve). The plateau of the learning curve was established when no statistical difference was shown between trials.
Performance standard
The performance of the experts was compared to the results of the trained novices (final test) in order to analyze whether experts’ basic skills level is feasible as perfor-mance standard for laparoscopic novices on our simulator.
Statistics
Collected data were analyzed by SPSS 12.0 software package (SPSS, Chicago, Illinois, USA). Statistical analyses were performed using Chi-square test, Paired t-test, Fisher’s Exact test, Mann-Whitney test, Friedman test, Pearson’s correlations coefficient and Wilcoxon rank-sum test. P-values below 0.05 were considered statistically significant.
Results
Table 1 shows the demographics of the training group (n=8) and the control group (n=10) were comparable (median age 22 years and 21 years (Mann-Whitney test: p=0.14). The training group consisted of 3 male and 5 female students, and the control group of 6 male and 4 female students (Fisher’s Exact Test p=0.63).
There were no dropouts throughout the study.
Table 1 | Demographics of novices in study.
Training group Control group
n=8 n=10 p
Median age (range) 22 (20-25) 21 (21-23) NS
Male n (%) 3 (37.5) 6 (60) NS
Female n (%) 5 (62.5) 4 (40) NS
Mean year of study (range) 3.5 (2-5) 2.8 (2-4) NS
Construct validity
The median scores of participants are stated in table 2-4 and figure 4. Comparison between experts (n=5) and novices’ baseline testing (n=18) demonstrated significant difference for all 5 tasks and sumscore in favor of the experts (Mann-Whitney test: pipe cleaner p<0.001, rubber band p=0.007, beads p<0.001, circle cutting p=0.001, knot tying p<0.001, sumscore p<0.001). This means that experts performed the tasks significantly faster and more accurate than novices during their baseline testing.
Effect of training
The training group improved significantly in all 5 tasks and sumscore (Wilcox-on’s signed-rank test: all p<0.02). The final testing scores of the training group were significantly better than final testing scores of the control group for all the 5 tasks (Mann-Whitney test: pipe cleaner: p<0.001, rubber band: p=0.04, beads: p=0.02, circle-cutting: p=0.009, intra-corporeal knot tying: p<0.001 and sumscore (p<0.001).
The training group had a total of 7 trials, which represents the novices’ learning curve on our simulation model. Their individual sumscore of these 7 trials are
Table 2 | Median scores of the control and training group
Control group Training group
Baseline test Final test p* Baseline test Final test p* score (range) score (range) score (range) score (range) Pipe cleaner 250 (155-892) 170 (76-620) 0.17 314 (126-900) 48 (33-105) 0.002 Rubber band 151 (79-488) 142 (35-488) 0.48 190 (89-484) 49 (22-72) 0.002 Beads 608 (371-1454) 363 (205-859) 0.009 1558 (474-1558) 227(168-420) 0.001 Cutting circle 455 (217-754) 254 (169-469) 0.03 426 (310-520) 148 (89-244) <0.000 Knot tying 416 (279-1363) 366 (232-605) 0.17 597 (383-930) 194 (105-298) <0.000 Sumscore 1978 (1278-4745) 1386 (841-2940) 0.75 2403 (1947-2931) 720 (497-971) <0.000
Table 3 |
Training group’s median scores.
Trial 1 Trial 2 Trial 3 Trial 4 Trial 5 Trial 6 Trial (baseline test) (training 1) (training 2) (training 3) (training 4) (training 5) (final test) score (range) score (range) score (range) score (range) score (range) score (range) score (range) Pipe cleaner 314 (126-900) 82 (49-177) 73 (58-118) 76 (42-132) 54 (33-148) 62 (29-150) 48 (33-105) Rubber band 190 (89-484) 149 (44-182) 113 (82-182) 99 (65-166) 79 (42-107) 41 (31-114) 49 (22-72) Beads 734 (474-1558) 434 (227-1163) 348 (208-582) 281 (189-510) 216 (181-363) 300 (136-487) 227 (168-420) Cutting circle 426 (310-520) 314 (158-427) 238 (145-358) 208 (146-373) 147 (123-245) 148 (97-269) 148 (89-244) Knot tying 597 (383-930) 520 (272-951) 373 (178-598) 284 (187-539) 344 (172-542) 254 (125-398) 194 (105-298) Sumscore 2403 (1947-2931) 1511 (1064-1965) 1179 (897-1395) 992 (714-1500) 911 (643-1232) 825 (641-995) 720 (497-971)
To reach a plateau in the learning curve, the novices had to perform a mean of 2 trials for the pipe cleaner task, 5 for the rubber band, 7 for the beads and 5 for the circle-cutting. The only task where plateau was not reached within 7 trials was intra-corporeal knot tying (Paired t-test).
Performance standard
Within 7 trials, the trained novices reached the experts’ skills level in all tasks and
Figure 4 | Trained novices’ individual sumscore during the 7 trials.
Table 4 | Median scores of the control and training group. Final test training group Experts score (range) score (range) p Pipe cleaner 48 (33-105) 62 (49-100) 0.13 Rubber band 49 (22-72) 62(35-195) 0.43 Beads 227(168-420) 271 (111-318) 0.9 Cutting circle 148 (89-244) 189 (76-240) 0.8 Knot tying 194 (105-298) 118 (50-177) 0.09 Sumscore 720 (497-971) 705 (351-878) 0.9
Score = time + penalty points, lower scores represent better performance, * = Mann-Whitney test.
Discussion
Our inanimate simulation model can distinguish reliably between the perfor-mance level of expert laparoscopists and novices (construct validity) on the
simulator. Furthermore, our simulation model is a successful device for training and measuring skills objectively. Experts’ basic skills level on the simulator as perfor-mance standard is feasible, given that laparoscopic novices can be trained to reach experts’ level on a simulator after a short simulator training program.
The tasks in this study, as well as the scoring system were based on the studies of Derossis et al. [Derossis 1998] In her studies these tasks on the inanimate laparoscopic simulator were validated and it was established that practice in the simulator resulted in improved performance in vivo. Therefore, for current study it was decided not to repeat these investigations, since the beneficial aspects of simulator training for ‘live’ laparoscopic skills were previously shown. In current study construct validity was analyzed in order to establish performance standards of the simulator.
Of the 5 tasks on our simulator, intra-corporeal knot tying was considered the most difficult task. Although intra-corporeal knot tying is not a frequently performed clinical procedure, it is required to be able to perform advanced lapa-roscopic procedures. Furthermore, this task mimics all skills mentioned above. Therefore, it is our opinion that intra-corporeal knot tying is an outstanding task to train on a simulator.
Statistical analyses showed that after 7 trials of intra-corporeal knot-tying the novices had a higher (worse) mean score than the experts. However, these scores showed no significant difference. In addition, the novices did not yet reach plateau in their learning curve for intra-corporeal knot-tying after these 7 trials. From these data we conclude that statistically novices can reach experts’ level after 7 trials, however more training is needed to establish plateau in the learning curve and to actually reach experts’ level on the simulator.
The pipe cleaner and rubber band tasks were chosen as the first two tasks, since they practice hand eye coordination and the lack of depth perception by working with two instruments, as well the camera. Of the other exercises the beads task resembles laparoscopic sterilization, in which one hand is used to operate the camera and the other to complete the task or procedure and the circle cutting tasks resembles a cystectomy where precision is required.
Skills are expected to improve with increased training and repetition. We found that novices’ performance improved and they reached the experts’ skills level on the simulator. In order to reach maximum results, it is our opinion that residents should be trained early in residency training, when they are still inexperienced in laparo-scopic surgery. To obtain experts’ level of basic laparolaparo-scopic skills on the simulator, a short training course as is described in this study is certainly feasible during residency. The frequency of training necessary to maintain skills and the retention of skills after training are not well established yet. [Stefanidis 2005, Stefanidis 2006] Obviously, continuous training, either in the operating room or on a simulator, is essential to preserve laparoscopic skills.
