The design of a hysteroscopy simulator

Hele tekst

(1)UNIVERSITEIT•STELLENBOSCH•UNIVERSITY jou kennisvennoot. •. your knowledge partner. The Design of a Hysteroscopy Simulator by. Gerrit V. Kruger. Thesis presented at the University of Stellenbosch in partial fulfilment of the requirements for the degree of. Master of Science in Engineering (Electronic Engineering with Computer Science). Department of Electrical and Electronic Engineering University of Stellenbosch South Africa. Study leader: Dr. M.M. Blanckenberg. December 2007.

(2) Declaration I, the undersigned, hereby declare that the work contained in this thesis is my own original work and that I have not previously in its entirety or in part submitted it at any university for a degree.. Signature: . . . . . . . . . . . . . . . . . . . . . . . . . . G.V. Kruger. Date: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. c 2007 University of Stellenbosch Copyright All rights reserved..

(3) Abstract The Design of a Hysteroscopy Simulator G.V. Kruger Department of Electrical and Electronic Engineering University of Stellenbosch South Africa. Thesis: MScEng (Electronic Engineering with Computer Science) December 2007 This thesis documents the research, design and implementation of a virtual hysteroscopy simulator. A hysteroscopy is a medical procedure used in the field of obstetrics and gynaecology. The system consists of a mechanically modelled hysteroscope connecting to any personal computer using USB, and is plug and play compatible on any modern Windows operating system. The scope interfaces with custom-designed software, which simulates a patient’s anatomy and thus creates a virtual operating environment. The software incorporates features like simulated soft tissue and an ablation algorithm. This system is developed with the aim to aid the young or inexperienced surgeon with hands-on training, developing his hand-eye coordination and visual and surgical skills used in this procedure.. ii.

(4) Uittreksel Die Ontwerp van ’n Histeroskopie Simulator (“The Design of a Hysteroscopy Simulator”). G.V. Kruger Departement Elektries en Elektroniese Ingenieurswese Universiteit van Stellenbosch Suid Afrika. Tesis: MScIng (Elektroniese Ingenieurswese met Rekenaarwetenskap) Desember 2007 Hierdie tesis handel oor die ontwerp en implementering van ’n virtuele histeroskopie simulator. ’n Histeroskopie is ’n mediese prosedure wat in die veld van ginekologie beoefen word. Die sisteem bestaan uit ’n meganiese histeroskoop wat aan ’n persoonlike rekenaar deur middel van ’n USB konneksie verbind word. Die beweging van die meganiese histeroskoop word gemeet en na die sagteware gestuur. Die sagteware word dan gebruik om ’n pasient se anatomie na te boots, asook om die beweging van die skoop vas te vang. So word ’n virtuele operasie dus gesimuleer. Die sagteware maak gebruik van ’n sagte weefsel- asook van ’n sny algoritme. Die sisteem is daarop gemik om jong of onervare dokters op te lei deur hulle hand-oog koördinasie asook hulle shirurgiese vermoëns te verbeter.. iii.

(5) Acknowledgements I would like to express my sincere gratitude to the following people and organisations: Dr. Blanckenberg for his guidance throughout this project My parents, Thinus and Sanderina Kruger, for all their support Abraham Coetzee for proof reading The online communities for all the help I received at the forums for both Blender and Ogre My fellow students at the Electronic Systems Lab for all their help, comments and suggestions; in particular: Eric Baker Janto Dreijer Edré janse van Rensburg Izak Marais Henk Marais Johan Schoonwinkel Johannes van der Horst Stephan van Ellewee. iv.

(6) Dedications. This thesis is dedicated to my loving grandparents, former Judge of Appeal Gerrit Viljoen, Wilma Viljoen, Piet Kruger and Joan Kruger.. v.

(7) Abbreviations IUD D&C 3D PC USB CPR POT AMR CPI IPS LED DIP 2D IAS DSP CPU A/D IO RISC SRAM EEPROM PCB HID SCK. Intrauterine device Dilation and curettage Three-dimensional Personal computer Universal serial bus Cardiopulmonary resuscitation Potentiometer Anisotropic Magneto-Resistive Counts per inch Inches per second Light emitting diode Dual inline package Two dimensional Image Acquisition System Digital Signal Processor Central processing unit Analog-to-digital Input/output Reduced instruction set computer Static random access memory Electrically erasable programmable read only memory Printed circuit board Human interface device Serial port clock. vi.

(8) ABBREVIATIONS. SDIO FPS CSG CAD FEM GLSL HLSL API Cg. serial data in/out line Frames per second Constructive solid geometry Computer aided design Finite Element Method OpenGL Shading Language High Level Shader Language Application programming interface C for graphics. vii.

(9) Contents Declaration. i. Abstract. ii. Uittreksel. iii. Acknowledgements. iv. Dedications. v. Abbreviations. vi. Contents. viii. List of Figures. xii. List of Tables. xiv. 1 Introduction 1.1 Hysteroscopy Defined 1.2 Problem Description 1.3 System Requirements 1.4 Design Overview . . 1.5 Thesis Overview . . .. . . . . .. 1 1 4 4 6 6. 2 System Level Design 2.1 Existing Systems . . . . . . . . . . . . . . . . . . . . . . . .. 7 7. . . . . . . . . . . . . . . . . . . . . . . and Specifications . . . . . . . . . . . . . . . . . . . . . .. viii. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . ..

(10) ix. CONTENTS. 2.2 2.3. System Innovations . . . . . . . Complete System Description . 2.3.1 Mechanical Hysteroscope 2.3.2 Electronic Hardware . . 2.3.3 Simulation Software . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. . . . . .. 8 9 9 10 11. 3 Hardware Design 13 3.1 Electronic Hardware Overview . . . . . . . . . . . . . . . . . 13 3.2 Sensors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.2.1 Sensor requirements . . . . . . . . . . . . . . . . . . 14 3.2.2 Sensor Technology . . . . . . . . . . . . . . . . . . . 14 3.2.3 VISHAY Spectrol Model 357 Conductive Plastic Potentiometer . . . . . . . . . . . . . . . . . . . . . . . 16 3.2.4 Avago Optical Sensor ADNS-2610 . . . . . . . . . . . 16 3.3 Processing Unit . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.3.1 CPU requirements . . . . . . . . . . . . . . . . . . . 17 3.3.2 CPU technology . . . . . . . . . . . . . . . . . . . . . 17 3.3.3 Microchip’s PIC18F2550 . . . . . . . . . . . . . . . . 18 3.4 Subsystem Integration . . . . . . . . . . . . . . . . . . . . . 18 3.5 Mechanical Hardware . . . . . . . . . . . . . . . . . . . . . . 19 3.6 Modelled Anatomy . . . . . . . . . . . . . . . . . . . . . . . 20 3.6.1 Requirements . . . . . . . . . . . . . . . . . . . . . . 20 3.6.2 Construction Process . . . . . . . . . . . . . . . . . . 20 3.7 Completed Hardware . . . . . . . . . . . . . . . . . . . . . . 21 4 Firmware Design 22 4.1 A Brief Overview of USB and its Implementation . . . . . . 22 4.2 Firmware Overview . . . . . . . . . . . . . . . . . . . . . . . 25 4.3 Calibration between Hardware and Software . . . . . . . . . 26 5 Software Design 28 5.1 Software Overview . . . . . . . . . . . . . . . . . . . . . . . 28 5.2 Related Research . . . . . . . . . . . . . . . . . . . . . . . . 29 5.2.1 A Brief Introduction to Computer Graphics . . . . . 29.

(11) x. CONTENTS. 5.3 5.4. 5.5 5.6. 5.7. 5.8. 5.2.2 Three-Dimensional Object Representations Software Requirements . . . . . . . . . . . . . . . Component Library Overview . . . . . . . . . . . 5.4.1 3D Content Creation . . . . . . . . . . . . 5.4.2 Graphics Rendering Engine . . . . . . . . 5.4.3 Input System . . . . . . . . . . . . . . . . 5.4.4 Graphical User Interface . . . . . . . . . . 5.4.5 Collision System . . . . . . . . . . . . . . Frame Overview . . . . . . . . . . . . . . . . . . . Soft Tissue Simulation . . . . . . . . . . . . . . . 5.6.1 Background Study . . . . . . . . . . . . . 5.6.2 Soft Tissue Algorithm . . . . . . . . . . . Surgery Simulation . . . . . . . . . . . . . . . . . 5.7.1 Background Study . . . . . . . . . . . . . 5.7.2 Algorithm Employed . . . . . . . . . . . . 5.7.3 Texturing a cut . . . . . . . . . . . . . . . Special Effects . . . . . . . . . . . . . . . . . . . . 5.8.1 Animation . . . . . . . . . . . . . . . . . . 5.8.2 Particle System . . . . . . . . . . . . . . . 5.8.3 Lighting and Shading . . . . . . . . . . . . 5.8.4 Vertex and Fragment Programs . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . . . . . . . . .. 30 33 33 34 36 37 38 38 38 39 39 40 44 44 44 46 48 48 49 50 51. 6 Testing and Results 55 6.1 Mechanical Hysteroscope . . . . . . . . . . . . . . . . . . . . 55 6.2 Virtual Surgery Application . . . . . . . . . . . . . . . . . . 56 7 Conclusions and Recommendations 59 7.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 7.2 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . 60 List of References. 62. Appendices. 64.

(12) CONTENTS. xi. A Sensor Technology. 65. B Hardware Schematics 67 B.1 Electronic Schematics . . . . . . . . . . . . . . . . . . . . . . 68 B.2 PCB Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 B.3 Technical Drawings . . . . . . . . . . . . . . . . . . . . . . . 71 C USB Descriptors C.0.1 Device Descriptor . . . . C.0.2 Configuration Descriptor C.0.3 Interface Descriptor . . . C.0.4 HID Descriptor . . . . . C.0.5 Endpoint Descriptor . . C.0.6 String Descriptors . . . . C.0.7 Report Descriptor . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. . . . . . . .. 73 73 74 74 74 75 76 77. D Firmware 79 D.1 Interfacing with the Optical Sensors . . . . . . . . . . . . . . 79 D.2 Analog to Digital Conversion . . . . . . . . . . . . . . . . . 80 D.3 Firmware Code . . . . . . . . . . . . . . . . . . . . . . . . . 81 E Code Structure. 83. F Software Scripts 84 F.1 Example Material Script . . . . . . . . . . . . . . . . . . . . 84 F.2 Vertex and Fragment Programs . . . . . . . . . . . . . . . . 87 G Rendering Pipeline. 92.

(13) List of Figures 1.1 1.2 1.3 1.4. Uterus . . . . . . . . . Hysteroscope . . . . . Front of hysteroscope. Different instruments .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. 2.1 2.2 2.3 2.4. System overview. . . . . . Mechanical hardware. . . Gimbal with POTs. Left: slot visible. . . . . . . . . Captured movement. . . .. 3.1 3.2 3.3 3.4. VISHAY Spectrol potentiometer. . . . . . . . . . . . . . . Sectional view of PCB assembly. . . . . . . . . . . . . . . Microprocessor (left) and optical sensor (right) PCBs. . . The external boundary shell of the fibreglass uterus model.. . . . .. 16 17 19 21. 4.1 4.2 4.3. Hierarchy of USB descriptors. . . . . . . . . . . . . . . . . . . . Flow diagram of main program loop. . . . . . . . . . . . . . . . Format of data sent to the PC (byte 0 left, byte 6 right). . . . .. 24 25 26. 5.1 5.2 5.3 5.4 5.5 5.6. Graphics rendering pipeline. . . . . . . . . . . . . . . . A polygon mesh. . . . . . . . . . . . . . . . . . . . . . Union operation between solids in CSG modelling. . . . A voxel representation of a pig’s head. . . . . . . . . . A wireframe, solid, and textured uterus polygon model. A wireframe, solid, and textured ablation tool model. .. 30 31 31 32 35 36. . . . . . . . . optical . . . . . . . .. xii. . . . .. . . . .. . . . .. . . . .. . . . . . . . . sensor . . . . . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . .. . . . . . . . . . . . . . . . . board visible. . . . . . . . . . . . . . . . .. . . . .. . . . .. 2 2 3 4. . . . . . . . . . . Right: . . . . . . . . . .. 9 10. . . . . . .. . . . .. . . . . . .. . . . .. . . . .. . . . . . .. . . . .. . . . .. . . . . . .. . . . . . .. 11 11.

(14) xiii. LIST OF FIGURES. 5.7 5.8 5.9 5.10 5.11 5.12 5.13 5.14 5.15 5.16 5.17 5.18 5.19 5.20. The constructed 3D scene at the start of the application. . Operations per frame. . . . . . . . . . . . . . . . . . . . . The constraints between element V0 and its neighbour V1. Flow of algorithm through mesh. . . . . . . . . . . . . . . The format of the hardware buffers. . . . . . . . . . . . . . Data structure setup. . . . . . . . . . . . . . . . . . . . . . Projecting vertices with the ablation algorithm . . . . . . Before and after new vertices are added to the mesh. . . . Data structure after subdivision. . . . . . . . . . . . . . . A textured plane with armature visible. . . . . . . . . . . Screenshots of generated particles . . . . . . . . . . . . . . Lighting . . . . . . . . . . . . . . . . . . . . . . . . . . . . A texture that has no intrinsic shading property. . . . . . Normal mapping . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. . . . . . . . . . . . . . .. B.1 B.2 B.3 B.4 B.5 B.6. Schematic of optic sensor PCB . . . . . . . . . . . . . . . . . . Schematic of PIC PCB . . . . . . . . . . . . . . . . . . . . . . . Layout of PIC PCB . . . . . . . . . . . . . . . . . . . . . . . . . Layout of optic sensor PCB . . . . . . . . . . . . . . . . . . . . Technical drawing of modelled hysteroscope: gimbal . . . . . . Technical drawing of modelled hysteroscope: optical sensor fitting. 37 39 41 41 42 43 45 46 47 48 49 51 54 54 68 69 70 70 71 72.

(15) List of Tables 6.1. Average FPS during simulation . . . . . . . . . . . . . . . . . .. 7.1. Total cost of prototype system (estimate) excluding PC hardware. 60. E.1 Application source files . . . . . . . . . . . . . . . . . . . . . . .. xiv. 57. 83.

(16) Chapter 1 Introduction This chapter begins with a background study that leads to the formulation of the project and the initial system specification. The chapter also briefly covers the basic design of the system and concludes with an overview of this thesis.. 1.1. Hysteroscopy Defined. Hysteroscopy is a minimally invasive procedure used in the field of obstetrics and gynaecology [1]. A Hysteroscopy is performed on a female patient as a diagnostic modality, in order to determine causes of infertility, as well as a treatment tool to rectify any identified problems. During a hysteroscopy, the physician inserts a hysteroscope through the cervical canal to inspect the cavity of the uterus. The hysteroscope is an extremely thin telescope-like instrument. It contains a small light, lens and camera which is connected to a television monitor to give the surgeon a view to work from. Different instruments can be inserted through the scope to perform the required functions (see Fig. 1.3). The modern hysteroscope is so thin that it can fit through the cervix with only minimal or no dilation. Diagnostic hysteroscopy can be used to help determine the cause of infertility, dysfunctional uterine bleeding, and repeated miscarriages. It. 1.

(17) CHAPTER 1. INTRODUCTION. 2. Figure 1.1: Uterus. Figure 1.2: Left: disassembled hysteroscope. Right: Assembled scope.. can also help locate polyps and fibroids (irregular tissue growths in the uterine cavity), as well as intrauterine devices (IUDs). The procedure is also used to investigate and treat gynaecological conditions, often done instead of performing a dilation and curettage (D&C). A D&C is a procedure that expands the cervical canal (dilation) so that the lining of the uterus can be scraped (curettage). A D&C can be used to take.

(18) CHAPTER 1. INTRODUCTION. 3. Figure 1.3: Front of hysteroscope. Left: camera visible. Right: ablation tool protruding.. a sample of the lining of the uterus for analysis. However, hysteroscopy has advantages over a D&C because the physician can take tissue samples of specific areas and view any fibroids, polyps, or structural abnormalities. In addition, small fibroids and polyps may be removed via the hysteroscope, thus avoiding more invasive and complicated open surgery. This approach is also used to remove IUDs that have become embedded in the wall of the uterus. Before inserting the hysteroscope, an anaesthetic is administered to the patient. Once it has taken effect, the doctor dilates the cervix slightly, and then inserts the hysteroscope through the cervix to reveal the inside of the uterus. Ordinarily, the walls of the uterus are touching each other. In order to get a better view, the uterus may be inflated with carbon dioxide gas or fluid. Tiny surgical instruments may be inserted through the hysteroscope to remove polyps or fibroids (see figure 1.4). Different instruments that can be used are an ablation tool, scissors or biopsy pliers. Some tools, such as the ablation tool is activated by means of a foot pedal, while others like the scissors and pliers are controlled solely by the surgeon’s hands. A hysteroscopy takes approximately 7 to 30 minutes to perform, depending on the specific procedure. Treatment involving the use of hysteroscopy is usually performed as a short-stay hospital procedure..

(19) CHAPTER 1. INTRODUCTION. 4. Figure 1.4: Different instruments. From left to right: ablation tool, scissors and pliers.. 1.2. Problem Description. The current method of training is firstly, by observing a procedure being performed by a senior surgeon. The next step would be to let the junior surgeon assist a few procedures until finally he or she is able to perform the procedure on their own. This way of training is often done over an extended period of time and therefore takes a while for the surgeon to develop the necessary proficiency, not only with the instrument, but also with the various procedures that they must be able to perform. Although the current training method won’t be replaced with the introduction of a simulator, it would speed up the training process by helping the young surgeon to develop self-confidence and skill with extra practice.. 1.3. System Requirements and Specifications. With the need for a more practical training method in mind, the initial system specifications were drawn up. The aim of this project is to provide a simulated surgical environment for young or inexperienced surgeons in the field of gynaecology, using it to practice their surgical skills and developing their hand-eye coordination, specifically using the hysteroscope. The basic real-world system consists of a patient, hysteroscope, and a television monitor next to the surgeon, which gives him or her a view to.

(20) CHAPTER 1. INTRODUCTION. 5. work from. To simulate this system, the hysteroscope will be modelled mechanically with sensors attached to capture the movement. The television monitor will be replaced by a personal computer (PC) running custom software, simulating a procedure with interactive three-dimensional (3D) graphics. The scope will connect to the computer via the universal serial bus (USB). A USB connection was chosen as it is the current standard for connecting peripherals. Through live observation of a hysteroscopy being performed, it was determined that the pitch, yaw, translation and roll of the scope is very important for simulation. The translation and roll of the instruments inserted through the scope would also be required if an accurate simulation was to be achieved. The actual vertical and horizontal movement of the scope was relatively small, and could safely be ignored without compromising the accuracy of the simulator. Next, the imagery a surgeon would expect to see, would have to be recreated with software on a computer screen. In broad terms the project can thus be divided into hardware and software. The hardware specifications are as follows: • Mechanically modelling a hysteroscope • Detecting the predetermined required movements of the scope (pitch, yaw, translation and roll of the scope, as well as the roll and translation of the instrument inside the scope). Total angles for pitch and yaw was estimated to be no more than 70◦ each. • Transferring the information via USB to a PC The software specifications are: • Accurately model the appropriate anatomy • Simulate soft tissue • Incorporate surgical algorithms for interactive surgery capability.

(21) CHAPTER 1. INTRODUCTION. 1.4. 6. Design Overview. Using the problem statement and the specifications mentioned in the previous section, a prototype system was developed. It consists of a modelled hysteroscope attached via USB to a computer. The main six movements of the modelled hysteroscope are captured through different sensors and sent to the computer. The hysteroscope is enumerated as a USB gaming device or joystick, and interfaces with custom-designed software, which creates a simulated surgical environment.. 1.5. Thesis Overview. The rest of this thesis details the research and design that went into the implementation of a prototype hysteroscopy simulator. Chapter 2 details the system level design which gives a good overview of the more detailed design to follow. Chapter 3 then begins with the design of the hardware and is followed by the firmware design in chapter 4. The development of the software is detailed in chapter 5 followed by testing and results and then conclusions and recommendations in the last two chapters respectfully. Each chapter detailing design choices starts with a literature study of the available technology to fulfil the specific system requirements. After the necessary research has been done, the design choices are made and the implementation is documented..

(22) Chapter 2 System Level Design This chapter begins by looking at existing commercially available surgery simulators. Section 2.2 describes how this design differs from existing systems. The last section gives a broad overview of the complete system design and functioning thereof.. 2.1. Existing Systems. The field of medical simulators and trainers is relatively new, but it’s also expanding at a very fast pace. Devices ranging from medical training manikins, used to practise cardiopulmonary resuscitation (CPR), intravenous insertion or teaching anatomy, to devices such as a haptic suturing simulator 1 have been developed. Although the applications of these devices vary widely, they are all used to educate and train medical personnel in one way or another. The focus here will be on devices simulating a hysteroscope or similar instrument. The first system being discussed is developed by Simulab Corporation2 . They have a range of products designed for laparoscopic surgery training and simulation models. The “Laptrainer” is a laparoscopic video trainer and uses a mounted digital camera which allows the user to reposition the field of view within the operative cavity. Modelled tissue pads are used to 1 2. http://cs.millersville.edu/ webster/haptics/suture/ http://www.simulab.com/LaparoscopicSurgery.htm. 7.

(23) CHAPTER 2. SYSTEM LEVEL DESIGN. 8. operate on during the training. The other products are based on the same technology and are basic variations of the “Laptrainer”. The “Laptrainer” is currently available for just under US$2000. Another virtual endoscopic training simulator or VEST3 , also known as VSOne has been developed in Germany4 . The system consists of a complete stand alone module with built in controls, haptic feedback, a display screen, and all the relevant hardware needed to run the software simulations. The price of the system is only available on request. Immersion Medical5 designs, manufactures and markets computer-based medical simulation systems worldwide. They combine custom designed software and haptic feedback robotics with high powered computers to produce realistic training simulators. The company has a number of available simulators, including vascular access, endoscopy, endovascular, laparoscopy R System is and also hysteroscopy simulators. The Hysteroscopy Accutouch used to practice myomectomy (removing fibroids from the uterus) and basic hysteroscopy skills. The cost of this system can be obtained by requesting a quote. For the sake of comparison an estimate of cost is needed. The price R System can only be of both the VSOne and Hysteroscopy Accutouch obtained by requesting a quote, therefore making a conservative estimate for both systems, based on the cost of the more basic system from Simulab, would be at least more than $5000 thousand dollars each.. 2.2. System Innovations. The main advantage of this system is the cost. Using inexpensive components, the cost of the system is kept relatively low compared to the commercially available systems. The system is very compact, consisting only of the custom designed hysteroscope, and the accompanying simulation software. It’s also very portable, and can be set up near any modern PC workstation 3. http://iregt1.iai.fzk.de/KISMET/VestSystem.html http://www.select-it.de/ 5 http://www.immersion.com/medical/ 4.

(24) CHAPTER 2. SYSTEM LEVEL DESIGN. 9. as desired. With these advantages, the hope is that such a system would be readily adopted by most medical training institutions and practices.. 2.3. Complete System Description. The system consists of a mechanically modelled hysteroscope and physical anatomy. Different sensors enable a microprocessor to acquire the movements of the scope. The movements as mentioned before are the roll, pitch, yaw and translation of the scope itself, and also the independent roll and translation of the shaft inserted through the scope. The information is transmitted through the USB to a personal computer. The information is then used by the custom designed software to navigate the virtual scope and instrument in the simulated operation. A brief description of the different parts of the system are discussed in the following sections.. Figure 2.1: System overview.. 2.3.1. Mechanical Hysteroscope. The entire mechanical structure is machined from aluminium and consists of a base plate with an attached gimbal structure and a separate probe. The gimbal contains a slot where the probe is inserted. The attached sensors can then measure the pitch, yaw and translation of the probe as it is manoeuvred. The probe itself consists of a hollow tube to allow another shaft to be inserted through it. A sensor attached to the probe can then detect.

(25) CHAPTER 2. SYSTEM LEVEL DESIGN. 10. the translation and roll of the inner shaft. This shaft simulates the instrument inserted through a hysteroscope. Further details about the hardware can be found in chapter 3.. Figure 2.2: Top left: disassembled hardware. Top right: shaft (representing an instrument) inserted into modelled scope. Bottom: modelled scope inserted into gimbal.. 2.3.2. Electronic Hardware. Attached to the gimbal axes are two potentiometers (POTs), measuring the pitch and yaw. On top of the slot where the probe is inserted, is an optical sensor, capable of measuring two axes of motion, and is used to detect the roll and translation of the probe. Another optical sensor is attached to the probe to measure the roll and translation of the inner shaft. The microprocessor polls each sensor in order to gather the data. The information is then transmitted via the on-chip USB transceiver to the PC. A detailed design of the hardware and the different sensors are discussed in chapter 3, while the firmware is discussed in more detail in chapter 4..

(26) CHAPTER 2. SYSTEM LEVEL DESIGN. 11. Figure 2.3: Gimbal with POTs. Left: optical sensor board visible. Right: slot visible.. Figure 2.4: Captured movement.. 2.3.3. Simulation Software. The software aims to simulate a hysteroscopy as closely as possible. The cervix, uterus and an ablation tool are modelled to create a virtual operating environment. The camera in this virtual scene represents the lens of an actual hysteroscope. Movement of the mechanical probe and its inner shaft is reflected by the movement of the camera and the ablation tool in.

(27) CHAPTER 2. SYSTEM LEVEL DESIGN. 12. the software. The simulation of soft tissue, as well as a surgical algorithm for an ablation tool is incorporated into software. To create a more realistic environment, features such as lighting, particle systems and animation were used to enhance the scene. A detailed discussion of the design of the software can be found in chapter 5..

(28) Chapter 3 Hardware Design This chapter will discuss the hardware design for the project in detail. The first section gives a brief overview of the electronic hardware. The rest of the chapter discusses the individual components that were selected and justifies the choices made for the design. Each subsection starts with the requirements that have to be fulfilled. A brief look at the available technology is then given, whereafter the component is chosen. Each subsection ends with an overview of the selected component and its functions. The Mechanical Hardware section discusses the design of the modelled scope and anatomy. The chapter then ends with an overview of the completed and assembled hardware.. 3.1. Electronic Hardware Overview. The hardware for this system is divided into three groups namely: data capture, data processing and data transfer. Data capture is examined in section 3.2. The other two groups are discussed under section 3.3.. 3.2. Sensors. Sensors are used to capture the physical movement of the mechanical scope and convert it into electronic signals. The different requirements, avail13.

(29) CHAPTER 3. HARDWARE DESIGN. 14. able sensor technologies and the sensor selection process is discussed in the following sections.. 3.2.1. Sensor requirements. From the system requirements it was determined that the pitch, yaw, roll and translation of the scope should be detected. With a gimbal structure in mind to capture the pitch and yaw, it was apparent that a rotational sensor would be required to determine the position of the gimbal. The sensor should be able to measure angles up to plus and minus 35◦ (for a total of 70◦ ) as specified by the system requirements. The remaining requirements were the roll and translation of the scope through the gimbal as well as the roll and translation of the instrument shaft through the scope. The roll of the probe can be measured with a linear sensor and then converted to a roll motion in software. The translation and roll can then be grouped as a two dimensional movement. Sensors capable of measuring two axes could then also be considered in the selection process.. 3.2.2. Sensor Technology. There are several types of sensors available that would fulfil the requirements for the measurement of the angles of the gimbal axes, such as rotational position sensors, tilt sensors, inclinometers, digital encoders and potentiometers. There are a multitude of technologies available to measure angular position and movement. For a more detailed description of these technologies please refer to appendix A. With the different available technologies, there exists a wide range of devices suitable for this project. Rotational position sensors, tilt sensors and inclinometers were quickly disregarded since most of these were unnecessarily complex or too expensive for the system being designed. More often than not, they also require external electronics to interface with and therefore the development time would be greatly increased if such sensors were.

(30) CHAPTER 3. HARDWARE DESIGN. 15. chosen. The two main types of sensors left being considered were digital decoders and potentiometers. There were a few digital encoder sensors considered for this project. The main disadvantages were the cost, and also ease of use. Most of these sensors were also too large. There are smaller inexpensive absolute encoders, but it needs 8 pins on the microprocessor to decode the position of the shaft. If two of these sensors are used then the required amount of connections would amount to 16. This amount of connections would require extra logic components to be able to interface with the processor since there are limited connections available on the processor chip. After a comparison of the different sensor technologies, it became apparent that POTs would be the best sensor to use in this situation. Contrary to most of the sensors considered, potentiometers are relatively low cost and unbeaten in their simplicity and ease of use. There are a lot of POTs available on the market. An average size potentiometer with a good lifespan, continuous rotational shaft and good linearity was selected; the VISHAY Spectrol conductive plastic potentiometer. The remaining measurements are translation and roll of the scope, and the instrument inside the scope. The roll could be measured by some of the aforementioned sensors, but translation of the scope would be a little more complex. An optical sensor used in computer mice was suggested. Further study into the sensor indicated that it could measure approximately 400 counts per inch (CPI) and at a rate of 12 inches per second (IPS). It can also capture complete 2-dimensional motion. If this sensor is placed close enough to the measuring surface and if this surface is large enough, then the roll of the surface underneath the optical sensor would be detected as sideward movement. This would result in only two optical sensors needed to measure the roll and translation of both the scope and its inner shaft. The differences between the available LED based optical sensors were size, resolution, speed and power usage. A sensor that fulfilled the requirements of the project with regards to these attributes was chosen: the ADNS-2610..

(31) CHAPTER 3. HARDWARE DESIGN. 3.2.3. 16. VISHAY Spectrol Model 357 Conductive Plastic Potentiometer. Model 357 has a rotational life of 5 million shaft revolutions. The tolerance is specified as 15% with a 1% linearity. It features a 360◦ continuous shaft rotation. A 10k model was selected for this application.. Figure 3.1: VISHAY Spectrol potentiometer.. 3.2.4. Avago Optical Sensor ADNS-2610. The ADNS-2610 is housed in an 8-pin staggered dual inline package (DIP). It is a complete 2D motion sensor, without using any mechanical moving parts. It consists of an Image Acquisition System (IAS), a Digital Signal Processor (DSP) and a two wire serial port. It is designed to be used in conjuction with a lens, LED, and LED clip. The IAS acquires surface images with the help of the illumination system and lens. The images are then processed by the DSP to determine the direction and distance of motion. It uses a single 5V power supply, conforms to USB suspend mode specifications and switches to power conservation mode during times of no movement..

(32) CHAPTER 3. HARDWARE DESIGN. 17. Figure 3.2: Sectional view of typical PCB assembly for the optical sensor.. 3.3. Processing Unit. The central processing unit (CPU) is used to process the captured data from the sensors and transmit the information to the next suitable component. The different requirements and the selection process are discussed in the following sections.. 3.3.1. CPU requirements. From the selection of sensors made, it was determined that the CPU must be able to have at least two inputs for analog-to-digital (A/D) conversion, to accommodate the two POTs. It should have enough input/output (IO) ports to support the connections to the two optical sensors, a few buttons, LEDs and perhaps future extensions. A USB transceiver, either built in or as a separate module is also needed for USB communication.. 3.3.2. CPU technology. There are several processors on the market with built in USB functionality. On board USB functionality means fewer components and easier design, therefore there were two main families of microchips considered during the design stage of the microprocessor. Cypress Semiconductor offers a range of USB functional microcontrollers as well as Microchip Technology who offer 8-bit RISC (reduced instruction set computer) microcontrollers with low-.

(33) CHAPTER 3. HARDWARE DESIGN. 18. and high-speed USB functionality. The only option available from Cypress which satisfied all the requirements mentioned above was the CY7C64215. Microchip’s PIC18 family of microcontrollers also satisfied all the requirements. Both options support full-speed USB, analog-to-digital conversion and general IO pins. Either option would therefore suffice in this design. The PIC18F2550 microcontroller was chosen due to the fact that it was readily available.. 3.3.3. Microchip’s PIC18F2550. The PIC18F2550 is an 8-bit RISC microcontroller with Harvard architecture. It has 32Kbytes of Flash program memory, 2048 bytes of SRAM (static random access memory) and 256 bytes of EEPROM (electrically erasable programmable read only memory) data memory available. There are 24 general IO ports, 10 of which can be used as 10-bit A/D conversion channels. The chip features a USB 2.0 compliant transceiver which supports both low speed (1.5 Mb/s) and full speed (12Mb/s) as well as different types of USB data transfers such as control, interrupt, isochronous (streaming, real-time transfers at a constant rate, or by a specific time, where occasional errors can be tolerated) and bulk transfers.. 3.4. Subsystem Integration. The schematic and printed circuit board (PCB) layout were prepared using Protel DXP. The complete schematic diagram is available in Appendix B. The design consists of two separate boards. One board consists of the optical sensor, together with the lens, LED, LED clip and the required circuitry as outlined by the datasheet. The PIC processor is housed on its own board, together with the supporting circuitry, LEDs, a reset button, programming and USB connectors as well as four connectors for connecting the two POTs and the two optical sensor boards. The board also has two extra connectors for optional exten-.

(34) CHAPTER 3. HARDWARE DESIGN. 19. sions, specifically designed for another POT and a button. The board is housed in its own plastic enclosure for protection.. Figure 3.3: Microprocessor (left) and optical sensor (right) PCBs.. The POTs and optical sensor boards are mounted on the aluminium scope and connected by ribbon cables to the processor’s board. The main board is then connected to a PC via the universal serial bus.. 3.5. Mechanical Hardware. The detailed mechanical drawings were drafted by a mechanical technician. Several specifications were outlined for the design: • A two axis gimbal, capable of measuring at least plus and minus 35◦ on each axis • A mounting space for the two POTs on the two axes • A mounting space for the optical sensor PCB to measure the roll and translation of the scope.

(35) CHAPTER 3. HARDWARE DESIGN. 20. • A place to mount the second optical sensor PCB to measure the movement of the shaft inside the scope PCB measurements as well as the sensor information and photos of an actual hysteroscope were used to guide the technician. Special attention had to be given to the distance between the lens of the optical sensor, and the surface that it had to measure. This would ensure optimal performance by the sensor. After approval was given for the design, the parts were manufactured. For the technical drawing of the scope refer to appendix B.3.. 3.6 3.6.1. Modelled Anatomy Requirements. Since the freedom of movement in the software will be limited to the modelled anatomy, the scope’s movement should be limited as well. Failure to do so would cause the software to become desynchronised with the physical scope, as the one would be able to move freely, while the other is constrained to the virtual anatomy. There are two options considered to solve this problem. The first would be to use haptic feedback on the mechanical scope, but limited time and the complexity of this option hindered it from being a viable solution for this project. The other option was to construct a physical boundary that correlates closely with the anatomy in the software. This option was chosen for its relative ease and time to implement.. 3.6.2. Construction Process. Numerous materials and sculpting techniques exist to construct custom moulds. Modelling clay and fibreglass were used to construct the mould and cast since these are widely available. The clay was used to sculpt the basic uterine form. Average measurements of a normal uterus was obtained by consulting a gynaecologist as well as anatomy textbooks [5][6]. Measurements were adjusted to be 1 cm larger, to accommodate an inner lining of.

(36) CHAPTER 3. HARDWARE DESIGN. 21. sponge material. Once the clay had hardened, it had to be prepared with a releasing agent to prevent the resin sticking to the clay mould. Fibreglass was used to lay over the mould in order to create a cast of the uterus. This cast would then serve as a physical boundary for the scope.. Figure 3.4: The external boundary shell of the fibreglass uterus model.. To add a sense of softness to the modelled uterus, it was lined with a soft sponge material together with a protective membrane to minimize wear and tear.. 3.7. Completed Hardware. Both the scope and modelled uterus are housed in a large enclosure. This serves a double purpose by effectively creating a more professional presentation to the user as well as protecting the hardware from unnecessary handling or damage..

(37) Chapter 4 Firmware Design This chapter discusses the development of the firmware used in the project. It starts with a brief introduction to USB. For a more in depth study of USB please refer to the USB specification1 . An overview of the firmware together with a flow diagram of the main function can be found in section 4.2. The code of the main function is listed in appendix D.3. The chapter concludes with a look at different data formats and calibration issues in section 4.3.. 4.1. A Brief Overview of USB and its Implementation. It has been more than 10 years since the release of the first USB specification in January 1996. April 2000 saw the release of USB 2.0 with the new high-speed option. USB was developed to replace conventional RS232, be easy to use, and support port expansion. Up to 127 devices can be connected to any one USB hub at a time. There are a lot of benefits for users of USB. Firstly it is easy to use, for example: devices are easy to connect and are automatically configured, with no settings required (port addresses, IRQ lines etc.). It is flexible by design, having one interface for many devices. USB is fast and has reliable data transfers. It is also low 1. http://www.usb.org/developers/docs. 22.

(38) CHAPTER 4. FIRMWARE DESIGN. 23. cost and has low power consumption due to power saving circuits and code that automatically powers down peripherals when not in use. Besides the fact that USB is a very popular with users, there are other benefits from a developer’s point of view [2]. USB has four transfer types and three speeds that makes it suitable for many different types of peripherals. USB also enjoys support from all modern operating systems. Microsoft has added class drivers with each release of Windows, supporting human interface devices (mice, keyboards, joysticks), audio devices, modems, cameras, scanners, printers and mass storage devices. There are a wide range of USB devices intended for a wide range of purposes, and this means that implementation details can vary widely. For this reason, the USB specification makes use of classes. In this instance, a class is a group of devices with common attributes or services. Grouping devices together in classes and specifying their characteristics allows for easier development of device drivers and firmware. It is not difficult to make a comparison between a joystick, for use in gaming software, and the mechanical hysteroscope for the simulation software being developed. Thus, for easy integration into the simulation software, the scope is developed as a USB joystick. Joysticks fall under the human interface device (HID) class [3]. As the name suggests, a human interface device, is a device that interacts directly with the user. The device may detect the press of a button or movement of the device. HID drivers included with the operating system will automatically be loaded for the device during enumeration, making the device truly plug-and-playable. After a device is plugged into the USB port, the host PC needs to learn about the device and assign a device driver, after which applications can communicate with it. Enumeration is the initial exchange of information that accomplishes this. It includes assigning of an unique device number (1-127) to the device, reading data structures from the device (descriptors), assigning and loading a device driver, and selecting a configuration based on the options presented in the retrieved descriptors. After this process, the device is configured and ready to transfer data. The descriptor is a data structure which contains information about.

(39) 24. CHAPTER 4. FIRMWARE DESIGN. the device and its properties. This information enables the host to learn about the newly connected device. The USB standard defines a hierarchy of descriptors. Some classes, like the HID class, contains class specific descriptors. For the device to enumerate successfully, the descriptors should be set up correctly [4]. For an explanation of the different descriptors used, please refer to appendix C. All descriptors were set up according to the deDevice descriptor. Configuration descriptor. Interface descriptor. Endpoint descriptor. HID descriptor. Report descriptor. Physical descriptor. Figure 4.1: Hierarchy of USB descriptors.. vice’s parameters. A report descriptor was created, describing the 6 axes, a button, and the size of each. The source code for the USB enumeration was originally supplied by Microchip Technology and ported to C by Bradley Minch. The code was then adapted to the HID device used in this project..

(40) 25. CHAPTER 4. FIRMWARE DESIGN. 4.2. Firmware Overview. On power up, the PIC’s registers are configured. After all the registers are set up correctly the main program loop starts. First the USB module is initialised followed by the A/D converter module. The program then enters a loop until the USB peripheral is enumerated. Once the USB peripheral has successfully enumerated, the final continuous loop is started. At the start of the loop, each sensor is polled in turn to obtain data from it. The two POTs are accessed by performing an A/D conversion on the ports they are connected to. Each optical sensor is then polled twice, once for each axis (x and y). The data acquired by polling is stored in an array of 7 bytes (figure 4.3), and sent via USB to the PC. The loop then repeats itself until the USB cable is disconnected. Begin Main. Initialise USB. Initialise AD Module. No Service USB Requests. Is Peripheral Configured?. Yes. Poll the two AD pins. Send Data via USB. Poll the two Optical Sensors. Figure 4.2: Flow diagram of main program loop..

(41) CHAPTER 4. FIRMWARE DESIGN. 26. Figure 4.3: Format of data sent to the PC (byte 0 left, byte 6 right).. For a more technical look at the workings of the data transfer between the CPU and the optical sensor, see appendix D.1. A more detailed discussion of the A/D conversion can be found in appendix D.2.. 4.3. Calibration between Hardware and Software. During simulation, the position of the physical scope and fibreglass uterus should be synchronised with their respective software counterparts. There are two ways to synchronise the system. The first is to use absolute position information. This method calculates the absolute position of the software counterpart, based on the values received from the sensors. The second method is to use relative values. The position of the physical scope would have to be positioned correctly at the start of the simulation by means of on screen instructions. Once the scope is in position, the hit of a button can start the simulation. The sensor data then contains the relative movement from the previous point and the software positions are updated accordingly. The type of data that is used (relative or absolute), must be specified in the USB report descriptor. This is obviously dependent on the type of sensor used, but the data can be converted to any format in firmware or software as desired. In this project it was decided to calibrate the physical scope at the start of the simulation and to use relative values from the sensors. This method.

(42) CHAPTER 4. FIRMWARE DESIGN. 27. simplifies the use of the received sensor data, by simply applying the relative values to the current mesh position. Data from the POTs were converted in firmware to relative values, while the optical sensor data were in relative format by default..

(43) Chapter 5 Software Design This chapter discusses the development of the surgery simulation software. The chapter starts with an overview of the software, followed by related research and the software requirements. The rest of the chapter discusses the different components that were used, as well as the algorithms that were developed and the special effects employed during the development of the program.. 5.1. Software Overview. The software developed for this projects aims to simulate a hysteroscopy as closely as possible. It makes use of several open source libraries to achieve this goal and is object oriented and written in C++. The program creates a 3D scene with a modelled uterus and ablation tool. It is designed to be used in conjunction with the physically modelled hardware previously discussed and it incorporates soft tissue, ablation and subdivision algorithms together with other effects like animation to create a graphically pleasing surgery environment.. 28.

(44) CHAPTER 5. SOFTWARE DESIGN. 5.2 5.2.1. 29. Related Research A Brief Introduction to Computer Graphics. Computer graphics is a vast field that encompasses any graphical aspect in the field of computer science. Computer imagery has applications in computer-aided design processes, presentation graphics, computer art, education, training, simulation, image processing and entertainment. For the scope of this project we are mainly interested in interactive real-time rendered graphics which represents 3-dimensional objects and scenes, such as those found in the latest gaming software. Real-time rendering [7] is concerned with generating images rapidly on the computer screen. Interactivity implies action or reaction to the image from the viewer. This feedback determines what is generated next. The rate at which these images are generated is measured in frames per second (FPS). To achieve an adequate level of frames per second for interactivity, the use of special graphics acceleration hardware is required. Although hardware dedicated to 3D graphics has been available on professional workstations for a while, it is only in the last couple of years that graphics accelerators have become accessible for the average consumer. With continuing advances in this market, 3D graphics accelerators are becoming standard peripherals for home computers, enabling the use of real-time 3D graphics applications. Different methods and techniques of representing these 3D objects and scenes are discussed in the following section. After a 3D scene has been constructed it has to be displayed on a 2D screen, or rendered. This task is accomplished by the graphics rendering pipeline. As stated, the main function of the pipeline is to generate or render, a 2D image, given a virtual camera, 3D objects, light sources, lighting models, textures, and more. The basic construction of a rendering pipeline consists of three stages: application, geometry and rasterizer. Each of these stages may be divided into smaller linear or parallelized stages. For more detail on the different stages of the rendering pipeline, refer to appendix G..

(45) CHAPTER 5. SOFTWARE DESIGN. 30. Figure 5.1: Graphics rendering pipeline.. 5.2.2. Three-Dimensional Object Representations. The main purpose of 3D computer graphics is to produce a 2D image of a scene or object from a description or model of the object. Several representation methods [8] exist, each with their own advantages and disadvantages. Particular modelling methods have evolved for particular contexts. Representation schemes for solid objects are often divided into two broad categories. The first is boundary representations which describe a 3D object as a set of surfaces that separate the object interior from the environment. Secondly space partitioning representations are used to describe interior properties, by partitioning the spatial region containing an object into a set of small, non-overlapping, contiguous solids. The most commonly used boundary representation for three-dimensional graphics objects is a set of surface polygons that enclose the object volume. A polygon is a closed plane figure bounded by three or more line segments. The line segments are described by connected points or vertices in three-dimensional space. In general, triangles are used most of the time for simplicity. An object is therefore approximated by a set of connected polygons, or a polygonal mesh. Each polygon can then be described by the 3D coordinates of its list of vertices. Another method to define objects is through the use of 3D lines and surfaces, generated from a set of mathematical functions or a set of user specified data points. When functions are specified, the application can project a curve from the defining equation and plot pixel positions along the path of the projected function. Advantages of this method are simplicity of manipulation of the object and minimal storage requirements. A technique for solid modelling is to represent a solid object as a com-.

(46) CHAPTER 5. SOFTWARE DESIGN. 31. Figure 5.2: A polygon mesh.. bination of the volumes of simpler objects using set operations. This is known as constructive solid geometry (CSG). New volumes can be created by applying the union, intersection, or difference operation to two specified volumes. CSG is popular in interactive computer aided design (CAD) software, because they facilitate an intuitive interface for the interactive design of complex industrial objects.. Figure 5.3: Union operation between solids in CSG modelling.. A voxel is a volume element, similar to a pixel, or pictorial element. The purpose of this scheme is to represent volumetric information. Voxel tech-.

(47) CHAPTER 5. SOFTWARE DESIGN. 32. niques therefore aim to represent an object through its spatial occupancy. Sampling devices developed to automate creation of synthetic descriptions of real-life objects produce discrete sets of sampled points. These devices can be used in conjunction with voxel techniques to produce a volumetric representation of an object. Techniques include using a 3D matrix, each. Figure 5.4: A voxel representation of a pig’s head.. cell of which models a small, cubic volume of space. It can then be marked as occupied, in which case the entry in the matrix would be taken up by a value representing some property of that region of space; for instance a colour or density. The spacial occupancy matrix representation requires a lot of storage and is wasteful at times (for unoccupied spaces). An effective alternative to the spacial occupancy matrix is octal trees (also known as octrees), which is a tree data structure with a fanout of eight. In short, a region of space is divided into eight equal portions. All the empty portions in the corresponding leaf nodes are marked with a zero while portions that are completely occupied are marked with a “2”. The partially occupied portions are marked with a “1”. The algorithm is applied recursively for all partially occupied portions at the current level. This technique offers considerable saving in storage, especially for sparsely occupied scenes. Medical.

(48) CHAPTER 5. SOFTWARE DESIGN. 33. imaging and other applications that require displays of object cross sections commonly use these type of techniques.. 5.3. Software Requirements. The encompassing requirements are that the software should be able to accurately simulate a surgical environment with the purpose of using it for training, and also that the simulation should be rendered in real-time. To simulate surgery accurately, this requirement can be broken down into four sub-requirements. Firstly, the probe in the software should mimic the movement of the mechanical probe. Developing the motor skills of the surgeon is very important in training. Secondly, the images presented in the three dimensional scene should look as real as possible. What the surgeon sees on the computer screen should be very close to what he would expect to see in a live operation. Thirdly, the anatomical models should mimic soft tissue when touched. It is no use if the models seem like a solid structure to the surgeon. Lastly, algorithms should be used to simulate the cutting of the tissue accurately. If all these requirements are fulfilled, the software should be able to simulate the procedure accurately, and give the surgeon a feeling of realism when using the simulator. This would greatly improve the training efficiency of the simulator as a whole.. 5.4. Component Library Overview. It would be no use to create 3D content without the proper software to render it with. Computer games or other interactive applications with realtime graphics make use of a game engine [9] as the core software component. The functionality typically provided by a game engine includes a rendering engine for 2D or 3D graphics, collision detection or physics engine, sound, animation, artificial intelligence and networking capabilities. It would be impossible, given the time limitations, to write a game engine from the ground up for this project. Luckily there are open source.

(49) CHAPTER 5. SOFTWARE DESIGN. 34. libraries available that help to accomplish this task. This software could not have been developed if libraries such as these were not used. They enable the developer to focus on the specific problems of the application by providing him with the tools necessary to accomplish the task. By using open source libraries, development time is considerably shortened. The following libraries or software packages were used in the development of the software: • Blender - a 3D content creation package • The Gimp - image manipulation software • Ogre 3D - the main graphics rendering engine • OIS or Open Input System - a direct input wrapper, used to manage the mouse, keyboard and joystick inputs • CEGUI - used to create and manage the graphical user interface • OgreOpcode - used for object collision detection. Each component, including the 3D content creation is discussed in more detail in the following sections.. 5.4.1. 3D Content Creation. Selecting a modelling scheme depends on many factors such as the particular goals of an application, the quality of the images, and the rendering method the application is using. Since most modelling applications, rendering software and hardware are designed and optimised to render polygons, and since speed for real-time interaction is an important factor in the design, it was decided to use the polygon modelling method to model the required anatomy and operating environment. The algorithms simulating soft tissue and cutting will be attempted by manipulating the data structures of the polygon mesh..

(50) CHAPTER 5. SOFTWARE DESIGN. 35. Figure 5.5: A wireframe, solid, and textured uterus polygon model.. When it comes to 3D content creation, there are several options available to choose from. Maya and 3D Studio Max are two well known commercial applications capable of creating high quality 3D content. On the open source side of the market is Blender. Blender can be used for modelling, texturing, animation and rendering of 3D content amongst other things. Blender has a robust feature set similar in scope and depth compared to other high-end 3D software packages mentioned previously. Since Blender is freely available, it was chosen as the modelling software for this project. A 3D model of a uterus was constructed by the same means as the fibreglass counterpart. The ablation tool were modelled using photographs. Textures were created by taking snapshots of video footage and editing these pictures with the help of open source image editing software. After textures were applied to the model it was exported to a format compatible with the rendering software. Calibrating the software with the physical constraints imposed by the fibreglass uterus was achieved by using the same sources for modelling and keeping the scale between the uterus and hysteroscope consistent in both realms. The final step for calibrating the constraints is to simply scale the meshes in software until a satisfying result is obtained..

(51) CHAPTER 5. SOFTWARE DESIGN. 36. Figure 5.6: A wireframe, solid, and textured ablation tool model.. 5.4.2. Graphics Rendering Engine. A freely available rendering engine, called Ogre was used for the rendering of the 3D content. Ogre is an acronym which stands for object-oriented graphics rendering engine. It’s a scene-oriented, flexible 3D engine written in C++. It is designed to make it easier and more intuitive for developers to produce applications utilising hardware accelerated 3D graphics. The class library abstracts all the details of using the underlying system libraries like Direct3D and OpenGL and provides an interface based on world objects and other intuitive classes. One very important reason why Ogre was chosen to be used, was because it provided read and write access to the hardware buffers which contain a mesh’s data. Access to these buffers were necessary to be able to implement the soft tissue, ablation and subdivision algorithms discussed in sections 5.6 and 5.7. Ogre is not a complete game engine, but rather a graphics solution. To use other features like collision detection and physics, other libraries have to be integrated into the application. It is designed this way to provide the.

(52) CHAPTER 5. SOFTWARE DESIGN. 37. Figure 5.7: The constructed 3D scene at the start of the application.. developer with choices regarding the libraries he wishes to use according to his specific requirements.. 5.4.3. Input System. For a system to be interactive, it needs to be able to receive data from a user and react to it in an appropriate way. One way to access input data from the system is to use DirectInput. DirectInput is part of the DirectX library, written by Microsoft. It enables the programmer to access the input data from the user, via the system input devices such as the mouse, keyboard and joystick. An open source library, called OIS, that acts as a wrapper for DirectInput was used in this software. It provides a simple interface for the developer to access DirectInput. OIS stands for object-oriented input system. The main goal of OIS is to shield the application programmer from having to write an input system from scratch for each project. OIS is used in the software to capture input from the keyboard as well.

(53) CHAPTER 5. SOFTWARE DESIGN. 38. as handling the movement of the hysteroscope as input from a joystick.. 5.4.4. Graphical User Interface. Every piece of software needs some form of user interface where the user can change settings, save, load, or exit the application for instance. CEGUI (Crazy Eddie’s graphical user interface) is the open source library used for the GUI system, and provides windowing and widgets to the programmer. CEGUI is written for graphics engines or API’s that don’t have built in GUI support. The library is object-oriented and written in C++.. 5.4.5. Collision System. An integral part of any game or simulation is the need to detect collisions between different objects. This data is essential in simulating physics or in this case, to simulate soft tissue when the hysteroscope collides with the surrounding environment. A library that provided this functionality called OgreOpcode was used in this project. OgreOpcode is a wrapper for Opcode, a small collision detection library, developed by Pierre Terdiman. OgreOpcode itself is thus used as a collision detection library, as opposed to a fully functioning physics library and is specifically designed for Ogre users.. 5.5. Frame Overview. After the application has done the necessary scene setup and data structure initialisation, it is ready to start rendering. As previously mentioned, the rate at which real time graphics is rendered is measured in frames per second. For every frame, a series of operations are performed. A basic flow diagram of the operations can be seen in figure 5.8..

(54) 39. CHAPTER 5. SOFTWARE DESIGN. Frame Started. Input Data Captured via OIS. Update Position of Camera and Ablation Tool. Check for Collisions via OgreOpcode. Collisions Detected?. Yes Apply Soft Tissue Algorithm. No No. Has Tool been activated?. Yes. Render Next Frame via OGRE. Apply Ablation Algorithm. Figure 5.8: Operations per frame.. 5.6 5.6.1. Soft Tissue Simulation Background Study. The two main approaches to simulate soft tissue over the last few years have been to use a mass-spring model and the Finite Element Method (FEM) or variations of these [10]. Both of these methods however apply to volumetric meshes. Since this project examines the use of surface-based models for modelling object deformation, the aforementioned methods were not applicable. Another method, called the 3D ChainMail [11] algorithm, has been.

(55) CHAPTER 5. SOFTWARE DESIGN. 40. designed for volumetric objects, in which deformations on the surface as well as the interior are modelled. There are two main processes involved: the ChainMail process itself which imposes certain constraints on movement, similar to chainmail armour; and a relaxation process where energy minimization is applied to refine the shape. A variation on this technique has been described by Ying [12]; still using the ChainMail idea, but only considering surface elements and not modelling interior behaviour. This method is specifically designed for speed, and shows promising results for soft tissue simulation. Upon researching these last mentioned two techniques, a variation on the ChainMail algorithm was developed, similar to Ying’s SurfaceChainMail algorithm, and implemented to simulate soft tissue.. 5.6.2. Soft Tissue Algorithm. 5.6.2.1. Algorithm Employed. The algorithm uses the same idea as the ChainMail and SurfaceChainMail algorithm, but differs slightly in its implementation. For ease of understanding, the basic workings of this algorithm is explained in two dimensions. Certain constraints are applied to each element. When an element is moved, its neighbours become possible candidates to be moved if the constraints have been violated. This is done recursively in a chain reaction. Using a collision detection library, the closest vertices involved in the collision between the uterus and probe are identified and moved in the direction that the probe is pointing. Starting from this vertex and spiralling outwards, each connected element is checked to see if it violates any constraints. If it does, it is moved towards the centre in the direction of the connected edge until the constraint is met. Its neighbours are then marked to be checked. Any element is checked and moved no more than once. This continues until there are no more vertices that violate the constraints. The softness of the mesh is controlled by the values of the constraints. The effectiveness of this algorithm depends on the resolution or amount of.

(56) CHAPTER 5. SOFTWARE DESIGN. Figure 5.9: The constraints between element V0 and its neighbour V1.. Figure 5.10: Flow of algorithm through mesh.. 41.

(57) CHAPTER 5. SOFTWARE DESIGN. 42. vertices in the mesh. If the mesh contains very few vertices, then there are much less points to move around, creating an irregular effect, but with enough vertices the effect becomes smooth and a good result can be obtained. A good rule of thumb is to have at least three vertices per diameter of the probe’s mesh for an adequate deformation effect. 5.6.2.2. Data Structures Employed. To be able to apply the algorithm described above, information regarding all the vertices, faces and edges have to be available with quick lookup time.. Figure 5.11: The format of the hardware buffers. Every three entries in the index buffer forms a face and contains the different offsets to the appropriate vertices in the vertex buffer..

(58) CHAPTER 5. SOFTWARE DESIGN. 43. At the initialisation stage of the application, the data of the uterus mesh is extracted from the hardware buffers (figure 5.11), analysed and inserted into vector lists. The lists contain the data of all the vertices, edges and faces as well as the connections between the structures. In other words, a vertex entry will contain the offsets to all the edges and faces that it is connected to and vice versa. Given a face, edge or vertex, this structure therefore allows quick traversal between all connected entities and the mesh as a whole. An illustration of the data structures used to contain the mesh’s information can be seen in figure 5.12.. Figure 5.12: Data structure setup..

(59) CHAPTER 5. SOFTWARE DESIGN. 5.7 5.7.1. 44. Surgery Simulation Background Study. A problem with surface models, besides their non-solid behaviour, is the lack of a defined interior when surgical procedures are modelled. This complicates the effectiveness of methods proposed to simulate surgical procedures. The cutting of volumetric meshes is relatively straight forward since a cut can simply be modelled by removing the required connections between the elements. More complex and sophisticated algorithms for volumetric meshes have been developed of course, such as a state machine for realtime cutting of tetrahedral meshes [13]. An article by Voss [14] describes deformation templates for triangular faces for deformation. Elements, or vertices are created in real time depending on the position of the tool relative to the face. New edges are formed to preserve the surface triangulation and this new element is used to transform the surface of the mesh. The cutting part of the algorithm is based on the usage of different templates to create structures generated by the cutting operation of a knife tool. An article by Basdogan et. al. [15] also briefly describes a method for cutting by determining the closest vertices to the line of cut and duplicating them. A polynomial model is used to separate the duplicated vertices from each other to visually expose the cut. Bruyns et. al. [16] also discusses several methods of different interactions with certain tools. They describe interaction with a roller ablator (a type of ablation instrument) by progressively blackening the contacting tissue as well as using blended textures to achieve the desired result.. 5.7.2. Algorithm Employed. For the scope of this project, only one instrument, namely an ablation tool had to be simulated. Unlike a scalpel or knife tool, which creates small line cuts, an ablation tools burns small holes into the tissue. If enough.

(60) CHAPTER 5. SOFTWARE DESIGN. 45. control points exists at the point of the cut1 then a small hole can be formed by pushing the vertices in an outwardly direction. Texturing the hole with an appropriate texture would create the effect that the mesh has been cauterised by the ablation tool. For the algorithm to work effectively, the mesh would have to contain enough vertices for manipulation in the area of the cut. Code was implemented to add vertices to the mesh in realtime. This enabled the algorithm to cut any part of the mesh, regardless of the resolution at that specific point. An invisible sphere is positioned at the tip of the ablation tool. Once a collision is detected and the tool is activated, the ablation algorithm is applied. Firstly the amount of vertices inside the sphere is calculated. If there are enough points available, they are projected outwardly into their new positions onto the surface of the sphere.. Figure 5.13: From top to bottom: probe approaching mesh, soft tissue algorithm applied, ablation algorithm applied.. 1. In the following sections, “cut” refers to the cauterising effect of the ablation tool on tissue.

(61) CHAPTER 5. SOFTWARE DESIGN. 46. If there are not enough control points available to create a good looking hole then all the faces within the spherical area are subdivided. New vertices, edges and faces are added in real time to the mesh and all the corresponding data structures are updated accordingly (see figure 5.15), after which the points are moved into the appropriate position. The subdivision algorithm ensures that the ablation algorithm can be applied anywhere on the mesh, regardless of the polygon resolution at that point.. Figure 5.14: Before and after new vertices are added to the mesh.. 5.7.3. Texturing a cut. To complete the effect, the hole has to look like ablated tissue. There are two methods employed to achieve the desired effect. The first one is to blend the cut with the appropriate ablated tissue texture by using a method called render-to-texture. The second method is changing the vertex colours. Vertices have certain attributes that can be manipulated such as their position, normals and texture coordinates. There is also a vertex colour attribute and this colour is blended together with the texture that is applied to the mesh. By adjusting the vertex colour with regards to the time it is in contact with the ablation tool, it can be continually darkened, going from yellow to brown to black for instance..

(62) CHAPTER 5. SOFTWARE DESIGN. Figure 5.15: Data structure after subdivision.. 47.

(63) CHAPTER 5. SOFTWARE DESIGN. 5.8. 48. Special Effects. The term special effects is relative in the field of computer graphics. In this section, special effects refer to techniques that are employed to create a more immersive atmosphere or realistic look to the 3D scene. Techniques that were researched or employed are discussed in the following subsections.. 5.8.1. Animation. To simulate rapidly moving tissue on the walls of the uterus, as seen in the video footage, some animation had to be incorporated into the software. To make use of animation, a mesh had to be animated in Blender and then exported to Ogre. A plane, or grid of vertices, was constructed in Blender. The next step was to attach a skeleton, or armature to the mesh. This is called rigging. Each bone is then associated with certain groups of vertices. To create an animation sequence the position of the armature is manipulated at certain keyframes. The timeline is advanced and another keyframe is inserted with a new pose of the armature. The user can then select the way in which Blender interpolates between the keyframes. Once a sequence is set up it can be exported into a compatible format for use in Ogre. This method of animation is known as skeletal animation.. Figure 5.16: A textured plane with armature visible. Left: at rest. Right: posed for a keyframe..

No results found