• No results found

Model-based interface framework : an extensible framework for automatic generation of test cases and wrappers

N/A
N/A
Protected

Academic year: 2021

Share "Model-based interface framework : an extensible framework for automatic generation of test cases and wrappers"

Copied!
151
0
0

Bezig met laden.... (Bekijk nu de volledige tekst)

Hele tekst

(1)

Model-based interface framework : an extensible framework

for automatic generation of test cases and wrappers

Citation for published version (APA):

Martinez Marquez, A. D. (2016). Model-based interface framework : an extensible framework for automatic generation of test cases and wrappers. Technische Universiteit Eindhoven.

Document status and date: Published: 28/09/2016 Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:

www.tue.nl/taverne

Take down policy

If you believe that this document breaches copyright please contact us at: openaccess@tue.nl

providing details and we will investigate your claim.

(2)

Model-based interface

framework

Aldo D. Martinez

September 2016

(3)

Model-based interface framework

Aldo D. Martinez

September 2016

(4)
(5)

Model-based interface framework

An extensible framework for automatic generation of test cases and wrappers

Eindhoven University of Technology Stan Ackermans Institute / Software Technology

Partners

FEI Company Eindhoven University of Technology

Steering Group Andrei Radulescu Paul Janson Tim Willemse

Date September 2016

Document Status Public

(6)

Contact Address

Eindhoven University of Technology

Department of Mathematics and Computer Science

MF 5.097b, P.O. Box 513, NL-5600 MB, Eindhoven, The Netherlands +31402474334

Published by Eindhoven University of Technology Stan Ackermans Institute

Printed by Eindhoven University of Technology

UniversiteitsDrukkerij

SAI report no. Eindverslagen Stan Ackermans Instituut ; 2016/043

Abstract Automation through model-based development is becoming more and more popular between companies with big software systems. Due to the innate complexity of the systems a

significant amount of time has to be invested for the development activities, such as,

maintenance, updating, and testing. These activities imply repetition and tend to have an error prone nature. This report describes a model-based framework project for modeling software interfaces with a high-level homogenous representation that can be later used for the

automatic generation of test cases and wrappers. A formal modeling language is defined based on the analysis of implemented software interfaces at FEI focusing on the description of the main elements and behavior of an interface and removing technology-related values. A testing algorithm is proposed in which test cases are generated based on state transitions of the interface model, described through a state machine. All these features have been implemented into a prototype that demonstrates the feasibility of testing and wrapping automation.

Keywords Model-based development, state machine testing, behavioural testing, model-based testing, black-box testing, software quality, wrapping technologies, Domain Specific Languages, code generators, modeling languages, testing automation, model transformations, software technology, PDEng

Preferred reference

Model-based interface framework: An extensible framework for automatic generation of test cases and wrappers. , SAI Technical Report, September 2015. (Eindverslagen Stan Ackermans Instituut ; 2016/043)

Partnership This project was supported by Eindhoven University of Technology and FEI Company. Disclaimer

Endorsement

Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the Eindhoven University of Technology or FEI Company. The views and opinions of authors expressed herein do not necessarily state or reflect those of the Eindhoven University of Technology or FEI Company, and shall not be used for advertising or product endorsement purposes.

Disclaimer Liability

While every effort will be made to ensure that the information contained within this report is accurate and up to date, Eindhoven University of Technology makes no warranty, representation or undertaking whether expressed or implied, nor does it assume any legal liability, whether direct or indirect, or responsibility for the accuracy, completeness, or usefulness of any information.

Trademarks Product and company names mentioned herein may be trademarks and/or service marks of their respective owners. We use these names without any particular endorsement or with the intent to infringe the copyright of the respective owners.

Copyright Copyright © 2016. Eindhoven University of Technology. All rights reserved.

No part of the material protected by this copyright notice may be reproduced, modified, or redistributed in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the Eindhoven University of Technology and FEI Company.

(7)

Foreword

Within FEI, we partition the software into server software, providing the microscope control and safety, and applications, implementing advanced functionalities and user interfaces. We formalize the server interfaces as the contract between the server and the applications. We have defined this project to automate the tedious, time-consuming activities around the interfaces:

─ Tests, ensuring the interface correctness while the software evolves, and ─ Wrappers, bridging the technologies boundaries between the server (C++),

the interface (COM) and the applications (C++. C#, python).

Fortunately, Aldo has shown interest in this project and has picked it for his graduation thesis. After an initial exploration phase, we have chosen to develop our own language to have the full flexibility of concisely covering our requirements. Aldo has quickly become familiar with both the FEI server technologies, and xtext / xtend, in which the Model-Based Interface Framework has been developed. He has created a language for modelling interfaces, capturing both their semantics and behavior. Further, he created language extensions targeting the specifics of testing and wrapper generation. Aldo has created a test framework providing full functional coverage. The wrapper generators can cover not only technology translations, but also other interface artifacts, such as multi-client arbitration or tracing.

We have run the project in an agile manner. Aldo has constantly kept a running version of the framework, over which we have iterated multiple, defining and (re)prioritizing tasks as we have learned more. We have aimed at maximizing the types of features we could cover with the framework, and applying them to concrete server interfaces with increased level of complexity. Aldo ensured the project run smoothly, was able to adapt to changing requirements, and successfully tackled all challenges arising over time.

The results are very promising. We have involved a team in a test trial, which has shown that the framework is ease to use and complete enough to enable modelling and testing interfaces in a short time. A few missing features were identified, which can easily be added to the framework. Surprisingly, the main challenge faced during the trial was actually not related to the framework, but to modelling and testing the behavior of one of their own legacy interfaces.

Within FEI, we plan to further extend the framework, make it part of our build and test processes, and use it by all our teams. With the increased productivity and increased test coverage it provides, we believe it will quickly be adopted by our teams, which will be able to spend more time on features and to guarantee the quality of their interfaces.

PROJECT MANAGER Dr. Andrei Radulescu September 2016

(8)
(9)

Preface

This document summarizes the “Model-based interface framework: An extensible framework for automatic generation of test cases and wrappers” project. The project addresses the challenges of testing and wrapper automation using model-based technologies and applying them into a high-complex software system. The project has been executed by Aldo D. Martinez from the Stan Ackermans Institute, Software Technology Programme of the Eindhoven University of Technology. This project is the nine-month final assignment for the aforementioned two-year Profesional Doctorate in Engineering (PDEng) programme, known by its duch name as Ontwerpers Opleiding Technische Informatica (OOTI). This project has been implemented within FEI Company in Eindhoven.

This document is primarily intended for readers with a technical background in disciplines, such as automation, model-based techniques, testing, wrapping, domain specific languages, derivative text-based languages, and general software engineering. However, no specialized knowledge in these disciplines is needed. For readers with a non-technical background or those who are interested in knowing the basis and results of this project, they should read chapters 1 to 5. These chapters introduce and explain the essential points in which the system is founded and we were aiming with this project.

Readers with an interested in the implemented solution, how it was designed, planned, and the main development decisions, should read chapters 6 to 10. These chapters cover the requirements, high-level architecture, solution’s design, and the project verification and validation. In specific, the chapters 7 to 9 include the main topics regarding the implementation of a domain specific language as a model-based solution for automating testing and wrapper generation.

Additionally, chapters 12, 13, and 14 cover the conclusions and achieved results through the project, the project managements, and a retrospective from the perspective of the author.

Aldo D. Martinez September 2016

(10)
(11)

Acknowledgements

I heartily want to express my gratitude to all the people that helped me, guided me, led me, and supported me during this project. Without them, the success of this project would not be possible.

First of all, I am deeply grateful with the people that made this project possible from FEI Company. Special thanks are given to Andrei Radulescu and Paul Janson, my supervisors and mentors at FEI. They made possible my participation in the company and guided me throughout the whole project. Their support, enthusiasm, and collaboration have been essential for the success of this project. Their valuable advices and opportune suggestions helped me to further develop myself professionally. I would also like to thank to all the FEI engineers that supported me and collaborated with the project during these nine months. Martijn Kabel and Paul Van Gorp helped me during my integration to the company. David Van Luijk, Tim Bertholet, Santosh Zope, Vasily Galkin, Vladimir Ivin, Laurens Oosterhof, and Davy Kox supported me and facilitated material and knowledge for understanding FEI interfaces. Andrea Pasqualini, Peter van Merkerk, Erwin de Groot, and Joost Dierkse introduced to me the current wrapper technologies and implementations.

I would like to expresses my thankfulness to my university supervisor Tim Willemse who always gave pertinent advices and valuable feedback of my work. With his expertise in model-driven and testing, he opened my vision to new topics and subjects that I added to my work in order to improve it. Our communication was excellent and he was always there to answer my questions. I am deeply grateful to a lot of people within the OOTI program and TU/e that supported me during the last two years. Special thanks to Ad Aerts, Maggy de Wert, and Desiree van Oorschot, and all the OOTI coaches for all the advices, reviews, and fruitful talks.

Furthermore, I would like to thank my friends from the OOTI program, during the past two years we shared experiences, culture, knowledge, and we also helped each other to make this journey a success.

Last but not least, I have been able to reach this point in my life thanks to my family. They have always given me continuous support to follow my dreams and make them possible.

Aldo D. Martinez September 2016

(12)
(13)

Executive Summary

Repetitive activities tend to be tedious, mechanical, and error prone nature. Additionally, these activities consume a significant time, decreasing development efficiency. Removing repetitive activities is not an option when these activities are related with essential elements of quality (creating test cases) and code interoperability (creating wrappers). The recent advances in automation and model-based technologies present an alternative to reduce time consumption and ensuring the quality of the produced artifacts of these repetitive activities. This project focuses on investigating and prototyping a possible solution in the domain of automation of test case and wrapper generation through model-based technologies.

Ensuring interface behavior and avoiding repetitive activities is of great importance to FEI. FEI invests significant time in manually creating tests for validating the interfaces behavior. In this project, we have investigated the feasibility of automating the test generation by using a model-based approach. We have conducted a study of the FEI IOM interfaces in which we have analyzed interface behavior and testing techniques, then we have implemented our results into a prototype that automatically generates tests based on a behavioral model of the interface. We have developed an algorithm that generates test cases for an interface depending on the modeled interface behavior. The model contains a state machine and a simple interface description.

On the code interoperability domain, we have studied the current implemented approaches for wrappers at FEI and have defined an automation process. The automation process requires a wrapper model which is defined by the wrapping technology (boost.python and COM-ATL have been researched). This wrapper model together with the simple interface model allow us to generate wrapper files ready to be compiled and used within the FEI software system.

This report presents the results of the previous two subjects (testing and wrappers) through a model-based solution that is extendable, easy to use, and flexible. This is achieved by the implementation of a component-based architecture with a modular separation of concerns keeping a clear distinction between the modeling language and the code generators. For modeling the interface and artifact generation, we have used Xtext/Xtend, a technology that is becoming the de-facto standard for model-based frameworks in academia and industry.

The prototype is a first instance of a framework that validates the automation of test case and wrapper generation by the definition of models (interface and artifact). The models have proven to be easy to be read and created in the initial trials (with FEI developers). The generated artifacts (test cases and wrappers) are easily integrated and validated within the FEI code base.

As a prototype and a first approach for an automation framework, the results are very promising and the investigation can now move to the next phase, which is the extension of the framework for covering a larger number of interfaces and engineering to improve the current features of the solution.

(14)
(15)

Table of Contents

Foreword ... i

Preface ... iii

Acknowledgements ... v

Executive Summary ... vii

Table of Contents ... ix

List of Figures ... xiii

List of Tables ... xv

1.

Introduction ... 1

1.1

About FEI ... 1

1.2

Transmission Electron Microscope ... 1

1.3

Model-based interface framework ... 2

1.4

Model, behavior, and test-based development ... 3

1.5

Outline ... 4

2.

Stakeholder Analysis ... 5

2.1

FEI ... 5

2.2

Eindhoven University of Technology (TU/e) ... 6

3.

Problem Analysis ... 9

3.1

Complex software systems ... 9

3.2

Ensuring software quality through testing ... 10

3.3

Modeling interfaces ... 10

3.4

Black-box testing ... 11

3.5

Problem statement ... 11

3.6

Design Opportunities ... 12

4.

Domain Analysis ... 13

4.1

Microscope software system architecture ... 13

4.2

Instrument Object Model ... 14

4.3

COM interfaces ... 14

4.4

Synchronous versus asynchronous ... 14

(16)

4.6

Interface behavior ... 16

4.7

Using interfaces ... 17

5.

Feasibility Analysis ... 19

5.1

Challenges and alternatives ... 19

5.1.1. Model abstraction level ... 19

5.1.2. Standardizing testing and wrapping ... 19

5.1.3. Project deployment ... 19

5.1.4. Technology support for project requirements ... 20

5.1.5. Structural difference in the interfaces ... 20

5.1.6. Automation of repetitive work ... 20

6.

System Requirements ... 21

6.1

Requirement gathering process ... 21

6.2

Main requirements ... 21

6.2.1. Main use cases ... 23

6.2.2. Feature 1: Modeling interface definition ... 25

6.2.3. Feature 2: Test case generation ... 26

6.2.4. Feature 3: Interface wrapper generation ... 26

6.2.5. Feature 4: Providing interface documentation ... 27

6.3

Non-functional requirements... 28

6.3.1. Extensibility ... 28

6.3.2. Ease of use ... 29

6.3.3. Flexibility / Configurability ... 29

6.3.4. Effectiveness ... 30

7.

System Architecture ... 31

7.1

High level elements ... 31

7.2

Modeling approach decision ... 32

7.3

Component-based architecture ... 33

7.4

The modeling language ... 34

7.5

The editor ... 36

7.6

The parser ... 36

7.7

The generator ... 37

7.7.1. The artifact generators ... 38

7.8

The transformation view... 38

8.

System Design ... 41

8.1

Introduction ... 41

8.2

The metamodel layer ... 42

8.2.1. Grammar ... 42

8.2.2. Process view ... 47

8.2.3. Technology perspective ... 50

8.3

The interpretation layer ... 51

8.4

The generation layer ... 51

8.4.1. The testing algorithm ... 51

(17)

8.4.3. The generator ... 54

8.5

Layer interactions ... 58

9.

Implementation ... 61

9.1

Introduction ... 61

9.2

Modeling ... 61

9.2.1. Core modelling language aspects ... 62

9.2.2. State machine implementation ... 63

9.2.3. The custom method implementation ... 63

9.2.4. Visualization of the modeling feature ... 64

9.3

Artifact generation ... 66

9.3.1. Artifacts technologies ... 66

9.3.2. The generation gap pattern ... 66

9.3.3. Testing implementation ... 67

9.3.4. Wrapper implementation ... 73

9.4

Executing artifacts ... 73

10.

Verification & Validation ... 77

10.1

Introduction ... 77

10.2

Testing with FEI interfaces... 77

10.3

Trials with end users ... 80

10.3.1. Quality requirements ... 81

10.4

Capabilities and limitations... 82

10.4.1. State machine transition complexity... 82

10.4.2. Asynchronous calls ... 83

10.4.3. Time performance of produced artifacts ... 83

10.4.4. Maturity of testing ... 83

10.4.5. Interface implementation differences ... 83

10.4.6. Microscope configurations ... 84

10.4.7. Model limitations... 84

10.4.8. Logging and error detection ... 84

11.

Deployment ... 85

11.1

Deployment view ... 85

11.2

Deployment plan within FEI... 86

11.2.1. Deploying for usage ... 86

11.2.2. Updating and providing maintenance ... 86

11.2.3. Integrating within FEI build server ... 87

12.

Conclusions ... 89

12.1

Results & lessons learned ... 89

12.2

Future work ... 90

13.

Project Management ... 93

13.1

Way of working ... 93

13.2

Work-Breakdown Structure ... 93

(18)

13.4

Risk Analysis ... 95

13.5

Project execution ... 96

14.

Project Retrospective ... 99

14.1

Reflection ... 99

14.2

Design opportunities revisited ... 100

Glossary ... 103

Bibliography ... 107

References ... 107

Additional Reading ... 108

Technology references ... 108

Appendix A: Model-based approaches ... 109

Proposal no. 1 – Pros, cons, and effort activities ... 114

Proposal no. 2 – Pros, cons, and effort activities ... 115

Appendix B: The Technology decision... 118

Appendix C: The modeling language grammar ... 120

Appendix D: Model examples ... 125

(19)

List of Figures

Figure 1 – FEI current microscope product portfolio ... 1

Figure 2 – FEI Transmission Electron Microscope ... 2

Figure 3 – Internal composition of a TEM column... 9

Figure 4 – General view of the system under test ... 11

Figure 5 – High-level representation of microscope software architecture... 13

Figure 6 – Microscope PC representation ... 13

Figure 7 – Server exposing IOM interfaces of specific instruments ... 14

Figure 8 – Synchronous and asynchronous example ... 15

Figure 9 – Example of a COM Event interface ... 16

Figure 10 COM interface definition – methods and state enumeration ... 16

Figure 11 Common workflow for using an interface ... 17

Figure 12 Main use cases ... 22

Figure 13 Main features for the MBIF project ... 25

Figure 14 Main elements derived from the requirements ... 31

Figure 15 Component-based architecture for our project ... 35

Figure 16 Modeling language component ... 36

Figure 17 Editor component ... 37

Figure 18 Parser component ... 37

Figure 19 Generator component ... 39

Figure 20 Layered view architecture ... 40

Figure 21 Layered view architecture ... 41

Figure 22 Main elements in modeling grammar... 43

Figure 23 Interface grammar main structure and enumerations: ... 44

Figure 24 Testing grammar main structure and enumerations ... 46

Figure 25 Wrapper grammar main structure and enumerations ... 46

Figure 26 Documentation grammar main structure and enumerations ... 47

Figure 27 Sequence flow for textual model generation ... 48

Figure 28 Internal structure of the modeling language package – Validator class only shows two examples of validation rules ... 49

Figure 29 Creating the semantic mapper ... 50

Figure 30 Technology chosen for the metamodel layer ... 50

Figure 31 Interface model (defined using our grammar) including a state machine definition (visually represented in the right). ... 52

Figure 32 Additional transitions (red) that are not described in the interface model. ... 53

Figure 33 Resulting test table. ... 53

Figure 34 Wrapper process activities ... 54

Figure 35 Generator package structural view ... 56

Figure 36 Sequence diagram presenting the interaction in the generation layer for test case generation ... 57

Figure 37 File template sections ... 59

Figure 38 Full sequence for generating test cases including the three layers ... 60

Figure 39 Example of the implementation of an interface state machine ... 64

Figure 40 A fragment of the CMOS protector interface model in which custom methods are defined and then used in the state machine transitions ... 65

Figure 41 Example of Eclipse IDE editor showing feedback to the end user ... 65

Figure 42 Generation gap pattern explained by applying inheritance. ... 67

Figure 43 The process of creating test cases for a specific testing framework. ... 67

Figure 44 The logic behind the parameterized testing strategy. In this case specific definitions are provided by our test table in the form of methods/triggers to execute. ... 68

Figure 45 Example of the skeleton of a test suite implementation. ... 70

Figure 46 Example of the skeleton of an interface handler. ... 71

Figure 47 Test case execution workflow – IStemDetector example . ... 72

(20)

Figure 49 The global view of the process for artifact execution ... 75

Figure 50 Test suite execution for the IStemDetector interface ... 79

Figure 51 Deployment view of our solution prototype from the usage perspective. ... 85

Figure 52 The incremental-iterative approach. ... 93

Figure 53 Work-breakdown structure of the project. ... 94

Figure 54 Global view of the 9-month period planning and high-level activities. ... 95

Figure 55 General view of the main components that our solution required. ... 109

Figure 56 Our solution deployed with a generic model-based tool. A generic model-based tool is the kind of tool that only generates implementation code from a common model. ... 110

Figure 57 Our solution deployed with a DSL approach. In this case the DSL extends our solution to develop generators for specific artifact generation. ... 111

Figure 58 Our solution deployed with a model-based testing tool. The tool takes responsibility for generating and executing test cases. ... 112

Figure 59 Solution proposal no. 1: DSL + MBT tool approach. The DSL takes responsibility for interface modeling and artifact generation regarding documentation and wrappers. The MBT tool takes care of creating, generating, executing, and reporting test cases. ... 113

Figure 60 Proposal no. 2: The DSL approach. All the modeling and code generation is covered by developing a DSL while test compilation, execution, and result management is a independent process led by the end user. Accepted proposal. ... 117

Figure 61 Comparison table between DSL tools. ... 119

Figure 62 Main interface elements. ... 120

Figure 63 The state machine grammar. ... 121

Figure 64 The operations grammar. ... 122

Figure 65 The testing grammar. ... 123

Figure 66 The wrapper (python wrappers) grammar. ... 124

Figure 67 The ISource interface. ... 125

Figure 68 A test model for the ISource interface. ... 126

Figure 69 A python wrapper model for the ISource interface. ... 126

Figure 70 The IDetector interface. ... 127

(21)

List of Tables

Table 1 – High-level project management ... 5

Table 2 – Project management group ... 5

Table 3 – Knowledge sources – technical group – potential users ... 6

Table 4 – TU/e Stakeholders... 6

Table 5 – Model interface use case specification ... 23

Table 6 – Model behavior use case specification ... 23

Table 7 – Model test settings use case specification ... 23

Table 8 – Model wrapper settings use case specification ... 23

Table 9 – Generate test cases use case specification ... 23

Table 10 – Generate wrappers use case specification ... 24

Table 11 – Generate documentation use case specification ... 24

Table 12 Functional requirements for the modeling interface definition feature . 25 Table 13 Test case generation functional requirements ... 26

Table 14 Interface wrapper generation requirements ... 27

Table 15 Providing up-to-date and complete documentation requirements ... 27

Table 16 – Extensibility requirement description ... 28

Table 17 – Ease of use requirement description ... 29

Table 18 – Flexibility / Configurability requirement description ... 29

Table 19 – Effectiveness requirement description ... 30

Table 20 – Elements to model through the DSL language ... 62

Table 21 – Testing the solution for generating test cases ... 77

Table 22 – State machine coverage results ... 78

Table 23 – Testing the solution for generating wrappers ... 79

Table 24 – Testing the solution with end users ... 80

Table 25 – Global results of the trials with end users ... 80

Table 26 – Testing the solution results from the quality requirements... 81

(22)

1.Introduction

This chapter introduces FEI as a company and Transmission Electron Microscopes (TEM) produced by them, followed by the context definition in terms of methodologies and approaches related to the project. A brief explanation of how these elements match the FEI situation is introduced. The next section (1.4) states the main goal and high level objectives for this project. Finally, the outline section gives a brief overview of what is discussed in the rest of the document.

1.1

About FEI

FEI products in the microscopy field are primarily scanning electron microscopes (SEM), transmission electron microscopes (TEM), as well as focused ion beam (FIB) and dual beam (FIB/SEM) systems. All FEI products are employed in a wide range of applications that contemplate areas such as materials science, semiconductors, oil and gas, life sciences, minerals and mining, as well as industrial manufacturing [1].

Figure 1 – FEI current microscope product portfolio

FEI is a leading technology company that designs, manufactures and supports transmission-electron microscopes providing ultra-high resolution to sub-Angstrom levels. Software plays a key role in controlling and ensuring a safe operation of the microscope, and in creating a range of workflows and applications. FEI’s vision is creating nanoscale imaging and analysis workflows that enable existing and new customers to innovate in science, technology, and production.

1.2

Transmission Electron Microscope

A TEM is a type of microscope that injects electrons into a sample (specimen) onto a fluorescent screen. The specimen usually has to be specially prepared and held inside a vacuum chamber from which the air has been pumped out. A TEM produces a high-resolution, black and white image from the interaction that takes place between prepared samples and electrons in the vacuum chamber.

(23)

Figure 2 – FEI Transmission Electron Microscope

A TEM is integrated from several different hardware elements. The software implementation for controlling these hardware elements is managed by a server. This server offer interfaces as a method for usage of the hardware and software functionalities. This project intends to work with those interfaces. The following are the representative set of interfaces employed during the project:

Stem detectors: Detect direct or scattered electrons bounced from the

specimen surface.

Field emission gun: Provides an intense beam of high energy electrons.

Cmos protector: Protects a sensitive camera from being exposed to a direct

electron beam with a high dose.

Vacuum chamber: Creates and supports the vacuum inside the microscope

to minimize the interaction between the electrons in the beam and particles in the column that are not part of a specimen

The FEI software server has a complex structure that contains the required elements to control the configurable hardware elements and to offer functionality needed by the applications running on the microscope. To achieve high quality of the features offered by the microscopes, testing is crucial in the software process.

Microscopes by themselves are extremely complex machines containing high cost hardware. It is mandatory that the interaction between software and hardware works in a manner that ensures proper functioning with a minimum risk of damage to the hardware. Effective and efficient testing of the software is an absolute necessity to guarantee reliability.

1.3

Model-based interface framework

Testing and automation are the main keywords for this project. Currently unit and system tests are written manually. Unit tests are run by the build server during continuous/nightly builds. System tests are invoked on a selected configuration running on virtual machines that simulate a microscope to allow executing tests that depend on specific behavior. Testing is a process that happens after interface implementation and is not always performed by the developer who coded the interface thereby increasing the complexity for test definition due to miscommunication in the interface functioning. An inherent issue with manual written tests is that having a full coverage of an interface behavior becomes complex as the interface extends its interaction with other parts of the system. Even more, maintaining a manually written test can escalate and be costlier as long as the system grows. Nevertheless, all interfaces share a common point, the definition of a state

(24)

machine that describes the expected behavior of the interface. Having this state machine as an input, FEI envisions automating test case generation.

A typical situation in high complex systems is the lack of or out-of-date documentation. Especially in systems in constant evolution and personnel changes, the necessity of updating the existing documentation is a high priority. This is a second process in which automation is essential.

Finally, in complex systems it is often required to make the functionality available to different technologies and programming languages to support a variety of clients. Adapters and wrappers are software elements that make achieving this goal possible. However, creating different adapters is a repetitive activity. In addition, existing adapters are a maintenance responsibility, which can be removed by automation. The model-based interface framework project (MBIF) aims to improve the FEI interfaces quality by:

 Guaranteeing its behavior through testing

 Providing up-to-date and complete documentation

 Generating wrappers automatically to other target languages

A parallel goal of the project is to demonstrate the feasibility of using specialized automation technology oriented in model-based development in order to achieve the previously introduced main goals.

1.4

Model, behavior, and test-based development

The MBIF project is based on the combination of existing methodologies applied to the specific FEI software infrastructure. Those methodologies are:

Model-based software development (MBSD)

A model is an abstraction of a system [4] and the key element in MBSD approaches. MBSD is focused on achieving code reuse as well as performing maintenance and development by employing software modeling technologies. Models are created to serve particular purposes and usually represent a simpler human understandable representation of a system or a software component. One of the goals of MBSD is the automatic generation of code (including source code, wrappers, and adapters), documentation, and other artifacts according to the model description.

Behavior-based testing (BBT)

This is a specialization of test-driven development that focuses on monitoring specific expected interactions [5]; these interactions occur when certain methods/functions are executed and can involve multiple objects. BBT has been developed as an option to validate that a system is behaving as expected. As a result, BBT is strongly centered on testing an existing implementation by describing steps/sequences and their expected output [6].

Model-based testing (MBT)

This approach is derived from model-based techniques and covers the modeling, designing, generating, and executing artifacts to perform software or system testing [7]. In this approach, the system under test (SUT) is modeled (a part of) including the behavior and testing strategies/environment. MBT aims to keep development and testing time in balance by automating the test generation/execution [8]. MBT incorporates different testing strategies that allow a variety in the test case generation for a SUT.

All these techniques aim to automate code/artifact generation to maximize coverage, and, thus, ensure code quality, with minimal effort. Even when they are relatively new in the field, they are becoming more and more popular amongst complex and/or big software systems. Commonly, for these software systems, developing new code

(25)

or testing existing code is a hard task due to the interconnections that are created as the amount and complexity of code increases. Therefore, a model-based solution will speed up the development process and will increase the understanding of the system by adding an abstraction level based on the domain of the system.

1.5

Outline

The next chapter introduces the main stakeholders from the involved parties (TU/e and FEI) and gives a brief analysis of their role in the project. Their goals, interests, and influence towards the project are presented. Following this analysis, Chapter 3 states the problem analysis that describes in detail the aspects and expectations of the project. After this, (Chapter 4) a domain analysis is presented in which an inquiry of the current software situation at the company is conducted in order to define how our project fits and to define boundaries and limitations.

The next two chapters go into the formal requirements of our system. Chapter 5 presents a feasibility analysis of our problem by defining potential issues and risks that may arise during the lifespan of the project. Mitigation and contingency strategies are also documented. Chapter 6 states the formal system requirements for the project based on the company’s requirements and the performed analysis in previous chapters. Functional and non-functional requirements are defined in this chapter.

The following three chapters offer a view of the solution for this project. First, Chapter 7 addresses the system architecture that contains all the high level elements developed and included for our solution. Chapter 8 shows the system design based on the previously chosen architecture and finally, Chapter 9 goes into the specific implementation details of our solution. This chapter also explains the links between architecture, design, and implementation. Design decisions regarding technology and software are also covered.

Chapters 10, 11, and 12 are mostly focused on the results of the projects. Chapter 10 discusses the deployed methods for validation and verification of our solution and how they connect the original requirements and expectations. Chapter 11 elaborates on the deployment plan for our solution and Chapter 12 presents the conclusions and results of this project.

The last two chapters are focused on the management of the project. Chapter 13 addresses the executed process during the nine months of the project and Chapter 14 relies on the project retrospective and reflection from the author’s point of view.

(26)

2.Stakeholder Analysis

In this chapter, we present stakeholders involved in the project and their interests and goals. As a starting point, we have FEI Company and Eindhoven University of Technology as the parties involved, each of them with specific interests towards the project. The following sections present each party in detail and introduce the stakeholders and people involved.

2.1

FEI

FEI as a company is the owner and initiator of the project. They represent the source of knowledge, requirements, and expectations for this work.

FEI aims at ensuring a high quality of their software interfaces because interfaces play a key role providing access to applications for controlling the microscope and acquiring images.

To minimize the effort of creating and maintaining tests, wrappers, and documentation, FEI envisions employing a model-based technology to cover this part of the development process.

The company stakeholders can be grouped in three categories:

1) High-level project management: These are stakeholders with enough influence to make significant changes during the project.

Table 1 – High-level project management

Name Role

Martijn Kabel

Platform Software Manager

Monitoring progress of the project and planning project future and deployment.

Paul van Gorp

Control Server Software Manager

Monitoring progress of the project and providing support for mitigating risks and threats.

2) Project management group: Members of the company that influence the project directly. Requirements, scope, modifications, boundaries, and functionalities were defined with them.

Table 2 – Project management group

Name Role

Andrei Radulescu Software Technical

Lead

Key stakeholder with the role of project supervisor and owner. Managing project progress and defining priorities for requirements. As a software architect, he contributes with knowledge and defines the paths to follow during the project development. Active Progress Steering Group (PSG) member and involved in major decisions. Paul Janson

Software Engineer

Project mentor. Responsible for monitoring daily progress of the project and connecting with other related

stakeholders. Provided expertise in testing topics and influenced daily decisions about the direction of the project. Active PSG member and involved in major decisions.

(27)

3) Knowledge sources – technical group – potential users: This group covers stakeholders who provide information during the research phase about FEI interfaces and topics related to the project, as well as testers of the project and potential final users.

Table 3 – Knowledge sources – technical group – potential users

Name Role

David van Luijk

Detectors Software Tim Bertholet FEG Software Santosh Zope Optics Software Vasiliy Galkin Server Software Vladimir Ivin HT Software Laurens Oosterhof Camera Software Joost Dierkste COM wrappers Peter van Merkerk

Python wrappers Davy Kox

Motion – IStage interface

Different developers and members across FEI software teams that collaborated with the project by providing specific information about interface behavior. Those interfaces were the core element for defining and building a common testing framework as the solution for the project.

Andrea Pasqualini Software Architect

Responsible for providing requirements and information about the wrapper generation process. Involved in decisions related to this topic.

IOM Interface Developers

They will play the role of final users once the project is extended to real usage. For the scope of the project, they are testers of the solution.

2.2

Eindhoven University of Technology (TU/e)

As an educational program, the Professional Doctorate in Engineering (PDEng) in Software Technology (ST) is conducted and assessed by the Eindhoven University of Technology. TU/e dictates certain standards that have to be met. Those standards are mainly related to the design process, project management, and project implementation.

Table 4 – TU/e Stakeholders

Name Role

Ad Aerts

Program Director of PDEng in Software Technology

Ensuring that the quality of the project artifacts and deliverables are in line with the program standard to grant the PDEng in Software Technology degree.

Tim Willemse TU/e Supervisor

Monitoring the process of the project and involved in decisions regarding quality of the assignment and documentation according to the technical report specification.

Aldo D. Martinez PDEng Candidate

Coordinating, designing, and developing the project are his main tasks within the project timeframe. In parallel, responsible for matching project results with company and university standards.

(28)
(29)
(30)

3.Problem Analysis

In this chapter, we introduce a detailed analysis of the domain by presenting the essential aspects needed for a formal definition of the problem statement. The chapter is mainly focused on addressing the common problems with ensuring quality by testing in high-complex systems. Specific concepts from the company software system are also introduced in order to make a clear problem statement.

3.1

Complex software systems

Software in industry is growing every day. The necessity of connecting hardware through software applications and offering the biggest number of features with an acceptable quality is a high-priority demand. As a consequence, the software behind these applications is becoming complex by connecting all the involved parts. Additionally, the complexity tends to grow every year the system is developed because of new additions in terms of hardware and technology. The following are most common ways in which a system becomes complex:

Complexity by definition: The system solves a problem that is complex by

itself such as biological systems.

Complexity by legacy: Previous developments of the system have collected

old code, technologies, and design patterns. Technology and good practices change overtime.

Complexity by hardware: New hardware elements and/or software

features for the existing ones are added though time.

Complexity by development location: Different teams, companies, and

people from different countries have developed parts of the system

Complexity by interdependencies: Parts of the system that depends on

inputs from other components, hardware, or systems.

(31)

Independent of the reason, complexity affects every phase of the software development cycle, producing the necessity of specific processes to ensure software quality and reduce time costs. As Figure 3 shows (which only depicts a few of the TEM elements), FEI products have an inherited complexity due to the amount and configuration of internal hardware they contain. Each hardware element has its own software implementation that specifies connection, acquisition, and expected behavior. In terms of software, these elements are represented as interfaces. This project is primarily focused on ensuring quality of implemented interfaces through testing and it aims for automation of this process. In parallel to this, interfaces persist over time, which leads to leaks of knowledge or understanding if they are not properly documented; enriching the interface quality by providing documentation is intended. Finally and due to the interdependency complexity, interfaces require availability for different programming languages and technologies; generating adapters and wrappers for specific target technologies is required.

3.2

Ensuring software quality through testing

Testing is a process in which software is exposed to a set of situations with the intention of finding defects. Testing is also stated as the process of validating and verifying that a software, application, or product meets the business, technical, and functional requirements as well as works as expected. Finally, testing is a subset of quality assurance because ensuring functional requirements provides metrics to determine the software quality.

For this project, testing is the core element for ensuring quality, but testing by itself has several different approaches that are completely valid in terms of improving quality. This is the main reason why it is essential to set a scope for the project in terms of testing. As we mentioned in the previous section, the project is centered in ensuring the quality of software interfaces, which are part of the total FEI system. Interfaces are defined in terms of methods as well as behavior, and for our problem analysis, testing the behavior of the interface is the main focus.

For the sake of interface behavior testing, we have to introduce a new element that is also essential in our problem, and this is defining the interface. Previously, we stated the fact that this project is aiming for ensuring quality of implemented interfaces and in order to meet that goal, we have to consider modeling or defining the interface as a necessary step for our solution.

3.3

Modeling interfaces

A model is an abstract representation of an object and for our specific problem a model is a representation of an existing software interface. The necessity of modeling the interfaces leads our project into the model-based development field. Even though several notations for modeling interfaces are available on the market, e.g., UML, picking the proper one for the context of this project depended on the testing necessities and existing interfaces. Three main elements were considered for the decision:

Modeling methods: Interfaces are composed of methods with specific

signatures and parameters. For this project, methods are considered triggers for the interface behavior. In addition, wrappers and adapters are built around these modeled methods as a bridge for using them with different technologies.

Modeling behavior: The expected behavior of the interface is represented

by a state machine. This behavior represents the expected calls and states as well as interactions amongst interfaces we want to ensure through testing.  Modeling specific settings: A last element to consider is the possibility to

model specific settings that allow or complement the solution. Specific values for testing, documentation, and wrapper generation are included in this element.

(32)

A final consideration for the modeling is to clarify that the ultimate goal was not to model all the elements that comprise an existing interface, but modeling all the necessary elements to describe the interface behavior as well as testing and wrapper generation. Modeling the full interfaces will be redundant considering that definition exists in an Interface Definition Language (IDL). An IDL is a specification language used to describe a software component's application programming interface (API).

3.4

Black-box testing

After a model of the interface has been defined, the next element to consider in the analysis of our problem is the testing method. For this, it is important to introduce a general view of the system as Figure 4 shows.

Server

Interface Logic

Hardware

(Simulator)

Client Test case Interface method call Interface method response

Client influencing simulator

Figure 4 – General view of the system under test

The client (in this case, our application/test case) has access to the interface that is implemented in the server side. The interface offers its methods to the clients that connect to the server, but without exposing the real implementation of the interface. This situation matches with the definition of black box testing [3] in which a system is tested by providing inputs and analyzing the outputs without taking into consideration implementation details. However, the project is aiming for a step further in the abstraction by executing interface method calls in terms of transitions and analyzing the responses in terms of state changes (This concept is covered in detail in the following two chapters).

3.5

Problem statement

The assignment focuses on creating a model-based testing framework for a FEI part of the system (Instrument Object Model (IOM) interfaces are covered in the next chapter). The way to achieve this goal is by generating test cases, documentation, and wrappers automatically from an interface model. The interface model has to be able to include the essential information for describing the expected behavior of the interface. The behavior is depicted as a state machine with a set of transitions which are mainly composed of triggers (from the same or external interfaces), conditions (guards), and an expected state as a result of the transition.

The testing part of the assignment is focused on creating real test cases from the interface model that are ready to be compiled and later executed. These test cases have to map the existing implementation and automate as much as possible to keep user modifications to the minimum. Compilation, execution, and result analysis are used as methods to validate the artifacts generated from the model, but these activities are not included in the solution scope.

(33)

The documentation part of the assignment is centered on the automatic generation of documents for the interface. By definition, the interface model is one of these elements, but further documentation has to be generated to cover all the generated artifacts.

The wrapper part of the assignment is focused on creating wrapper code automatically to provide consistent and always up-to-date and matching wrappers for target languages (C# and Python).

3.6

Design Opportunities

In this chapter we have introduced all the elements that surround our project from a global point of view. We envision that our solution will be based on a process which starts with modeling the interface and after that the generation of test cases, documentation, and wrappers conclude the process. We also stated that our system is a complex system in which interfaces are interconnected with each other. Based on these two facts, three potential design opportunities arise:

Elegance/Usability

As a model-based solution, modeling can become complex and hard to follow/read if the modeling tool provides many options or complicated ways to define elements. One of the main aspects for this project is keeping the model easy to read and understand so that it can also serve as documentation. Strategies such as splitting models or keeping the model dynamic allowing the user to define only the elements he/she needs, have to be considered in the final solution. At the end, our tool has to make producing test cases, documentation, and wrappers simpler than writing them manually.

Methodical approach

The test cases have to be generated following testing technologies for state machines and established testing frameworks. It is also important that our solution executes the transition with an order defined in the model and taking care of events and expected outputs. In the case of interconnected interfaces, there has to be control over how the interfaces are called and evaluated for the test cases. Finally, the solution has to follow a methodical approach from modeling until test/document/code generation.

Genericity

The construction of the solution is based on a predefined set of IOM interfaces. These interfaces are a representative sample for which test suites, wrappers, and documentation is generated in order to define the specific requirements for the project. However, the final solution has to provide a generic testing framework that works for the IOM interfaces with minor changes required.

Based on the system vision, the next two design opportunities are not applicable:

Impact

As a pilot project, this is the first approach for building a testing framework with a model-based perspective. The integration of the project into the real development/testing process is a future step that is not included in our current scope. However, a deployment plan is considered as part of the deliverables.

Complexity

Even when the global topics of testing and model-based solutions are potentially complex, this project is aims to demonstrate the applicability within the company. As a new project, keeping the concepts simple but open to future improvement has to steer the solution.

(34)

4.Domain Analysis

The previous chapter discusses the domain for this project, which is model-based testing as well as code and documentation generation for microscope interfaces. The objective of this chapter is to introduce the domain while identifying the most relevant parts that are involved in our problem. The following sections give a detailed view of the microscope system architecture and the software elements to consider for an optimal model description.

4.1

Microscope

software

system architecture

The Microscope Software System (MSS) is a set of software utilities that make it possible to employ all the functionalities of a TEM microscope. This system enables and connects the microscope hardware to software interfaces for controlling and performing applications within the microscope. In the high-level abstract representation (Fig 5) it is visualized as a microscope PC that is connected to several pieces of embedded hardware, which vary depending on the microscope model.

The microscope PC is composed of a set of applications and a TEM server (Fig. 6). The

former are elements that provide user-interfaces to operate the microscope and the latter provides software interfaces for controlling the microscope hardware. Applications are defined in terms of specific use cases abstracting the functionality of the system to a convenient workflow for the end-user. The TEM server is designed to abstract hardware and software differences for the different microscope configurations while maintaining the relevant capabilities of the subsystem.

Embedded Hardware Microscope PC

TEM Server

App 1 App 2 App n

Figure 6 – Microscope PC representation

Figure 5 – High-level representation of microscope software architecture Microscope PC Direct embedded hardware * 1

(35)

For the scope of this project, applications represent test cases and wrappers that connect to the exposed interfaces in the TEM server. This connection happens in the layer level known as object model, for which we need to formally introduce the concept of Instrument Object Model (IOM).

4.2

Instrument Object Model

IOM is the interface for the TEM server software. It is the main interface for microscope control and acquisition. The IOM is allocated in the object model layer that provides access to instrument functionality for automation applications and user interfaces (UIs). IOM interfaces have a well-defined structure (based on a tree perspective) and semantics with respect to the instrument server. These interfaces do not allow bypassing as a measure for ensuring system safety, which means that the only way to access the interfaces is through an instrument. IOM interfaces are the exposed (public) part of the TEM server and they are employed as a starting point for creating clients that connect and employ the interfaces. All the interfaces are implemented in a different layer that is hidden for the external application that is using them (black box).

Application TEM Server IOM Instrument Detector StemDetector FluScreen

Figure 7 – Server exposing IOM interfaces of specific instruments

4.3

COM interfaces

The TEM server provides software interfaces to user interface applications for controlling the microscope hardware. These interfaces define a set of methods that an instrument can support, without dictating anything about the implementation. In order to achieve this goal, Component Object Model (COM) is used as RPC technology for inter-process communication. COM is a binary-interface standard for software components introduced by Microsoft that includes the ability to marshal interface cross processes. Additionally, it provides the functionality to generate .NET runtime-callable-wrappers from type libraries. COM maintains a strict separation between interface and implementation. A COM interface is defined using an IDL that is processed by an IDL compiler, which generates a header file.

Interfaces implemented within COM technology influence activities such as accessing, employing, and testing these interfaces.

4.4

Synchronous versus asynchronous

Once COM interfaces are accessed, methods are executed to perform a specific functionality on the system (microscope). Depending on the instrument and implementation style, this execution is supported in one of the following two approaches:

(36)

Synchronous: The server interface allows parallel access by multiple clients

(for example, while inserting a STEM detector, another client can change optics settings). The client waits until the completion and this creates a sequential workflow.

Asynchronous: The server creates a second thread to execute the method

allowing the client to employ other operations while the method is executed. The completion of the action requested by an asynchronous call is reported using events.

Figure 8 – Synchronous and asynchronous example

Due to the complexity of TEM microscope instruments, both kinds of methods are defined within the IOM interfaces. FEI is striving to use synchronous methods, and then asynchronous methods will be gradually removed. Meanwhile, common techniques for handling asynchronous methods are using events to inform the caller about system changes or employing wait-for-completion methods that wrap asynchronous calls.

4.5

Events

Inside the microscope software, interfaces and hardware are related (for example turning on the high voltage requires a FEG source powered on). This means that a change in one module can affect one or more of its associated modules. The current MSS propagates this change by implementing events. Events are notifications associated with state changes of a specific instrument. These state changes are value changes in interface properties triggered by a software or hardware module. These triggers can be part of the interface or come from a different interface. For both cases, they produce a notification that is received by all the registered clients of the interface. A client needs to register its event listener for handling these notifications. Events are an essential part of interface behavior because they represent how interfaces have to notify state changes. Similarly to the interface definition, events are also defined in COM technology.

(37)

Figure 9 – Example of a COM Event interface

4.6

Interface behavior

IOM interfaces follow a state machine approach. The key concept of this approach is that every interface owns an internal state machine implementation, which is influenced by internal or external triggers. These triggers are generated by methods or events from IOM interfaces as well as hardware changes. The triggers that are produced by methods of the same interface are known as external triggers while triggers coming from hardware changes are defined as internal triggers.

Figure 10 COM interface definition – methods and state enumeration

As we mentioned before, COM interfaces use encapsulation to hide the implementation from the interface definition. As public elements, the interface exposes two main components that are essential to track the implemented behavior: the state enumeration and the methods. The state enumeration contains all the valid states for an interface and the methods are the operations of the interface that can externally stimulate state transitions.

(38)

As we can see in Figure 10, a typical COM interface defined through an interface definition language (IDL) does not include a state machine; then, an approach to describe and model a state machine’s behavior is required for representing behavior. Describing this behavior enhances the interface definition and makes it possible to test the implemented behavior against this definition.

4.7

Using interfaces

The last element to take into account is how to get access to the existing server interfaces. The only way to use interface functionalities in a client is by following the next steps:

1. Connecting to the server: Creating an instrument and enabling connection to the server.

2. Traversing the interface tree: Accessing a specific interface requires navigating within the tree. A connected instrument has access to all the existing interfaces in the server, which are organized in a hierarchical tree. 3. Executing interface methods: Executing the methods from the selected

interface. In this point, any method execution flows through the server and interacts with related hardware and/or interfaces.

Client IOM specificInterface Related HW

creatingInstrument connecting connectionEstablished getInterface getInterface Interface Interface executeMethod method result result

TEM server

Figure 11 Common workflow for using an interface

(39)

Referenties

GERELATEERDE DOCUMENTEN

This is exploited in a proposed design technique where the cut-off frequencies throughout the horn are used to synthesise the ridge taper profile, in order to achieve the desired

Algemeen kan uit de resultaten van de prospectie door middel van landschappelijke boringen worden afgeleid, dat bij een prospectie met ingreep in de bodem binnen 1,5 m

Het publiek gebruik van dit niet-officiële document is onderworpen aan de voorafgaande schriftelijke toestemming van de Algemene Administratie van de Patrimoniumdocumentatie, die

Tegen het licht van de assumptie dat afhankelijkheid van de organisatie van zijn omgeving onzekerheid kan opleveren met,een bedreigende invloed op de bedrijfsvoering ('die

Als u met een klein project begint, benoem dan personen; start u groot, benoem dan functiegroepen en eventueel personen die opvallen of invloed hebben binnen die functiegroep..

Bij optellen en aftrekken van grootheden moet je de absolute fouten volgens de KOWmethode samenstellen om de absolute fout in het resultaat te krijgen.. Iemand wil de massa van

[r]

While the functional behavior of a circuit in the frequency domain may not be linear, or even in the dc domain as a result of a nonlinear function between output and input