Metrics for model transformations

(1)

Metrics for model transformations

Citation for published version (APA):

Amstel, van, M. F., Brand, van den, M. G. J., & Nguyen, H. P. (2010). Metrics for model transformations. In S. Ducasse, L. Duchien, & L. Seinturier (Eds.), BENEVOL 2010 (9th Belgian-Netherlands Software Evolution Seminar, Lille, France, December 16, 2010. Proceedings of Short Papers) (pp. 1-5). Université Lille 1.

Document status and date: Published: 01/01/2010 Document Version:

Publisher’s PDF, also known as Version of Record (includes final page, issue and volume numbers) Please check the document version of this publication:

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website.

• The final author version and the galley proof are versions of the publication after peer review.

• The final published version features the final layout of the paper including the volume, issue and page numbers.

Link to publication

General rights

Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain

• You may freely distribute the URL identifying the publication in the public portal.

If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, please follow below link for the End User Agreement:

www.tue.nl/taverne

Take down policy

If you believe that this document breaches copyright please contact us at: openaccess@tue.nl

providing details and we will investigate your claim.

(2)

Metrics for Model Transformations

$ M.F. van Amstel, M.G.J. van den Brand, P.H. Nguyen Eindhoven University of Technology, P.O. Box 513, Eindhoven, The Netherlands

Abstract

One of the key concepts of model-driven engineering (MDE) is model transformation. Because of the crucial role of model transformations in MDE, they have to be treated in a similar way as traditional software artifacts. It is therefore necessary to deﬁne and assess their quality. In this paper we present three metric sets for assessing the quality of ATL, QVTO, and Xtend model transformations. We show the overlap between the metric sets, as well as their diﬀerences.

Keywords: MDE, model transformation, ATL, QVTO, Xtend, metrics, quality

1. Introduction

Model-Driven Engineering (MDE) is a software engineering paradigm in which models play a central role through-out the entire development process [1]. MDE combines domain-specific languages for modeling at a higher level of abstraction and model transformations for the automated generation of various artifacts from these models. Since MDE is becoming increasingly important, so are model transformations. Model transformations are in many ways similar to traditional software artifacts, i.e., they have to be used by multiple developers, have to be changed according to changing requirements and should preferably be reused. Therefore, it is necessary to define and assess their quality. Quality attributes such as modifiability, understandability and reusability need to be understood and defined in the context of MDE, in particular for model transformations. The goal of our research is to make the quality of model transformations measurable. The type of quality we refer to here is the internal quality of model transformations, i.e., the quality of the transformation artifact itself [2]. In this paper, we focus on model transformations created using ATL [3], QVT operational mappings [4], and Xtend [5].

An approach that is often employed for assessing the quality of software artifacts is metrics. We will use metrics for assessing the quality of model transformations as well. For each of the three aforementioned model transformations formalisms we have defined a set of metrics. The metrics are defined on the model transformation artifact only, i.e., the input and output models of a model transformation are not considered. This is referred to as direct assessment [2]. Since all three model transformation formalisms are designed for a similar task, i.e., transforming models, it is to be expected that there will be overlap between the sets of metrics. We will address these similarities and we will also point out the differences among the metrics sets.

The remainder of this paper is structured as follows. In Section 2, we shortly introduce the model transformation formalisms for which we have derived metrics. Section 3 describes the metrics we have derived. Conclusions and directions for further research are given in Section 4.

2. Model Transformation Formalisms

ATL. The Atlas Transformation Language (ATL) is a hybrid model transformation language [3]. It is hybrid because

it supports both a declarative and an imperative programming style. We chose to study metrics for ATL because ATL

$_{This work has been carried out as part of the FALCON project under the responsibility of the Embedded Systems Institute with Vanderlande}

Industries as the industrial partner. This project is partially supported by the Netherlands Ministry of Economic Aﬀairs under the Embedded Systems Institute (BSIK03021) program.

Email addresses: M.F.v.Amstel@tue.nl (M.F. van Amstel), M.G.J.v.d.Brand@tue.nl (M.G.J. van den Brand),

(3)

is currently one of the most popular model transformation languages around.

QVT Operational Mappings. The Query/View/Transformation (QVT) speciﬁcation includes three languages for

per-forming model transformation [4]. QVT Operational Mappings (QVTO) is one of these languages. QVTO is an imperative language. This means that the complete procedure of generating target models from source models has to be speciﬁed explicitly. In contrast, the declarative part of ATL for example does not require specifying how model elements should be matched. We chose to study metrics for QVTO because it is part of OMG’s standard for model transformation and it is widely used as well.

Xtend. Xtend is the model-to-model transformation language that used to be part of the openArchitectureWare

project [5]. Xtend can best be characterized as an imperative language with some traits of a functional language. This means that also for Xtend the complete procedure of generating target models from source models has to be speciﬁed explicitly. We chose to study Xtend because it is frequently used by one of our industrial partners.

3. Metrics

This section describes a selection of the metrics we deﬁned for assessing the quality of model transformations created with ATL, QVTO, and Xtend. We acquired the metric sets by performing brainstorm sessions. In the end we want to establish which metrics relate to quality attributes. Therefore we do not want to restrict our metrics sets on beforehand. This is also why we did not use an approach like the Goal-Question-Metric approach [6]. The metrics can be divided into eight categories. In the remainder of this section, we will address each of these categories and elaborate on the metrics belonging to them. For more information on the metrics presented here and other metrics as well, the reader is referred to [7, 8].

3.1. Common Metrics

The ﬁrst seven categories of metric are generic, i.e., they apply to all of the three studied formalisms and most likely to other model transformation formalisms as well.

3.1.1. Size metrics

The first category contains metrics for measuring the size of a transformation. We established in an earlier study that the size of a transformation correlates negatively with its understandability and modifiability [9]. The size metrics are shown in Table 1. The size of a model transformation is determined by the number of transformation functions it comprises. In ATL they are called rules, in QVTO mappings, and in Xtend extensions. All three formalisms have different types of transformation functions. For each of them we defined metrics for measuring the number of trans-formation functions for each of these function types (not shown in the table). Besides the number of transtrans-formation functions, the size of a transformation is also affected by the number of helper functions. Again, metrics have been defined to measure the number of helper functions for each of the helper function types there are in ATL and QVTO. Note that in Xtend all functions are extensions.

ATL QVTO Xtend

# Rules # Mappings # Extensions

# Helpers # Helpers

Table 1: Size metrics

3.1.2. Function Complexity Metrics

The complexity of a model transformation is affected by the complexity of its parts, i.e., transformation and helper functions. We also established in an earlier study that the complexity of a transformation function correlates negatively with the understandability and modifiability of a transformation [9]. The metrics we defined for measuring the complexity of these functions are shown in Table 2. The complexity of a transformation function is among others determined by the number of model elements it takes as input and returns as output. In ATL this is measured using the metrics # Elements per in-/output pattern. In QVTO this is measured using the metric # Parameters per mapping. For input model elements only in and inout parameters should be considered and for output model elements only out and

(4)

inout parameters. Transformation functions in Xtend can return only one output model element, there is no metric for

this. The number of input model elements is measured by the metric # Parameters per extension. The complexity of helper functions for both ATL and QVTO can be measured using the metric # Parameters per helper as well.

# Elements per in-/output pattern # Parameters per mapping/helper # Parameters per extension

# Parameters per called rule/operation helper # Variables per mapping/helper # Operations on collections per extension # Bindings per rule # Operations on collections per mapping/helper # Local variables per extension # Operations on collections per rule/helper Mapping/helper cyclomatic complexity Extension cyclomatic complexity # Variables per rule/helper

Helper cyclomatic complexity

Table 2: Function complexity metrics

In the body of a transformation function the assignment of transformed source model elements to target model elements is performed. In ATL, target model elements are typically assigned values using bindings. Therefore, the

number of bindings per rule aﬀects the complexity of a rule in ATL. Since model transformations often need to

deal with collections, i.e., sets of model elements, we propose to measure the number of operations on collections separately. Another inﬂuence on the complexity of a function related to statements, is its cyclomatic complexity.This metric measures the number of decision points in a function.

All three formalisms allow defining local variables. These are typically used to provide separation of concerns, i.e., to split a calculation in orderly parts. Therefore, the number of local variable definitions affects the complexity of a transformation function as well.

3.1.3. Modularity Metrics

All three formalisms have support for structuring model transformations by packaging transformation rules in modules, albeit that ATL has some restrictions on this. Therefore, the third category of metrics contains metrics related to modularity. These metrics are shown in Table 3. The metrics that measure the number of modules/units that a transformation consists of can be used as a size metric as well. The import metrics can be used to assess which modules rely on other modules as well as on which modules is heavily relied on. The number of transformation

and helper functions per module can be used to determine the balance of the modules that comprise a transformation.

When the standard deviation of these metrics is high, the modules are not properly balanced. This could be intentional, but it could also be an indicator that the division of functions over modules should be reconsidered.

# Units # Modules # Modules

# Imported units # Imported modules # Extensions per module

# Times a unit is imported # Times a module is imported # Imported modules

# Helpers per unit # Mappings/helpers per module # Times a module is imported

Table 3: Modularity metrics

3.1.4. Inheritance Metrics

ATL and QVTO have support for rule inheritance. The use of inheritance may affect the quality of a model transformation in a similar way as it affects object-oriented software [10]. A rule deeper in the rule inheritance tree may be more fault-prone because it inherits a number of properties from its ancestors. Moreover, in deep hierarchies it is often unclear from which rule a new rule should inherit from. To acquire insights into the rule inheritance relations in ATL and QVTO transformation, we defined the metrics shown in Table 4. QVTO has different forms of inheritance, i.e., inherit, merge, and disjunct. We defined metrics to measure all three forms.

# Abstract rules # Abstract mappings # Overloaded extensions

# Children per matched rule # Mapping inherits # Extensions per extension name (overloadings)

# Rule inheritance trees # Mapping merges

Depth of rule inheritance tree # Mapping disjuncts

Width of rule inheritance tree # Mappings per disjunct

# Overloaded helpers # Overloaded mappings/helpers

# Helpers per helper name (overloadings) # Mappings per mapping name (overloadings) # Helpers per helper name (overloadings)

(5)

Related to inheritance is overloading. Therefore, metrics regarding overloading are also included in this category. All formalisms support overloading of functions, except for ATL rules. We deﬁned metrics to measure the number of

overloaded functions and the number of overloadings per overloaded function. 3.1.5. Dependency Metrics

Transformation functions generally depend on other transformation functions. To measure this dependency, we measure fan-in and fan-out of transformation functions. Fan-in of a transformation function is the number of times it is invoked by other transformation functions. Fan-out of a transformation function is the number of times it invokes other transformation functions. We consider fan-in/fan-out metrics for four types of transformation elements, i.e., transformation functions, helper functions, library functions, and modules. The metrics are shown in Table 5.

# Calls to rules # Calls to mappings per mapping # Calls to extensions

# Calls from rules # Calls from mappings per mapping # Calls from extensions

# Calls to helpers # Calls to helpers # Calls to extensions in other modules

# Calls from helpers # Calls from helpers # Calls from extensions in other modules

# Calls to helpers in other units # Calls to mappings/helpers in other modules # Calls to extensions in the standard library # Calls from helpers in other units # Calls from mappings/helpers in other modules

# Calls to built-in functions

Table 5: Dependency metrics

3.1.6. Consistency Metrics

Metrics can be used to detect all kinds of inconsistencies in transformations. The metrics we defined for that purpose are shown in Table 6. A typical inconsistency is an unused model transformation element. This can be a helper function, a transformation function, or an entire module. On a smaller scale there are parameters and local variables that may have been defined but are unused. We have defined metrics to detect all of these unused elements.

# Unused units # Unused modules # Unused extensions

# Unused rules # Unused mappings # Unused parameters

# Unused helpers # Unused helpers # Unused local variables

# Unused input pattern elements # Unused parameters # Extensions with anullbody

# Unused parameters # Unused local variables # Variables with same name but different types

# Unused local variables # Variables with same name but different types # Calls to logging expressions

# Variables with same name but different types # Calls tolog() # Calls tosyserr()

# Calls toprintln() # Calls toassert()

# Calls todebug()

Table 6: Consistency metrics

All three formalisms allow defining local variables. This means that a variable needs to be redefined if it is to be used in other functions. This may lead to inconsistencies in variable naming, i.e., a variable name in one module can be related to a different type in another function. To detect such inconsistencies we defined the metric number of

variables with same name but diﬀerent types.

A number of functions are available in all three formalisms that support logging of some kind. Typically, this is used for debugging purposes. The occurrence of calls to these functions may therefore indicate that the model transformation is still under development.

3.1.7. Input/Output Metrics

The last category of metrics is aimed at providing insight in the context of the transformation. They are shown in Table 7. It is to be expected that transformations involving more models and metamodels are more complex. Therefore we deﬁned metrics that measure the number of models and metamodels involved in a transformation.

# Input/output models # Input/output models # Imported metamodels

# Involved metamodels # Imported metamodels # Imported metamodels per module

Percentage of rules that change the model # Input models per operational transformation # Output models per operational transformation # Imported metamodels per module

Table 7: I/O metrics

(6)

3.2. Formalism-speciﬁc Metrics

The last category of metrics contain the metrics that measure aspects of model transformations that are speciﬁc for each of the formalisms. We will address only a few of them.

In ATL, a transformation function can have multiple in- and output model elements. The metric rule complexity

change measures the amount of output model elements that are generated per input model element. The input pattern

of an ATL matched rule can be constrained using ﬁlter conditions. The metric number of rules with a ﬁlter condition

on the input pattern measures this. Using such ﬁlter conditions a rule matches only on a subset of the model elements

deﬁned by the input pattern. This may inﬂuence understandability since it may not be immediately clear which rule matches on a model element.

In QVTO it is possible to define intermediate classes and properties. These are used in the scope of the transfor-mation only. These intermediate classes and properties may be useful for splitting calculations. Therefore we measure the number of intermediate classes/properties. In QVTO and in ATL there are constructs for explicit trace resolution. These constructs may affect the understandability of a model transformation. Therefore we defined metrics to measure the number of trace resolution calls.

Xtend has support for aspect-oriented programming using so-called arounds. We measure this using the metric

number of arounds.

4. Conclusions and Future Work

We presented sets of metrics for ATL, QVTO, and Xtend and a classification of these metrics. Even though the formalisms have different characteristics, a lot of metrics are similar. Nevertheless, each of the formalisms also has some specific metrics. These metrics are used to measure aspects of a transformation that are specific for the paradigm and the range of ideas implemented in the respective formalism. This most likely holds for other model transformation formalisms as well.

Extracting metrics by hand is infeasible and also error-prone. Therefore, tools have been implemented for each of the three formalisms to extract metrics from transformations automatically [7, 8].

Being able to measure a model transformation using a set of metrics alone is not enough for assessing its quality. Metrics should be related to quality attributes [11]. Therefore empirical studies should be conducted to relate each of the three metrics sets to these quality attributes. For ASF+SDF, a formalism that can be used for model trans-formations, such an empirical study has already been performed [9]. Performing similar empirical studies for the formalisms treated in this paper is a point for future work. In [8], metrics have been related to quality attributes based on our expectations.

References

[1] D. C. Schmidt, Model-Driven Engineering, Computer 39 (2) (2006) 25 – 31.

[2] M. F. van Amstel, The Right Tool for the Right Job: Measuring Model Transformation Quality, in: Proceedings of the Fourth IEEE Interna-tional Workshop on Quality Oriented Reuse of Software (QUORS’10) in proceedings of 34th Annual InternaInterna-tional Computer Software and Applications Conference (COMPSAC’10), 2010, pp. 69–74.

[3] F. Jouault, I. Kurtev, Transforming Models with ATL, in: MoDELS 2005 Satellite Events, no. 3844 in LNCS, Springer, 2005, pp. 128–138. [4] Object Management Group, Meta Object Facility (MOF) 2.0 Query/View/Transformation Speciﬁcation, version 1.0, Document

--formal/2008-04-03, OMG (2008).

[5] A. Haase, M. V¨olter, S. Eﬀtinge, B. Kolb, Introduction to openArchitectureWare 4.1.2, in: Model-Driven Development Tool Implementers Forum (MDD-TIF’07) (co-located with TOOLS 2007), 2007.

[6] V. B. Basili, D. M. Weiss, A Methodology for Collecting Valid Software Engineering Data, IEEE Transactions on Software Engineering 10 (6) (1984) 728–738.

[7] M. F. van Amstel, M. G. J. van den Brand, Quality Assessment of ATL Model Transformations using Metrics, in: 2nd International Workshop on Model Transformation with ATL (MtATL’10), 2010.

[8] P. H. Nguyen, Quantitative Analysis of Model Transformations, Master’s thesis, Eindhoven University of Technology, Eindhoven, The Netherlands (2010).

[9] M. F. van Amstel, C. F. J. Lange, M. G. J. van den Brand, Using Metrics for Assessing the Quality of ASF+SDF Model Transformations, in: Proceedings of the Second International Conference on Model Transformation (ICMT’09), Vol. 5563 of LNCS, Springer, 2009, pp. 239–248. [10] V. R. Basili, L. C. Briand, W. L. Melo, A Validation of Object-Oriented Design Metrics as Quality Indicators, IEEE Transactions on Software

Engineering 22 (10) (1996) 751 – 761.