Capgemini
SharePoint code quality analysis
Author:
Thom Koenders
University supervisor:
Apostolos Ampatzoglou Company supervisor:
Pascal van Alphen
March 11, 2015
Abstract 6
Preface 7
Acknowledgements 8
1 i n t r o d u c t i o n 9
1.1 SharePoint . . . 9
1.2 Motivation . . . 9
1.3 Capgemini . . . 10
1.4 Problem . . . 10
1.5 Contribution . . . 11
1.6 Terminology . . . 11
1.6.1 Code quality . . . 11
1.6.2 Rule violation . . . 12
1.6.3 Defects . . . 12
2 c a s e s t u d y d e s i g n 13 2.1 Research Objectives & Research . . . 13
2.1.1 Research objective . . . 13
2.1.2 Research questions . . . 14
2.2 Data Collection . . . 14
2.2.1 SPCAF . . . 14
2.2.2 Methods . . . 15
2.3 Data Analysis . . . 17
2.4 Validity and Reliability . . . 17
2.4.1 Replication strategy . . . 17
2.4.2 Quality assurance, validity and reliability . . . 17
3 r e s u lt s 18 3.1 What are the most frequently violated rules in SharePoint? . 18 3.1.1 Used projects . . . 18
3.1.2 Interpreting the intermediate project results . . . 19
3.1.3 Interpreting the intermediate combined results . . . . 21
3.1.4 Sorting the data . . . 22
3.1.5 Conclusions . . . 22
3.2 What aspects of SharePoint are most prone to produce defects? 26 3.2.1 Question 1 . . . 26
3.2.2 Question 2 . . . 28
3.2.3 Question 3 . . . 29
3.2.4 Question 4 . . . 30
3.2.5 Question 5 . . . 32
3.2.6 Question 6 . . . 33
3.2.7 Conclusion . . . 34
3.3 What rule violations have a probability of resulting in defects? 36 3.3.1 Groups of rule violations . . . 36
3.3.2 First assessment . . . 37
3.3.3 Second assessment . . . 38
3.3.4 What is the general expert opinion on the undeter- mined rules? . . . 39
3.3.5 Conclusion . . . 55
4 d i s c u s s i o n 56 4.1 Interpretation of results . . . 56
4.1.1 What are the most frequently violated rules in Share-
Point? . . . 56
4.1.2 What aspects of SharePoint are most prone to produce defects? . . . 56
4.1.3 What rule violations have a probability of resulting in defects? . . . 57
4.2 Is it possible to detect defects using rule violations? . . . 58
4.2.1 Theoretical approach . . . 58
4.2.2 Practical approach . . . 58
4.2.3 Conclusion . . . 59
4.3 Implications for researchers and practitioners . . . 59
4.3.1 Implications for researchers . . . 59
4.3.2 Implications for practitioners . . . 59
5 c o n c l u s i o n 65 5.1 Future work . . . 65
5.1.1 Custom rules . . . 65
5.1.2 Dynamic code quality analysis . . . 65
Appendices 66 a b u i l d s e r v e r 67 a.1 Choice for Windows Azure . . . 67
b p r o j e c t s c o m b i n e d 68 b.1 Correctness . . . 69
b.2 SharePoint Supportability . . . 70
b.3 Deployment . . . 71
b.4 Security . . . 72
b.5 Design . . . 73
b.6 Best Practices . . . 74
b.7 Memory Disposal . . . 75
b.8 CSSLint . . . 76
b.9 Localization . . . 76
b.10 Naming . . . 77
b.11 JSHint . . . 77
b.12 Summary . . . 77
c c o m p l e t e v i o l at i o n o c c u r r e n c e r e p o r t 79 d r u l e a s s e s s m e n t 84 d.1 Argumentation . . . 85
d.1.1 Shared Argumentation . . . 85
d.2 Correctness . . . 87
d.2.1 Critical errors . . . 87
d.2.2 Errors . . . 91
d.2.3 Critical warning . . . 97
d.2.4 Warnings . . . 101
d.3 SharePoint Supportability . . . 101
d.3.1 Critical errors . . . 101
d.3.2 Errors . . . 101
d.4 Deployment . . . 102
d.4.1 Critical errors . . . 102
d.4.2 Errors . . . 102
d.4.3 Critical warnings . . . 104
d.4.4 Warnings . . . 105
d.5 Security . . . 106
d.5.1 Errors . . . 106
d.5.2 Critical warning . . . 107
d.6 Design . . . 109
d.6.1 Critical warnings . . . 109
d.6.2 Warnings . . . 111
d.7 Best Practices . . . 112
d.7.1 Critical warnings . . . 112
d.7.2 Warnings . . . 115
d.8 Memory Disposal . . . 116
d.8.1 Critical warnings . . . 116
d.8.2 Warnings . . . 119
d.9 CSSLint . . . 119
d.9.1 Critical warnings . . . 119
d.9.2 Warnings . . . 120
d.10 Localization . . . 124
d.10.1 Warnings . . . 124
d.11 Naming . . . 124
d.11.1 Warnings . . . 124
d.12 JSHint . . . 125
d.12.1 Warnings . . . 125
e u n d e t e r m i n e d r u l e s 132 f e x t e n d e d c o n c l u s i o n r u l e s 144 f.1 Correctness . . . 144
f.1.1 Critical errors . . . 144
f.1.2 Errors . . . 147
f.1.3 Critical warning . . . 151
f.1.4 Warnings . . . 153
f.2 SharePoint Supportability . . . 153
f.2.1 Critical errors . . . 153
f.2.2 Errors . . . 153
f.3 Deployment . . . 153
f.3.1 Critical errors . . . 153
f.3.2 Errors . . . 154
f.3.3 Critical warnings . . . 155
f.3.4 Warnings . . . 155
f.4 Security . . . 156
f.4.1 Errors . . . 156
f.4.2 Critical warning . . . 156
f.5 Design . . . 157
f.5.1 Critical warnings . . . 157
f.5.2 Warnings . . . 158
f.6 Best Practices . . . 158
f.6.1 Critical warnings . . . 158
f.6.2 Warnings . . . 160
f.7 Memory Disposal . . . 160
f.7.1 Critical warnings . . . 160
f.7.2 Warnings . . . 163
f.8 CSSLint . . . 163
f.8.1 Critical warnings . . . 163
f.8.2 Warnings . . . 163
f.9 Localization . . . 165
f.9.1 Warnings . . . 165
f.10 Naming . . . 165
f.10.1 Warnings . . . 165
f.11 JSHint . . . 165
f.11.1 Warnings . . . 165
6 b i b l i o g r a p h y 167
A high quality codebase is an important requirement to deliver high qual- ity software. A high code quality can be achieved in various ways, one of which is static code quality analysis, which is the focus of the thesis. More specifically, this thesis is focused on SPCAF, a static code quality analysis tool for SharePoint. This tool scans a codebase, line by line, and applies a set of rules to each of these lines to determine if these lines do not violate any of the quality standards. Each violation of these rules is listed, and indicates a problem with the quality of the corresponding piece of code. Problem with the quality of the code can manifest in different ways, ranging from making it more difficult to maintain the codebase to having a negative impact on the stability of the software. While all problems can be considered equally important, in practice it is most important that the software is stable and functions as intended. Therefore, the emphasis in this thesis is put on the detection of defects.
The thesis is conducted as a case study and features three research questions.
The first research question is focused on determining the most frequently violated rules in SharePoint. The second research question is focused on determining what aspects of SharePoint are most prone to produce defects.
And the third and last research question is focused on determining what specific rule violations could indicate defects. The results of the thesis are useful both academically as well as in practice. Academically, it offers a thorough insight into static code quality analysis in SharePoint. Practically, it offers general directives and a list of specific rules that can be directly used to detect potential defects.
The main topic of this thesis has always been code quality analysis in Share- Point. Nevertheless, over the course of this thesis, the precise focus has shifted multiple times. The thesis was conducted for Capgemini, that was interested in a code quality analysis tool to maintain a high quality in their projects. At the time, SPCAF, a full fledged code quality analysis tool specifi- cally designed for SharePoint, was a well known tool with a good reputation at Capgemini, but naturally it was important to research the alternatives as well. Therefore, the thesis started out as a comparison study, which was focused on finding a worthy alternative for the SPCAF tool.
It quickly became apparent that there were no alternatives of significance to SPCAF. There were tools available that at least provided part of the re- quired functionality. However, only a few of these tools were specifically designed for SharePoint, and these tools were nowhere near as thorough or complete as SPCAF. Apart from these tools, other tools that were not specifically designed for SharePoint were researched. These tools were for example developed for the code quality analysis of C# code, which resulted in SharePoint specific aspects not being addressed by these tools. Because of this, these tools were not in the slightest able to provide the required standards that were set, making further research into this topic redundant.
Considering the lack of alternatives to SPCAF, it was decided to focus the thesis on the SPCAF tool, to determine its capabilities. At this point, the research method had been changed to case study, since this facilitated the research into a single specific tool better. Later on, the goal of this case study into SPCAF was focused further onto the rules that SPCAF uses to perform its code quality analysis, and how this tool and its predefined rules could be used to detect and locate defects.
As described, the thesis evolved multiple times, which is not uncommon for a case study. The further the thesis progressed, the more focused the research became. This resulted in the thesis it is today, providing both aca- demic and practical results on static code quality analysis in SharePoint using the SPCAF tool.
First, I would like to sincerely thank my supervisor of the University of Groningen, dr. Apostolos Ampatzoglou, whose guidance helped me make the thesis what it is today. His mentorship and expertise allowed me to suc- cessfully complete this thesis. Thanks a lot for all of the valuable and much appreciated guidance.
Second, I would like to show my sincere gratitude to Capgemini, especially Pascal van Alphen, Marc Schaaks and Donald Hessing. Thanks a lot for providing the interesting assignment, for the opportunity to conduct this thesis at Capgemini, and for providing all of the assistance and expertise.
Third, I would like to thank the company that developed SPCAF, Rencore, especially Matthias Einig. Thanks for all the support and all the additional information on unclear aspects of the tool.
And last but not least, I would like to thank my proofreaders for all the valuable feedback they provided, which allowed me to make the final ad- justments.
One last thanks to all of you for all the support you gave me, it was highly appreciated. My sincere gratitude to all of you, for enabling me to make this thesis what it has become.
1
I N T R O D U C T I O N
There are still organizations that have such unique business processes that the facilitating software needs to be custom made. However, in most com- panies the business processes are fairly standard. Because of this, most organizations can be accommodated using modular standardized solutions, e.g. SharePoint. This has led to a shift from custom made solutions to standardized solutions based on modular development.
1.1 s h a r e p o i n t
SharePoint1, started out in 2001 as SharePoint Team Service, to replace Ex- change Public Folders, and was focusing on team collaboration. In MOSS 2007 (Microsoft Office SharePoint Server 2007), also known as SharePoint 2007, real web content management capabilities were introduced. Over time, SharePoint evolved into the modular web application framework and plat- form it is today.
With SharePoint, software systems are built by combining standardized out- of-the-box modules to develop the system that provides the required func- tionality. These standardized modules can be configured to target and ac- commodate the specific needs of the organization. These configurations can be written using a declarative language, to describe the set up of the module. Declarative programming is a programming paradigm in which the goal that should be achieved is declared, instead of using primitives to specifically describe how this should be done.
Furthermore, SharePoint can be extended with custom modules and code to provide functionality that is not found in the out-of-the-box modules. This way, solutions can be created by combining standardized out-of-the-box and organization specific modules and code to develop software systems that tar- gets the specific requirements. These custom modules are written using an imperative language, with which the module is developed to provide the re- quired functionality. Imperative programming is the opposite of declarative programming, in which a sequence of the programming language primitives are used to specifically describe how an operation should be performed.
1.2 m o t i vat i o n
Just like in all branches of software engineering, a high quality of software developed using SharePoint, is essential. The quality of the codebase is directly related to the quality of the resulting end user software. A high quality codebase is more likely to function as intended and expected. The quality of the codebase also has a large indirect influence. The quality of the
1 http://office.microsoft.com/nl-nl/sharepoint-help/
codebase is closely related to the amount of effort it takes to perform addi- tional work on the codebase, i.e. the maintainability. An easy to maintain codebase makes it easier to work on for example bug fixes, new functional- ity and other types of support.
The maintenance of software is a large item of expenditure during the entire life cycle. Studies show that 50 to 75 percent of the overall project expen- ditures are spent during the maintenance phase [7]. Research prior to that showed that up to even 80 percent of the overall project expenditures were spent on maintenance[4]. So, even though significant progress has been made, there is still a lot to gain in this area. There is evidence that the qual- ity of software reduces the overall maintenance expenditure [1]. Therefore, a high quality of the codebase can result in remarkable expenditure reduc- tions, making it a financially interesting opportunity.
Code review is often considered a cost-effective method to improve the code quality and discover bugs and other code convention violations in an early stage of the development [2]. Code review can be performed both manually, which is referred to as code inspection, as well as with the use of tools, which is referred to as tool assisted code review. Tool assisted code review enables not only distributed reviewing, but also improves both the quality as well as the quantity of reviews [3]. It strongly decreases the amount of manual work that is involved with the reviewing process, which means it directly decreases the amount of required man-hours.
1.3 c a p g e m i n i
Capgemini, a multinational corporation primarily focused on providing IT services, wants to implement its SharePoint solutions using a thorough and industrialized approach, including a high-grade code quality analysis. This to shorten the throughput and deliver higher quality end products. At this point, the high-grade code quality analysis is yet to be implemented. There- fore, it was decided to perform this thesis at the dutch office of Capgemini, on the topic of code quality analysis in SharePoint.
1.4 p r o b l e m
There are different types of code quality analysis. Two of these major types are dynamic code quality analysis and static code quality analysis. Where dynamic code quality analysis is devoted to the analysis of software being executed, static code quality analysis revolves around analyzing the code- base. The field of code quality analysis is too big to research as a whole in this thesis. Therefore, it was decided to focus this research on static code quality analysis. In this thesis, code quality analysis is interpreted as the quality of the codebase on a line-by-line basis, i.e. the quality of each line of code has to match the set quality standards.
This can be achieved using various methods. For this thesis it was decided to do this using a static code quality analysis tool, SPCAF, which is a static code quality analysis tool for SharePoint. The analysis features SPCAF of- fers are divided across four tools, each providing its own specific sort of analysis. The kinds of analysis that are offered through this tool are further described in section 2.2.1 on page 14. For this thesis, the SPCop compo-
nent of SPCAF, which detects and locates rule violations (disapproved code constructions) based on a set of rules, was used to assist the code quality analysis.
Different sets can be selected in the tool. In this thesis the complete set was used, containing all of the predefined rules. Each of these rules is dedicated to detect a single violation when running this code quality analysis tool.
The rule-sets focus on maintaining a high quality of the codebase. This set of rules looks for all code constructions that are bad for the codebase. This ranges from code constructions that have been deemed obsolete to code con- structions that have a negative impact on the stability of the software and therefore may result in defects, i.e. crashes and other unwanted and un- intended behaviour. Correcting all faulty code constructions is important when trying to maintain a high code quality. Nevertheless, the detection of defects can be considered more pressing, to make sure the software func- tions as intended. Because of this importance, detecting and locating defects has been made the main topic of this thesis.
1.5 c o n t r i b u t i o n
This thesis can be considered a start into the area of code quality analysis in SharePoint. It will provide a practical approach at discovering defects, which can be applied in organizations to assist in the bug fixing process in their projects. Apart from these practical results, the thesis will also offer valuable insights in code quality analysis in general in SharePoint. To get to the practical result, i.e. the assistance on discovering defects, code quality analysis in general will be thoroughly researched, accompanied by the intermediate results that come with it.
1.6 t e r m i n o l o g y
Part of the used terminology may be interpreted in a number of different ways. To make sure the intended interpretation is clear to the reader, the used interpretation of the important terminology will be clarified in this section.
1.6.1 Code quality
Code quality and code quality analysis may be interpreted in different ways.
A much used interpretation for code quality is for instance related to the metrics, e.g. the complexity and the LOC (lines of code) per class or func- tion. This interpretation may be considered limited, due to only covering the quality on a global level.
In this thesis, a different interpretation of code quality is used. The used interpretation can be considered far more complete. It does not only covers aspects such as the number of lines per class or function, but also provides a more thorough understanding of the code base. Apart from the global information, the code base is analyzed line-by-line, to determine if each line meets the code quality requirements. This provides a more thorough understanding of the quality of the code base. Therefore, the definition of
code quality in this thesis refers to a complete code quality including both the global code quality as well as a line-by-line code quality.
1.6.2 Rule violation
The SPCop tool analyzes the line-by-line quality of the code by applying rules. Each rule is applied to each line in the codebase that is scanned by the tool to determine if the requirements are met. The rules are provided to the tool in the form of a tool set. This ruleset holds the rules that are to be applied to the solution during the scan. When a line in the codebase does not meet the requirement that is found in the rule, a rule violation is added to the resulting report.
1.6.3 Defects
The term defect can be interpreted in different ways, the emphasis being on what is considered a defect and what is not. In this thesis, the term defect refers to an aspect that does not function as intended. However, even more important is what this includes in practice. As in general, the most promi- nent defects are the ones that cause crashes and other forms of application breaking errors.
However, the term defects also refers to all other forms of unwanted and unintended behaviour. This includes security related defects, e.g. access for accounts to locations that should be prohibited. Memory related de- fects, e.g. object destruction that should be performed manually since it is not automatically performed by the garbage collector. And performance re- lated defects, e.g. inefficient methods and constructions that take too long to perform the task at hand, and therefore put too much pressure on the performance of the solution.
2
C A S E S T U D Y D E S I G N
This chapter covers the design of the case study that is used in this thesis.
The chapter has been divided across different sections, each discussing a specific aspect in detail.
c a s e s t u d y Because of the nature of this research, it was decided to perform a case study since it best fit this thesis. The case study has been conducted based on the guidelines on case studies that have been provided by Runeson and H ¨ost [6].
u s e d r e s e a r c h m e t h o d o l o g y The purpose of this thesis can be clas- sified as exploratory, based on Robson’s classification [5]. Exploratory re- search includes relatively unstudied areas or new topics and generates ideas and hypothesis for future research. Therefore, it was decided to use the ex- ploratory research methodology since the topic of code quality analysis in SharePoint is relatively new. The exploratory research methodology is used to research the complete set of rules and determine its capacity to detect defects. However, the results of this thesis are by no means limited to future research, since it offers practical and directly applicable results.
t h e c a s e The case is defect detection through code quality analysis in SharePoint. The goal of this thesis is to research the SPCop tool, which is part of SPCAF, to determine the potential to detect defects based on the set of rules that is applied. Since determining the potential of this tool to detect defects is the case, it was decided to perform this study as a holistic case study as defined by Yin [8], since it focuses on a single unit.
t h e o r y Code quality analysis in SharePoint is a relatively new and unre- searched field of expertise. Because of this, the theoretical frame of reference on this specific subject is very limited. To fill this void, research from ad- jacent fields has been used. Even though this research was not specifically performed for this SharePoint, it still had overlapping aspects.
This has led to the literature study mostly revolving around research done in the code quality analysis field. The research was not language or environ- ment specific and was therefore not limited to a single technology. Instead this literature was focused on code quality analysis in general.
2.1 r e s e a r c h o b j e c t i v e s& research 2.1.1 Research objective
The purpose of this study is to evaluate and determine the defect detection capabilities of the SharePoint code quality analysis tool SPCAF based on a
set of rules, from the point of view of the Capgemini SharePoint division in the context of a production environment.
2.1.2 Research questions
The thesis is focused on the answering the following research questions.
1. What are the most frequently violated rules in SharePoint? SPCop comes with 400 predefined rules that can be used to scan SharePoint projects, but what rule violations are most commonly occurring? This research ques- tion is focused on determining what the most common rule violations are, using real-world projects.
2. What aspects of SharePoint are most prone to produce defects? Share- Point solutions consist of a wide array of different aspects and areas. Before specific defects can be detected, first it is important to determine what areas are most prone to produce defects. Therefore, this research question is focused on the orientation of the different areas.
3. What rule violations have a probability of resulting in defects? SPCAF offers extensive documentation on the rules that are used in the SPCop tool.
This research question is focused on determining what rules have the potential to detect potential defects, based on the documentation. The potential for rules to detect defects can not always be derived from the documentation.
Therefore, the remaining rules for which the defect detection potential remains undetermined, are presented to a group of experts.
2.2 d ata c o l l e c t i o n 2.2.1 SPCAF
Even though code quality analysis for SharePoint is still relatively new, ini- tial research was quick to show there is only one tool that is way out in front, SPCAF1. SharePoint Code Analysis Framework, or in short SPCAF, is a set of four tools, each focused on its own aspect of code quality in SharePoint.
In this thesis, version 4 of SPCAF was researched.
All four tools use WSP and APP files to perform their analysis on. The WSP and APP files are the binaries that result from building SharePoint code.
WSP files are packages that hold SharePoint solutions. APP files are pack- ages that contain SharePoint apps. The WSP and APP files consist of the code, the configuration files and other resources that are used.
The four tools of which SPCAF consists are
• SPCop
• SPMetrics
• SPDepend
• SPInventory SPCop
This first tool, SPCop2, is the tool on which the thesis is focused. In a simpli- fied approach, the tool can be considered to consist of two main components;
the engine that provides the functionality to scan a solution, and the set of
1 http://www.spcaf.com/
2 http://www.spcaf.com/analyzers/spcop/
rules, that are applied by the engine to the solution. The tool is used to detect and locate rule violations. This is done by scanning the WSP and APP packages and applying the rules that need to be satisfied by validating each line. In this scan, both the imperative code, e.g. the C# code files, as well as the declarative code, e.g. the XML configuration files are validated.
The tool is offered with several out-of-the-box rulesets, each set consisting of a certain set of predefined rules. For example the ”All Rules” set that includes all of the rules, or the ”Minimum Recommended Rules”, which includes the rules that the developers consider the recommended minimum.
Custom rules and rulesets can be created by the supplied SDK.
If a violation of one of the rules is encountered, the occurrence is added to the rule violations report. The rule violations are listed using the title of the rule that is violated and the location the violation was discovered. The lo- cation consists of the WSP or APP container the violation was found in, the file that contained the violation and the line number in which the violation was discovered.
SPMetrics
SPMetrics3is the tool that calculates the metrics. Its reports provide statis- tics on the WSP or APP file that was scanned. It counts the number of occurrences of all different sorts of aspects on different levels and scopes.
This is done on a project scale, meaning it provides its information based on the entire project rather than the line-by-line approach of the rule violation tool. The report it produces provides information on, among other things, the complexity and maintainability.
SPDepend
SPDepend4 is the tool that determines the dependencies in the project. Its reports provide detailed information about the dependencies, both Share- Point dependencies as well as external dependencies, that are found in the project. This data can also be visualized in a dependency diagram to show the dependencies between the WPS packages and apps.
SPInventory
SPInventory5 is the tool that documents all deployed content. Its report provides a detailed and complete documentation on the artefacts that are found.
2.2.2 Methods
The data will be collected using three methods, i.e. the reports from the SPCop tool, surveys and the SPCop rule documentation.
SPCop reports
Out of the four tools, SPCop is focused on detecting rule violations based on a set of rules. Therefore, even though all of the four tools offer valuable
3 http://www.spcaf.com/analyzers/spmetrics/
4 http://www.spcaf.com/analyzers/spdepend/
5 http://www.spcaf.com/analyzers/spinventory/
analysis reports, the SPCop tool is the only tool that is used in this thesis.
To determine the capabilities of the tool, the rule violations reports are an- alyzed, which is the output of the SPCop tool. This is done by analyzing SharePoint solutions. Based on the resulting rule violations report, it can be determined what the tool detects.
This method of data collection is primarily used in answering the first re- search question, which is focused on determining what the most occurring rule violations are. This is based on the analysis of actual real life projects, to determine the most occurring rule violations that are found in a production environment.
SPCop rule documentation
The answer of the first research question, and the SPCop reports in general, give a good view of the rules that are found in the SPCop tool. However, these rule violation reports only list the rules that were actually violated.
This means that only the rules that were violated in the analysed projects will be found.
Therefore, to include all of the rules that are applied by the SPCop tool, the documentation on these rules was used. This rules documentation lists the rules using the title of the rule and a description of the rule. This descrip- tion clarifies what code constructions triggers the rule violation and why this code constructions should not be used.
The SPCop rule documentation will be primarily used to answer the third and fourth research question. In the third research question, the rules will be analysed to determine if defects can be detected using these rules. The SPCop rules documentation will be used in the fourth research question to determine what rules have a probability to result in a defect.
Surveys
The behaviour that follows from using certain constructs in SharePoint, can be difficult to predict, because some aspects are hard to assess, unless they specifically have been previously encountered. Because of this, it was de- cided to collect knowledge from experts to fill in these uncertainties, and on top of that, to validate the information and conclusions. In the research questions, it became clear that the thesis should contain two surveys. The first survey being focused on the orientation while the second survey was to be more in depth.
The first survey was focused on orientation, featuring a small amount of questions. In each of these questions, participants were asked their opinion on the defect potential of a certain aspect of SharePoint, and its importance.
The survey had two goals, exploration and validation, to determine the po- tential problem areas that have a higher probability to result in defects. The orientation survey is the topic of the second research question, in which the survey and its results are thoroughly discussed.
The second survey was more in depth as it focused on gathering the expert opinion on the defect potential in specific rules. This survey featured 25 questions, each focused on the defect potential of a single rule. It discussed
the rules of which the potential to result in defects could not be determined based on the rule documentation. This second survey is the topic of the third research question, in which the survey and its results are thoroughly discussed.
2.3 d ata a na ly s i s
The thesis makes use of qualitative data analysis to interpret data and de- rive conclusions. In the derivation of the conclusions, the emphasis lies on providing a clear chain of evidence, enabling the reader to follow the reason- ing behind the intermediate results and conclusions. This can be considered important since this thesis has been performed by a single person, where at least two people may be preferable to reduce bias in a qualitative case study.
By providing a clear chain of evidence, offering all of the intermediate re- sults and the corresponding chain of evidence, the reader is able to validate the reasoning behind the results.
2.4 va l i d i t y a n d r e l i a b i l i t y 2.4.1 Replication strategy
The conclusions are derived from hard numbers and statistics. The way in which it is performed is comprehensively described. Therefore, the study should be easy to replicate. The results per study may differ, because the outcome of the reports depends to some extent on the projects that are used to gather the data. Apart from that, the data resulting from the survey is also prone to have some differences. The two main reasons being that it involves developers, which might produce differences for the obvious rea- sons, and because the developers are also influenced on a professional level since different companies have different conventions and ways of working.
Nevertheless, in theory the overall results should be very similar and more importantly, differences should be easy to justify and explain.
2.4.2 Quality assurance, validity and reliability
There is a strong chain of evidence in the case study, which leaves little room for interpretation. Apart from that, as the previous section on the topic of replication described, the study can be easily replicated. Therefore, the quality of this case study is assured considering it is easily verifiable.
3
R E S U LT S
In this third chapter, the results to the three research questions are pre- sented.
3.1 w h at a r e t h e m o s t f r e q u e n t ly v i o l at e d r u l e s i n s h a r e- p o i n t?
It was decided that the best way to find the most frequently occurring rule violation in SharePoint projects was by collecting the data from real world situations. This was done by using the data that resulted from generating rule violation reports using the SPCop tool. The process and its results have been divided across several sections.
3.1.1 Used projects
Three projects were used as samples to collect the data from. The codebases of the used projects were all in different stages of their life-cycle. Each stage in the life-cycle offers different sorts of challenges and problems, resulting in different violations. Therefore, using different projects in different stages offers the broadest range of violations. The three sample projects were se- lected based on their variety, together covering most of the possible rule violations found in practice, therefore providing a good representation of the real world.
The three used projects were anonymized real world projects provided by different SharePoint divisions in Capgemini. Capgemini has different sorts of activities concerning SharePoint. The activities range from in-house- development to management of projects that were developed elsewhere.
This provided a good spread of the rule violation that were made because of the projects originating from different backgrounds.
The combined ruleviolation reports can be found in Appendix C, starting on 79. This appendix offers the rule violation reports on the three projects.
This combined report contains both the data on all three reports, as well as the overall value, which is the sum of the three individual values. This is done by presenting four count columns per rule violations. The first column holding the count for project A. The second column holding the count for project B. The third column holding the count for project C. And the last column holding the combined count.
3.1.2 Interpreting the intermediate project results
The intermediate results for the three projects are discussed in this subsec- tion. Each project is discussed individually, starting with a small introduc- tion, followed by a table in which the results are presented, and ending with a bar graph, to visually present the results. The tables present the rule vio- lations of the project, which are divided across four levels of severity. These levels are sorted by decreasing severity, i.e. Critical error, Error, Critical warning and Warning. A bar graph is also created based on the data found in the tables, to visually represent the results.
Project A
Project A was being developed internally by Capgemini. At the time of this rule violation analysis, the project was still in its early stages of develop- ment.
Severity Count CriticalError 28
Error 150
CriticalWarning 1193 Warning 2517
Table 1.: Project A Summary
28 150
1193
2517
0 500 1000 1500 2000 2500 3000
Critical Error Error CriticalWarning Warning
Count
Severity
Figure 1.: Project A Summary Bar Chart
Table 1 presents the number of rule violations in Project A in a table and Figure 1 present a visual presentation of the results. Compared to the re- sults of the other two projects, the numbers over all four severity levels are relatively high. Especially the number of Critical errors and Errors is high considering the impact it may have on the system. Therefore, it can be con- cluded that the results of the code quality analysis do indeed confirm the immaturity of the code-base, which was still in the early stages of develop- ment when this analysis was performed.
Project B
Project B had been developed by an external organization, and Capgemini was managing the code-base. At the time of this rule violation analysis, this code-base was ready for use in a production environment according to the external organization that was responsible for the development.
Severity Count CriticalError 2
Error 3
CriticalWarning 215 Warning 2421
Table 2.: Project B Summary
2 3
215
2421
0 500 1000 1500 2000 2500 3000
Critical Error Error CriticalWarning Warning
Count
Severity
Figure 2.: Project B Summary Bar Chart
Table 2 presents the number of rule violations in Project B in a table and Figure 2 present a visual presentation of the results. While the preferable number of Critical errors and Errors should be zero, this number of errors is still low. However, on the other hand the number of Warnings is high.
These results clash with the fact that this code-base is used in a production environment, considering the relatively high number of rule violations. It may function as intended due to its low number of Errors, so on that ground it might be sufficiently ready for use in a production environment. However, due to the high amount of Warnings, the general quality of the code-base leaves a lot of room for improvements.
Project C
Project C had been developed internally by Capgemini. At the time of this rule violation analysis, the code-base was being used in a production envi- ronment.
Severity Count CriticalError 1
Error 2
CriticalWarning 173
Warning 276
Table 3.: Project C Summary
1 2 173 276 0
500 1000 1500 2000 2500 3000
Critical Error Error CriticalWarning Warning
Count
Severity
Figure 3.: Project C Summary Bar Chart
Table 3 presents the number of rule violations in Project C in a table and Figure 3 present a visual presentation of the results. Out of the three projects, project C clearly has the most mature code-base. It has a low amount of Critical Errors and Errors and, compared to the other projects, also a relatively low amount of Critical warning and Warnings. These re- sults confirm that the code-base was ready to be used in a production envi- ronment. Nevertheless, it can easily be concluded that even in these mature code-bases, the SPCop tool with its rule violation detection could offer a lot of valuable assistance on improving the quality of the code base even more.
3.1.3 Interpreting the intermediate combined results
Severity A B C All
CriticalError 28 2 1 31
Error 150 3 2 155
CriticalWarning 1193 215 173 1581 Warning 2517 2421 276 5214
Table 4.: Projects combined Summary
28 150
1193
2517
2 3 215
2421
1 2 173 276
0 500 1000 1500 2000 2500 3000
Critical Error Error CriticalWarning Warning
Count
Severity Project A Project B Project C
Figure 4.: Combined Projects Bar Chart
Table 4 presents the number of rule violations of the combined project in a table and Figure 4 present a bar chart of the results. Based on this figure, it can be concluded that the three researched samples are very different from one another, providing broad and representative data. Another conclusion that can be derived is that no matter the maturity of the code-base, the tool
was able, in all three cases, to provide valuable data on how to improve the quality of the codebase.
The bar graph also shows that there are a lot of Critical warnings and Warn- ings. Even though the severity levels indicate that Critical errors and Errors have a higher probability to have a negative influence on the system, that does not mean Critical warnings and Warnings could not have a similar negative impact. Therefore, the defect potential of violating any of the rules in all four of the severity levels will be researched.
3.1.4 Sorting the data
After the rule violation reports had been combined, of which the result can be found in Appendix B, the data was sorted. The results of this step, the fully sorted rule violation occurrence report, can be found in Appendix C, starting on page 79.
This report was created by processing each rule found in the combined re- port and placing it on its rightful position in the list based on its occurrence.
Rule violations with different counts are ranked numerically in a descend- ing order. Rule violations with an equal count are ranked based on their severity, in a descending order, the order of severity being CriticalError, Error, CriticalWarning, Warning. In case of an equal count and an equal severity, the rule violation was added after the last rule violation that had the same count and severity.
3.1.5 Conclusions
As discussed in the previous section, the complete sorted rule violation oc- currence report can be found in Appendix C. The conclusions will be based on three snippets from this complete sorted rule violation occurrence report.
Legend
The conclusion sections use the same table abbreviations as used in Ap- pendix C. Table 5 contains the abbreviations that are used on the categories as well as a short description on the category as taken from the documen- tation of the tool. Table 6 holds the abbreviations that have been used to denote the severity. Apart from that, the conclusion tables, table 7 and ta- ble 8 each hold the abbreviated columns Cat, Sev Cnt, which respectively stands for Category, Severity and Count.
Ten most occurring rule violation
Id Category Description
cor Correctness Correctness rules check the Share- Point XML code for syntax errors.
This includes check for all required XML attributes, correct values and data types of attributes.
sup SharePoint Supportability Checks if solutions endanger the supportability of SharePoint.
dep Deployment The deployment process of Share- Point customizations is often a crit- ical part. Deploying the wrong way or the wrong files can harm the SharePoint farm or make the farm inaccessible. Deployment rules check the code for these risks or po- tential problems.
sec Security Checks if solutions pose security is- sues.
des Design Warnings that check proper solution
design.
pra Best Practices Rules to warn if best practices are not used.
mem Memory Disposal Rules to warn for potential memory leaks due to wrong object disposal.
css CSSLint Runs CSSLint analysis tool.
loc Localization Localization is the process of cus- tomizing an application, webpage, or website for a given culture or lo- cale. The localization rules check if all attributes in XML which support localization are localized in a proper way.
nam Naming Checks files and artefacts for viola- tions against naming conventions.
jsh JSHint Runs JSHint analysis tool which
checks for issues in JavaScript files.
Table 5.: Violation occurrence table category legend
Id Severity ce CriticalError e Error
cw CriticalWarning w Warning
Table 6.: Violation occurrence table severity legend
Ten most occurring rule violations
Cat Sev Violation Cnt
loc w Use resources for localizable attributes 1642
jsh w Use curly braces around blocks 745
jsh w Use correct === and !== 700
jsh w Declare variable before it is used 430
jsh w Avoid trailing whitespaces 379
jsh w Do not exceed max length of a line in code 324 sec cw Avoid usage of ’RunWithElevatedPrivileges’ 196
jsh w Remove unused variables 189
nam w Files or Folders should contain the name of the par- ent solution
146 sec cw Avoid setting ’AllowUnsafeUpdates’ on SPWeb 142
Table 7.: Ten most occurring rule violations
Table 7 shows the ten most occurring rule violations. Conclusions on three different topics can be derived from this table, i.e. the most occurring rule violation, the high number of rule violations in the JSHint category and the number of warnings.
m o s t o c c u r r i n g r u l e v i o l at i o n s There are two things that make the number one most occurring rule violation distinctive. The first conclu- sion becomes immediately apparent from the table; the count is a lot higher than the rest of the rule violations. The second reason is found in the com- bined rule violation report in Appendix B, specifically B.9 on page 76. This data shows that the most occurring rule violation is the sole rule violation in the Localization category. On top of that, a large share of the number of rule violations is found in project A. Nevertheless, the rule violations are well represented in all three projects.
j s h i n t c at e g o r y The first column of the table shows that no less than six out of the the ten most occurring rule violations are found in the JSHint category. This means that a lot of the rule violation are found in the Javascript code. However, this does not necessarily means that violating these rules has a negative impact on the stability of the system, some of these rules might be focused on convention. Nevertheless, it still is something worth to take into account, when optimizing the code and improving the code quality.
wa r n i n g s The last conclusion that was derived is regarding the severity of these ten most occurring rule violations. Two out of ten rule violations are critical warnings, while the remaining eight are warnings. This makes the highest severity a critical warning. When looking at the first table of the complete sorted rule violation occurrence report, table 54 on page 80, it becomes apparent that as many as the first twenty-two rule violations are either critical warnings or warnings. The question remains, how much impact these warnings have on the system. Perhaps there is impact found in the critical warnings. Especially since both critical warnings are found in the security category, which increases the potential impact. The hypthesis is that these warnings generally will not have an impact on the stability of
the system. Nevertheless, no matter what the impact is in the area, all the rule violations can be used to improve the quality of the code.
Cat Sev Violation Cnt
cor e Define attribute ’ID’ in FieldRef in correct casing 50 dep e Do not deploy assembly multiple times 28 cor e Declare required attributes in schema of ListTemplate 27 dep e Do not deploy assembly with DEBUG mode 24 cor ce Define unique value for ’Id’ in CustomAction 22 dep e Do not deploy TemplateFile multiple times 10 sup ce Do not access SharePoint API via reflection 6 sup e Do not read ConnectionString from SPContent-
Database
6 cor e Declare required attribute in CustomAction 4 cor e Declare required attributes in SiteDefinition 4
Table 8.: Ten most occurring (critical) errors
t e n m o s t o c c u r r i n g(critical) errors Table 7 shows the ten most occurring rule violations with a severity of either critical error or error. It was decided to show this list of rule violations as well considering its higher severity, i.e. higher potential to result in defects when violated. There are two aspects that immediately become apparent, the relatively low number of occurrences and the categories these rule violations are found.
o c c u r r e n c e s The number of occurrences of the ten most occurring crit- ical errors and errors are relatively low. Especially when compared to the amount of occurrences of the ten most occurring rule violations. However, each of these errors have a heightened potential to have a direct negative influence on the stability of the system. Therefore, the information that is provided by these reports can be considered valuable, since they directly provide information on aspects that require immediate attention.
c at e g o r i e s The other aspect that is hugely different between the ten most occurring rule violations and the ten most occurring errors are the cat- egories of the violations. Where the top ten most occurring rule violations was dominated by the JSHint front-end category, the top ten most occurring errors is representing the SharePoint backbone categories. Perhaps this is also the difference between the JSHint category rule violations, that do not have a large negative influence on the system, but rather result in visual problems, and the SharePoint backbone categories, that potentially have a huge impact on the ability of the system to function as intended and the overall stability of the system.
3.2 w h at a s p e c t s o f s h a r e p o i n t a r e m o s t p r o n e t o p r o d u c e d e f e c t s?
SharePoint is a very broad platform, which consists of a lot of different as- pects. Before zooming in on what specific rules have the potential to result in defects when violated, it was first determined what general aspects of SharePoint were most prone to result in defects. To achieve this, it was decided to ask experts about these aspects, by means of a survey. The ques- tions are based on two sources of information. First, on the results of the first research questions, in which it became clear what rules were most vio- lated in the sample projects, to determine their impact. And second on the preliminary results of the analysis of the rule documentation, in which it became clear what the rules covered.
The orientation survey was distributed on the SharePoint related groups on the yammer platform of Capgemini, targetting all employees working with SharePoint. The number of responses to this survey was rather low.
Out of all of the international employees working at Capgemini with Share- Point, who were active on the yammer community, only ten responses were returned. This low number of responses can be considered a negative in- fluence on the validity. Nevertheless, the results are still considered useful.
Even though it may not be entirely representative, it still offers valuable insight into what is considered most problematic, by the experts that did respond.
In this section, responses to the six survey questions will be discussed in- dividually. Each question will be discussed starting with the question that was presented to the participant to offer context. The question is followed by the raw data, that was collected from the responses. This includes both the multiple choice results and, if provided by the participants, the com- ments that were given. This raw data is interpreted in the interpretation section that follows. Finally, the discussion is completed with a conclusion, to determine the outcome of the results and what this means for the thesis
3.2.1 Question 1
SharePoint projects are built using two types of code. The first type being the imperative code, which is the C# code. The second type being the declarative code, which is the XML code. Which of the two types of code do you consider more prone to produce defects?
Raw data
Answer Count Percentage
Imperative code, e.g. C# 5 50%
Declarative code, e.g. XML 4 40%
Equal 1 10%
Total 10 100%
Table 9.: Multiple choice results question 1
m u lt i p l e c h o i c e r e s u lt s
c o m m e n t s Each comment is accompanied by the multiple choice answer the participant choose, to offer context.
• Imperative code, e.g. C#: Since most of the times we will try to code in C#, XML will the case to communicate between the two websites.
• Declarative code, e.g. XML: XSL
• Imperative code, e.g. C#: XML is more sensitive to typo’s, so small mistakes have big consequences. But usually those are easier to fix.
• Declarative code, e.g. XML: It’s more error prone. But impact on stability and performance is significant less.
Interpretation
The first question was designed to determine what type of code, i.e. im- perative code or declarative code, was considered most prone to result in defects. The Correctness category, one of the categories used in the rule doc- umentation, is focused on detecting the violation of declarative code rules.
An example of this is the ”Fix schema errors” rule, which enforces that the XML files within a WSP solution should not contain schema errors. There- fore, it was interesting to determine how defect prone both types of code were considered.
10
50 40
0 10 20 30 40 50 60
Equal Declarative code e.g. XML Imperative code e.g. C#
Percentage
Answer
Figure 5.: Graph bar displaying the results of question 1
The results are displayed in Figure 5. the figure shows that 10 percent of the participants considers imperative code and declarative code equally prone to have the potential to cause defects. The remaining 90 percent chose the imperative code and declarative code options nearly the same amount of times, making it an indecisive difference.
Question 1 was the only question that allowed participants to leave a com- ment, on top of their choice for one of the multiple choices answers that were provided. This was done because of the unavailability of the ”Other”
option that allowed the participant to leave their own answer, which was left out due to the nature of the question.
In total, 4 comments were given to provide additional insights to this ques- tion. The first comment stated that coding was primarily done in C#, and XML was mostly used to facilitate communication between websites. The second comment consisted of a single word: XSL, which is a form of declar- ative code. The third comment stated that XML is more sensitive to syntax related mistakes, and that these small mistakes may have big consequences.
However, these are supposed to be easier to fix, leaving choice on the mul- tiple choice part to the imperative option. And to conclude, the fourth
comment stated that XML is more error prone but the impact on the stabil- ity and the performance is significantly less, resulting in the choice for the declarative option.
Conclusion
Only a single person considered both types of code to have an equal poten- tial to result in defects. Most of the experts decided on one of the two types of code, to have the most potential to result in defects. Nevertheless, these participants were being almost equally divided over the two options target- ing a single type of code. Therefore, it can be determined that the overall conclusion is that both types of code are considered equally prone to result in defects. Because of this, in the rest of the thesis, both types of code will be considered equally prone to result in defects.
3.2.2 Question 2
The absence of referenced resources will most probably be followed by defects. How- ever, a missing resource should be easily detectable. From this perspective, how severe do you consider this kind of defects?
Raw data
Answer Count Percentage
Not severe at all considering it is easily detectable. 2 20% Moderately severe, it may be easy to detect, but
that does not make the defect less important.
6 60%
Highly severe, it is important such a defect is de- tected and solved as soon as possible.
2 20%
Other, 0 0%
Total 10 100%
Table 10.: Multiple choice results question 2
m u lt i p l e c h o i c e r e s u lt s Interpretation
The second question was designed to determine the severeness of leaving out referenced resources, the emphasis being on the influence, considering the reason behind the defect should be relatively easy to detect. The deploy- ment category, contains rules that are focused on determining if required files are missing. An example of this is the ”Deploy missing image for Fea- ture” rule, which enforces that when a custom icon for a Feature is defined in attribute ’ImageUrl’, the file should be deployed to the SharePoint system.
0
20
60 20
0 10 20 30 40 50 60 70
Other, Highly severe, it is important such a defect is detected and solved as
soon as possible.
Moderately severe, it may be easy to detect, but that does not make the defect less important.
Not severe at all considering it is easily detectable.
Percentage
Answer
Figure 6.: Graph bar displaying the results of question 2
The results are displayed in Figure 6. The responses to this survey ques- tion were somewhat divided across the three different levels of severity, not severe, moderately severe and highly severe. However, most of the partici- pants considered the potential for defects moderately severe.
Conclusion
Only 20 percent of the participants considered the potential to result in de- fects to be highly severe. On top of that, 60 percent considered the potential to result in defects to be moderately severe. Overall, this means that a total of 80 percent considered the potential for defects at least moderately severe.
This is a good indication that this aspect of SharePoint has to be monitored when analyzing the code quality.
3.2.3 Question 3
On the contrary, having too many resources might also result in defects. In some cases SharePoint deploys prohibited assemblies, e.g. ’ssocli.dll’, or includes the same assembly in different WSPs. How severe do you consider this kind of defects?
Raw data
Answer Count Percentage
Not severe at all considering it is easily detectable. 1 10% Moderately severe, it may be easy to detect, but
that does not make the defect less important.
7 70%
Highly severe, it is important such a defect is de- tected and solved as soon as possible.
0 0%
Other, 2 20%
Total 10 100%
Table 11.: Multiple choice results question 3
m u lt i p l e c h o i c e r e s u lt s o t h e r,
• Don’t know. Seems to be a problem with wspbuilder? Never experi- enced this.
• Moderately severe, because this is not Always easily detectable. The code may compile fine, but if you have some assemblies duplicated in several places, it could lead to strange, unexplainable effects after deployment..
Interpretation
This question was designed to determine through the experts, the negative influence of adding too much assemblies, i.e. prohibited assemblies or in- cluding the same assembly twice. This kind of rule was also part of the deployment category. An example of such a rule is ”Do not deploy pro- hibited assemblies”, which states that in some case WSPBuilder includes prohibited assemblies into the package, e.g. ’ssocli.dll’.
20 0
70 10
0 10 20 30 40 50 60 70 80
Other, Highly severe, it is important such a defect is detected and solved as
soon as possible.
Moderately severe, it may be easy to detect, but that does not make the defect less important.
Not severe at all considering it is easily detectable.
Percentage
Answer
Figure 7.: Graph bar displaying the results of question 3
The results are displayed in Figure 7. Out of the 80 percent of the par- ticipants that chose one of the multiple choice answers, only 10 percent considered the defect not severe. The remaining 70 percent considered the potential to result in defects to be moderately severe. That leaves 20 per- cent of the participants, that choose the Other option, and gave their own response to this question. One of the remarks that was made, was that the participant never encountered this particular defect, which might indicate that it is rather uncommon. The Other answer from the second participant stated that even though the code may compile fine, it could lead to strange and unexplainable effects, which makes it difficult to locate and fix this kind of defect.
Conclusion
Based on both the 70 percent that choose the moderately severe option, and the 10 percent that indicated through comment that it could result in strange and unexplainable effects, that also labeled it moderately severe, a total of 80 percent considers the potential to result in defects to be moderately severe.
Therefore, it was concluded that this kind of rule violation in SharePoint will be considered moderately severe, meaning this kind of rule violations will be monitored when analyzing the code quality.
3.2.4 Question 4
In the XML configuration files, a lot of required attributes have to be defined. Not filling in these required attributes might result in defects. How would you value assistance on this kind of defects?
Raw data
Answer Count Percentage
Very important, I consider these defects to be se- vere.
4 40%
Moderately important, assistance would help me solve these defects faster.
3 30%
I do not need assistance, these defects are easy enough to solve on my own.
2 20%
Other, 1 10%
Total 10 100%
Table 12.: Multiple choice results question 4
m u lt i p l e c h o i c e r e s u lt s o t h e r,
• Mostly filled via Visual Studio (?). Don’t understand the question.
Interpretation
After initial research into the SPCAF tool, it soon became clear it offered good detection and assistance on missing required attributes in the XML configuration files. This question was designed to question the experts on the severity of this kind of defects, to determine its impact on the stability, and to ascertain the importance of this feature. This was also an aspect of the Correctness category. An example for such a rule is ”Declare required attributes in SiteDefinition”, which states that Required attributes Title and ListDir must be declared in SiteDefinition.
10 20
30 40
0 10 20 30 40 50
Other, I do not need assistance, these defects are easy enough to solve
on my own.
Moderately important, assistance would help me solve these defects faster.
Very important, I consider these defects to be severe.
Percentage
Answer
Figure 8.: Graph bar displaying the results of question 4
The results are displayed in Figure 8. The majority, 40 percent of the partic- ipants considered the assistance very important, since they considered the potential defect to result from this severe. 30 percent considered the assis- tance moderately important, mainly because it allowed them to solve the problematic code faster. Only 20 percent of the participants stated that they did not find the assistance valuable.
There was also one participant that responded with the Other, options, ac- companied by text that stated that he did not understand the question. Con-
sidering the other 90 percent had no problem answering the question, it is difficult to determine if the question was weak or if it was due to the lack of knowledge on this particular subject.
Conclusion
Based on the results, it can be gathered that 70 percent of the participants considered the assistance to be valuable. Out of this 70 percent, the majority, 40percent, considered the potential defects so threatening they considered it very important. In conclusion, the severity of this type of defect will be regarded as highly severe since 70 percent considered it at least moderately important and most participants, 40 percent, considered it very important.
3.2.5 Question 5
A possible aspect of security related defects is that they may not be as detectable as other defects. This because security related defects do not have to cause crashes but rather result in unwanted behaviour that is more difficult to discover. Also, these defects might pose a big threat. How do you value security related defects that produce this unwanted behaviour?
Raw data
Answer Count Percentage
Highly severe, any assistance in this field would be highly helpful since these defects are hard to detect.
7 70%
Moderately severe, these kind of defects are not that much of a problem.
0 0%
Moderately severe, these kind of defects are un- common.
3 30%
Not severe, these kind of defects are no problem at all.
0 0%
Not severe, testing is done in such a way that these kind of defects are easily found and solved.
0 0%
Other, 0 0%
Total 10 100%
Table 13.: Multiple choice results question 4
m u lt i p l e c h o i c e r e s u lt s
i n t e r p r e tat i o n Even though security related defects might not have a huge impact on the stability of the software, possible defects may have an even bigger impact on the system. Software carrying security related defects may appear to function as intended, but would malfunction in the security area, e.g. it may provide entrance to users to parts that should not be ac- cessible. This question is designed to determine how valuable the experts consider detection and assistance in the security area. This question was focused on the Security category. An example of such a rule was ”Do not
nest calls to RunWithElevatedPrivileges”, which states that RunWithElevat- edPrivileges should not be called inside an existing RunWithElevatedPrivi- leges.
0 0 0
30 0
70
0 10 20 30 40 50 60 70 80
Other, Not severe, testing is done in such a way that these kind of
defects are easily found and solved.
Not severe, these kind of defects are no problem at all..
Moderately severe, these kind of defects are uncommon.
Moderately severe, these kind of defects are not that much of a problem.
Highly severe, any assistance in this field would be highly helpful since these defects are hard to detect.
Percentage
Answer
Figure 9.: Graph bar displaying the results of question 5
The results are displayed in Figure 9. It shows that 70 percent of the par- ticipants consider the potential to result in defects, highly severe, and that assistance on this type of defects is highly appreciated. The other 30 per- cent of the participants consider the defects moderately severe on grounds of them being uncommon.
Conclusion
This means that a total of 100 percent considered the defect to be at least moderately severe. Considering that the vast majority, 70 percent, consid- ered the defect to be highly severe, it can be concluded that this is a very important type of defect. Therefore, it is important that this type of defect will receive the proper attention when analyzing the code quality.
3.2.6 Question 6
The questions have covered the following defects:-Missing resources defects-Having too many resources defects-Missing attributes in XML defects-Potentially difficult to detect security related defects. Is, or are there any categories of defects that have not been covered by the previous five questions? If so, what categories or types of defects do you feel are not represented?
Raw data a n s w e r s
• Correct
• Moderately severe, these kind of defects are uncommon.
• – Memory Defects (disposing instances of SPWeb, SPSite, etc...) – ”Performance” defects ( getting all items/fields in the list instead
of selected items/fields. ...)
• – Common coding mistakes (wrong syntax, improper use of vari- ables, endless loops).
– Performance related defects (code causing memory leaks, table scans, endless loops).
– Partially related to having to many (duplicate) resources: not us- ing unique GUIDs for all of your components.
