How to ensure secure and up to date configuration at customers that use a web application of a COTS-supplier

(1)

How to ensure secure and

up to date configuration at

customers that use a web

application of a

COTS-supplier

Sander van Hulst

24-08-2017

Master’s Software Engineering

Supervisors UvA: Wil Leeuwis

Hans Dekkers

Supervisor company: Remko Nienhuis

Organisation: ChipSoft BV

University of Amsterdam

(2)

2

3 Acknowledgements

I would like to thank everyone who helped me to write this thesis.

This would never have been possible without the feedback I received from my supervisors, Wil Leeuwis and Hans Dekkers.

Wil called me weekly to discuss the progress and to provide feedback, even when he was on holiday.

I want to thank my company supervisor Remko Nienhuis, for his flexibility when the topic of this thesis changed, for providing clear feedback, and for allowing me to spend more time on this thesis than I initially thought I needed.

I also want to thank Dominique Stoverink for keeping me company during the countless weekends that we worked on our theses together, and for convincing me to not give up.

(4)

4 Abstract

Incorrect security configuration is prevalent in web applications.

It was analysed just how prevalent these issues are for ChipSoft’s eHealth solution, CS-Zorgportaal.

Support tickets related to security, that ChipSoft received in the period from April 2009 until May 2017, were analysed. A search was conducted in the literature to find proposed

solutions for the reported configuration issues in these support tickets. Of the articles found, not one study proposed a satisfactory solution for the issues in the ChipSoft case.

The articles, however, did lead to a part of the requirements for a prototype solution. Security-related configuration issues that were commonly reported in the analysed support tickets lead to specific requirements on what the prototype should detect.

Subsequently, a prototype software application was developed in Python that detects

common configuration issues in customer implementations of ChipSoft’s CS-Zorgportaal web application.

A test run was performed on 65 CS-Zorgportaal web applications of 39 customers. On 25 of the 65 web applications, security configuration issues were detected.

Besides these issues, it was also found out that preproduction and acceptation environments were exposed to the internet. This was reported to the customers.

Using this prototype in production would help ChipSoft to get clear overview of the state of configuration of its customers. Fast reporting of these issues would then reduce the time that the incorrect configuration can be abused in the web application.

(5)

5 1. Introduction

1.1. The company

ChipSoft is an IT company that develops solutions for the healthcare sector. The company is headquartered in Amsterdam, the Netherlands, and has offices in

Hoogeveen, Heerenveen, and Niel (Belgium). ChipSoft’s main product is an EHR system (Electronic Health Record), which is called HiX (Health Information X-Change).

ChipSoft is the market leader in the Netherlands for EHR systems, with hospitals being the main customers.

Besides an EHR system, they also develop eHealth solutions. Among the eHealth solutions that the company offers are a self-service kiosk, a web portal for patients and/or physicians and a mobile app.

During this internship, I worked with the team responsible for development of the web portal, CS-Zorgportaal. The customer organisations that use this software are hospitals or specialist treatment centres.

1.2. CS-Zorgportaal

CS-Zorgportaal is a web application built on the Microsoft SharePoint platform. Microsoft SharePoint is a framework commonly used for internal web sites to share documents within an organization.

Microsoft SharePoint however also allows for the development of custom web applications, called SharePoint solutions. [1]

Functionalities of the web frontend of CS-Zorgportaal are developed as custom ‘web parts’. Web parts can be seen as Lego bricks that each provide a functionality.

The web parts can exchange information with each other, for example an ‘Appointment overview’ web part can send the selected Appointment to an ‘Appointment details’ web part which then displays information about the specific appointment.

This example is shown in the figure below.

Figure 1: Appointment web parts in CS-Zorgportaal

The fact that the web parts provide small functionalities and that they can be dragged and dropped on webpages to the customers liking, allows for a different user experience in different hospitals without requiring more development.

Architecture

The information that the web parts display needs to be retrieved from the HiX database that contains patient data. In earlier versions of CS-Zorgportaal, the web application

(6)

6

With time, this was perceived as in conflict with security best practice.

Therefore, in the newest versions CS-Zorgportaal communicates with a backend called HAS (HiX Application Services). Requests by CS-Zorgportaal are validated and reauthorized by HAS. HAS communicates with the HiX SQL database.

This trust boundary between CS-Zorgportaal and HAS makes the new model inherently safer. The data flow of CS-Zorgportaal is displayed in the figure below.

Figure 2: Data flow of CS-Zorgportaal

1.3. Audit process

In the development life cycle of CS-Zorgportaal, measures have been implemented to prevent software errors and incorrect configuration.

For example, software is tested manually by a developer and a software tester to verify if the software works as intended. Unit tests are executed during each build of the software. But after the deployment phase, ChipSoft can no longer verify if software, which is now configured by a customer organisation, is working as intended.

The only verification of the software along with its configuration, is performed by external auditors.

The reason that these audits are performed is that one of the supported authentication methods by CS-Zorgportaal is ‘DigiD’, the Dutch digital passport. The Dutch government has decided, due to the discovery of multiple vulnerabilities within web applications belonging to

(7)

7

Dutch municipalities, to make third party assessments required if organisations want to use DigiD in their web application.

Customers that have a CS-Zorgportaal web portal which is connected to DigiD, thus have to face third party audits every year. Such an audit needs to be performed by an independent auditor who has to be registered in the Dutch Register EDP. [2]

This auditor not only performs a vulnerability scan of the internal network of the hospital, but also thoroughly tests the software functionality of the web portal.

As ChipSoft does not permit third parties to have access to the source code of their software, the vulnerability scans are limited to a black box approach. In order for the software to

successfully pass this audit, it needs to conform to subset of a list of security guidelines issued by the Dutch National Cyber Security Centre (NCSC). [2]

This list of security guidelines is revised periodically, and the demands on web applications have gotten stricter in recent years. The figure below shows one of the points from the latest version of the NCSC guidelines.

Figure 3: NCSC Guideline U/PW.03

The NCSC guidelines are divided into three different domains: respectively the policy domain, the implementation domain and the control domain.

The policy domain consists of requirements on the policy and organisational procedures of the organisation that hosts CS-Zorgportaal.

In the implementation domain, security requirements are described on the web application itself and the supporting infrastructure (web servers and other hardware and software that are in use by the customer organisation).

In the control domain, requirements are specified related to maintenance and monitoring of the CS-Zorgportaal web application and the infrastructure by system administrators of customer organisation.

(8)

8

Many of the findings that are reported related to these criteria are influenced by configuration of the software and hardware within the customer organization.

The findings are usually reported to ChipSoft by the customer through ChipSoft’s support ticket system.ChipSoft then replies within a few days or weeks, with an official reaction to the findings.The reaction time differs depending on the complexity of the vulnerabilities that are reported. From experience of the team members in the teams responsible for CS-Zorgportaal, issues related to security misconfiguration are common and take much time to analyse and solve.

1.4. Problem statement

Auditors brief findings based on vulnerability scans of the software made by CS-Zorgportaal and of the infrastructure of the customer organisation.

Many findings in audit reports could have been prevented with correct configuration, but neither ChipSoft nor the system administrators were aware of incorrect configuration until this was reported by the auditor.

This problem is related to the fact that ChipSoft’s consultants are responsible for the

implementation and configuration when a hospital goes live with the software. But, when the hospital is live, further configuration will be done by the system administrator of the hospital. ChipSoft currently has no overview of the state of configuration of its customers.

In the most recent version of the DigiD assessment norms, incorrect security configuration is however a requirement to get permission for the web application to keep using DigiD

authentication.

Since 1 January 2016, a new law has also come into effect in the Netherlands: the reporting obligation for data breaches (meldplicht datalekken). [3]

This law states that when confidential personal data is leaked, organisations must report this within 72 hours after the event occurred.

When a data breach is not reported, the responsible organization can get fined up to the amount of €820.000 or 10% of the yearly revenue. [4]

The same fines also apply when the organisation’s security was not adequate.

The software supplier can also be held accountable for these fines, when it is shown that the software supplier is (partly) responsible for the data breach.

And even if this was not the case and there was no such thing as chain responsibility in law, customer organisations can either sue or avoid the software supplier in the future, should they become subject to these fines and the resulting reputation damage that a data breach brings along.

The software supplier must thus show that customer organisations take adequate countermeasures by using their software.

The goal that ChipSoft has, is to implement the system in an optimal secure way, but also to ensure that the system maintains in that state after the initial implementation.

This goal brought me to the following research question:

How can it be ensured that configuration of ChipSoft software at customer organisations is up to date and secure and maintains in that state?

(9)

9

1.5. Research method

The objective of this study was to get an overview of security configuration issues in the ChipSoft case and to look for a solution for these issues.

To also get a feeling of the occurrence of similar issues in other companies and to learn from similar issues, I researched literature to find out how prevalent incorrect software

configuration actually is.

Following from this information, I analysed and described what forms of incorrect

configuration there are and to what issues these forms of incorrect configuration can lead. I then searched literature to find out what measures are commonly taken against these forms of incorrect configuration.

After this, I zoomed in on the ChipSoft case, so that I could get a solution for the problem statement from which this study was derived. I described what measures ChipSoft already took against security configuration errors and how these measures are implemented. Subsequently, I analysed and described the prevalence of configuration issues within ChipSoft, despite the existing measures.

To achieve this, I researched the support tickets that have been submitted in the past years. These support tickets are stored within a SQL-database.

I collected the relevant support tickets using a query that filters for security-related keywords. A number of common configuration issues were found. These issues are suitable for

automatic detection. A software prototype was developed to perform automatic detection of these issues.

The prototype was tested in practice on the CS-Zorgportaal portals of 39 customers. I described the results of the prototype test run and how ChipSoft and its customers dealt with these results. Finally, I described how the prototype software solution can be improved and extended for usage in production.

The information gathered from the earlier found literature about proposed solutions for these issues was analysed and considered for implementation within ChipSoft.

(10)

10 2. Background & context

To understand the origin of the issues described in this thesis, it’s important to know some background. In this chapter, I will therefore describe some information on different relevant topics to give the reader some context.

Lastly, I will also describe some terms the reader should know to grasp the essence of this thesis.

2.1. Background

System administrators and security risks

System administrators are responsible for software and hardware configuration, but also have other responsibilities.

An analysis by Anderson reveals that the primary tasks of a system administrator include maintenance, configuration and end user training. [6]

Gagne states based on interviews that system administrators typically only focus on security when something bad happens. They perceive security professionals as ‘enforcers’ who make their work harder, in contrast to other IT professionals who work to make it easier. [7]

While security may be an aspect of the job of a system administrator, it is fundamentally different than the maintenance of systems.

Because of this it may be hard for a system administrator to identify whether a log message actually signals a security risk as this often requires tacit knowledge. [7]

Collaboration between developers and administrators

Traditionally, collaboration between developers and operations teams in software development was not described in software process methodologies.

Virmani states that IT organisations have adopted principles which have led to an increase in development speed, such as continuous integration. A lack of emphasis on the operations side of the development process has however lead to operations teams being unable to keep up with this increased development speed. [8]

DevOps is a relatively new way of working which lays focus on collaboration between the development and the operations teams.

The term DevOps refers to “a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes”. [9]

Lwakatare et al. describe that the focus on this collaboration between Development and Operations teams leads to systems that are designed to expose relevant information through logging functionalities which can make monitoring for errors more effective. [10]

However, literature about applying the DevOps approach describes collaboration between development and operations teams within the same organisation.

In ChipSoft’s case, the CS-Zorgportaal software running at customer organisations is not operated by ChipSoft.

It is the responsibility of the operations team of the customer organisation to update and maintain the software.

Obviously, the operations team of the customer organisation is not paid by ChipSoft which makes intensive collaboration, as is done in a DevOps approach, difficult.

Effectivity of automatic vulnerability scanning versus manual audits

Vulnerability scanners are often marketed as point-and-click scanners which can run and detect vulnerabilities on any kind of web application.

Doupé et al. compared the effectiveness of 11 vulnerability scanners on a web application they developed themselves which purposely contained 16 different vulnerabilities. [11] The scanners ran with default out-of-the-box configuration, with specific configuration options and manually.

Even with the sum of results of those three test runs combined per scanner, not one scanner detected more than 40% of the vulnerabilities in the web application.

(11)

11

The authors identified issues with the vulnerability scanners in the proxy mechanisms used, which caused the scanners to not be able to upload a picture in the web application.

Often a vulnerability scanner did also not understand Flash or JavaScript, which meant that the vulnerabilities on webpages that used these technologies could not be detected.

HTML with a slight typing mistake also resulted in an inability to parse the web page and thus to detect a vulnerability which was present on that page.

They state that besides these obvious bugs, vulnerabilities in the application logic that were included in the web application often required contextual knowledge, which the vulnerability scanners lacked.

This is knowledge that should come naturally to users of a specific application: for example ‘What actions do I need to perform to get from this page to a different page?’ and ‘What text fields are required to submit this form?’

Some of the vulnerabilities were eventually detected by manually making the vulnerability scanners perform actions in the way they needed to be performed, but the remaining 60% could still not be detected. This is shown in the figure below. [11]

Figure 4: Detection and false negative rates of vulnerability scanners, taken from Doupé et al.

As web applications become more complex, Doupé et al. predict that vulnerabilities which require an understanding of contextual knowledge to detect will become more and more prevalent, making this a pressing issue to solve for developers of vulnerability scanners. Another problem is the amount of false positive vulnerabilities that are reported.

Fonseca et al. reported false positive rates between 20 and 77% (!) when they compared three vulnerability scanners to scan a web application in which they purposely injected vulnerabilities that are commonly found in web applications. [12]

These issues raise doubts about both the sensitivity and specificity of automatic vulnerability scanners. Usage of these scanners can therefore not simply lead to satisfactory conclusions about the security of a web application.

Some form of manual vulnerability scanning will still need to be performed or specific rules about the needed domain knowledge will need to be included in an automated vulnerability scan.

Kupsch et al. confirmed the findings described above in a study on the effectiveness of automatic versus manual vulnerability assessments. [13]

The tools that the authors found only a few of 15 vulnerabilities they tested for, none of which needed an understanding of the working of the software.

(12)

12

2.2. Terminology

Configuration

In version 3 of the SWEBOK Guide, configuration is defined as the “functional and physical characteristics of hardware or software as set forth in technical documentation or achieved in a product”. [14] [15]

There are however more definitions of configuration, and it can also mean “the manner in which the hardware and software of an information processing system are organised and interconnected”. [15]

From these definitions, we can derive that there are three kinds of configuration:  Physical configuration

Examples of physical configuration are locks on the doors of the server rooms and power backups.

 Hardware configuration

Examples of hardware configuration are separation of networks with a Demilitarized Zone (DMZ) and defining firewall rules.

 Software configuration

Software configuration includes, but is not limited to, the settings of webservers, settings of (web) applications, and authorization levels of user accounts.

Hardening

Securing a system configuration is called ‘hardening’.

The goal of hardening is to reduce the attack surface that is exposed by the system. An attack surface is determined by the features or services that are running on the system; these may be vulnerable and can thus help attackers to achieve a system compromise. In IT security guidelines it is thus recommended to disable all unnecessary services. A service is unnecessary simply when it is not needed to fulfil the business needs. [16] A hardware or software firewall can ensure that even if unnecessary services are running, they are not exposed and therefore the attack surface does not increase.

Vulnerabilities in essential services may however also be discovered, and this is why patch management is crucial.

User accounts should also not have more rights than needed for their use case.

If a database-account only has to read data, it should not have the role of database owner but an account with only reading rights. This is called the principle of least privilege. [17] Other terms and abbreviations

 Business impact: the impact a successful attack has on the business, expressed in terms of financial damage, reputation damage and privacy violations.

 EHR: Electronic Health Record

 IIS: Internet Information Services, a web server developed by Microsoft.

 NCSC: Nationaal Cyber Security Centrum, or translated to English ‘National Cyber Security Centre’. A Dutch sub organisation of the NCTV (Nationaal Coördinator Terrorismebestrijding en Veiligheid) that is tasked with improving the resilience of the Dutch society with regard to cyber security.

 STS: Security Token Service

 Technical impact: the impact a successful attack has on the system, related to availability, integrity, accountability and confidentiality.

(13)

13 3. Research

The objective of this thesis is to answer the following question:

‘How can it be ensured that configuration of ChipSoft software at customer organisations is up to date and secure and maintains in that state?’

This is however a very broad question and it contains many smaller topics. To get to an answer, I will therefore first answer sub-questions following from this calling:

 Why is this an important thing to do?

 What does ChipSoft already do to ensure this?  What issues arise because of misconfiguration?

 What possibilities are there to ensure that configuration remains up to date and secure?

What can ChipSoft do to ensure up-to-date and secure configuration at all times at customer configurations?

3.1. Why is this an important thing to do?

So, CS-Zorgportaal wants to ensure that configuration of ChipSoft software at customer organisations is up to date and secure and maintains in that state. They want to find out how to do so, but first I want to explain; why should they want to?

To answer this question, we differentiate between three forms of configurations: physical configuration, hardware configuration and software configuration.

Each of these categories of configuration relates to a different category of risks.

Configuration affects both the hardware and software of an information system and therefore multiple layers of the OSI-model.

This model is displayed below.

Figure 5: OSI model [18]

Each of the layers of the OSI-model can suffer from vulnerabilities.

In the table below I will describe this using common vulnerabilities that can occur in each layer, as described by Reed. [19]

(14)

14

Layer Vulnerabilities

Application Logic flaws, inadequate security controls, design flaws, poor implementation of complex security controls

Presentation Crashes due to poor handling of unexpected input, manipulation or information leakage, cryptographic flaws

Session Weak or non-existent authentication mechanisms, passing credentials in clear text, spoofing or hijacking of sessions, leakage of information based on failed authentication attempts, brute forcing due to unlimited failed sessions

Transport  Mishandling of undefined, poorly defined, or “illegal” conditions differences in transport protocol implementation allow “fingerprinting’ and other enumeration of host information  Overloading of transport-layer mechanisms such as port numbers limit the ability to

effectively filter and qualify traffic

 Transmission mechanisms can be subject to spoofing and attack based on crafted packets and the educated guessing of flow and transmission values, allowing the disruption or seizure of control of communications

Network  Route spoofing - propagation of false network topology

 IP Address Spoofing- false source addressing on malicious packets

 Identity & Resource ID Vulnerability - Reliance on addressing to identify resources and peers can be brittle and vulnerable

Data Link  MAC Address Spoofing  VLAN circumvention

 Spanning Tree errors may be accidentally or purposefully introduced, causing the layer two environment to transmit packets in infinite loops.

 In wireless media situations, layer two protocols may allow free connection to the network by unauthorized entities, or weak authentication and encryption may allow a false sense of security.

 Switches may be forced to flood traffic to all VLAN ports rather than selectively forwarding to the appropriate ports, allowing interception of data by any device connected to a VLAN.

Physical  Loss of power

 Loss of environmental control  Physical theft of data and hardware

 Physical Damage or destruction of data and hardware

 Unauthorized changes to the functional environment (data connections, removable media, adding/removing resources)

 Disconnection of physical data links  Undetectable interception of data  Keystroke & other input logging

Table 1: OSI-model layers and their vulnerabilities [19]

We can derive from this mapping of vulnerabilities on layers of the OSI-model that physical configuration can prevent some risks that have a very high impact with simple measures such as locks on the doors of the server room, or installing backup power generators. Such measures are clearly the responsibility of the party that hosts the physical hardware components; in our case this party is almost always a healthcare organization.

This category of vulnerabilities is not within the scope of this research.

The physical layer of course also includes hardware configuration to some extent, namely the physical placement of servers and network cables.

The vulnerabilities that may occur in the data link layer can be addressed by measures in the hardware configuration.

The first major hardware configuration decision that is typically taken is the deciding of the system infrastructure.

A common security measure that is taken in this process is the separation of network segments. Often not all components of the system need to be accessible through the internet. Components that do need to be connected to the internet can be placed in a DMZ (Demilitarized Zone).

(15)

15

This approach reduces the attack surface to only those components that are located in the DMZ. The other components can then only be attacked when components in the DMZ are breached. To make this harder, strict firewall rules should be in place to limit the traffic flows between the DMZ and the other network segments.

The network, transport, session, presentation and application layers of the OSI-model are affected by vulnerabilities that can be addressed by software configuration. Software configuration is a very broad term here; it includes configuration of routers, firewalls, web servers and user applications.

The software of routers and hardware firewalls is nowadays much more complex than the hardware in the devices, and offers lots of configuration options.

This form of configuration can be divided further into configuration related to system infrastructure and configuration related to user applications, but security risks with user applications can often be addressed by measures in both configuration categories. Incorrect software configuration is mentioned in the OWASP top 10 of most common vulnerabilities in web applications as ‘Security Misconfiguration’.

In the edition from 2010 it was on the sixth place in the top 10 and in both the editions of 2013 and 2017 it has risen to the fifth place. [20]–[22]

Security misconfiguration can happen at any level in the application stack and because of the variability in possible forms of incorrect configuration, resulting risks can be as diverse as information disclosure, unauthorized access or data breaches. [23] [24]

One form of security misconfiguration is the use of out-of-date components.

In web applications this also includes older versions of JavaScript libraries such as jQuery. Web applications often rely on functions of older versions of such libraries and therefore simply updating to the most recent version may introduce breaking changes.

These older library versions might have known vulnerabilities such as Cross Site Scripting (XSS), as has been the case with versions of jQuery. [25]–[27]

If web applications depend on the functions of these third party libraries in which a vulnerability has been discovered, this can make the web application itself vulnerable.

XSS has been reported as the most frequently encountered vulnerability in web applications. [28] XSS attacks are often used to steal cookies and thereby hijack user sessions. [29] Other types of common XSS attack usage are for phishing and keylogging users. [28]

Another form of misconfiguration is exposing features that are not necessary. This can mean amongst other things: entire web applications which are not used but are accessible through the internet, services that are meant for internal use but are exposed to the internet.

This form of misconfiguration enlarges the attack surface that malicious users can abuse. [16]

Depending on the nature of the unnecessary features that are accessible, this could lead to a system compromise. Especially because unused features or services are often not

maintained, and therefore might not be up-to-date and secure.

Leaving default credentials unchanged, as happens often with consumer routers, also counts as security misconfiguration. [23]

In the CS-Zorgportaal infrastructure, configuration occurs in different places for the different components.

For the HiX Application Services, the backend, configuration is in the form of XML-documents.

The configuration of HiX Application Services is defined in a ‘farm configuration’ document which contains references to a number of other XML-documents: the services configuration, authentication configuration and the settings configuration.

(16)

16

Figure 6: HAS farm configuration file

In the farm configuration document, Windows account, client applications and servers are defined.

A reference is made to a different file where the used variables are defined.

Modifications to the HAS farm configuration need to be made by hand by the system administrators of the customer organisation as ChipSoft does not provide tools to manage these configuration settings.

The services configuration lists all services that are in use on the server.

In this configuration file, services are specified along with the endpoint addresses on which they are reachable.

System administrators can for example choose to run the Antivirus service on a separate server, next to a server that runs all other HAS services.

For both CS-Zorgportaal and Microsoft SharePoint, settings can be configured in SharePoint’s Central Administration.

Part of the settings related to CS-Zorgportaal are stored in the HiX EHR database; these settings can be configured in HiX.

Conclusion

Security configuration issues can occur in a number of very different domains. Incorrect software configuration is a major cause of security vulnerabilities in web applications.

Configuration that affects CS-Zorgportaal can be adjusted in multiple places. So, what does ChipSoft actually do to prevent misconfiguration of their software?

(17)

17 3.2. What does ChipSoft already do to prevent misconfiguration?

Before we elaborate on what can be improved concerning software configuration, it is essential to understand what ChipSoft already tries to achieve optimal configuration in customer organisations. To answer this question, I will describe existing approaches that are in use by ChipSoft and analyse their usefulness.

At the start of a CS-Zorgportaal project, a kick-off session is held. During this session, a technical consultant is present at the customer organisation. The technical consultant explains the implementation process. This is also described in detail in a document that the customer organisation receives.

Before an implementation of the CS-Zorgportaal software is allowed to start, the environments on which the software will be deployed and configured in the customer organization, need to conform to a number of requirements.

These requirements are enforced by ChipSoft using a checklist that system administrators have to fill in and send to ChipSoft’s technical consultants.

ChipSoft for instance requires from its customers to follow the DTAP1_{-approach to software}

deployment.

This DTAP-approach entails that customer organisations should have at least an acceptation and a production environment. If they wish to develop functionalities for their portals

themselves, it is recommended that they also set up separate development and testing environments.

The checklist also specifies software requirements, including minimal versions of software dependencies for CS-Zorgportaal, such as SharePoint 2013 server and the DOTNET framework.

ChipSoft refuses to implement CS-Zorgportaal if the version of Microsoft Windows that runs on the servers is not supported by Microsoft anymore.

The verification of these requirements during the implementation phase is however a one-time snapshot: ChipSoft does not verify the installed software components periodically. Whether the software components are up to date or not, is only verified to a very basic level during the initial installation: are the software components supported by the producer, and are they somewhat recent?

Using this checklist, it is also verified whether ChipSoft has received a network topology schema which conforms to a set of requirements, including a separation between the internal domain and the DMZ in which the servers that run CS-Zorgportaal are located.

In the documentation describing the preparation for the implementation of CS-Zorgportaal and HAS, ChipSoft explicitly describes the required rights for accounts that are needed. This is done with the intent of preventing accounts to have more privileges than needed for the software to function.

When the technical installation starts, the technical consultant connects through the RDP-protocol to a terminal server in the environment of the customer organization.

Each customer organization therefore has to set up a VPN-connection with ChipSoft, to make the terminal servers accessible to ChipSoft but not to the internet.

This remote connection is also used to provide support to customers.

In the checklist, one of the checks therefore specifies if all servers that are part of the CS-Zorgportaal infrastructure are accessible from a terminal server that is accessible by ChipSoft.

Internally within the ChipSoft network, an overview webpage lists all customer organisations along with the credentials needed to connect to their terminal servers.

(18)

18

While some customer organisations only accept connections manually after they receive a phone call specifying the reason and the time required for the remote support, in most cases connections are accepted without verification.

The Windows accounts that may be used by ChipSoft have local administrator rights on all servers that they are allowed to access.

Pointing out the need for configuration changes

Due to the design of the CS-Zorgportaal and HiX Application Services software, new functionalities often also require new configuration settings.

Because the software is in the hands of the system administrators during the maintenance phase, they have to make the needed configuration changes.

When a new service is added to the HiX Application Services framework, configuration files have to be modified in order for instances of HAS to listen for service calls for the new service. The necessity to make these configuration modifications has to be described in release notes for the software update. In the past this has been forgotten a few times, resulting in broken software in customer organisations.

Support ticket system

ChipSoft has a support ticket system in place to become aware of these and other problems and to help customer organisations to resolve them.

Customers can create a ticket which describes a problem that occurs or a question that they have.

Figure 7: ChipSoft’s support ticket system

In ChipSoft’s case, the people that have access to this system in customer organisations often have the role of application or system administrator.

(19)

19

The different required steps that a user needs to perform in order to submit the support ticket are described in the table below:

Step # Description of step

1 Choose the software module to which the problem is related 2 Write a short description of the problem

3 Specify whether the support ticket concerns a problem or a wish for new functionality

4 Indicate whether the problem can be reproduced, and describe how 5 State which priority the support ticket has on a scale of 1 to 5 Table 2: Required steps to submit a support ticket with ChipSoft

The priority levels that can be chosen in step 5, and to what extent the functionality of the software is influenced, are related to each other.

The priority levels and their meaning are described in a separate table below.

Priority Meaning

Level 1 The problem that occurs makes the application completely unusable.

Level 2 The functionality of the application has become limited, but it can still be used to some extent.

Level 3 The working of the software is not influenced, but the working process is disrupted.

Level 4 The working of the software is not influenced, but the working process is disrupted to a limited a degree.

Level 5 The functionality of the software is not influenced, and there is not really a problem, but the customer wishes for a new functionality or an adjustment of existing functionality.

Table 3: Priority levels in ChipSoft's support ticket system and their meaning

Looking at this priority ranking through the eyes of someone who cares about security, it seems odd.

When a data breach has occurred, the system is probably still up and running like before, meaning that this issue has no impact on the functionality of the software.

Would the customer thus have to create a support call with a priority of low or ‘wish’? According to the priority description he does.

The priority levels might have an effect on how fast the support call will be processed and data breaches must be reported within 72 hours of occurrence.

A very obvious improvement in ChipSoft’s support call system could be a category for security issues or an altering of the existing prioritization of support tickets.

Other than this priority system, the input necessary for the creation of a support ticket at ChipSoft is comparable to the input needed to file a bug in Mozilla Firefox. [30]

(20)

20

Figure 8: Mozilla’s bug submission system

Such support ticket systems however, rely on knowledge from the user that a certain problem occurred.

If no one reports the issue, the software supplier does not become aware of the problem and the issue can lay dormant and unresolved for extended periods of time.

Automatic error reporting systems

To become aware of issues that are not encountered by end users, certain software

companies have mechanisms built into their software that can automatically report software errors that occur during usage by end users to the software companies.

A well-known example of such a system is that which is in use by Microsoft for its Windows operating system.

(21)

21

When an application crashes in certain versions of the Windows operating system, the user is asked if he or she wants to report the error to Microsoft. In newer versions like Windows 7 and 10 the report is sent automatically with the default settings, unless the user disables this. [31]

In ChipSoft’s EHR suite HiX, a comparable functionality has been introduced.

When the application encounters an exception, a popup window is shown to the user. The user can describe in this popup window what he was doing exactly before the error occurred.

In the background, a lot of information is sent along with the report such as the call stack, the user that was logged on, the date and time and sometimes a screenshot of the users’ HiX window in which confidential data is automatically censored.

Figure 10: Error reporting in ChipSoft HiX

These error reports are called JIP’s (Joint Information Package) and are sent from the customer organisation to ChipSoft through a dedicated communication server called CS-Comez.

On ChipSoft’s side, these error reports are grouped together by software module and shown in an overview on an internal web page. The responsible developers for the concerning HiX-module can comment on error reports and refer to software changes which resolve the error in the future.

While this error reporting approach can work for desktop applications, web browsers tend to automatically block popups, making the use of showing report popups to users less reliable. Displaying error messages to end users also may reveal unnecessary information about the application’s security. This is a common weakness in web applications that can be abused by malicious users to conduct further attacks using their newly gained knowledge of the system. [32]–[34]

Automatic error reporting mechanisms that require user interaction or that reveal information about software errors to users are therefore not the right approach for web applications. However, there are ways to capture usage information without user interaction, and without revealing too much information to the user.

(22)

22 Error reporting mechanisms within the customer organisation

Due to the fact that as a system administrator or developer you don’t want to reveal too much information in error messages, it is common practice for web applications to silently log (sensitive) error messages and, perhaps, report these within the organisation. [35, pp. 28– 29]

Microsoft SharePoint, the underlying web framework upon which CS-Zorgportaal is built, comes out of the box with such a functionality.

From SharePoint 2010 onwards, a feature called the ‘SharePoint Health Analyzer’ has been included in the product. [36]

This Health Analyzer checks periodically for SharePoint-related issues based on preconfigured rules.

The system administrators can then see these ‘health issues’ in a dashboard page

accessible through the Central Administration web application, where they are informed with a warning text if health issues are present. [37, p. 23]

The figure below shows how SharePoint informs system administrators of (critical) issues.

Figure 11: The warning bar that informs system administrators of issues in Microsoft SharePoint 2010

In theory, this may sound like a good way to get the attention of a system administrator when a critical issue is at hand.

The experience at the CS-Zorgportaal team is however that the warning bar is present so much of the time in customer organisations that it is perceived as being normal.

Recent research has found out that warning messages often interfere with a task a user was doing. [38] A system administrator might have navigated to SharePoint’s Central

Administration to configure a certain setting, but is disturbed by the red warning message shown in the screenshot. This interference teaches the system administrator that critical warning messages often interfere with his primary task and that these warning messages should be ignored. [38]

This of course nullifies the effectiveness of this form of error reporting as the reported issues are never solved.

Another question is if the SharePoint health issues reach the right audience in an organisation. The system administrator might be concerned with the technical impact whereas the higher management is concerned with the business impact.

While the technical impact that results from a configuration issue may not be that big, the business impact could be. However, as Central Administration is intended to be used by system administrators, the higher management will never become aware of these issues. CS-Zorgportaal comes with a custom log viewer which is also integrated in the Central Administration interface of Microsoft SharePoint.

It does however not show warning messages on the home page, and users have to voluntarily navigate to the log viewer. There is thus no interference with the system

(23)

23

administrator’s primary task; having navigated to the log viewer, it can be assumed that the task at hand is looking at log messages.

A screenshot of the interface of this log viewer is displayed in the picture below.

Figure 12: The log viewer in CS-Zorgportaal

While system administrators can filter the logs to some extent, based on the seriousness of the log messages, and search through the messages that are displayed, the CS-Zorgportaal team is used to seeing the log viewer filled with warnings and errors at customer

organisations.

Yet, how user friendly is it to be able to search only through the 100 log messages that are already loaded? Fortunately, scrolling down until the bottom expands the search area each time with 100 more log messages.

Manually scanning these log messages without a specific idea of what to look for, is of course a very repetitive activity. [39]

The task of monitoring and manually auditing log files suffers from inherent problems. As each log message is stored and displayed as text, the system administrator must (partially) read each message to understand what it is about, making it a very time consuming activity. [40]

Girardin and Brodbeck propose the use of visualisation techniques to counter this, along with the use of automated anomaly detection techniques. [39]

Work has been started in the CS-Zorgportaal team on a dashboard that visualizes the log messages. System administrators would then be able to quickly see a difference in the number of reported errors or warnings, per category or per type of event, between time periods.

The expectation is that this will allow system administrators to become aware of serious issues that have resulted in a significant rise in the number of log messages.

This approach looks promising in that aspect, but how can this dashboard make both parties aware of incorrect configuration issues?

Malicious users look from the outside at the parts of the application that are accessible on the internet.

(24)

24

Configuration issues are not limited to SharePoint or CS-Zorgportaal but can also occur in the web server, the backend or the firewall or reverse proxy.

To an attacker this makes no difference; incorrect configuration of any component in the architecture can be abused when it is noticeable.

Initiatives for better collaboration between developers and system administrators

The CS-Zorgportaal team hired a technical consultant to address the collaboration between system administrators and the developers.

The role of this technical consultant is to support healthcare organisations with the technical implementation of the CS-Zorgportaal software.

This involves maintaining a library of best practice procedures, and keeping both the developers and the system administrators of the customer organisations up to date about these best practices.

The current best practices consist of a recommended IT architecture model, scripts to automate processes such as installation, but also guides on how to survive DigiD-enrolment, how to monitor, how to request support and how to ‘harden’ the infrastructure.

Because the technical consultant has a high level of knowledge about IT infrastructure and about ChipSoft’s software, he assesses the customers’ environments on which the software is to be deployed. These assessments are called ‘IT readiness’-assessments.

As the software’s requirements on the environment may change over time, the technical consultant maintains close contact with the developers.

When this job vacancy was written, the intent was to also make the job include responsibilities related to security assurance.

The technical consultant was supposed to assist others within the company to put in place a software security assurance policy.

The idea was also that the technical consultant would (co-)create company policy with a goal of obtaining a TPM-certificate to the benefit of the DigiD-assessments and assist customers whose organisations are under audit.

In practice however, the technical consultant is busy enough with technical implementations and migrations to newer versions of the software at customer organisations.

This responsibility has thus not been taken up by the technical consultants yet.

Another initiative that ChipSoft has taken is organising knowledge sharing sessions to which both developers and system administrators attend.

ChipSoft organises these in the form of user sessions about different software modules. Each year a user session about eHealth is also organised.

Such a user session is a whole day filled with presentations about different subjects with the goal of sharing knowledge with customer organisations.

The people from customer organisations that attend are mainly system and application administrators but also people that fulfil other roles such as physicians and higher management.

Conclusion

To conclude, ChipSoft already takes multiple measures to become aware of incorrect configuration. Among these existing measures is the enforcement of a hardware configuration that supports the DTAP-approach and fulfils some basic requirements, a support ticket system, the standard SharePoint Health Analyzer, a custom log viewer and a dashboard that visualizes the number of log messages that were generated.

A number of remarks have been made about possible improvements in the existing measures. ChipSoft has also taken steps to improve the collaboration between their developers and the system administrators in customer organisations.

(25)

25

This chapter shows that ChipSoft already does a lot to assure correct configuration, but apparently there are still issues. In the next chapter I will dive into which issues are, despite ChipSoft’s goodwill, still prevalent because of misconfiguration.

3.3. What issues arise because of misconfiguration?

As described in the previous chapter, ChipSoft helps in multiple ways to verify the state of configuration in customer organisations.

Incorrect configuration however still causes a lot of work for both ChipSoft and the customer organisations.

A search was done through ChipSoft’s support tickets to investigate exactly how often incorrect configuration led to problems or discovered vulnerabilities reported by customers. A SQL-query was used to find all support tickets that were relevant, on the database of ChipSoft’s support ticket system.

This query resulted in 108 unique support tickets in the period of April 2009 to May 2017. The support tickets were put into an Excel sheet and carefully read, after which they were assigned categories.

Category Number of security-related support tickets Total 108 Configuration issues 28 XSS vulnerabilities 3 CSRF vulnerabilities 1 Other 6 No issue 76

Table 4: Reported vulnerabilities per category

As can be seen in the table above, configuration issues are the most often reported vulnerability category.

These issues mainly come forward in vulnerability reports from third party auditors, and in some cases the issues are discovered by customer organisations themselves.

These issues are unfortunately reported long after the mistakes have been introduced: the DigiD vulnerability audits are only conducted once a year!

Neither ChipSoft nor the customer organization has a clear overview of the state of the configuration in the components that are part of the CS-Zorgportaal infrastructure.

In the previous chapter it has been described that the support ticket system is not optimal for the reporting of security-related configuration issues.

This attributes to the lack of overview that ChipSoft has on the state of configuration. Among the 28 configuration issues are two frequently occurring issues.

The first common issue is that the webserver that hosts the CS-Zorgportaal applications sends response headers that reveal information about versions of software components. A screenshot of the common response headers is displayed below.

(26)

26

Figure 13: Common response headers of a CS-Zorgportaal web application

From the response headers shown in this screenshot we can deduce a number of things.  The ‘Server: Microsoft-IIS/8.5’ header tells us that the webserver that is serving

CS-Zorgportaal is Microsoft IIS 8.5.

IIS 8.5 is included within Windows 8.1 and Windows Server 2012 R2.

The machine on which the webserver runs, thus uses one of these versions of Windows.

 The header ‘MicrosoftSharePointTeamServices: 14.0.0.7006 tells us that the CS-Zorgportaal web application runs on top of Microsoft SharePoint 2010.

 The header ‘X-Powered-By: ASP.NET’ reveals that the ASP.NET technology is being used by the webserver.

 ‘X-AspNet-Version: 2.0.50727’ then gives us the exact version of the .NET framework that is installed on the machine.

As an attacker, I can now search for vulnerabilities in IIS 8.5, SharePoint 2010 and in version 2 of the .NET framework.

And in fact, there seems to be a critical and unpatched vulnerability in this version of the .NET framework. [41]

The second common issue is that web pages of the SharePoint runtime web services are accessible in subdirectories of the web application.

(27)

27

Figure 14: One of the webpages belonging to the SharePoint web services

These web pages describe how to send POST requests to the SharePoint web services. By default, anonymous users do not have the required rights to actually abuse these web services.

If SharePoint is configured in such a way that anonymous users or authorized patients have the required rights, they can request the collection of Windows accounts that are known by the SharePoint instance, and view or change permissions, amongst other things.

When a configuration-related issue like this is found, the customer organisation can either choose to fix the issue themselves based on advice from ChipSoft, or choose to let ChipSoft fix the problem remotely.

Most customers choose to let ChipSoft fix the incorrect configuration. Conclusion

Despite existing measures, security misconfiguration is a prevalent issue in support tickets that ChipSoft receives from customer organisations.

The configuration issues that occur the most frequent in these support tickets are related to unneeded response-headers with version information and to the exposure of unneeded services.

The frequent occurrence of these same configuration issues leads to additional work for ChipSoft and often to additional costs for the customer organisations.

In the next chapter I will look for proposed solutions in literature that address incorrect software configuration.

A previous attempt by ChipSoft to monitor for incorrect configuration will also be described in the following chapter.

Where applicable, I will implement the proposed solutions in a software prototype. The frequent occurring issues described in this chapter will have to be detected by the prototype.

(28)

28 3.4. What possibilities are there to ensure that configuration remains up to date

and secure?

Now that we know that incorrect configuration is a prevalent issue and the current measures that ChipSoft has implemented do not prevent all related issues, the question is what earlier studies have attempted to verify that configuration is up to date and secure.

As explained earlier, configuration is a very broad term and I will therefore look at multiple angles for a solution in literature.

Out of date software components

A common form of incorrect software configuration is the usage of out-of-date software components. In reports from EdgeScan, the authors state that respectively 37% and 29% of the discovered vulnerabilities could have been prevented by proper patch management in 2015 and 2016. [42], [43] This means that a patch was already out for the vulnerability, but not yet installed by the organisation. As the reports from EdgeScan note, this can be addressed by proper patch management. But how do you know if a patch management policy is actually followed in an organization?

Microsoft has developed a software tool to check if software within an organisation is up-to-date. [44] This application, which is called Microsoft Baseline Security Analyzer, can be used to scan computer systems belonging to one’s organization for missing security updates for Microsoft components such as IIS, Internet Explorer and SQL Server. This can be useful for system administrators, because the report shows for all systems in their organization

whether the installed components are up to date or not. A disadvantage is that the tool only reports on the status of Microsoft components. Within a Microsoft Windows system, it can already be enforced that these components automatically update, without requiring user interaction.

For third party software, no such automatic update functionally is available in Microsoft Windows, in contrast to the major Linux distributions.

Linux distributions typically work with package managers, allowing one to update all software that was installed using said package manager with one click or terminal command.

As such functionality is lacking within Microsoft Windows, it would perhaps have been more useful if Microsoft had thought about integrating package management in newer versions of Windows. A third party tool that attempts to provide package management is however

available, called Chocolatey. [45] This package manager is similar in usage to Linux package managers like apt or yum. Currently their repository contains 4987 packages, which are maintained by the community. Chocolatey enabled the user to upgrade all software which has been installed using the package manager. While this is great, software which is installed in the ‘normal’ Windows way of installing software, by executing an installation wizard, can of course not be kept up to date using Chocolatey.

This package manager is similar in usage to Linux package managers like apt or yum. Currently their repository contains 4987 packages, which are maintained by the community. Chocolatey enabled the user to upgrade all software which has been installed using the package manager. While this is great, software which is installed in the ‘normal’ Windows way of installing software, by executing an installation wizard, can of course not be kept up to date using Chocolatey. Because the advantages of package managers are not widely known to Windows users, adoption of package management will probably remain low, along with the percentage of maintained packages in repositories.

To make patch management easier by enforcing package management, the only really effective scenario is when Microsoft integrates this in the Windows ecosystem.

They have recently taken the first step towards doing this, by including a package management feature with Windows 10.

(29)

29

This feature is actually a package manager aggregator that can be used to install packages with existing package managers, if these package managers provide a compatible ‘package provider’. [46], [47]

Shadow configuration

Besides out of date components, an important form of configuration issues is caused by incorrect configuration settings.

To prevent such configuration issues, one of the common best practices is to first test configuration changes on an acceptance environment and then perform acceptance tests. Only when these acceptance tests are successful, should the configuration changes be performed on the production environment.

This is the DTAP approach (see p. 17), where software is deployed to an acceptance

environment first. In the acceptance environment, the customer verifies if the software meets their expectations. [48]

The acceptance environment should be as similar as possible to the production environment in order for the acceptance tests to have the same results as they would in production.

However, when dissimilarities exist between the acceptance and production environment, the acceptance tests are not completely reliable.

Alimi et al. propose the use of ‘shadow configuration’. [49]

Shadow configuration is the ‘new’ configuration that is meant to replace the current configuration. They explain that using shadow configuration parallel to the actual configuration in the production environment, the configuration can be evaluated before deployment. The difference in the approach of Alimi et al. is that ‘shadow’ traffic is used in the production environment to test the shadow configuration, in parallel to actual traffic that uses the ‘old’ configuration.

This approach takes away the issue of having to mimic the production environment in the acceptance environment and subtle differences between the two environments are therefore no longer a problem. The results for this study are however aimed at network configuration, as authors are of the opinion that software configuration is already covered by testing practices and debugging whereas no such practices yet exist for network configurations.

Clear log messages

To identify the root cause of a configuration issue, it is often needed to perform troubleshooting of log messages.

System administrators perceive this troubleshooting to be very complex when dealing with security issues, according to a survey by Gagné et al. [7]

Xu et al. therefore propose logging very clear error messages when configuration errors are detected, and suggest that users could then fix the configuration errors themselves. [50] Yin et al. state that clear log messages reduce the time it takes to diagnose the problem 3 to 13 times. [51]

As already explained however, configuration errors can occur in multiple components of the platform infrastructure; most of which are not developed by ChipSoft.

That does of course not mean that this approach is not possible; it would only get a little more complex.

The different components have different methods to log configuration errors and this approach would thus need to analyse that logging and report specific types of errors. As we have already explained, system administrators are mainly concerned with the technical impact of risks. System administrators are also mainly concerned with security when something goes really wrong.

Making sure that clear messages are logged when misconfiguration is detected is thus of course a logical first step but is not a solution if the system administrators do not monitor and acts upon these log messages.

(30)

30 Operator mistakes

Oliveira et al. developed and evaluated a framework that attempts to prevent operator mistakes. [52]

Their framework, ‘Barricade’ works based on tasks with related steps, which are commands that can be executed by the operator in a terminal. In each task file, commands can be specified which will be blocked. An example of a blocked command is ‘apachectl stop’ which would stop the Apache webserver from running, causing downtime.

While the framework can block write-access to configuration files in certain situations, no other use cases are described related to ensuring up-to-date and secure configuration. In their conclusion, the authors stated that they were at that time studying how to extend the framework to security-related operator mistakes. As of this moment however, no follow-up study of that kind has been published.

3.4.1. Relevance for ChipSoft

No articles were found that described a software supplier that verified the configuration of their software in customer organisations.

So no requirements could be derived directly from existing research documents.

I first discussed the prevalence of a lack of patch management in organisations, and ways to address this problem.

ChipSoft can, as a software supplier, not really play a role in enforcing the usage of tools such as Microsoft Baseline Security Analyzer or package managers in order to stimulate proper patch management.

They do however already specify software version requirements for the implementation of CS-Zorgportaal to ensure that the software dependencies of CS-Zorgportaal are supported by the supplier and thus receive security patches.

ChipSoft could verify to if CS-Zorgportaal or underlying components are up to date periodically after the implementation.

As explained before, CS-Zorgportaal runs on top of Microsoft SharePoint and the Microsoft IIS webserver.

If these components are out of date, this may make the web application itself vulnerable. Recently, many vulnerabilities have been discovered in implementations of the TLS and SSL protocols.

Because of the POODLE vulnerability, SSL 3 is now seen as an ‘obsolete and insecure protocol’. [53]

ChipSoft could verify for all their customers whether the webservers that host the CS-Zorgportaal instances support the very insecure SSL 3 protocol, and if the web server is vulnerable for any other vulnerabilities that can be prevented by updating the webserver. The second concept that was found during the literature search was that of shadow configuration.

While the idea of testing configuration changes in parallel in the production environment sounds interesting, the configuration of CS-Zorgportaal is more than just the flow of network traffic.

The acceptance environment depends on a different database, different web servers and different physical or virtual machines than the production environment.

This makes parallel testing of configuration changes not feasible.

Xu et al. propose logging clear error messages, and Gagne et al. state based on interviews that system administrators find security troubleshooting to be difficult.

This can be used in the approach to address incorrect configuration.

When issues are detected, it should be very clearly communicated to the customer organization what has to be done to solve those issues.

(31)

31

Because ChipSoft deals with a lot of recurring configuration errors, they can make very clear documentation describing each issue along with the steps that need to be taken to solve them.

The last concept that was found in the literature search was that of the Barracuda framework to prevent operator mistakes.

Because ChipSoft’s software is not (only) configured using simple terminal commands, the idea that is proposed within this study is not applicable for a solution to ChipSoft’s problems with incorrect configuration.

If ChipSoft wanted to prevent operator mistakes that have impact on the security of Zorgportaal, they would have to monitor and warn for configuration adjustments in the CS-Zorgportaal web application, in Microsoft SharePoint and in HiX, which would be not feasible.

3.4.2. Earlier solution attempts within ChipSoft

At ChipSoft, it was already attempted to develop software that checks for configuration problems related to the CS-Zorgportaal platform at the customer organisation and reports these configuration problems to ChipSoft.

I will describe what we built, based on what design decisions, and why it ultimately was no success.

While the SharePoint Health Analyzer checks for SharePoint problems already, we experienced that often issues that it reported were ignored.

The service that we designed should thus also report these health issues to ChipSoft.

ChipSoft can then decide to inform the right people within the customer organization of these issues.

Another common misconfiguration is that the customer has installed different versions of CS-Zorgportaal and HiX on the same machine.

As HiX Application Services depends on methods in the HiX assemblies, a difference in versions of the two software components can cause issues; the two versions can be incompatible as breaking changes can have been introduced in one of them.

These version differences should thus also be reported.

In the diagram below the different components of this software, along with their communication flows are described:

How to ensure secure and up to date configuration at customers that use a web application of a COTS-supplier