Shared achievement or failed attempt? Factors affecting the perceived effectiveness of Public-Private Partnerships on cybersecurity in the Dutch Critical Infrastructure

January 2019

Shared Achievement or

Failed Attempt?

FACTORS AFFECTING THE PERCEIVED EFFECTIVENESS OF

PUBLIC-PRIVATE PARTNERSHIPS ON CYBERSECURITY IN

THE DUTCH CRITICAL INFRASTRUCTURE

THESIS OF TIM KENTER

S1429124

LEIDEN UNIVERSITY MASTER THESIS CRISIS & SECURITY MANAGEMENT

SUPERVISOR DR. J. MATTHYS

SECOND READER DR. L.D. CABANE

Acknowledgements

Before introducing my thesis, I would like give a special thanks to a number of people who have helped me to finish my thesis and make it something to be really proud of. First, I would like to thank my supervisor, Dr. Joery Matthys, who provided me with useful and critical feedback, even at Friday evening before Christmas. Your help has been really appreciated and for all the upcoming students you will supervise, they are lucky to have you as a guide and mentor.

Second, I would like to express my gratitude to the participating members of the ISAC-Port. Thank you for making time in your busy schedules. Without the possibility of conducting interviews with all of you, the results of this thesis would not have been as sufficient as they currently are.

Third, I would like to thank my colleagues. Without the use of your network, it would not have been possible to contact the relevant personal for this research. Hopefully, after my thesis, I will become a full member of the team to be able to help wherever is needed.

Finally, my special appreciation goes to my friends and family. The moments I kept on staring at my thesis without filling in any words, your stimulation and attention helped me to proceed. Also the feedback you have provided me was crucial in order to achieve higher quality thesis.

Table of Contents

2.3. Public Private Partnerships ... 15

2.4. Network effectiveness ... 16 2.4.1. Network Structure... 18 2.4.2. Network Culture ... 19 2.4.3. Network Policies... 20 2.4.4. Network Technologies ... 21 2.4.5. Network Relationships ... 22 3. Methodology ... 25 3.1. Conceptual Framework ... 25 3.2. Case Study ... 27

3.3. Data collection & analysis ... 29

3.4. Operationalization ... 31

3.5. Validity and reliability ... 33

4. Analysis ... 34

4.1. Information Sharing and Analysis Centres (ISACs) ... 34

4.2. Perceived effectiveness ... 36

4.2.1. Goal achievement ... 36

4.3. Structure ... 39 4.3.1. Governance model ... 39 4.3.2. Roles ... 40 4.3.3. Size ... 41 4.4. Culture ... 42 4.4.1. History ... 42

4.4.2. Reciprocity and stability ... 43

4.4.3. Shared beliefs, attitudes and values ... 43

4.5. Policies ... 45

4.5.1. Traffic Light Protocol (TLP) ... 45

4.5.2. Membership Guidelines ... 47

4.6. Technologies ... 48

4.6.1. Communication and administrative systems ... 48

4.6.2. Information systems and technical assistance systems ... 49

4.7. Relationships ... 49

4.7.1. Interpersonal relationships ... 50

4.7.2. Inter-organizational relationships ... 51

4.8. Interconnected relations of the different levels ... 51

5. Conclusion ... 54

5.1. Conclusion ... 54

5.2. Reflections ... 56

1. Introduction

1.1. Research problem

Since the start of the digital era, a world without its digital components is rather unthinkable. Apart from all the benefits the digital era provides us, it also comes with a set of negative effects. The size and dangers of digital threats have become a worldwide phenomenon, resulting in security studies increasing their interests in cybersecurity within the critical infrastructure. “Critical infrastructure protection is prominently concerned with objects that appear indispensable for the functioning of social and political life” (Aradau, 2010: 491). Attacks on these infrastructures could have a major impact on society as it disturbs crucial public resources (Boeke, 2017). Critical infrastructure disruptions are no longer myths. During the disruption of the Ukrainian electric grid operations of the power companies on 23th of December 2015, the first ever successful cyberattack on power grids presented itself (Dragos, 2017). More than thousand customers were left without electricity for at least six hours (Sullivan & Kamensky, 2017). The Netherlands also experienced cyberattacks within the critical infrastructure such as the APM terminals in the Port of Rotterdam and the Diginotar crisis. “To improve the security and resilience of their CI [critical infrastructure], states have drafted national cyber security strategies since the mid-2000s” (Boeke, 2017: 2). Cybersecurity protection within the critical infrastructure has also become a central security objective in the Netherlands (MinV&J, 2013; MinV&J, 2018).

The cybersecurity protection of the critical infrastructure is not entirely a public sector responsibility (Stoddart, 2016). Because private sector companies often operate the critical infrastructure in the Netherlands, their protection becomes a shared responsibility (Boeke, 2016; Boin & Mcconnell, 2007). Moreover, security provision is mainly a governmental task, while cybersecurity knowledge is primarily allocated within the private sector (Kshetri, 2015). Therefore, “governments are increasingly dependent on other parties when it comes to protecting critical infrastructures” (Luiijf & Klaver, 2004: 1189). Due to the evolving shared responsibility in cybersecurity, there is a need for both sectors to collaborate on the issue. Public-Private Partnerships are often mentioned as useful networks to address both traditional and non-traditional security threats (Carr, 2016). This has resulted in Public-Private Partnerships becoming an interesting solution to deal with cybersecurity problems. In the Netherlands, Public-Private Partnerships on cybersecurity are established by the National Cyber Security Centre (NCSC), stimulating the exchange of information among public and private organizations about cybersecurity threats, weaknesses and best practises.

The increased use of Public-Private Partnerships in tackling public issues has also raised the academic interest into this subject (Warsen, Nederhand, Klijn, Grotenbreg & Koppenjan, 2018). The academic literature debates on whether Public-Private Partnerships are an effective instrument in protecting the critical infrastructure against cyber threats. Some academics argue that Public-Private Partnerships are an effective instrument and describe them as a ‘best-practise’ (Dunn-Cavelty & Suter, 2009; Shore, Du & Zeadally, 2011), while others address their drawbacks of such networks to cybersecurity (Carr, 2016). For example, according to Proven & Milward (1995), it is still a premature idea to claim that networks, such as Public-Private Partnerships, are an effective way of dealing with policy issues. Carr (2016: 62) argues that “partnership arrangement cannot work effectively in its current form”. Members within Public-Private Partnerships on cybersecurity have differences in interests, lack common definitions and can have disagreements on the funding (Carr, 2016). Moreover, Public-Private Partnerships sometimes originate out of necessity, lack effective collaboration1 or have tried and failed to solve issues of ‘wicked problems’ (Malone & Malone, 2013). This has resulted in Public-Private Partnerships sometimes failing their initial attempt.

1.2. Research question

The importance of strengthening the resilience on cybersecurity in the critical infrastructure has become a priority within the Dutch Cybersecurity Agenda. To achieve this, Public-Private Partnerships are often addressed as a useful network to solve cybersecurity issues. But the previous section also addressed the possible barriers of Public-Private Partnerships. Therefore, it is important to assess whether Public-Private Partnerships in the Netherlands are perceived as effective by the members in the network. Moreover, insufficient attention is given those factors important for the members in achieving this effectiveness. This leads to the following research question:

What factors have an effect on the perceived effectiveness of Public-Private Partnerships responsible for the cybersecurity protection in Dutch critical infrastructures?

This study defined perceived effectiveness as the perceived capacity of Public-Private Partnerships to achieve their stated goals. Moreover, there are different Public-Private Partnerships on cybersecurity in the Dutch critical infrastructure. For this study, the

1 _{Ludden, G.J. (2018, March 28). Overheid en bedrijfsleven werken niet goed samen in crisissituaties. Retrieved from}

Information Sharing and Analysis Centre (ISAC) within Ports is applied as a case. The ISAC-Port is a Public-Private Partnership which goal is to share relevant information among different public and private organizations within the sector.

1.3. Academic relevance

The establishment of Public-Private Partnerships have already been noticed since the 90’s as a central element of public policy across the world (Osborne, 2000). Since then, academic interest on these networks increased significantly. Multiple studies have tried to look at the effectiveness of Public-Private Partnerships, especially within construction and the health industry (Hare, 2013; van Ham & Koppenjan, 2001). Public-Private Partnerships have been addressed as useful networks in these environments. However, even while the literature addressed Public-Private Partnerships to be an appropriate mean for collaborative networks, this is not always the case. Kenis & Provan (2006: 229) argued that “networks often fail to achieve their intended goals, the cost of network disruptions is often substantial, and they can even be harmful to the public at large”. This becomes particularly relevant within the world of cybersecurity where Public-Private Partnerships are frequently launched. According to Carr (2016: 43):

“The reluctance of politicians to claim authority for the state to introduce tougher cyber-security measures by law, coupled with the private sector’s aversion to accepting responsibility or liability for national security, leaves the ‘partnership’ without clear lines of responsibility or accountability”.

Only limited academic attention is given to the effectiveness of cybersecurity networks for critical infrastructure protection (Carr, 2016). Besides the narrow attention to network effectiveness on cybersecurity in general, the academic body of knowledge also paid limited attention to the factors affecting the effectiveness. It is observed that cybersecurity consists out of certain characteristics which differs from characteristics within security studies (Trimintzios et al., 2014). This could, by no other means, have an effect on the factors affecting the perceived effectiveness of Public-Private Partnerships. Here, a clear knowledge-gap can be found, addressing one of the elements making this study relevant for academic research. Therefore, this study contributes to the theory of network effectiveness both in general as within the field of cybersecurity.

Moreover, to examine network effectiveness and the factors possibly affecting this, the literature provides relevant information on networks in general. These networks are primarily focussing on either public or private networks, while limited studies addressed interconnected

public-private networks. This study examine the applicability of the theory of network effectiveness on public-private networks, such as Public-Private Partnerships.

1.4. Social relevance

Besides its academic relevance, studying the perceived effectiveness of Public-Private Partnerships on cybersecurity also provides valuable knowledge that is socially relevant. The Netherlands is the second country world-wide to digital connectivity2. This allows organizations with many different possibilities. However, it also increases the digital vulnerability. According to the Dutch government, strengthening public-private collaboration is essential to a continuously digitalised society (MinV&J, 2018). Especially information sharing platforms of these collaborative structures are necessary to enhance cybersecurity. Hence, the Dutch Cybersecurity Agenda (2018) emphasised that one of the central elements for the coalition was to achieve a safer environment in the Netherlands by addressing closer collaboration with the private sector. According to the agenda “security in the digital domain can only be shaped in cooperation with and in part of the business community. Public-private cooperation therefore forms the basis for the Dutch approach to cybersecurity” (MinV&J, 2018: 7).

Cybersecurity has become an increasingly important subject to security policies, especially the strengthening of cybersecurity in the critical infrastructure of the Netherlands. This study could provide extra attention to accomplish these security policies. But even while Public-Private Partnerships are frequently recommended in the Dutch approach to cybersecurity, it must be examined how the effectiveness of such networks are perceived. According to Hale & Mabuterol (2004), only less than half of the partnerships are monitoring their effectiveness. Moreover, within networks “actors attempt to realize their own interests, joint decision making can be difficult. Decision making is also hampered by institutional complexity or by the unwillingness of actors to share information, because they fear opportunistic behavior from other actors” (Edelenbos & Klijn, 2007: 26). If collaboration in its current form is not perceived as an effective network by the members, government ordered cybersecurity measures and strategies cannot be performed as expected and therefore need to be adjusted. Hence, it is essential to analyse the perceived effectiveness of Public-Private Partnerships.

2 _{Manyika, J. et al. (2016, February). Digital globalization: The new era of global flows. Retrieved from}

1.5. Overview

The present thesis is divided in five Chapters. Chapter 1 contains the introduction of this study, outlining the research problem, research question and both the academic and social relevance. Chapter 2 contains the theoretical framework, focussing on existing academic literature corresponding with the most important concepts of the thesis. In Chapter 3 a systematic presentation of the execution of this study is provided, the methodology chapter. This part will focus on the conceptual framework, the reasons for performing a single-case study, the data collection and analysis, the operationalization of the concepts and finally a section on validity and reliability. Chapter 4 concerns the analysis of this study, separated by sections corresponding Whelan’s (2012) levels of network analysis. Following the analysis, a conclusion in Chapter 5 is given, providing an answer to the research question and a reflection of the study.

2. Theoretical framework

The theoretical framework provides the bases for most academic research. In this chapter, the important concepts of this study have been reviewed, focussing on the existing body of knowledge. The theoretical framework first elaborates on the concept of cybersecurity as wicked problem. Thereafter, the concept of critical infrastructures and Public-Private Partnerships in policy making are described. Finally, the theoretical framework pays extensive attention to the concept of network effectiveness, focussing on the concept itself and the levels that could influence network effectiveness, following the methodological framework by Whelan (2012).

2.1. Cybersecurity

Malware, Ransomware, Phishing-mails, Distributed denial-of-service (DDoS) attacks and data-leaks are multiple threats the world is currently facing. New actors have entered the playground in the cyber domain. Besides the threat cybercriminals pose to society, state actors have become an even greater danger due to their capacity to take offensive actions on the critical infrastructure (NCSC, 2018). The motivations and techniques for these offenses vary widely, making each cyber-attack an almost unique exemplar (Maitra, 2015). This section elaborates on some of the conceptual and organizational challenges of cybersecurity.

2.1.1. Terminology

Before taking an in-depth examination on perceived effectiveness of Public-Private Partnerships, it is important to introduce cybersecurity in order to understand the context of this study. The most common definition of cybersecurity is found in the Oxford English Dictionary as “the state of being protected against the criminal or unauthorized use of electronic data, or measures take to achieve this” (Oxford University Press, 2014). While

cybersecurity was mainly rooted in computer science, the growing vulnerabilities among different fields (e.g. business, law and politics) have also gained their interest in the concept. As a consequence of this growing interest, the body of knowledge increased extensively. However, this also let to a mixture of terminology which is well-known among academics (Ramirez & Choucri, 2016; Barzilay, 2013; Stubley, 2013). Because of the various use in the terminology, it has an impact on how different public and private institutions use the concept. Moreover, according to Baylon (2014), a cybersecurity definition which lacks common understanding poses many difficult challenges in international treaties and arms control.

Nevertheless, different concepts of cybersecurity are used routinely. The study by Schatz, Bashroush & Wall (2017) argued that cybersecurity is often used identically to ‘computer security’, ‘information security’, ‘IT security’ or ‘cyber security’. According to the authors, the definition used in organizations depends on the category of study (e.g. government/public, infrastructure, academic). Ramirez & Chourci (2016) also mentioned differences in these categories as “a long-standing disconnect between traditional technological research in cybersecurity and public and private sectors’ nontechnical dealing with cyber security” (Ramirez & Chourci, 2016: 2221). For example, in the category infrastructure, Barzilay (2013) perceived cybersecurity as a sub discipline of information security. In the more governmental setting, Wamala (2012) argued that ‘cyber security’ is a branch of information security, similar to Barzilay (2013). The European Network Information Security Agency (ENISA) is providing a European standardized framework on cybersecurity, covered in different security norms. (ENISA, 2015). According to ENISA, information security implies a broader field of the cybersecurity, using the technical framework C-I-A trinity (Confidentiality, Integrity and Availability).

2.1.2. Wicked problem

There are more issues cybersecurity faces. Managing cybersecurity is different compared to normal security issues because of the different characteristics. Cybersecurity threats can continue for extended period of time before being discovered and have a cross-border nature, making collaboration between countries more evident (Trimintzios et al., 2014). Boeke (2018) adds that interdependency does not only exist between countries. On a national level, the responsibility lies within both the public and private sector, “and a government’s lead is by no means self-evident” (Boeke, 2017: 451). Moreover, decision-making procedures are difficult due to language barriers between the technical professionals and non-technical decision-makers (Trimintzios et al., 2014). Among others, Pal & Hui (2012) add to these characteristics:

the variety of intentions in the attacks; cyber risks are hard to measure; incentives between private companies and regulatory authorities differ; and the free-rider problem. Together, this leads to the agreed consensus among academics and practitioners that it is impossible to achieve a state that is entirely cybersecurity protected (Pal & Hui, 2012).

All of these characteristics confirm that cybersecurity consists out of multiple problems. Creating policies on such concept with a very complex nature are therefore rather difficult, implementing incomplete policies. Malone & Malone (2013) argue that this endorses the notion that cybersecurity is a ‘wicked problem’. Wicked problems are “complex, unpredictable, open ended, or intractable” (Alford & Head, 2017: 397). Singer & Friedman (2014) argue that cybersecurity is a wicked problem because of the people working behind the machine, rather than the technology itself. This corresponds with Rittel & Webbers’ (1973) perception of wicked problems. They note that problems are wicked when policy makers have to deal with societal problems, which are totally different than the technical problems of engineers. Hence, according to the literature, cybersecurity is rather a societal problem than a technical problem. To deal with wicked problems, O’Toole (1997) suggests public sector networks to bring public and private companies together in decision-making. This is acknowledged by Willem & Licidarme (2014).

2.2. Critical Infrastructures

For a society to function, some vital processes must be maintained. These processes are called the critical infrastructures. Critical infrastructures are often defined as a compromise of “systems and assets, whether physical or virtual, that are so essential to a nation that any disruption of their services could have a serious impact on the national security, economic well-being, public health or safety, or any combination thereof” (Alcaraz & Zeadally, 2014: 52). This definition is also used by other authors in security studies (Paletti, Joseph & Silvac, 2018; Moteff & Partfomak, 2004; Edwards, 2014; Harašta, 2018). Critical infrastructures enable us to live in our modern society. It includes, among other things, the communication on the internet, banking, (air-)ports, food and water supplies (Edwards, 2014). In the Netherlands, the indication of vital systems is conducted by the National Coordinator for Security and Counterterrorism of the Ministry of Security and Justice3. Although the classification of vital sectors are different around the world, the systems are mainly comparable.

The disruption of systems within the critical infrastructure could, as explained previously, cause serious damage to different processes in society. If one of the sectors is disturbed, this could influence all of the different sectors in the critical infrastructure. The connections are described as ‘interdependence’ (Harašta, 2018). “These interdependencies could trigger cascading effects in multiple critical infrastructures when one critical infrastructure is disrupted, damaged or destroyed” (Alcaraz & Zeadally, 2014: 54). This does not only occur within the physical borders of a country. Due to globalization, cross-border dependencies have become a serious issue (Paletti et al., 2018). Together with the increased dependence on information technology, critical infrastructures are increasingly vulnerable to new risks. Critical infrastructures mainly consist of Industrial Control Systems (ICS) such as SCADA (Supervisory Control and Data Acquisition), measuring and controlling the processes and corresponding machines within the system. Originally, these systems were protected against threats as they were locked out from the ICT domain. However, with the increasing use of internet and the ability to control systems from different locations, these systems became remotely connected through the internet (Nicholson, Webber, Dyer, Patel & Janicke, 2012). Although this has increased the efficiency of the working process, it also enhanced the possibility of cyberattacks (Thacker, Barr, Pant, Hall & Alderson, 2017). Paletti et al. (2018) describes these critical infrastructures as cyber-physical systems (CPSs). Not only malicious

3 _{National Coordinator for Security and Counterterrorism (2018, June 18). Critical Infrastructure protection. Retrieved from:}

attackers endanger the critical infrastructure, also human mistakes, technical errors and natural disasters are a serious concern with respect to the protection of the critical infrastructure (Singh, Gupta & Ojha, 2014). In protecting the critical assets in the Netherlands, three formal groups with own responsibilities are divided: owners and administrators of the critical infrastructure (both public and private); fellow public authorities (e.g. police, safety region); and national government (NCTV, 2018).

Figure 2: Different critical infrastructures in the Netherlands, sorted by direct vitality versus indirect vitality. (Luiijf, Burger & Klaver, 2003)

2.3. Public Private Partnerships

Public-Private Partnerships have been a central element in developing infrastructure and since the 1990s a key tool in public policies ((Brogaard & Petersen, 2018; Osborne, 2002). It has been widely embraced among academics and practitioners from both public and private sectors (Osborn, 2002). Public-Private Partnerships are assumed to be more cost-efficient and an effective mean in different policy areas (Zhang, 2005). In a Public-Private Partnership, governments work together with private sector companies to create collective and joint decision-making and is being understood as an appropriate mean to increase policy outcomes (Brogaard & Petersen, 2018). Nevertheless, the increased implementation of Public-Private Partnerships models does not mean that there is a common understanding of the concept. While the concept is widely used, a great variety of partnerships are being considered as Public-Private Partnerships (Forsyth, 2010), making the implementation more difficult (Jamali, 2004). Based on an extensive literature review, Brogaard & Petersen (2018) developed a cohesive conceptualization of Public-Private Partnerships, including its most relevant characteristics, leading to:

“More or less a formalized collaborations between government, business and/or third sector organizations based on shared knowledge, competencies and risks, and developed with the purpose of accomplishing long-term social and/or economic developments in developing countries.” (Brogaard & Petersen, 2018: 735)

However, this definition does not include the accomplishment of security. The studies on Public-Private Partnerships are mainly covering the construction of infrastructure, paying limited attention to security. Public-Private Partnerships are used frequently in the field of security, especially cybersecurity. The privatization of the critical infrastructure led to a shared responsibility in cybersecurity protection, both found is cyber security policies (Carr, 2016) and in cyber crisis management (Boeke, 2017). First in the US, later on in the Netherlands, Public-Private Partnerships were estimated to be the primary mean to address cybersecurity. The Dutch National Cyber Security Centrum (NCSC) explained that Public-Private Partnerships are essential to cybersecurity critical infrastructure protection because “it needs intensive cooperation to maintain a robust resilience of the Netherlands against cyber security threats. This cooperation ensures that the Netherlands is well informed about opportunities and challenges”4.

4 _{National Cyber Security Centrum (2018). Public-Private Partnerships. Retrieved from}

2.4. Network effectiveness

There is a wide recognition on the importance of networks in the academic literature (Milward & Provan, 1998; Provan & Milward, 2001; Provan & Kenis, 2008). These networks can be defined as “three or more legally autonomous organizations that work together to achieve not only their own goal but also a collective goal” (Provan & Kenis, 2008: 231). When examining networks, it is important to evaluate the effectiveness in order optimize the results. According to Provan & Milward (2001: 414-415), “evaluating network effectiveness is critical for understanding whether networks—and the network form of organizing—are effective in delivering needed services to community members”.

During the start of this decade, network effectiveness was an understudied concept (Provan & Milward, 2001). Nowadays, academics provided a considerable amount of time studying network effectiveness (to mention some: Provan & Kenis, 2008; Whelan, 2012; Provan & Milward, 2001; Turrini, Cristofoli, Frosini & Nasi, 2010). However, network effectiveness is still being perceived as a problematic concept on both organizational and network level and no overall theory is being issued (Turrini et al., 2010). There is no consensus on network effectiveness because it fully depends on who is being measured within the network. Moreover, difficulties are observed in measuring the outcomes of the network. Achieving goals can be a long-term process (Bäckstrand, 2006). Network effectiveness must also be clearly differentiated from network efficiency. Network effectiveness relates to the quality of the outcome, while network efficiency is about the quantity of the outcome (Whelan, 2012).Provan & Milward (1995) defined network effectiveness as improving the well-being of their clients and the delivering quality and is used by many different academics, specifically focussing on client-level performance (individual actors) and community level performance (effectiveness for community network operates) (Turrini et al., 2010; Whelan, 2012). Later, Provan & Milward (2001) added to network effectiveness the network-level effectiveness (effectiveness for the network, collaborating partners). The effectiveness on network-level performance led to Provan & Kenis (2008) defining network effectiveness as “the attainment of positive network level outcomes that could not normally be achieved by individual organizational participants acting independently” (Provan & Kenis, 2008: 230). The authors argued that network effectiveness is specifically about the whole network and not about the individual actors. This has been the main approach towards network effectiveness (Provan & Sebastian, 1998). For non-profit or public sectors, the overall effectiveness is normally more important than the impact on individual organizations. However, each actor inside the network

has different individual goals and preferences which makes strategies to control these differences very important.

Bäckstrand (2006) also evaluated network effectiveness as the problem-solving capacity of a network, when the outcomes are crucial to assess whether the network has achieved effectiveness. She assessed network effectiveness through institutional effectiveness:

leadership, clear goal formulation and policy coherence.Ngamassi, Maitland & Tapia (2014) studied network effectiveness in the humanitarian field and defined network effectiveness as the level of activity, measured by the number of funded projects. They primarily focussed on the community level effectiveness.

The operationalising network effectiveness is a difficult process. As showed in the previous paragraph, there are different approaches towards network effectiveness. Turrini et al. (2010) acknowledged these difficulties. To better understand the concept, they tried to provide a preliminary framework on network effectiveness by reviewing many different articles network effectiveness was measured. The framework addressed five types networks effectiveness: client level effectiveness; network capacity to achieve goals; network sustainability and viability; community effectiveness; network innovation and change. According to Turrini et al. (2010), network effectiveness is characterised by network structural, network functioning and network contextual characteristics. They mentioned, for example, that network effectiveness goes beyond the client-level effectiveness until the community benefits as well. Whelan (2012) on the other hand observed that, to analyse and understand network effectiveness, a multi-faced methodological framework is required. He approached network effectiveness in specifically security networks, increasing the number of determinants to which network effectiveness is affected. In his book Network and National Security: Dynamics, Effectiveness and Organization he argued that network effectiveness is “the capacity of a network to achieve its goals” (Whelan, 2012: 38). This is also described by Yang & Jung (2016), observing that network performance can be evaluated to whether it attaints its goals. Whelan (2012) specifically noted that information sharing among the members is crucial to achieve effectiveness. In the field of security, network effectiveness is achieved when there is fast, clear and relevant information sharing (Whelan, 2012). In providing a methodological framework to network effectiveness in a macro environment, he referred to Provan & Kenis (2008) endogenous (factors managers can control) and exogenous (factors that cannot be controlled). The framework divided five independent levels of analysis: Network Structure; Network Culture; Network policies; Network technologies; Network relationships. These independent levels are closely related to Turrini et al. (2010) addressed

characteristics. In the next part of the section, the five different levels are described in more detail.

2.4.1. Network Structure

According to Whelan (2012), the structure of the network is related to effectiveness. This is acknowledged by multiple other researchers (Provan & Millward, 1995; Provan & Kenis, 2008; Klijn, 1996; Raab, Mannak & Cambre, 2013; O’Toole, 1997; Turrini et al., 2010). The network structure by Whelan (2012) addressed both the design and the development of a network. The design “views networks as static forms of organisation in that it examines network dynamics at a particular point in time” (Whelan, 2012: 43). The development refers to the establishment and evolution of the network regarding internal and external contingencies. Networks need to adapt to changes in the ongoing process of organising (Whelan, 2012). Provan & Milward (1995) were the first authors to conduct an analysis on the structural characteristics that might play a role in achieving network effectiveness. They observed that network effectiveness can be explained by “various structural and contextual factors, specifically, network integration, external control, system stability, and environmental resource munificence” (Provan & Milward, 1995). Afterwards, in a study by Provan & Sebastian (1998), centralized integration, strong integration among some network cliques and direct non-fragmented external control were added to these structural characteristics. One can also think of interconnectedness and cohesiveness among the members (Turrini et al., 2010).

Provan & Kenis (2008) examined the governance of networks and how it impacted network effectiveness, categorising governance models along two dimensions: brokered or shared (Provan & Kenis, 2008). Brokered organizations have few direct interactions between organizations and are therefore highly centralized. Shared organizations are directly linked and with each other and frequently interact, resulting in a dense and highly decentralized governance model (Provan & Kenis, 2008). Subsequently, a distinction can be made between participant governed or externally governed. Participant governed networks are managed by a collective or a single participant, while external governed networks are managed as a unique exemplar (Provan & Kenis, 2008).

Whelan (2012) also addressed in his study differentiation between brokered and shared governance models. He related this to networks having a hub-design (vertical) or an all-channel design (horizontal) which could influence the network effectiveness. Effectiveness can be influenced by both forms of governance design. Within regard to information sharing, Whelan (2012) explained that in a hub or centralised network design, information is exchanged

through a central actor who then provides the information to all other members. An all-channel design does not have such a central actor. Information is directly shared among the different members. Especially the all-channel approach of communication and cooperation have become more prevalent when searching for policy outcome solutions (Klijn, 1996). According to Provan & Kenis (2008) networks should be governed as non-hierarchical and without ownership. However, from an economic perspective it is argued that “the market is the only efficient system of non-hierarchical coordination” (Provan & Kenis, 2008: 233). What sort of design is then desirable? It is argued that smaller networks with a large goal consensus and general goals will benefit more from an all-channel design, while larger and more specific and important the goals of a network are, the better a hub-design fits (Provan & Kenis).

Although these studies claimed that the structure of a network has a significant impact on the network effectiveness, it must be mentioned that the differences in structures are difficult to explain. While some characteristics clearly correspond with different structural forms, it is hard to explain what the different characteristics and issues mean to the network effectiveness (Milward & Provan, 1998).

2.4.2. Network Culture

Within networks, different actors collaborate on multiple issues. All these actors have different goals and preferences concerning these issues (Kickert, Klijn & Koppenjan, 1997). Subsequently, members of networks are frequently confronted with different cultural mind-sets. This trend is confirmed by different authors (Zou & Ingram, 2013; Kickert et al., 1997; Schein, 2010; Whelan, 2012; Whelan, 2017). Whelan (2012) observed in his methodological framework on network effectiveness that cultures have an impact on achieving the goals within networks. In a more recent article, he argued that “culture is one of the most significant relational properties of security networks” (Whelan, 2017: 114). However, the concept of culture is seen as rather difficult to define. According to Alvesson (1989), four conceptual versions of ‘culture’ exist: (i) as a building block in organizational design; (ii) as the outcome of symbolic management; (iii) as a diagnostic instrument; (iv) and as a paradigmatic concept. In the organizational literature, a conceptual conflict is found between researchers, mentioning that culture is something an organization has or is (Kummerow & Kirby, 2013). Within this contradiction, cultures are perceived as something that exists within organisations (Whelan, 2017). Schein (2010) also argued that organizations have subcultures, leading to the notion of having both integrated (culture as a whole) as differentiated (cultures within cultures) cultures. Therefore, he used the terms organizations and groups. Groups are defined as “social unit that

has shared history” (Schein, 2010: 4). The author further explained that cultures are to a group what a personality is to an individual. These distinctions become very relevant to understand cultures within networks. When different organizations collaborate in a network, it cannot be seen as an organization but rather as a group. Therefore, Schein (2010: 18) defined group cultures as:

“A pattern of shared basic assumptions learned by a group as it solved its problems of external adaptation and internal integration, which has worked well enough to be considered valid and, therefore, to be taught to new members as the correct way to perceive, think, and feel in relation to those problems”

Whelan’s (2017) definition of network cultures correspond with the definition of Schein (2010). However, Whelan (2017) reduced the definition to “the beliefs, values and attitudes which form over the course of a group’s history and which influence how it thinks and acts in relation to specific problems” (Whelan, 2011: 280). Frequent interaction and shared experiences are observed as instruments influencing the shared attitude towards achieving a common goal (Whelan, 2017). This is confirmed by Turrini et al. (2010), arguing that the cohesion and support from the community, measured by previous collaborative efforts, have a positive relationship. Schein (2010) added that the length of existence and the stability of membership have a significant impact on the network culture.

Hence, cultures differ among both organizations and groups and are constantly changing. Regarding networks, the inclusion of new members also changes the culture, even leading to the existence of subcultures within the networks. The literature also argued that cultures must be approached from an empirical perspective. Although cultural differences are frequently addressed to have a negative effect on the performance (Pothukuchi, Damanpour, Choi, Chen & Park, 2002; Schein, 2010; Whelan, 2012; Whelan, 2017), this is not always the case. “Cultural variation can be positive, negative or neutral for various outcomes” (Whelan, 2017: 131).

2.4.3. Network Policies

The importance of network policies is an understudied object within the theory of network effectiveness. Whelan (2012) acknowledged the lack of academic interest to the policies on network effectiveness, but still included it the level in his methodological framework. Kenis & Provan (2006) have been the one of the single authors, before Whelan (2012), to study the effects of policies on the network effectiveness, using the concept of network control. Why network policy is an understudied subject is grounded by the perception that networks are created from a collaborative perspective, leading to “the idea of formal control mechanisms is

typically viewed as inconsistent with the whole point of having a network” (Kenis & Provan, 2006: 228). Kenis & Provan (2006) addressed two possible reasons for neglected studies on network control. First, networks are perceived as non-hierarchal structures. They are used as a solution for the problem of control. Control, in this sense, would work as contradiction to collaborate cooperation. This relates to what Whelan (2012) defined as the flexibility of organizations. “In other words, network governance and network management are modern (or even postmodern) forms of self-regulated control” (Kenis & Provan, 2006:230). The second reason is that networks are seen as uncontrollable. Agranoff and McGuire (2001: 308) argued that it is ‘‘difficult to establish accountability in public management networks’’, concluding that using control orientation is problematic.

Although network control is addressed in the literature as unwanted, they are however sometimes necessary. Networks are indeed intended to be non-hierarchical and have limited formal accountability, but this could also lead to less collaboration. Following rules and procedures are in this sense purely voluntary (Provan & Kenis, 2008). Having control mechanisms could ensure the achievement of both the network and the individual goals. It also increases the stability of the network (Whelan, 2012). These mechanisms refer to how network control could be exercised (Kenis & Provan, 2006). Kenis & Provan divided five forms of network control: personal-centralized control; formal bureaucratic control; output control; cultural or clan control; and reputational control. Network policies by Whelan (2012) referred to the formal bureaucratic control, rules or regulations to standardize behaviour and prescribing how the network must operate. These rules or regulations are introduced from above (government) or suggested by the network itself. Whelan (2012) defined network policies as “the use of mechanisms by actors to monitor the actions and activities of organizational networks to enhance the likelihood that network-level goals can be attained” (Kenis & Provan, 2006: 228). For example, in a stable network, there could be a need for information sharing procedures.

2.4.4. Network Technologies

The forth level in Whelan’s (2012) methodological framework on network effectiveness are technologies. Network Technologies are addressed as the use of technology to support networks such as information and communication systems. In our digitalized society, the possibilities of technical development need to be utilized. Nowadays, technology forms the basis in sharing information among network members. Technological infrastructures provide network members with more effective and efficient information sharing, minimizing the costs

(Roberts, 2000). Especially in information intensive networks, these technologies are indispensable because “the network’s collaborative information and knowledge management strategy is salient to the functioning of the network and the achievement of goals and objectives” (Desouza, 2009: 1221).

Access to information technologies within networks would benefit the performance (Agranoff, 2006). However, the benefits of information and communication technologies are also part of discussion (Whelan, 2012). Roberts (2000) observed that the use of certain technologies are criticized. Her criticism is explained by distinguishing between codified and tacit knowledge transfer. When knowledge has the capability of being recorded or transmitted in the form of symbols, it is assumed to be codified knowledge. Tacit knowledge is the know-how of knowledge, “acquired via the informal take-up of learning behaviour and procedures” (Roberts, 2000: 431). With the use of information systems, the emphasis is primarily on codified knowledge while tacit knowledge is evenly important. ICT transfer of knowledge often requires co-presence. Tacit knowledge is gained through human interaction, face-to-face participation, strengthening trust and mutual understanding (Roberts, 2000). When knowledge is solely shared through a codified process, it would endanger innovation and the reduction of trust.

Moreover, in the field of security, a paradigm exists between the information sharing and information protection (Desouza, 2009). As a great amount of information has a high level of confidentially, sharing information is not always wanted and guaranteed. Current privacy frameworks also form a barrier to information sharing. Moreover, if all the information gathered is being shared, this could lead to an information overload (Whelan, 2011). Information overload “is a systemic problem in information-intensive networks with high failure costs for not sharing information” (Whelan, 2011: 282). Network technologies are thus beneficial in networks to the extent of being designed in supporting knowledge sharing and other network properties.

2.4.5. Network Relationships

Network Relationships is the final level of analysis in the methodological framework by Whelan (2012). It refers to “relationships between actors in networks and the ways in which such relationships shape the effectiveness of networks” (Whelan, 2012: 119). Network Relationships can be divided between social networks (micro, informal and interpersonal relationships) and organisational networks (macro, formal and inter-organizational). Formal networks rely strongly on different policies and procedures, while informal networks are based

on interpersonal relations. According to Whelan (2012) informal networks are based on interpersonal relationships and trust. Both are clearly important in networks, but the way trust and distrust are developed in networks, is an understudied subject (Whelan, 2012). Trust is defined as “the extent to which one is willing to ascribe good intentions and have confidence in the words and actions of other people” (Cook & Wall, 1980: 39). Two forms of interpersonal trust enhance knowledge sharing more effectively: personal competence and person’s benevolence (Abrams, Cross, Lesser & Levin, 2003). Benevolence is the interest in a person’s well-being, goals and competence is the usefulness of the person’s expertise within the network (Abrams et al., 2003). Both dimensions also have shown to be important factors in peer and manager performance. Rousseau, Sitkin, Burt & Camerer (1998) provided other forms of trust: Deterrence-based trust; Calculus-based trust; Relational trust; Institution-based trust. When not elaborating too much on the different forms of trust, a short explanation will be given due to the use of the different forms in Whelan’s (2012) framework. Deterrence-based trust is suggested to develop quickly “because the costly sanctions in place for breach of trust exceeds any potential benefits from opportunistic behaviour” (Rousseau et al., 1998: 1997). Calculus–based trust is a rational choice based on economic exchange, doing things that are economically beneficial for one another. Relational trust is created by constant interaction among different member of a network. Institution-based trust “develops on the basis of institutional controls that promote trust or create the context for trust to arise” (Whelan, 2012: 126).

According to Rahman, Osman-Gani, Momen & Islam (2015), trust has a significant impact on knowledge sharing process, specifically for non-academic participants. Within organizational studies, the nature of relationships, which are built on trust, are more important than the nature of resources in networks (Lavie, 2006). Network effectiveness “will depend critically on an underlying network of social relationships based on face-to-face interaction” (Nohria & Eccles, 1992: 290). The assumption that trust has a positive effect on network effectiveness is acknowledge by many authors (Whelan, 2012; Das & Teng, 1998; Nohria & Eccles, 1992; Sydow & Windeler, 1998; Edelenbos & Klijn, 2007 and more). Willem & Licidarme (2014), on the other hand, argued that trust might be harder to achieve in the public sector. They confirmed the importance of trust in inter-organizational relations, but observed that purely professional trust is important (cognitive trust) instead of friendships and caring relationships. Whelan (2012) also argued that trust is hard to develop when networks do not have a shared history and a stable membership. Such networks depend on other forms of trust to achieve network effectiveness.

For Whelan (2012), network relationships is the most important level of analysis in his methodological framework. “A network based on high levels of trust is likely to be able to overcome problems relating to structure, culture, policy and technology in ways that other networks are not” (Whelan, 2012: 282).

As the literature review on network effectiveness revealed, there are different variables involved in the process of shaping network effectiveness. Variables relevant to network effectiveness mainly depends on the field of study. Moreover, considering the different levels and the variables, many of them are interdependent. When one variable is changed, others will decrease or increase at the same time. Therefore, the different levels by Whelan (2012) cannot be observed as separated components but have an interconnected relationship.

3. Methodology

The following chapter provides a transparent explanation of the research conducted. Firstly, the conceptual framework is addressed, explaining how this study explains the phenomenon addressed in the literature. Afterwards a description is given on the case, elaborating on why a case-study, a single-case study and the particular single-case study was selected. Moreover, the methodological framework addresses how the data is collected and analysed, and an operationalisation of the different concepts. The chapter finishes with an argumentation on the validity and reliability of the thesis.

3.1. Conceptual Framework

According to the literature, there are multiple factors that could influence network effectiveness, both in organizational networks and security networks. This study specifically focussed on security networks, a Public-Private Partnerships on cybersecurity within the Dutch critical infrastructure, the ISAC-Port. Whelan’s (2012) research on network effectiveness is considered as the basis of this study because of the emphasis on security networks. While previous literature, including Whelan (2012), have formulated their conclusions from the actual effectiveness of the network, this study addressed the perception of network effectiveness. While studying perception of effectiveness, conclusions are drawn from impression on different participants in the Public-Private Partnership instead of actual achievements of effectiveness.

Moreover, the theoretical framework in the previous chapter observed different explanations and approaches on (the perception of) network effectiveness. Most academics refer network effectiveness to capacity to achieve certain goals, such as Whelan (2012) and Yang and Yung (2016). To assess the perception of network effectiveness, goal achievement is considered as the prominent approach. Therefore, the first sub-question aims to explain how the members within the network perceive the achievement of their stated goals within the Public-Private Partnership:

How do members within the Public-Private Partnership responsible for cybersecurity protection in the Dutch critical infrastructure perceive the effectiveness?

However, this question on its own does not sufficiently explain the main research question included in this study. Hence, this study used the methodological framework by Whelan (2012) to explain the factors affecting the perceived effectiveness. The methodological framework by

Whelan (2012) divided five independent levels to analyse the factors of the network effectiveness: Network Structure; Network Culture; Network Policies; Network Technologies; Network Relationships. Each level consists of different elements that could influence the perceived effectiveness within the Public-Private Partnership. In evaluating these factors within the selected case, this study assessed what factors have an influence on the perception on effectiveness. Therefore, the second sub-question is:

What factors affect the members’ perception on effectiveness within the Public-Private Partnership on cybersecurity in the Dutch critical infrastructure?

This question lead to the following conceptual design. The central element in this study is the ‘perceived goal achievement’, the way in which perceived network effectiveness is researched. Moreover, the design presents the different levels according to Whelan (2012) that affects the perceived goal achievement. The different factors are explained according to the different levels.

3.2. Case Study

According to Yin (2017: 15) “a case study is an empirical method that investigates a contemporary phenomenon (the “case”) in depth and within its real-world context, especially when the boundaries between the phenomenon and context may not be clearly evident”. Case studies are found in many different social science disciplines and have explained many different complex social phenomena. It is an appropriate mean to exploratory research (Yin, 2017). This study consisted of such an exploratory research and therefore fits the requirements for case study research. Although case studies preferably answer ‘how’ or ‘why’ research

questions, case studies could also apply to some ‘what’ research questions (Yin, 2017). Yin (2017) argued that these occasions can be flexibly interpreted. This study applied a ‘what’ question, but also provided an in-depth examination of ‘how’ Public-Private Partnerships on

Network effectiveness Perceived goal achievement Independent variables Structure Culture Relationships Technologies Policies Perceived network effectiveness Dependent variable

cybersecurity in the Dutch critical infrastructure perceive the effectiveness. Moreover, it is preferred to use case studies as a research method when studying a contemporary event and when the behaviours cannot be manipulated (Yin, 2017). The ISAC-Port was such a contemporary case with experts who were still representing the partnership. The researcher had no control over the event, mainly because the researcher was not allowed to participate in the ISAC-port meetings.

Case studies are supposed to be single-case studies or multiple-case studies, both holistic or embedded (Yin, 2017). This study consisted of a holistic single-case study. Although there are more ISACs protection the critical infrastructure in the Netherlands, a critical test was executed on the theoretical foundations of the theory of network effectiveness. Yin (2017) explained that critical rationales are appropriate mean to which single-case studies are performed. Whelan’s (2012) methodological framework provided different levels affecting the network effectiveness. This study critically assessed these factors by observing whether they had an effect on the perception of effectiveness within an ISAC, contributing to knowledge and theory building. Hence, this study corresponded with the critical rationale by Yin (2017). Although Whelan’s (2012) approach primarily addressed security organisations (e.g. intelligence agencies, police, fire department), the members within the ISAC-Port only partially represented certain security organizations. Still, the methodological framework by Whelan (2012) is used in this case. Moreover, this thesis was the first research on the perception of effectiveness within the ISAC environment in the Netherlands. No previous scientific study addressed this topic, which makes this study part of a new phenomenon.

The ISAC-Port has been used as a case in this study due to numerous reasons. The ISAC-Port represents a Public-Private Partnership within the Netherlands on cybersecurity critical infrastructure protection. It builds on settling long-term collaboration between partners in more permanent institute. This surpassed the applicability of the ICT Response Board (Public-Private Partnership on cyber crisis management), because this institute solely collaborates on incident bases. However, this does not yet explain the reason for choosing a single-case study over a multiple-case study. The reason for choosing a single-case study contains both accessibility and time constraints. When conducting a multiple-case study, only limited respondents per ISAC could have been interviewed. Although a multiple-case study would have increased the external validity of this research, the volume of relevant information per ISAC would have been very limited due to access constrains. Direct access to relevant members of ISAC is crucial in answering the research question of this study. ISACs are working according to high levels of trust and confidentiality, creating barriers to ‘outsiders’.

Studying only one ISAC resulted in a deeper and more precise understanding of the effectiveness and the factors affecting this. Furthermore, the researcher possessed indirect connections to relevant members within the ISAC-Port compared to other ISACs due to previous working experience. The organization of the researchers’ internship had direct connections to organizations central to the Port of Rotterdam and Amsterdam. This increased the possibility to access different organizations within the ISAC-Port. Also, the time for conduction interviews was limited, therefore having close connections to organizations became an important consideration. Moreover, The ISAC-Port contains, compared to other ISACs, both public and private partners in the network. Other ISACs are emphasized by either public or private organizations, limiting the connection to the central element of this study, Public-Private Partnerships. Subsequently, due to the academic interest of the researcher, only limited scientific attention was given to the cybersecurity collaboration within the Dutch Port environment. According to Yin (2017), single-case studies are an interesting and useful research method for understudied subjects, such as the ISAC-Port. The final reason for choosing the ISAC-Port was the importance of cybersecurity in the functioning of the sector. On the 27th of June 2017 a cyber crisis appeared in the Port of Rotterdam5. APM Terminals, active within the Port of Rotterdam caught serious damage due to the cyber-attack. This led to increased attention on cybersecurity issues and strengthened the collaboration within the sector.

3.3. Data collection & analysis

This research solely consisted of a qualitative research. A triangulation of qualitative methods was used: in-depth interviews, document analysis and desktop research. Interviews have been the central data collection method used in this research. The interviews were conducted from

five members within the ISAC-Port. These members directly represented organizations responsible for the information sharing within the ISAC-Port.

Access to the respondents was created by using snowball sampling. Snowball sampling method “yields a study sample through referrals made among people who share or know of

others who possess some characteristics that are of research interest” (Biernacki & Waldorf, 1981: 141). Considering the level of confidentiality within the ISAC-Port, the researcher chose to, in agreement with the respondents, to include them anonymously. This study looked at contradictions and downfalls in the collaboration of the ISAC-Port. It was expected that this

5 _{Verschuren, E. (2017, June 27). Wereldwijde aanval met ransomware treft ook deel Rotterdamse haven en TNT. Retrieved from}

would only be obtained when respondents are able to answer the questions freely. Therefore, anonymity was guaranteed.

Although not all members of the ISAC-Port were interviewed in this study, all of the respondents had a central position within the ISAC-Port. Some possessed the position of

chairman within the ISAC-Port, others were facilitator and other respondents represented organisations that participate in the ISAC since the establishment. Moreover, the respondents represented both public and private organizations and had one to five years of experience in the ISAC-port.

The researcher executed semi-structuralized the interviews. First, the researcher was interested in the perceived effectiveness of the ISAC-Port. Secondly, the researcher examined what factors the respondents addressed as important to achieve the perceived effectiveness, matching them with the independent levels obtained from the literature. The interviews were executed in Dutch. Each interview was recorded, transcribed and then returned to the respondent. The returning transcriptions gave the interviewees the possibility to check whether if provided information was not allowed to be shared regarding the policies of the ISAC-Port (including names, data and practises). The references and citing corresponding the transcriptions were translated into English to be able to include the results in the study. To analyse the data collected through the interviews, the transcriptions were coded. The coding was used to perform the analyse technique Pattern Matching. Yin (2017) argues that Pattern Matching is useful technique in case studies. Pattern Matching reflects to what extent the theoretical foundation matches with the collected results of the case. The researcher expected this method to be most desirable. The results, more specifically sentences, obtained from the interviews have been coloured according to the different levels of Whelan (2012) (Perceived effectiveness; Structure; Culture; Policies; Technology; Relationships).

This study also used documents to provide an answer to the research question. Documents were only used to support the results gathered from the interviews. In the analysis, the sections structure, policies and technologies were primarily analysed with documents and

Respondents Public/Private organization Years of experience within the ISAC-Port

Respondent 1 Private 4 to 4.5 years

Respondent 2 Public 1 year

Respondent 3 Public / (Private) 2 years

Respondent 4 Private 3 years

Respondent 5 Public / (Private) 5 years

supported by the interviews. The selection of documents specifically addressing the ISAC-port were limited. Therefore, multiple documents related to ISAC-Port primarily addressed ISACs in general. This study selected the following documents for the analysis: Sharing and Analysis

Centres (ISACs) Cooperative Models (ENISA, 2018), Public-Private Partnerships Cooperative Models (ENISA, 2018), Dutch Cybersecurity Strategies (2011 and 2013) and the Dutch Cybersecurity Agenda (2018). Apart from interviews and documents, desktop research

was carried out to focus more specific on the case. Terms as ISAC Port, Port collaboration

cybersecurity, Cybersecurity Port Netherlands, Public-Private Partnership Port. These terms

were put into search-engine Google to provide specific knowledge on the ISAC-Port.

3.4. Operationalization

Measuring network effectiveness is claimed to be a difficult task (Provan & Kenis, 2008). However, to execute this research, an understanding of the concept is required. Turrini et al. (2010) observed in their study the different forms of effectiveness, depending on the field of study which form is most desirable (client level effectiveness, capacity to achieve stated goals, network sustainability and viability, community effectiveness, network innovation and change). As already explained in section 3.1 (conceptual framework), it has be observed that the capacity to achieve the stated goals within the Public-Private Partnership is the preferable operationalisation of effectiveness, an integrated approach that is shared among both public and private organizations. This corresponds with the framework of Whelan (2012) as the single author focussing on security networks.

Whelan (2012) also addressed in his book five analytical levels that influence network effectiveness in security networks: Network Structure; Network Culture; Network Policies; Network Technologies; Network Relationships. The next table (table 2) provides both the operationalisation of the concept perceived network effectiveness and the different levels. Thereafter, different indicators, complemented with the determinants by Turrini et al. (2010), are selected for each level. The final column addresses the data source used to gather the information.

Table 2: operationalization concepts

Concepts Definitions Indicators Data source

Perceived network effectiveness

The perceived capacity of Public-Private Partnerships to achieve its stated goals.

• Goals • Objectives • Outcomes • Targets • Interviews Structure

The form at both a point in time (design) and through the

establishment and evolution of Public-Private Partnerships. • Size • Complexity • Interconnectivity • External control • (a-)Symmetrical power • Document analysis • Desktop research Culture

Group with shared basic beliefs, values and attitudes which forms course of a group’s history and which influence how it thinks and acts in relation to specific problems and how to solve them.

• Prolonged history (age) • Reciprocity • Stability • Shared beliefs, attitudes and values • Interviews Policies

Control mechanisms from both internal and external entities that ensures the achievement of the goals stated by Public-Private Partnerships. • Rules • Guidelines • Procedures • Document analysis • Desktop research Technologies Technological infrastructures used to support Public-Private Partnerships on sharing codified knowledge among members. • Information systems • Communication systems • Technical assistance systems • Administrative systems • Document analysis • Desktop research Relationships

The formal and informal

connections between actors in a network. • Interpersonal relationships • Organizational relationships • Trust • Interviews