The accuracy of VAA-designs : how to match a voter with a party?

(1)

The accuracy of VAA-designs: how to match a voter with a party?

Jouke Huijzer

Bachelor Thesis University of Amsterdam

(2)

2

Acknowledgements

This bachelor’s thesis is the outcome of an extensive analysis and writing process to which many lecturers, friends and family members have made their direct or indirect contributions. First of all I would like to thank Prof. Dr. Wouter van der Brug for his supervision, his comprehensive feedback, his recommendations and his flexibility throughout the process. Second, I am very grateful to Dr. André Krouwel, my second supervisor, for his enthusiastic ideas and recommendations, his feedback and, not in the least place, the provision of the data. Third, I would like to thank my girlfriend Lianne Schmidt and my mother Carien Huijzer, for their support and critical review of the text. Fourth, I would like to thank my other friends and family for their intellectual support and ever-present readiness to help me improving my work throughout my bachelor’s. In particular Peter Huijzer and Tivadar Vervoort should be mentioned, since they were always willing to review my papers. Last, I would like to thank all other lecturers at UvA (for this study, in particular the teachers in statistics) that have initiated me into the wide field of research in political science. I hope many students after me can enjoy the same intellectual climate at the UvA as did so far.

(4)

4

I. Introduction

Over the past decade, voters in many different countries have consulted Voting Advice Applications (VAAs) to determine their vote during election times. In countries such as Finland, Switzerland and The Netherlands, the first online VAAs, that appeared in the 1990s, where only consulted about 6.000 to 8.000 times (De Graaf, 2010; Ruusurvita, 2010). But in the 2000s, the popularity of VAAs exploded so that in some of these countries about 20 to 40 percent of the electorate consulted a VAA during the most recent national elections (Garzia and Marschall, 2012). Analogous to increases in the numbers of VAA-users, the number of VAAs grew rapidly as well. The first VAAs appeared in advanced European democracies like The Netherlands (StemWijzer), Finland (Vaalikone) and Germany (Wahl-o-Mat). But over the past few years, also voters in Canada, USA, Mexico and even Egypt were able to discover which party matched their preferences best by consulting a VAA.

Typically, VAAs ask users to take a position on a set of policy statements. Subsequently the positions are compared with the positions of political parties or candidates and the level of agreement with every party or candidate are presented to the user as a voting advice. Yet, VAAs strongly differ from one another in the way their voting advices are calculated and presented to the user (Garzia and Marschall, 2012). The first VAAs only calculated the agreement level with all political parties. The VAAs advised the user to vote for the party that, on average, took the most proximate positions on all statements. But when the number of VAAs increased, new models of calculating and presenting the voting advice were developed. Currently some VAAs (like the Swiss VAA SmartVote and several VAAs that are developed by the Dutch company Kieskompas BV) use a two- or more-dimensional spatial model to calculate and present their voting advice.

On a more detailed level, VAAs also differ in the number of response categories users or parties can choose to indicate their positions on a statement. Additionally, some VAAs also give their users the possibility to put some extra weight on statements (or thematic clusters of statements) they consider important. But also when it comes to this allocation of saliency, VAA’s allow their users to assign the saliency scores in very different ways (Garzia and Marschall, 2012: 208).

The large differences in the design and method of the VAAs have resulted in an ongoing debate about which VAA-design is most accurate. This question is not only often addressed in the media, but also in academic literature. Already in the earliest literature on VAAs, scholars have theorized about why some designs are preferable to others (Lobo et al., 2010). Still, in the small but fast-growing amount of studies on this subject, only limited empirical research has been done to the consequences of the design and calculation method used by different VAAs.

So far, only a handful of articles have been written on the differences in outcomes of various VAA-designs. Walgrave et al. (2009) found that the selection of statements had profound consequences for the individual voting advice. On an aggregative level they found that certain sets of statements benefited particular parties over others. More closely related to this article, Louwerse and Rosema (2013) compared different spatial and non-spatial methods to calculate the voting advice. They concluded that the outcome of the voting advice is strongly determined by the design and method of a VAA. For that reason they advice users to consult more than only one VAA and to take more parties into account than only the party with the highest match. But, although their findings stress the importance of the debate, it is still impossible to make any empirically based claim about which VAA-design is most accurate. Hence, the debate about which design and calculation method yields the most accurate voting advice, remains foremost a theoretical one.

Building on the study of Louwerse and Rosema (2013), I want to bring the empirical debate one step further and attempt to provide at least some empirical grounds on which the accuracy of the VAAs can be compared. This study is primarily concerned with the extent to which the advice of different VAAs-designs matches the (pre-existing) party preferences of VAA-users. My

(5)

5

objective is not entirely new. Previous studies of Walgrave et al. (2008), Mendez (2012) and Wall et al. (2012) did similar attempts, but, for methodological reasons (that will be discussed in the next section), their results are quite crude. So to draw some more refined conclusions my method differs in some respects from the earlier ones. Concisely formulated, I expect that the better differences in pre-existing party preferences of the individual user are resembled by the differences in matches to each party, calculated according to a certain calculation method, the more accurate a VAA-design.

I acknowledge that the party preferences of the users can’t be regarded as a fully appropriate instrument to measure the accuracy of a VAA. In the next section I’ll deal with the main objections to this method. Yet, following most studies on VAA-users and issue voters, it can be expected that at least the politically well-informed users base their party preferences for a large part on issue-positions. In an accurate VAA, the party preferences of this group of users should at least to some extent be reflected in the voting advice, whereas in an inaccurate VAA this relation may be less evident. So in this respect, the party preferences of (at least) the politically well-informed users can serve as a criterion to judge (or actually compare) the accuracy of different VAA-designs. Hence, I’ll pay particular attention to the different levels of internal political efficacy and the political interest of the users (that can serve as proxy-variables of the political knowledge).

The remainder of this article is organized as follows. In the next section I’ll discuss the validity of the method that is used to measure the accuracy of VAAs and – building on existing literature on VAAs – deal with the main objections against this method. Subsequently, I’ll give an overview of some of the most consulted VAAs, their designs and calculation methods. The fourth section links the ideas behind most VAAs to broader theories about party evaluation and issue voting in political science. Particular attention will be paid to the ‘proximity model’ and the ‘directional model’ of issue voting. The fifth section contains a description of the data, the case selection and the operationalization. In section six the results are presented and interpreted. The last section recapitulates the most important findings and discusses the implications and limitations of this study.

(6)

6

II. Measuring the accuracy of VAAs

The rise of VAAs over the past two decades is increasingly gaining academic attention. Research into VAAs and their effects is still in its infancy, but quickly expanding. In a broad sense, the emergence of VAAs since the mid-1990s, fits perfectly in the trend of diminishing cleavages, the decrease in party loyalty (Dalton and Wattenberg, 2000; Mair, 2013) and the eventual rise of issue-voting as observed by many scholars (Downs, 1957; Key, 1966; Aardal and Van Wijnen, 2012). The voter data and data of party positions are, as Wagner and Ruusuvirta (2012) found, a useful and valid instrument to test and develop theories on voting behavior (see: Mendez, 2012). Walgrave et al. (2008b) place VAAs on a more distant place from mainstream political science, but VAAs remain an increasingly popular data source and research object in the field.

Recent studies focused on varying facets of VAAs: their users, the party positions or their impact. Wall et al. (2012), Rosema and Ruusuvirta (2009), Dumont and Kies, (2012) and Ladner et al. (2012) examined the effects of the voting advice on voting decisions and found that the voting advice has significant effects on the ultimate voting decisions, especially among swing voters. Ladner and Pianzola (2010), Marschall and Schmidt (2010), and Hirzalla et al. (2010) found that VAAs also have a positive effect on electoral turnout and increase the political knowledge of the users. Other studies focused more on the supply side and observed large differences in the performance of parties in varying VAAs. Van Praag (2007) found large differences between VAAs, with respect to the number of advices for a certain party. Moreover, the share in voting advices of each party strongly fluctuates between elections, even for the same VAA. According to Walgrave et al. (2009), the inconsistency could be explained by the differences in statement selection. Louwerse and Rosema (2013) emphasize that the differences in outcomes were highly influenced by the design and calculation method that VAAs use. A small difference in the calculation method of the matches can make a large difference for the outcome.

As noted in the introduction, Walgrave et al. (2008), Wall et al. (2012) and Mendez (2012) even went one step further and compared voting advice with electoral preferences of the users. Walgrave et al. (2008) compared the share of voting advices for each party of the Belgian VAA named Doe de Stemtest, with the share of votes every party won at the 2007 federal elections. But as Wall et al. (2009) remarked, users of VAAs differ politically and demographically from the total electorate. For that reason, Wall et al. (2009) compared the voting advice of the Irish VAA Pick Your Party with the pre-existing vote intention of its users. After comparing several calculation methods of different VAA-designs, they found that the outcomes of a multidimensional spatial model corresponds the most with the pre-existing vote intention. Still there was, particularly among the large center parties, little similarity between the vote intention and the advice.

A similar research design was used by Mendez (2012), who compared several models to evaluate parties and calculate the matches. Some of the methods Mendez (2012) examined will be discussed in more detail in section IV. Just like Wall et al. (2009), Mendez tested which of the calculation methods rendered the most matches between the voting advice and the pre-existing vote-intention.

Though, not only the study of Walgrave et al. (2008), but also the findings of Wall et al. (2009) and Mendez (2012) have their limitations. In their studies they only examine in how many cases the party with the highest match in the voting advice, corresponds to the vote intention of a user, but they leave all other parties out of their analysis. Therefore, the validity of their method to measure the accuracy of a VAA is limited. For instance, it would make no sense if the party that a user wants to vote for, has the lowest or the second-highest match in the VAA. The voting advice would be equally false, according to the methods of Wall et al. (2009) and Mendez (2012).

So, although all previous studies have clearly shown in that there is, due to differences in design and method, a high number of mismatches between the voting advice and the actual vote (intention), no study was able to make valid judgments about the accuracy of different sorts of

(7)

7

VAAs. In this study I attempt to do so by comparing the match1_{of a user to each party with its} probability to vote for each similar party. I expect that if a user has a high match with a certain party in a VAA, (s)he also has a high probability to vote for this particular party. The other way around, I expect that for a user with a low match with a certain party, the probability to vote for this party is low as well. In general I assess a (certain calculation method of a) VAA-design more accurate as the voting advice is better able to predict the differences in the pre-existing electoral preferences.

As noted, the pre-existing electoral preferences of a user are not a fully appropriate instrument to assess the accuracy of VAAs. One major objection to this method can be that users are uncertain of their preferences before (and maybe after) they consult a VAA. Therefore the probability to vote, which is so central to this method, is an unreliable yardstick. If a user would be entirely sure about his preferences, why would he consult a VAA at all?

Theoretically, this question indeed exposes one of the weaknesses of this study. One would expect users only to consult a VAA, when they doubt their electoral preferences. However, most empirical studies that focus on the characteristics and motivations of the users to consult a VAA, found the contrary. Studies of Van der Pol, 2013; Fivaz and Nadig, 2011; Marschall and Schulze, 2012; Boogers and Voerman, 2003; Marschall and Schmidt, 2010; Ruusuvirta and Rosema, 2009 and Hooghe and Teepe, 2007 found that the vast majority of VAA-users is characterized by a high level of political efficacy, political knowledge and political interest. Instead of forming an electoral choice, most users consult a VAA to get their pre-existing preference confirmed. So, particularly for the politically well-informed users that are likely to base their party preferences on issues (Hobolt, 2005; Zaller, 1992), the ultimate voting advice should at least to some extent reflect their initial preferences. Despite the change in preferences a VAA-advice can bring about, it is not evidently false to use pre-existing preferences as instrument to measure the accuracy of VAAs. For the majority of voters, these preferences remain the same.

Another objection to the method I use to measure the accuracy of VAAs can be that the probability to vote is an invalid instrument because it is influenced by more factors than only the differences between a certain party and a voter on a set of statements. One can argue that a VAA bases its voting advice only on the substantive agreement on a set of statements, whereas the electoral preferences can also be determined by social-structural characteristics, party loyalty, sympathy towards a party or party leader and strategic considerations (see: Mendez, 2012). Because VAAs disregard all these determinants, it would be incorrect to expect that, even if a user is completely sure about his preferences, an accurate voting advice would correspond to the party preferences. For instance, if the electoral preferences of a user are mainly determined by psychological factors such as the sympathy to the party leader, the relation between the electoral preferences and a voting advice is undermined, because the preferences are not only based on substantive agreement like the voting advice. This would even be the case if a VAA would be perfectly able to take all the issue stances of voters and users into account. So according to my method, a VAA can never be fully accurate if the preferences of users are not entirely determined by issue stances.

However, the central purpose of this study is not to assess the accuracy of VAAs in general (that is the extent to which the advices of VAAs correspond to the electoral preferences); my aim is rather to make a comparison between the accuracy of different calculation methods and VAA-designs that are used by different VAAs. The question is not whether VAAs are accurate, but which VAAs are most accurate. Non-substantive factors like sympathy to the party leader only affect one side of the relation that is central to my method, namely the probability to vote. But this study focuses more on the other side of the relation, namely the different ways to calculate the matches. In all examined relations, the probability to vote is the same for every case and only the match is calculated in different ways. It follows that the relation between the probability to vote

1_{The term ‘match’ refers to any calculated score that indicates the extent to which a party is recommended by a}

VAA for a user. The ‘match’ can be the average level of agreement, the distance between party and user in a two-dimensional political landscape or it can be calculated in any other design.

(8)

8

and the calculated matches is equally undermined by non-substantive factors in all the relations that are examined. The differences in the relations can only be explained by differences in the method that is used to calculate the matches, so a fair comparison is still possible.

The only required condition to use the probability to vote as a valid yardstick to compare the different calculation methods, is that the pre-existing electoral preferences are at least partly determined by substantive considerations. Following the studies mentioned above (e.g. Hobolt, 2005; Zaller, 1992; Van der Pol, 2014; Fivaz and Nadig, 2011; Marschall and Schulze, 2012; etc.), it is very likely that this condition is satisfied in practically all cases. In a more normative sense one can even argue like Macdonald et al. (1991: 1107) that: “For representation to be meaningful, the mass must evaluate parties at least partially on the basis of policy issues.” Additionally, one can reason that it is questionable whether voters that consider substantive positions irrelevant for their electoral preference, have any incentives to consult a VAA at all.

Of course there are significant differences between users. As mentioned, Hobolt (2005), Zaller (1992) and Carmines and Stimson (1980) have argued that especially well-informed voters, are most likely to base their electoral preferences on issues (issue-voting). Hence, I’ll pay particular attention to the political efficacy or the political interest of the examined users. This makes it possible to focus on the group of users which are well-informed and are most likely to be sure about their pre-existing preferences. Non substantive factors will also be potent for the group of users that are most likely to base their party preferences on issues. But as noticed above, this matters only if one judges the accuracy of VAAs in general, but it matters far less if the purpose is only to make a comparison between VAA-designs. Taken all arguments together, the pre-existing probability to vote of a user towards all parties can be regarded at least as an appropriate instrument to compare differences between VAA-designs. In what respect those designs differ from one another, will be discussed in the next section.

(9)

9

III. Differences in VAA-designs

VAAs have a long history before they became commonplace during elections. The pioneer in the field of VAAs was unmistakably StemWijzer, which was developed in the Netherlands already in 1989 by an organization named Stichting Burgerschapskunde2_{(De Graaf, 2010: 35). Initially the test} was a print-version, made for educational purposes. StemWijzer had placed all parties along a one-dimensional left-right continuum, based on their stances on the statements in the questionnaire. The ‘advice’ consisted of a place between the parties along the left-right continuum; the more proximate a party to the user, the higher the match. The test had to be filled in by hand, and students had to calculate their scores themselves.

The first edition of StemWijzer didn’t get much public attention. But when the IT-usage increased in the 1990s, StemWijzer developed a digital tool which became publicly available online for the first time at the Dutch national elections of 1998. From that point on also the number of users quickly increased. After the VAA was consulted only 6.500 times in 1998, over 2 million voters consulted StemWijzer during the campaign for the national elections in 2002 and 2003. At the national elections of 2006, StemWijzer peaked with a total number of 4.7 million users (ibid.: 41).

When StemWijzer became accessible online in 1998, it had also adapted a new VAA-design which became a standard for many VAAs that were developed in the years that followed. Users of StemWijzer were asked to position themselves on thirty statements in total. They could choose between four response options: ‘agree’, ‘disagree’, ‘neutral’ or ‘skip this question’. Parties could choose between the same options to position themselves. The voting advice was calculated using the so called ‘city-block’ method. If a party and a user take the same position, the party gets two points. If one of the two (party or user) takes a neutral position, whereas the other agrees or disagrees, the party earns one point. If party and user take opposing positions, the party earns no points at all. Statements that were skipped were not taken into account. Additionally, users could select statements they considered important (the so called ‘saliency score’ as it is more commonly known in political science). All scores on statements that were selected as important were multiplied by two. So the maximum score of a party was 120 (if a party had a similar position on all statements and all statements were selected as important), and the minimum score was zero3_. The voting advice was presented similar to figure 1.1 and consisted of the party with the highest agreement score. The agreement scores of all other parties were listed from high to low (Ibid.: 40-3).

This method remained unchanged until 2006, after which the answer categories and calculation method slightly changed. Currently, the agreement score of a party only increases if the party and the voter take exactly the same position (Ibid.: 44). Yet, the central idea remains the same. The method of StemWijzer became a sort of a standard for all other VAAs that were developed in the early 2000s. In Germany StemWijzer helped with the creation of the popular VAA Wahl-o-Mat. Also in Finland, Belgium (Flanders) and Switzerland VAAs were developed (Walgrave et al., 2008: 52) with broadly the same features.

VAAs became incredibly popular in election times, but – as popularity comes with a price – their validity, objectivity and consistency were often criticized by journalists, parties, users and academics (see: Walgrave and Van Aelst, 2005; Ruusuvirta, 2010; Van Praag 2007). For that reason, many VAAs adopted different methods to select the statements; position the parties; and to calculate and present the voting advice. Most notable in this respect is the (again) Dutch VAA named Kieskompas, which currently has the largest market share in the world and is StemWijzer’s main competitor in The Netherlands. Kieskompas was developed at the Dutch national election of

2_{Translated: citizenship-foundation.}

3_{Note that, using this calculation method, the maximum score can vary per user which makes it hard to compare}

(10)

10

Figure 1: Answercategories and presentation of the voting advice in StemWijzer, Kieskompas and SmartVote.

Figure 1.2: Response categories and two-dimensional spatial landscape used by Kieskompas 2012

Figure 1.3: Response categories and smartspider used by SmartVote 2014 Figure 1.1: Response categories and agreement scores in StemWijzer 2012

(11)

11

2006 in cooperation with the VU university in Amsterdam and a newspaper named Trouw. After the initial success, Kieskompas became a commercial organization that has developed many VAAs in over 40 different countries.

The design of Kieskompas differs from StemWijzer in several respects. First, Kieskompas allowes parties and users to indicate their position towards a statement on a five-point scale instead of the three-point scale as used by StemWijzer. The response categories ‘agree’, ‘disagree’ and ‘neutral’ are supplemented by the categories ‘completely agree’ and ‘completely disagree’, so users and parties can also indicate the intensity of their positions (see: figure 1.2). Additionally, users can skip a statement by selecting the answer ‘no opinion’ instead of ‘skip this question’. Furthermore, Kieskompas doesn’t allow their users to put more weight to a number of statements the user regards important (saliency score). Instead, users can, after the advice is presented, modify the outcome by deselecting several thematic clusters of statements. All statements that are related to a theme that is not selected, are left out of the calculation.

Yet, the most distinctive feature of Kieskompas is the way in which the advice is presented to the users. Instead of a list of parties ranking from the party with the highest level of agreement on all statements to the party with the lowest level, Kieskompas positions all parties and the user in a spatial two-dimensional political landscape (see figure 1.2). Inspired by the work of political scientists, all statements were scaled along a economic left-right dimension, and a socio-cultural progressive-conservative4_{dimension. The extent to which users and parties agree or} disagree on all statements determines their positions on the axes. The implicit message is that the more proximate a party is positioned to the user, the higher the match (but since 2010, Kieskompas deliberately refuses to give an explicit voting advice for one party, so one can also interpret the landscape different).

One of the main advantages of the spatial model is that even if one or more statements are left out of the calculation, the positions and distances remain broadly the same, whereas in the city-block calculation method, the advice strongly depends on the selected statements (see: Walgrave et al. 2009). On the other hand, scholars have argued that agreement scores are more accurate because in a low-dimensional model5_{, parties and user can, even if they disagree on many} statements, hypothetically, end up at the same position along the axes (Lobo et al., 2010: 168). Because users can only move in two directions when they position themselves on a statement, some information gets lost. So where Kieskompas emphasizes its consistency and nuanced approach, StemWijzer can theoretically argue that their method retains more information and is due to its simplicity, more transparent.

StemWijzer and Kieskompas¸ respectively the oldest and the largest VAA, can be regarded as two extremes with regard to their designs. Many other VAAs take an intermediary position between the two by combining the methods, or they present the advice in both ways. For instance, the Austrian VAA Wahlkabine gives voters only the opportunity to agree or disagree with a statement, but voters can indicate on a nine point scale the importance or intensity for each position on a statement. The advice is given in agreement scores similar to StemWijzer. The British VAA VoteMatch gives users the possibility to indicate their positions on a three-point scale. After the user has taken stance on all statements, (s)he can assign saliency scores to thematic clusters of statements. Again the advice is presented as a list of parties and agreement scores. PreferenceMatcher, a consortium of several universities that has developed VAAs in several different countries, allows users to indicate their position on a five-point scale similar to Kieskompas, but gives their advice in both ways (a spatial two-dimensional model and agreement scores). After the presentation of the advice, users can refine their stances by assigning saliency

4_{The terms progressive and conservative are used as a substitute for the GAL-TAN (green/alternative/liberal and}

traditional/authoritarian/nationalist) dimension (see: Hooghe et al. 2002) which is widely used in political science.

5_{Similar to Louwerse and Rosema (2013), I’ll refer to the methods of Kieskompas (and SmartVote) as}

‘low-dimensional’ models, and regard the method of StemWijzer as ‘high-‘low-dimensional’ because all thirty statements can be regarded as a separate dimension.

(12)

12

scores on a three-point scale for every statement. The score on important statements counts double, unimportant statements count half.

A last VAA design that needs to be discussed is the ‘smartspider’, developed by Switzerland’s most popular VAA SmartVote. In this VAA users were asked to indicate their positions on a four-point scale (‘agree’, ‘rather agree’, ‘rather disagree’ and ‘disagree’) and assign a saliency score on a five point scale (labeled by the symbols ++, +, =, - and --). The advice was presented in three ways. SmartVote not only placed all parties (and candidates) in a two-dimensional political landscape and calculated the high-dimensional levels of agreement; it also calculated the position of each user on a total of eight different thematic dimensions. All positions on statements that were related to one or more thematic dimension(s) such as ‘environmental protection’ or ‘law & order’, determined the position along this dimension together with all other related statements. For instance, if one agreed on the thesis “would you support the introduction of a monthly minimum wage of CHF 3,800 for everyone in Switzerland?”, the position on the dimension ‘expanded welfare state’ would move towards 100 percent, whereas the position on the dimension ‘liberal economy’, would shift in the direction of 0 percent. The implicit idea is that the user has the highest match with the party (or candidate) with the closest average position on all eight dimensions. The saliency score was not taken into account for the calculation of the smartspider and the voting advice was presented as a kind of spider web (see: figure 1.3).

In this study I will briefly examine the accuracy of the smartspider, but my main focus lies on the differences between the high-dimensional calculation methods and the two-dimensional spatial model. Furthermore I’ll pay particular attention to the effects of measuring opinions and saliency in the different ways that are discussed above. Yet, before I’ll turn to the operationalization, I’ll discuss some of the most important theories on issue voting and party evaluation, that are highly influential to the design and further development of VAAs.

(13)

13

IV. Proximity versus directional theories of issue voting

The central purpose of a VAA is to give a voting advice based on attitudes of parties and on a number of issues. In this respect they completely adhere to the main assumptions of the wide range of theories on voting behavior that can be grouped under the name of ‘Issue-voting’. Originally, issue voting referred to the idea voters and parties could take varying positions toward a certain issue (or a set of issues) ranging from completely agree to completely disagree with a neutral or indifferent position in the middle. Following this theory, voters would vote for the party that has the most proximate position to the voter (Downs, 1957; Davis et al., 1970; Enelow and Hinich, 1984). This proximity model is particularly suitable to predict voting behavior after voters and parties place themselves parties along a left-right continuum. Much of the ideas behind the proximity model are resembled in the way VAAs calculate and present their voting advice.

However, by the end of the 1980s the proximity model was challenged by a so-called ‘directional model’, developed by Rabinowitz and Macdonald (1989; see also: Macdonald et al., 1991). Following Rabinowitz and Macdonald (1989), the proximity model as used for the left-right continuum, is not equally applicable when it comes to stances on specific issues.6_Their criticism on the proximity model, as well as their alternative directional model are central to the debate about how voters evaluate party stances on issues, which is crucial to the development of an accurate VAA design. For that reason, I will discuss their theory in some more detail.

The directional model assumes that a position towards an issue entails two components: a direction (agree, disagree or neutral) and the intensity of the direction. So also in this case, the positions of voters and parties on a certain issue can be plotted on a continuum ranging from completely agree, to completely disagree with a neutral position in the middle (see figure 2). Following Niemi and Bartels (1985), Rabinowitz and Macdonald (1989) argue it is unnecessary to add a saliency component to the issue, because much of this aspect is already covered by the intensity component.

The difference between the directional and the proximity model can be explained most clearly by using the examples that are all illustrated in figure 2. For every example, a line is drawn along which parties and voters are positioned. The party positions are indicated by the letter A (for party A) and B (for party B), the positions of two voters are indicated by the letters X (for voter X) and Y (for voter Y). A party or voter that is placed in the middle has a neutral position and is indifferent towards the issue. Parties and voters on the left agree on the issue, parties and voters on the right disagree. The further a party or voter is positioned from the center, the more intensely it agrees or disagrees with the issue.

The problem that arises with the proximity model can be explained by taking a look at example 2.1. Following the proximity model, the voters would both vote for party B. In the case of voter Y, because party B takes the same position (same direction and intensity). In the case of voter X, because party B is closer to its position than party A, although it has a different direction. According to Rabinowitz and Macdonald (1989), it would be highly unlikely that a voter would vote for a party that takes an opposing position just because the spatial distance to the party is closer than to the party with the same direction. Hence, they propose a directional model in which a voter prefers all party positions with the same direction over the all possible positions with a different direction. Thus, following the directional model, voter X would prefer party A over party B because A has the same direction.

But the directional model even goes one step further. According to Rabinowitz and Macdonald (1989), a voter would prefer a party with a strong position (that is a position with high intensity) over a party with a moderate position if both parties have the same direction as the voter. This would even be the case if the voter itself has only a moderate position. So if party

6_{In later work, they even imply that their directional model is also able to give a better understanding of party}

(14)

14

B would change its opinion and take the same position as voter X (moderately agree) as in example 2.2, voter X would still prefer party A on this issue. If there is no other party that takes the opposite direction, all voters that disagree on this issue like voter Y, would still vote for party B.

Contrary to the proximity model, the directional model states that extreme party positions7 are favored over moderate positions. However, this is not always the case. If a party position is too extreme, it might lose at least some voters because it positions itself outside of what Rabinowitz and Macdonald (1989: 108) call the region of acceptability. As in example 2.3, party A is positioned outside the region of acceptability, so in this case voter X would vote for party B. It follows that the optimal position for a party is to take an extreme position on an issue while remaining within the region of acceptability.

The model of Rabinowitz and Macdonald (1989) has large implications for party strategy and voting behavior. Following the proximity model most scholars assumed in accordance with the game-theoretical work of the economist Hotelling (1929), that parties will always try to take the position of the median voter because this would render the most votes. Yet, the directional model prescribes on the contrary. Parties should rather take more extreme (though still not too extreme) positions and the directional model explains why they often do. In later work Macdonald et al. (1993) extended their directional model by also taking into account the uncertainty about voter preferences and about whether a party keeps its promises. Furthermore, they found some evidence that the directional model also provided a better understanding of party evaluations than the proximity model in multiparty systems like Norway (Macdonald et al., 1991) and the Netherlands (Aarts et al., 1999).

The directional theory of issue voting also has large implications for the calculation methods of VAAs (Mendez, 2012). VAAs can, for instance, calculate the matches following different methods. Figure 3 presents two matrices according to which agreement scores can be assigned following the proximity or the directional method. Just like in VAAs as Kieskompas and

7_{Just like Rabinowitz and Macdonald (1989) I use the term ‘extreme’ to refer positions with a high intensity. Perhaps}

it would be better to speak of ‘more radical’ positions or ‘strong stances’, because ‘extreme’ positions might wrongly be confused with the positions of extremist parties.

Agree Disagree A B X Y Agree Disagree A B X Y Agree Disagree A B Boundary X Y Example 2.3

Figure 2: Directional versus proximity model of issue voting Example 2.1

Neutral

Example 2.2

Neutral

Preferred party according to: Directional model Proximity model Both models

Boundary of the region

(15)

15

PreferenceMatcher, users and parties can indicate their positions on a 5-point scale ranging from 1 (completely agree) to -1 (completely disagree) with the intermediary positions scoring 0.5 (agree), 0 (neutral) and -0.5 (disagree). Using the city-block formula it is possible to assign scores according to the proximity method and by calculating the so-called ‘scalar product’, it is possible to assign scores according to the directional method:

City-block (proximity): ∑_𝑖=11 − |𝑃_𝑋𝑖 − 𝑃_𝐴𝑖|

Scalar product (directional): ∑_𝑖=1(𝑃_𝑋𝑖− 0) × (𝑃_𝐴𝑖− 0)

Wherein:

𝑃𝑋𝑖 is the position of voter X on statement i

𝑃𝐴𝑖is the position of party A on statement i.

Matrix 3.1 shows the results for the proximity model.8_{If the user and the party take the same} position, they score a full point; if they take completely opposing positions they lose a full point. As the scores in the matrix show, the score increases as the positions of the party and user become more proximate to one another. Most VAAs use the city-block method to calculate the agreement scores in a high-dimensional model. Only recently some VAAs, like StemWijzer and Kieskompas (in some of their VAAs), changed to a calculation method wherein the party and user only score a full point if they take a position with exactly the same direction.

Yet, if voters, like the directional model theorists state, rather evaluate parties on their strong positions, the scores can probably be better assigned according to matrix 3.2. In this case the party and user can only score a full point, if both take the same extreme position. As Macdonald et al. (1991) argue that voters don’t evaluate parties on their neutral positions, the agreement score cannot increase, if the user or the party (or even both) takes a neutral position. Note that the party and the user can only reach a full match if both take only extreme positions with the same direction and party and user can get no match at all if one of the two takes only neutral positions.9

8_{In the matrices, all response categories are abbreviated as follows: completely agree (CA); agree (A); neutral (N);}

disagree (D); and completely disagree (CD). One can find the assigned score in the cell where the row of the user position and the column of the party position cross each other. The total score can by calculated by taking the sum of the scores on the statements were the party and user took a position (and eventually divide it by the total number of statements on which the user and party took a position). Note that in the data all scores are rescaled so that they (potentially) reach from 0 to 100.

9_{To make the matches of different users comparable, the total scalar product is divided by ∑ |𝑃}

𝑋𝑖| so that parties can

also have a full match with users that take moderate positions. Yet, this additional calculation doesn’t change the results much (compared to the proximity scores).

CA A N D CD CA A N D CD CA 1 0.5 0 -0.5 -1 CA 1 0.5 0 -0.5 -1 A 0.5 1 0.5 0 -0.5 A 0.5 0.25 0 -0.25 -0.5 N 0 0.5 1 0.5 0 N 0 0 0 0 0 D -0.5 0 0.5 1 0.5 D -0.5 -0.25 0 0.25 0.5 CD -1 -0.5 0 0.5 1 CD -1 -0.5 0 0.5 1

Figure 3: Matrices to assign agreement scores for statements according to the proximity and the directional models

Party Answer Party Answer

User Answer

(16)

16

Figure 4: Proximity and directional model in a two-dimensional space

The proximity model and the directional model also have their spatial representation in a political space with two issue dimensions. Figure 4 illustrates the difference between both methods to calculate the match between a voter and a party. In this example voter X can again choose between party A and party B. Voter X has the coordinates -0.6 and 0.4 on respectively the horizontal issue dimension h and the vertical issue dimension v (both ranging from -1 to 1); party A has the coordinates 0.8, 0.8; and party B has the coordinates 0.2, -0.4. Following the proximity method, voter X would prefer the party that is most proximate over the party that is least proximate. The distance between the voter and the party can be calculated using the Pythagoras formula (or Euclidean distance):

Euclidean distance (proximity; spatial): √(𝑃𝑋ℎ− 𝑃𝐴ℎ)2+ (𝑃𝑋𝑣− 𝑃𝐴𝑣)2

After filling in the positions in the formula, the distance to party A is 1,46 and to party B 1.13, so voter X will prefer party B.10

Again this is remarkable because party B has an opposite direction on both issue dimensions, whereas party A only takes an opposite position on the horizontal issue dimension. For that

10_{Filling in the formula gives:}

Distance to party A = √(−0.6 − 0.8)2_{+ (0.4 − 0.8)}2_{= √1.96 + 0.16 = 1.46} Distance to party B = √(−0.6 − 0.2)2_{+ (0.4 − −0.4)}2_{= √0.64 + 0.64 = 1.13} Party A Party B Voter X -1 -0,5 0 0,5 1 -1 -0,5 0 0,5 1 ve rt ica l is sue da ime nsion v

Horizontal issue dimension h 105°

(17)

17

reason Rabinowitz and Macdonald (1989) suggest a different formula to calculate the distance to a party that writes down as follows:

Scalar product (directional; spatial): √𝑃_𝑋ℎ2_{+ 𝑃}

𝑋𝑣2× √𝑃𝐴ℎ2+ 𝑃𝐴𝑣2× cos⁡𝐴𝑋

First, one calculates the vectors to the party and the voter (that is the distance between the center point and the position of the voter or the party) using again the Pythagoras formula. Subsequently the vectors are multiplied by each other and by the cosine of the angle in between the two vectors. In the directional model, the score increases as the voter and the party take a more similar direction and can increase (or decrease) any further if the party and the user take more extreme positions. In this case, a higher score indicates a higher match than a lower score. Filling in the positions in the formula gives a score of -0.16 for party A and a score of -0.28 for party B;11_{so contrary to the proximity model, now party A is preferred over party B.}

Also the spatial methods to calculate the matches are particularly relevant for VAA-designs. VAAs like Kieskompas, SmartVote and PreferenceMatcher all present their voting advices using a two-dimensional political landscape. A problem with the spatial directional model is that it is complicated for a VAA-user to figure out with which of the parties (s)he has the highest match. Intuitively one would look to the proximity of each party in the landscape, rather than the length of the two vectors and the angle in between them. However, preliminary to the question about how to present the voting advice, lies the question about which of the calculation methods yields the most accurate voting advice.

Section VI, contains a comparison between the accuracy of low and high-dimensional VAA-designs in which the matches are calculated according to the proximity or the directional method. Yet, the directional theory of issue voting does not only provide new clues to examine possible improvements of VAA-designs; the other way around can the different VAA-designs that will be examined, also provide a better understanding of the way voters evaluate parties. Though, the main purpose of this article is to make a comparison between the different VAA designs. The implications of this study for theories on issue-voting are only briefly discussed in the conclusion.

11_{Filling in the formula for the directional model gives:}

Score for party A = √−0.62_{+ 0.4}2_{× √0.8}2_{+ 0.8}2_{× cos 105° = 0.72 × 1.13 × −0.20 = −0.16}

(18)

18

V. Data selection

To examine all different VAA-designs and calculation methods, I use (only) two datasets. One consists of data generated by the approximately 750,000 users that consulted Kieskompas 2012 during the run up to national elections of 2012 in the Netherlands. The other dataset was generated by almost 200,000 users that filled out the Dutch version of the pan-European VAA EUprofiler that was developed by Kieskompas and SmartVote for the European elections of 2009. Both datasets consist of Dutch users and their relation to respectively eleven and nine parties. Following the history of VAAs that was briefly described in section III, the Netherlands can be regarded as a nursery for VAAs and VAA-designs. Many voters consult a VAA in election times, so by 2009 or otherwise 2012, it can be expected that most users were already familiar with the principle of VAAs.

It needs to be taken into account that The Netherlands has a multiparty system with an extremely low electoral threshold of only 1 seat in parliament (0.67 percent). This makes it relatively easy for small movements or single issue parties with less ideological coherence to get elected (for instance, if the electoral threshold would have been 5 percent, only 6 of the 11 parties would have won enough votes to enter parliament in 2012). Due to the low electoral-threshold, practically all European party families are represented in the Dutch political landscape which makes the Netherlands a representative case for many other Western-European countries. Notice however, that the presence of many small parties with low levels of ideological coherence might favor some VAA-designs over others with regard to their accuracy. Hence, this study needs to be placed in its context and it should be emphasized that VAA-designs always need to be adjusted to the electoral structure of a particular country.

Kieskompas 2012 was completed about 750.000 times, but it is impossible to trace whether some users filled out the complete Kieskompas more than one time. After filling out the ‘questionnaire’, users could optionally answer several additional questions and assign saliency to certain clusters of statements. For this study, I’ve only selected users that filled out also the additional questionnaire. Not only because this allows me to do a more extensive analysis, but it also reduces the probability that some users are taken twice into account. From all users that filled out all (optional) questions, I took a random sample so that there are about 4000 users left.

The design of EUprofiler 2009 was quite similar to that of Kieskompas 2012, but additional to the spatial model, which consisted of a left-right and a pro- or anti-European integration dimension (instead of a GAL/TAN dimension), the advice was also presented in terms of (high-dimensional) agreement and as a smartspider. Furthermore, users of EUprofiler could optionally assign a saliency score on a 3-point scale to every statement separately (instead of thematic clusters of statements). Again I selected only the users that assigned a saliency score to some statements and filled out all relevant additional questions so that there are about 2200 users left.

For all selected users, both datasets contain the following information: their position towards each statement on a 5-point scale or in some cases a ‘no answer’ position; the probability to vote from all users for most parties12_{in Kieskompas 2012 or EUprofiler 2009 on a 11-point scale}13_{; for} the users of Kieskompas 2012 there is a saliency for thematic clusters of statements on a binary scale available; for the users of EUprofiler there is a saliency score on a 3-point scale per statement available14_{. Furthermore, the users of Kieskompas 2012 filled in some extra questions that provide}

12_{This study takes only parties into account that were got elected or were already elected in the previous election.}

Still, some users only filled in their probability to vote for some parties.

13_{After the users positioned themselves on the total of 30 statements (but before they got an advice), they were}

asked “How likely is it that you will ever vote for the following parties?” Users could optionally indicate this likelihood (or probability to vote as I will call it) for all parties in Kieskompas or EUprofiler on a scale ranging from 0 to 10.

14_{Users of Kieskompas 2012 could modify their advice by (de)selecting one or several thematic clusters of statements}

(eleven in total) after which they had to press ‘recalculate position’. All statements related to deselected clusters were then left out of the calculation of user and party positions. The data contains only information about the first recalculation of the positions.

(19)

19

a self-estimation of their internal political efficacy.15_{The additional questions of the EUprofiler 2009} lacked the estimation of the internal political efficacy; instead the data contains the self-estimation of the political interest.16

Together with the party positions, it is possible to calculate the matches using the matrices and formulas mentioned in the previous section.The scores for the high-dimensional methods, wherein every statement forms a separate dimension, can be calculated for the users in both datasets. Using the Kieskompas 2012 dataset, it is possible to calculate the proximity scores and scalar products to each party in the two-dimensional political landscape. The EUprofiler 2009 dataset makes it possible to calculate the average proximity and the scalar product of all seven dimensions in the smartspider. 17

With these datasets, the calculation methods of the most important VAA-designs are covered. As stated, I’ll pay particular attention to the political efficacy and political interest of the users. Hence, I’ll not only make a comparison between the high-dimensional VAA-design, the two-dimensional design and the seven-two-dimensional smartspider, but also split out the results for groups of users with different levels of political interest or efficacy.

Furthermore, the data consists of saliency scores, so it is possible to weight the statements accordingly. The saliency scores are only taken into account for the proximity models, because according to the directional model, the saliency question is superfluous (Rabinowitz and Macdonald, 1989: 94-6). With the saliency taken into account, the formula mentioned in the previous section is extended to:

City-block (proximity; saliency): ∑ (1 − |𝑃𝑋𝑖− 𝑃𝐴𝑖|) ×

𝑆𝑋𝑖

∑ 𝑆𝑋𝑖

𝑖=1

Wherein:

𝑃𝐴𝑖is the position of party A on statement i.

𝑃𝑋𝑖 is the position of voter X on statement i

𝑆𝑋𝑖 is the saliency score of voter X for statement i

∑ 𝑆_𝑋𝑖 is the sum of all saliency scores of voter X to all statements i

Lastly, it is possible to compress the 5-point scale to a 3-point scale by merging the answer categories ‘completely agree’ with ‘agree’ and ‘completely disagree’ with ‘disagree’. This makes it possible to calculate the agreement scores with each party more similar to the method that StemWijzer used (until 2006) and to the one it currently uses. These methods are discussed in appendix B and will be discussed in more detail, together with the results in the next section.

Users of EUprofiler 2009 could optionally assign the saliency score after they positioned themselves on all 30 statements, but before they got their voting advice. They could choose between the options –, =, and +. If they chose ‘–’, the scores on that statement were counted half; in the case of ‘+’, scores counted double; if users chose ‘=’ or left the options open, the scores were weighted normal.

15_{To measure the internal political efficacy, users were asked to what extent they agreed upon the following ‘thesis’: “I}

think that I have a good understanding of the current political issues in my country.” Also here users could indicate their position on a 5-point scale ranging from ‘completely agree’ to ‘completely disagree’ with an additional option to select ‘no opinion’. Because almost no users answered ‘completely disagree’, I’ve merged the this response category with the category ‘disagree’.

16_{To measure the political interest, users were asked the question: “To what extent are you interested in politics?” The}

interest is measured on a 4-point scale. User could answer this question by selection one of the following options: ‘much interest’, ‘quite a lot of interest’, ‘not so much interest’ or ‘no interest’. Also here the categories ‘no interest’ and ‘not so much interest’ are merged.

17_{Note that I deduce the direction of the statements in the dimension from the VAAs (see for the method Krouwel}

and Wall, 2014), so the statements might be scaled (statistically) suboptimal. On the other hand, Kieskompas is entirely in control of the statement selection, so it is very likely that the statements are selected with the two dimensional model in mind, since this was the only way the advice was presented. This was different with the development of the EUprofiler, because they presented the advice in three ways. Consequently, some statements were not taken into account in the calculation of the two-dimensional spatial model and the smartspider because they didn’t fit in one of the dimensions.

(20)

20

In total, the matches are calculated in 18 different ways. For 8 methods I’ll use the Kieskompas 2012 dataset, for the other 10 methods the data from EUprofiler 2009 is used. So for all examined users in the Kieskompas 2012 dataset, the proximity score or the scalar product is calculated in 8 ways for all eleven different parties. and in 10 ways for all nine parties in the EUprofiler 2009 dataset. In total 88 matches are calculated for each examined user in the Kieskompas 2012 dataset and 90 scores for each user of EUprofiler2009.

Because the initial purpose of this study is to compare the different calculation methods without making a distinction between the parties, it is necessary to stack the data (see: Van der Eijk et al. 2006). This means that all scores of a user on variables that are related to a specific party become part of a new case. So the total number of users is multiplied by the total number of parties and every case forms a user-party combination (see figure 6). In the new situation, all scores that are calculated according to the same method are placed in the same column in the data matrix. This makes it possible to estimate a separate regression for each method with the calculated matches (with each party) according to a certain calculation method as independent variable and the probability to vote (to each party) as dependent variable. Because all cases share some characteristics that relate to the user with a number of other cases, not all cases are independent of each other. Hence, I’ll run a multi-level regression to correct for eventual user-specific deviations.

As noted in section II, I asses a VAA-design with a certain calculation method more accurate, the better this method is able to predict the pre-existing party preferences (that is the probability to vote for each party). The regression equation is:

Probability⁡to⁡vote𝑖𝑗 =γ₀₀+γ₁₀∗ ⁡calculated⁡match𝑖𝑗+ 𝛿0𝑗+ 𝜀𝑖𝑗

In every model, the calculated matches depend on the VAA-design and the associated calculation method. All matches in high-dimensional models are rescaled so that they (potentially) range from 0 (no agreement) to 100 (full agreement). The relation with the probability to vote is expected to be positive. Also the possible average distance to each party on the dimensions in the smartspider ranges from 0 to 100, but in this case the relation is expected to be negative; the shorter the average distance on the dimensions, the higher the match, so the higher the probability to vote is expected to be.18_{The maximum potential proximity score for the two-dimensional spatial}

18_{The formula for the smartspider is quite similar to the formulas for the high-dimensional model:}

City-block: ∑𝑑=1|𝑃𝑋𝑑− 𝑃𝐴𝑑|

Scalar product: ∑𝑖=1(𝑃𝑋𝑑− 0) × (𝑃𝐴𝑑− 0)

The city-block formula with saliency is: ∑𝑑=1|𝑃𝑋𝑑− 𝑃𝐴𝑑|× 𝑆̅_𝑋𝑖𝑑 ∑ 𝑆̅𝑋𝑖𝑑

Wherein:

𝑃𝐴𝑖is the position of party A on dimension d

𝑃𝑋𝑖 is the position of voter X on dimension d

𝑆̅𝑋𝑖𝑑 is the average of the saliency scores of voter X for statements i that are related to dimension d

∑ 𝑆̅𝑋𝑖𝑑 is the sum of all averages of saliency scores of voter X for statements i that are related to dimension d

One can calculate the average score for a user to a party by dividing the result by seven

(21)

21

model is 2.83 (= √22_{+ 2}2 _{= √8) and the minimum 0. Similar to the smartspider, the relation} between the proximity score and the probability to vote is expected to be negative. The maximum score for the directional spatial model is 2 and the minimum -2 (= √2 × √2 × ±1); the relation is expected to be positive. The next section discusses the results.

(22)

22

VI. Results

Because not all independent variables are measured on the same scale, it is only possible to make a fair comparison about which independent variable in a model predict the probability to vote best, by looking at the R-square. Table 1 shows the R-squares of eight models in which the scores on the independent variable are calculated in six different ways. In model 1 to 4, the main independent variable consists of proximity scores (calculated according to the methods and formulas that were discussed in the previous section); in model 5 to 8, the scores on the independent variables are scalar products that follow from the directional model. The precise method that is used to calculate the matches is indicated beneath the R-square coefficients; the design is indicated by the symbols (that apply to all models in the same column). In model 1 and 4, the calculated matches (independent variable) consist of spatial scores that are calculated in the two-dimensional political landscape. The matches of model 2, 3, 6 and 7 are calculated for high dimensional models in which every statement forms a separate dimension. Note that the matches in model 2 and 3 and in model 6 and 7 are calculated in exactly the same way, only the dataset is different.19_{The scores in model 4 and 8 are based on the party and user positions on the seven} dimensions of the smartspider of EUprofiler 2009.

As the results clearly show, the high-dimensional models (2, 3, 6 and 7) yield the most accurate voting advice of all proximity models and all directional models. It remains unclear whether the proximity or the directional model renders the most accurate voting advice. Based on the Kieskompas 2012 data, the directional model yields a more accurate voting advice than the proximity model; for the EUprofiler 2009 users, it’s the other way around.

The differences in accuracy between the two ways to calculate the match in a two-dimensional space are more robust (model 1 and 5). The R-square for the proximity model (1) is not even half as high as that of the high-dimensional proximity model (2). But the matches become way more accurate if the positions in the two-dimensional landscape are interpreted following the directional model (5). As noted, it remains counter-intuitive (or at least very complex) for a user to interpret the result this way. For the seven-dimensional smartspider (model 4 and 8) is the proximity model most accurate, although the differences are not as large as in a two-dimensional space.

From the results follows that the most accurate method to calculate the match, strongly depends on the design of a VAA. If a VAA (only) uses a two-dimensional political landscape to calculate the match like Kieskompas, it would be most accurate if the results are interpreted following the directional model. In the smartspider, on the other hand, is the match most accurate if the proximity model is used. For the high-dimensional designs, after all, the most accurate ones, there are no robust differences between the proximity and the directional model with regard to their accuracy. This finding has also interesting implications for the way users evaluate parties; those will be discussed in the conclusion.

19_{Both models are presented so that they can be used as reference to compare all three different VAA-designs.}

Table 1: Proximity and directional models

Model 1 Model 2 Model 3 Model 4

R² proximity models 0.077 0.167 0.172 0.143

Method Eucledian distance City-block City-block City-block

Model 5 Model 6 Model 7 Model 8

R² directional models 0.145 0.177 0.164 0.114

Method Scalar product Scalar product Scalar product Scalar product

Design

(23)

23

Following most studies mentioned in section II, it could be expected that voting advice of VAAs is most accurate for well-informed users since they are most likely to evaluate parties on their issue stances (just like VAAs do). Unfortunately, it is hard to find an instrument that can adequately measures to what extent a user is well-informed. For that reason, this study makes use of the self-estimation of the political efficacy (in Kieskompas 2012) or political interest (in EUprofiler 2009) as a proxy-variable to get an indication of the political knowledge of the user.

Table 2 presents the results of the models 1 to 8 of table 1 by separate groups of users with different levels of political efficacy. The pattern for the proximity models and the directional models is broadly the same and for all different groups, the high-dimensional models remain to yield the most accurate matches. In this respect there is no need to focus only on the group of users with highest level of political efficacy/interest to compare the accuracy of the different VAA-designs.

One can even better focus on users with intermediary levels of political interest or political efficacy to find which VAA-design is most accurate. Because, surprisingly, in none of the models is the advice most accurate for the group of users with the highest level of political efficacy or political interest. In model 4 and 8 where the match is calculated using a smartspider, the group with high political interest has even the lowest R-square. In all other models is the advice, in accordance with the expectations, least accurate for the group with the lowest levels of political interest and political efficacy. In the high-dimensional models 2 and 6 are the pre-existing voter preferences best predicted by the matches for the group with the second highest level of political efficacy. In all other models are the matches most accurate for the group with a medium level of political efficacy or interest.

The results suggest that the extent to which parties are evaluated on the basis of issues by voters, does not always increase with the level of political information. Of course this analysis is based on a self-estimation of the political efficacy or interest, so possibly users overestimate themselves. Another explanation might be that VAAs aim to help the mass-public with their voting advice and VAA-designers adjust their VAAs to the characteristics of the majority of the users (with regard to the formulation of the statements for instance). However, the question why some groups of users get a more accurate voting advice than others remains foremost an empirical one that lies beyond the scope of this article. But the results challenge one of the most widely accepted (and examined) views about which voters are most likely to evaluate parties on issue positions, so more research is required to find a more definite explanation.

As mentioned in section III, many VAAs give their users the option to modify their voting advice by making it possible to weight or deselect statements they regard (un)important (the so-called saliency question). Table 3 presents six models in which the voting advice is modified by taking the saliency score into account. As noticed, only the proximity models are modified by the saliency scores, since the directional model theorists state that the saliency question is already Table 2: Proximity and directional models by level of political efficacy/interest

Proximity models Directional models

Model 1 Model 2 Model 3 Model 4 Model 5 Model 6 Model 7 Model 8

R² very high efficacy 0.071 0.163 0.125 0.177

R² high efficacy/interest 0.082 0.177 0.170 0.129 0.154 0.188 0.167 0.102 R² medium

efficacy/interest 0.085 0.165 0.187 0.157 0.157 0.174 0.177 0.126

R² low efficacy/interest 0.047 0.125 0.158 0.146 0.113 0.131 0.147 0.119 Method Eucledian _distance _blockCity- _blockCity- _blockCity- _productScalar _productScalar _productScalar _productScalar Design

The accuracy of VAA-designs : how to match a voter with a party?