What makes hashtags effective on Twitter?
How relevance and length influence the effectiveness
Master thesis
University of Amsterdam
Faculty of Economics and Business
MSc. in Business Administration- Marketing Track
Student name：Jiahuan Zhang
Student number: 11089318
Supervisor: Dr. Hsin-Hsuan Meg Lee
Statement of Originality
This document is written by Student Jiahuan Zhang who declares to take full responsibility for the contents of this document.
I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.
The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.
Abstract
Recently, social medias such as Twitter have become as effective platforms on which marketers are motivated to conduct marketing activities. In order to stand out from these information-flooded platforms and ensure the desired marketing outcomes, the use of hashtags has become prevalent. A hashtag is a string of characters started with the hash (#) symbol and it serves as a topical marker to facilitate users’ navigation on social medias (Yang et al. 2012). While there is a lay belief that hashtags should be short and congruent to the brands (LePage, 2014), this has never been tested. Therefore, this study dialectically analyzes the effects of hashtag relevance and length on user engagement. It also investigates the association among these effects and brand familiarity. Using data from 255 Twitter users in Netherlands, it is identified that hashtag relevance and length both have effect on engagement and their interaction effect is also clear. Specifically, for unfamiliar brands the congruent hashtag is indeed more effective to increase user engagement than the incongruent hashtag. In contrast, for familiar brands the incongruent hashtag is somewhat better. Furthermore, for both familiar and unfamiliar brands the moderate length hashtag containing two or three words always performs the best compared with the too long or short hashtags. These results hold even when the brand preference, people’s habits and daily twitter behavior are controlled. Taken together, these findings shed light on what type of hashtags can increase the user engagement and how to design more effective hashtags for the marketing campaigns.
TABLE OF CONTENTS
1. Introduction... 1
2. Literature Review... 4
2.1 Hashtags and its effectiveness... 4
2.2 Relevance of hashtags and user engagement... 7
2.3 Length of hashtags and user engagement... 10
2.4 Moderating role of Relevance on Length... 12
2.5 Brand familiarity and User Engagement... 13
2.6 Conceptual framework... 16
3. Methodology... 17
3.1 Sample... 18
3.2 Experiment design... 18
3.3 Stimuli development and Procedure... 19
3.4 Measures... 23
3.5 Data analysis... 27
4. Results and Analysis... 28
4.1 Reliability and manipulation check... 28
4.2 Descriptive analysis ... 3
4.3 Hypotheses testing... 36
5. Discussions ... 44
5.1 Academic Implications... 50
5.2 Managerial Implications... 51
5.3 Limitations and Future research... 53
6. Conclusion ... 55
7. References...
..57
I. The Treatments... 65 II. The Experiment Survey... 67
TABLE OF FIGURES AND TABLES
Figure 1. Conceptual framework... 17
Figure 2. The interaction effect of hashtag relevance and length ... 40
Figure 3. The interaction effect of brand familiarity and hashtag relevance... 42
Table 1. 2 × 3× 2 between-subjects Design... 19
Table 2. Stimuli design... 22
Table 3. Cronbach’s alpha... 29
Table 4. Results of One-sample t-test... 30
Table 5. Demographic statistics... 31
Table 6. Descriptive statistics of variables for per condition... 33
Table 7. Correlations between main variables... 37
Table 8. ANCOVA, MANCOVA summary and Mean values for relevance effect.... 38
Table 9. ANCOVA, MANCOVA summary and Mean values for length effect... 39
Table 10. ANCOVA, MANCOVA summary and Mean values for interaction effect... 41
Table 11. ANCOVA, MANCOVA summary and Mean values for moderating effect on relevance... 44
1. Introduction
In recent years, many social medias such as Twitter and Facebook have been increasingly used as the marketing tool by brands, through which brands are able to maintain regular interaction with customers and gain feedbacks about what people think and feel about their products and services (Kouloumpis et al. 2011). Take Twitter as an example, there are around 75% Fortune 500 companies conducting their marketing activities on Twitter (Swani et al. 2014). The widely use of Twitter is changing the business landscape and redefine the marketing activities (Rapp et al. 2013). Besides that, using Twitter can improve the brand awareness, build and retain good relationships with consumers, create and strengthen brand loyalty as well (Kumar and Mirchandani, 2012). However, it is difficult for a brand to stand out from Twitter where is flooded with information. In order to solve this problem, the use of hashtag becomes increasingly prevalent, because a hashtag links relevant topics and events together, making the tagged tweets much more remarkable and searchable (Tsur and Rappoport 2012).
In the marketing context, hashtags can be viewed as an effective tool from perspectives of consumer and brand. From consumer’s perspective, an effective hashtag can make consumers search brand-related information more efficiently (Yang et al. 2012). From brand’s perspective, an effective hashtag enables brand to increase the user engagement such as the number of followers, number of retweetings (Martin, 2013). However, sometimes hashtags still are highly possible to be ignored or worse be misused for negative purposes from the public. McDonald’s, for instance, tried to
promote its brand by creating a hashtag (#McDStories) and expected people to share the heartwarming stories happened in McDonald’s restaurants around the world. But things did not go as planned and this hashtag backfired in the end, because people kept describing the bad personal experiences about McDonald’s (Zimmer, 2013). For those brand marketers who pursue the desirable marketing outcomes on Twitter, thus, it is very important to keep some thumb rules of the truly effective hashtag in mind, which is to also the main purpose of this study.
Although it is confirmed that the hashtag can be used as an effective tool for brands to conduct the marketing activities, less is known about what type of hashtag can be called an “effective hashtag” and ensure its desired marketing outcomes (e.g. Yang et al. 2012; Martin, 2013). This study, thus, will examine how hashtag characteristics influence its effectiveness from the brand’s perspective. On the one hand, the effectiveness of the hashtag might be explained by the perceived fit between the hashtag and the brand. In this study, the relevance is evaluated by the level of fit between the hashtag relevance and the brand image (i.e. brand associations). With regard to the effect of relevance, there are two theories supporting contradicting views. One is congruency theory claiming that a high fit between two objects could be more impressive and cause more favorable outcomes than a low fit (Cornwell et al. 2005; Brian and Busler, 2000). In contrast, the scheme theory holds opposite opinion positing that the incongruent information will cause more curiosities and result in greater recall than congruent information (Hastie, 1980). It should be noted that when scholars study the effect of relevance in the marketing context, most of them tend to
analyze the effect for familiar brand and unfamiliar brand respectively (e.g. Törn, 2012; Lee and Thorson, 2008). Given that the brand familiarity moderates the impact of relevance, this study also suggests that the brand familiarity has moderating effect on hashtag relevance and in turn, influences the effectiveness of the hashtag.
On the other hand, the effectiveness of the hashtag might also be explained by the hashtag length (i.e. number of words). There are also two opposite views for the optimal length of hashtag. Some practitioners take the view that shorter is better because of the decreasing trend of people’s attention spans (e.g. LePage, 2014). In contrast, other scholars argue that the somewhat longer hashtag is effective because it can contain more detailed information, which helps people to well understand (Singh et al. 2000; Bunting, 2012). The same as relevance effect, it is predicted that brand familiarity can also moderate the length effect because for the relatively unfamiliar topics the short and brief message is likely to be unreadable and confused, while the longer messages enable to provide detailed information through reiteration of key points, which will improve the message comprehension and increase the accuracy of cue recognition (Singh et al. 2000; Deborah et al. 1991). Therefore, this study is conducted to examine how hashtag relevance and length influence the user engagement, and whether the brand familiarity will be the moderator for these effects.
Overall, this study contributes to the current literature in the following ways. First, previous researches on hashtag have focused on its impact (i.e. on user engagement; Martin, 2013; Suh et al. 2010). So far, however, it has been unclear what type of hashtag is effective to increase the user engagement. This article demonstrates
the effects of characteristics of hashtags (i.e. relevance and length) on its effectiveness and clarifies the possible moderator that influences these effects. Second, this study is of practical value for brands. The findings will give brands insight about how to design and use the hashtag effectively to draw public attention and increase the user engagement, and in turn to reap benefits of the online conversation on Twitter.
2. Literature Review
This chapter provides a comprehensive review of the literature about the key concepts in this study to analyze what has already been studied about the effects of hashtag relevance and length on its effectiveness. First, some basic information about hashtag is given. Next, the previous studies about the main effects of relevance and length are reviewed. Then, the related studies about the interaction effect of relevance and length are also reviewed. Furthermore, the detailed analysis of the moderating role of the brand familiarity on both relevance and length effects is provided. Based on this review, hypotheses and a conceptual framework are developed and will be tested in the following chapters.
2.1 Hashtags and its effectiveness
Hashtag, a user-composed keyword preceded by a hash (#), is widely adopted by users to facilitate their navigation in this deluge of information on Twitter (Lu and Lee, 2015). In general, the hashtag plays a dual role. On one hand, the hashtag serves as a bookmark or headline of content, which links the tweet with a similar topic. On the other hand, the hashtag serves as the symbol of a community membership, which bridges a virtual community of users (Yang et al. 2012). Many scholars have proven
that readers and bloggers both can benefit from the use of hashtags, but this study will only discuss the effectiveness of the hashtag from the perspective of brands (i.e. bloggers). For brands, hashtags relate a message to a topic, which can promote conversations among their consumers, improve searches for related information and enable the recognition of events and trends (Lu and Lee, 2015). It is validated that there is a positive relationship between the use of hashtags and the number of followers, which means tweets containing hashtags can produce a higher increase in the number of followers than tweets without hashtag (Martin, 2013). Moreover, the use of hashtags can also increase the number of retweetings for the tagged tweets (Suh et al. 2010).
The industrial data from RadiumOne reporting that about half of the respondents claimed they would click on a hashtag to learn more about a brand or product, and 35 percent of them mainly use hashtags to search or follow categories and brands of personal interest (Gesenhues, 2013). Therefore, brands can take advantage of this trend and largely benefit from using hashtags appropriately. Kitkat, as a good example, did a good job with the #HaveABreak hashtag since this hashtag enhanced the user engagement successfully. More specifically, consumers are motivated to share their photos with the #HaveABreak community and Kitkat gets lots of authentic photos of people enjoying their product (Bunskoek, 2014). Therefore, it is clear that the hashtag can serve as an effective marketing tool when the brand uses it properly and the effectiveness of the hashtag can be proved by the user engagement.
definition of engagement on Twitter should be introduced. Lagun and Lalmas (2016) have defined the user engagement as the psychological and behavioral connections existing between a user and a resource. In this study, the psychological engagement is reflected by three dimensions, namely focused attention, curiosity, and intrinsic interest (Webster and Ho, 1997). Moreover, the behavioral engagement is reflected by people’s usual practices on Twitter, which include clicking “like”, retweeting and re-embedding other people’s hashtag in own tweets (Suh et al. 2010).
Indeed, as mentioned, the hashtag enables to create and increase the user engagement, but sometimes it is still ignored by the public when the hashtag is used ineffectively. For the ineffective hashtag, LePage (2014) provides an example that even though the #NewYorkCronutLovers may help the brand to attract a very specific consumers, no one will be expected to notice and use this hashtag because it contains too many characters. Therefore, it is important for brands to identify the effective hashtags that can really increase the user engagement. However, what type of hashtags is effective and how hashtag characteristics contribute to its effectiveness have remained underexposed. This study thus aims to examine how hashtag characteristics affect its effectiveness. Although the hashtag consists of several characteristics, the relevance and length are mentioned most by practitioners (e.g. LePage, 2014; Calero, 2015). Generally, they share a common view that the hashtag should be short and relevant to the brand. But whether the short and relevant hashtag can necessarily ensure its effectiveness will be dialectically discussed in this study. In the following subchapters the previous studies regarding the effects of relevance and
length will be reviewed respectively.
2.2 Relevance of hashtags and user engagement
The definition of perceived relevance is to what extent two concepts fit or similar with each other (Till and Busler, 2000). Frist, for the evaluation of the level of fit, most previous researches on endorser’s match-up effect used to focus on the physical attractiveness or expertise of the endorser (e.g. Kahle and Homer, 1985; Ohanian 1991). However, Törn (2012) provides a better demonstration who evaluates the level of fit between brand and endorser from a more holistic view of brand image using brand associations, showing that match-up studies need not be restricted to specific aspects of endorsement image such as attractiveness or expertise. In this article, similarly, the hashtag relevance is defined as the fitness or similarity between the hashtag and the brand image reflected by the brand associations held in consumer memory.
Second, with the regard to the effect of relevance, there are two opposite theories existing in the academic area (i.e. Congruence theory vs. Schema congruity theory). Congruence theory suggests that the retrieval and recall of information are influenced by relatedness or similarity between two objects. The congruent information is more easily understood and remembered (Cornwell et al. 2005). In contrast, the schema congruity theory claims that incongruent information requires more elaborated processing and results in greater recall than the congruent information (Hastie, 1980). According to these competing views concerning with the relevance effect, it seems plausible to claim that the hashtag relevance can also influence the user engagement,
thus, I hypothesize that:
H1. The relevance of hashtag has an effect on user engagement.
Many scholars have conducted their studies on the basis of the congruence theory. The match-up effect claims that endorsers are more effective when there is a high "fit" between the endorser and the endorsed product (Till and Busler, 2000). Higher the degree of such fit, the greater the likelihood of audience acceptance (Chakraborty et al. 2000). Moreover, for the event-sponsor fit, some researchers also suggest that the high fit between the event and sponsor will ensure the memory for sponsorship stimuli as well as facilitate more positive sponsorship outcomes (Cornwell et al. 2005; Becker-Olsen and Hill, 2006). LePage (2014) also holds the similar view that if an effective hashtag is embedded in a tweet representing the brand’s image exactly, then the hashtag can be easily understood and accepted by most customers. Herschel that mainly manipulates backpacks and travel goods has created a congruent hashtag called #welltravelled and encouraged people to share beautiful travel photos featuring the products. Seeing photos from the public on the official Herschel account quickly prompted more of followers to embed the hashtag and share their own Herschel photos (LePage, 2014). Considering form this angle, it can also be argued that a high fit between the hashtag and the brand image (i.e. brand associations) will make the hashtag more effective to increase the user engagement. I therefore hypothesize that:
H1a. In general, when the level of fit between the hashtag and brand image is
In contrast, turning to the schema theory many researchers have focused on the effect of incongruity (i.e. low fit) in different contexts (e.g. Goodstein, 1993; Hastie, 1980; Törn, 2012). In the marketing context, particularly, studies have been
conducted on incongruities between pictures and words in ads, between an ad and general viewer expectations, between features of a product and product category schemas (Lee and Schumann 2004). For example, it is shown that the perceived low fit between brand and celebrity endorsers will generate greater endorsement outcomes than the high fit endorsement. Because when consumers can easily anticipate what the advertising will entail for a brand, their curiosities and ad’s communication effects will be reduced (Törn, 2012). To put it another way, if the information is similar to an existing schema or conforms to expectations, people will interpret it effortlessly. But when information does not fit the schema, people will feel uncomfortable due to the resulted tension then they will try to relieve it (Mandler, 1982). The incongruent information, thus, will cause people to pay more attention and make them more motivated to think about the information so that incur deeper cognition and memory than traditional information (Fiske et al. 1983). Marketers are suggested to take the advantage of incongruity to design the advertisements in order to arouse people’s curiosities and capture their attention (Lee and Schumann, 2004). Similarly, it can be argued from this perspective that brands can benefit from the incongruent hashtags.
H1b. In general, when the level of fit between the hashtag and brand image is low,
2.3. Length of hashtags and user engagement
In addition to the hashtag relevance, the length of the hashtag will also be studied in this article. Firstly, the hashtag length in this study is evaluated by the number of words contained in the hashtag, which is developed by Tsur and Rappoport (2012). More specifically, the short length is defined as a hashtag consists of one word, the moderate hashtag consists of two or three words, and the long hashtag consists of more than three words. Tsur and Rappoport (2012) also claim that long hashtags are more difficult to type and interpret because there is a sequence of words without white spaces for each hashtag. Even though less researches on hashtag length, there is a lay belief that people are more willing to engage with short hashtags (LePage, 2014). A recent Microsoft study might provide evidence for this lay belief, which reports that online readers have shorter and shorter attention spans and do not want to read and pay attention to the long messages (Khan, 2015). One of the human behavior facts shows that people prefer conciseness and simplicity (Calero, 2015). Based on these views, it might be argued that brands do not try to merge too many words in a hashtag, because the longer the hashtag the more complex it would be so that the user engagement will be reduced. Moreover, alternative explanation for people do not want to engage with the long hashtag is that the long hashtag will take away too much space from main content or link in each tweet under the 140-character per tweet limitation (Tsur and Rappoport, 2012). Considering from this perspective, maybe it can be argued that the short hashtag is more effective.
always (e.g. Bunting, 2012) so the long-form blog post is recommended for many reasons. First, the long message enables the sender to provide detailed information through reiteration of key points so that improve the message comprehension (Singh et al. 2000). Second, such long message including detailed explanation will prime the information in reader’s working memory, thereby increasing the speed and accuracy of cue recognition in subsequent presentations (Feustel et al. 1983). Furthermore, it is found that word skipping is negatively related to word length during reading, which means as the length of a given word increases, the probability that it will be skipped decreases (Rayner et al. 2011). Taking the advertising headline as an example, there is a suggestion that “a headline should be as long as it has to be to get the job done. Brevity is desirable, but not at the expense of clarity or persuasiveness” (Wesson, 1989, p.466). Turning to the hashtag, similarly, the view seems plausible that marketers do not avoid using the longer hashtag because it enables them to provide more detailed information for readers. Based on these two contradicting views, it cannot be asserted that the short hashtag is absolutely more effective than the long hashtag, or vice versa, but the effect of hashtag length seems to be confirmed, I therefore hypothesize that:
H2. The length of hashtag has an effect on user engagement.
In order to cope with this confusion on the length effect, Bennett (2013) conducts an experiment on click-through data of over 100,000 English headlines for around 8 months, and the result shows that moderate length performs best on the readership and the communication outcomes. Because engagement will decline when
the headline approaches either the shorter or longer end of the spectrum (Bennett, 2013). Similarly, the hashtag serving as a topic or a headline for the tagged tweet (Yang et al. 2012), thus, a U-shape relationship for the effect of hashtag length is developed:
H2a. The moderate length of hashtag is more effective to increase the user
engagement than too short or too long hashtag.
2.4 Moderating role of Relevance on Length
Importantly, the effect of hashtag length may be moderated by the perceived relevance, which means there is an interaction effect of relevance and length on user engagement. As mentioned above, schema theory confirms that the incongruent information can encourage people to pay more attention and make them more motivated to think about the information than traditional information (Fiske et al. 1983; Hastie, 1980). Scholars also highlight the role of articulation, which is defined as “the act of explaining the relationship between entities to support the development of meaning in the mind of the individual” (Cornwell et al. 2006, p.312). Specifically, the articulation of sponsorship fit can enhance the public memory for an incongruent sponsor-event pairing (Cornwell et al., 2006). Based on these findings, if there is perceived low fit between entities, then providing the somewhat detailed explanation for the fit or creating a meaningful context for it will ensure the effectiveness of incongruence (Cornwell et al., 2006). Furthermore, it is found that the sentential context enables the paring of two unrelated words to be learned and recalled more effectively (Prior and Bentin, 2003). The long-form message, thus, is generated when
people link the irrelevant words in a detailed context to articulate the reasons of the incongruent relationship. Similarly, for the hashtag, it can be argued that if brands have to tweet with the irrelevant hashtag in order to achieve certain communication objectives then the hashtag should contain more number of precise words to enrich the hashtag and make it perceived well-understood. That way, the more number of words are used to explain the fit, the longer the hashtag will be. In contrast, if there is perceived high fit between the hashtag and brand, then the detailed explanation is redundant, which means a short and brief hashtag is more effective. I therefore hypothesize that:
H3. For incongruent hashtag the effect of long hashtag on user engagement will be
greater than short hashtag, while for congruent hashtag the effect of short
hashtag will be greater than long hashtag.
2.5 Brand familiarity and User Engagement
When it comes to the relevance and length effect in the marketing context, many scholars have taken brand familiarity as a moderator to completely argue their studies. For example, studies confirm the moderating role of brand familiarity in consumers’ attitudes towards brands placed in movies (Verhellen et al. 2015), the effect of ad-brand incongruence (Lange and Dahlen, 2003), the effect of brand-incongruent celebrity endorsements (Törn, 2012), etc. There is sufficient evidence in the literature showing that familiar brands would have a communicative advantage over unfamiliar brands (Lange and Dahlen, 2003). Brand familiarity primes consumers’ brand knowledge structures, which forms the brand associations in a consumer’s memory
(Campbell and Keller, 2003). For a familiar brand, customers already store multiple associations about the brand in terms of brand’s core business, product lines, and the brand image. In contrast, consumers lack these associations for the unfamiliar brands (Low and Lamb, 2000). Moreover, brand familiarity will also influence the information storage (Kent and Allen 1994) and the information retrieval (Heckler and Childers 1992). When consumers are exposed to a congruent ad for a familiar brand, the ad will be easy to store in the existing schema of the brand, concurrently, the information will be easily retrieved from the schema (Lange and Dahlen, 2003). Based on these findings, I firstly predict that brand familiarity will moderate the hashtag relevance and length effects:
H4. Both effects of hashtag relevance and length on user engagement will be
moderated by brand familiarity.
For the relevance effect numerous literatures have categorized the research subjects (i.e. brands) into established brands (familiar) and new brands (unfamiliar) when discuss the effect of congruence (e.g. Lange and Dahlen, 2003; Lee et al. 2008; Törn, 2012), and all findings prove the moderating role of the brand familiarity. Specifically, the study conducted by Lange and Dahlen (2003) shows that ad-brand incongruence enhances brand attitude and brand memorability for the familiar brand, because the incongruence reinforces the existing associations for a familiar brand by increasing elaboration of the brand message. However, the congruence cannot play the similar role in familiar brands. For example, selecting famous athlete (congruent) for Nike that already has high brand awareness and clear associations may not be very
attention grabbing for most consumers, because customers can easily predict what its ad will show. Therefore, people’s curiosity and interest in the brand and the communication outcomes will be reduced (Törn, 2012). Similarly, it can be argued that irrelevant hashtag will be more effective for familiar brands (H4a). In contrast, customers do not have a sophisticated and well-established schema about unfamiliar brands, which makes it harder to retrieve the brand-related information (Lange and Dahlen, 2003). A congruent context, thus, can ease comprehension of the advertising (Gupta et al., 1994). That means for the new and unfamiliar brands, the ad’s main task is to strengthen the brand awareness, so the ad should be congruent with the brand rather than incongruent with the brand (Lange and Dahlen, 2003). Considering from this angle, for hashtag, the unfamiliar brand should tweet with the high-fit hashtag since it may be helpful to understand and receive the tweet and then customers will be more likely to engage with such hashtag (H4b).
H4a. For familiar brands, the incongruent hashtag is more effective to increase
user engagement than the congruent hashtag.
H4b. For unfamiliar brands, the congruent hashtag is more effective to increase
user engagement than the incongruent hashtag.
Turning to the hashtag length, the moderator of brand familiarity may also work. As mentioned, online readers have shorter and shorter attention spans they do not want to read long message. In addition, consumers tend to have a good understanding of the familiar brands (Campbell and Keller, 2003), so that the short and brief hashtags can meet the decreasing attention span and easily capture customers’
attention (Bunting, 2012). I thus hypothesize that the short hashtag is more effective for the familiar brands (H4c). However, for the unfamiliar brands, the prediction may be different. Customers know little about a new and strange brand and do not have clear association about the brand (Low and Lamb, 2000). It is shown that when customers are exposed to an ad for an unfamiliar brand, they mainly aim at learning about this new brand and form an accurate set of associations for the brand (Hilton et al. 1991). Considering from this perspective, it can be argued that the somewhat long and detailed hashtag enables customers to learn and form the associations of the new brands. In this case, using the somewhat long hashtag with more detailed information may be more effective for the unfamiliar brands (H4d).
H4c. For familiar brands, the short hashtag is more effective to increase user engagement than the long hashtag.
H4d. For unfamiliar brands, the long hashtag is more effective to increase user
engagement than the short hashtag.
2.6 Conceptual framework
Figure 1 is an overview of the conceptual framework of this study, which is developed to test the hypotheses, to analyze to what extent hashtag relevance and length influence the hashtag effectiveness, besides, to test the moderating roles of relevance and brand familiarity on this relationship.
Figure 1. Conceptual framework
3. Methodology
Overall, the data for this study is collected by an online survey and the main reasons for choosing this method will be explained below. The survey has been created on www.qualtrics.com and distributed via e-mail, and social medias. For the data collection, a pilot study will be conducted at small scales first to pre-test the research design and ensure all manipulations are working as expected. After the pilot study, the main study will be done to test the proposed hypotheses of this study. The following subchapters describe the experiment and how the data will be collected in detail.
Hashtag Effectiveness
Hashtag Relevance
The “fit” between hashtag and the brand
image
Hashtag Length
The number of words in a hashtag Brand familiarity H1&H1a&H1b H2&H2a H3 H4a H4b H4c H4d H4
3.1 Sample
It is reported that the Twitter users in Netherlands accounts for approximately 2.6 million people (Pieters, 2016). Besides, the most active users are young people under the age of thirty (CBS, 2013), specifically, 98 percent of Dutch Twitter users were between 18 and 25 years old, and 87 percent among 25 to 35-year-old as well (CBS, 2013). Therefore, the population of this survey will focus on Twitter users in
Netherlands between the ages of 18 and 35. Non-probability sampling will be used to select the sample for this survey. In order to make sure the chosen sample can
represent the population precisely, the minimum sample size of this survey will consist of data from at least 240 Twitter users in Netherlands, that is, at least 20 participants for per experimental condition (Saunders and Lewis, 2012).
On May 20, 2016 the experimental survey was distributed and it was deactivated on May 30, 2016. In total 352 questionnaires have been filled in, however, 97 of these have been excluded because of some non-answered questions. After excluding these 97 unqualified data, the remaining 255 are used to analyze. Among these 255
participants, 90 are male (35.3%) and 165 are female (64.7%). Most of the participants are between the ages 18 and 24, accounting for 68.2% of the total respondents.
3.2 Experiment design
This study mainly investigates the main effects of hashtag relevance and length on user engagement, the interaction effect of relevance and length as well, and the moderating effect of brand familiarity. To test all proposed hypotheses, an online
experiment will be conducted by the manner of web-based questionnaire. There are several reasons for choosing online experiment. First, the Internet allows me to access to a large population with diverse backgrounds around the world. Therefore, the data can be collected faster via the Internet. Second, online survey is a money-saving method to collect from the target population. Moreover, the respondents can use their own devices in their familiar place, which will increase the external validity of the study (Reips, 2000).
Additionally, the proposed hypotheses have been tested in an experimental survey using a 2 (relevance) × 3 (length) × 2 (familiarity) between-subjects Latin square design (Table 1). In total, therefore, there are twelve treatments (Appendix I) for the experiment, and for each treatment there is a survey provided. In each treatment, two independent variables and moderator are manipulated in different levels.
Table 1. 2 × 3× 2 between-subjects Design
Familiar brand Unfamiliar brand High fit Low fit High fit Low fit Short
Moderate Long
Treatment1 Treatment4 Treatment7 Treatment10
Treatment8 Treatment11 Treatment2 Treatment5
Treatment3 Treatment6 Treatment9 Treatment12
3.3 Stimuli development and Procedure
and the last parts will ask respondents some personal information, such as the age, gender, habits, etc. For the second part, one mock brand-initiated tweet with a certain hashtag will be shown to respondents first, followed by several sets of questions about their attitude towards the presented tweet. As mentioned before, there are twelve treatments for this study, which means twelve self-created tweets are needed. The reason why the tweets would be created by myself is because in order to test the pure hashtag effect, other possible factors such as the tweet content need to be controlled. Thus, for all self-created tweets the tweet contents are constant, but differ in the levels of hashtag length and relevance as well as the brand familiarity.
Before the introduction of stimuli design, it should be noted that this experiment would focus on the coffee market. Because marketers consider the coffee market is highly profitable and the social media is a fertile brewing ground for its competition (Stadd, 2013). Therefore, the coffee market is worthy to be focused on for my study, which also means the involved brand accounts and the stimuli design in this
experiment are related to the coffee industry. Next, for the selection of two different brands in terms of familiarity, I will follow the idea of Warren and Campbell (2014), which means ‘Starbucks coffee’ is selected as the familiar brand and ‘Sabbarrio coffee’, a fictitious brand, is selected as the unfamiliar brand. Turning to the stimuli design, this study needs two sets of hashtags (high fit vs. low fit), and each set of hashtag consists of six different hashtags in terms of length for two coffee brands respectively. The designs of hashtag relevance and length will be introduced below.
the hashtag and the brand image of the involved brand. Thus, two sets of hashtags with different level of fit will be chosen according to the general brand image of a café. As a result, the hashtags are searched from Hashtagify.me, which is a Twitter Hashtags search engine through which the widely used hashtags that are highly related to the searched topic can be found easily. Firstly, for the selection of the high-fit hashtag, when the “café” topic is searched the data indicates that most popular hashtag is “#Coffee” with the highest correlation from the “café” topic. Therefore, the #Coffee is chosen as the relevant hashtag for this study. On the contrary, “#Jogging” is one of the most popular hashtags related to the “sport” topic, which is perceived irrelevant to a coffee retailer, so it is chosen as the low-fit hashtag for this study.
Furthermore, with the regard to the evaluation of hashtag length, the length can be evaluated by the number of words contained in a hashtag, which is developed by Tsur and Rappoport, (2012). More specially, a hashtag containing one word is
perceived as a short hashtag; a hashtag containing two or three words is perceived as a moderate hashtag; and a hashtag containing more than three words is perceived as a long hashtag. In order to test the main effect of hashtag length and the interaction effect of relevance and length, thus, #Coffee (relevant) and #Jogging (irrelevant) will continue to be used as the short hashtags because each of them contains just one word. And then the moderate and long hashtags are lengthened based on these two original hashtags, specifically, the moderate hashtags containing two or three words are #MorningCoffee (relevant) and #MorningJogging (irrelevant); the long hashtags
containing more than three words are #CoffeeToRefreshTheMorning (relevant) and #JoggingToRefreshTheMorning (irrelevant). Six different hashtags are summarized in Table 2.
Table 2. Stimuli design
Relevant Irrelevant
Short (1 word) #Coffee #Jogging
Moderate (2-3 words) #MorningCoffee #MorningJogging
Long (>3 words) # CoffeeToRefreshTheMorning # JoggingToRefreshTheMorning
These hashtags are respectively embedded in one mock tweet sharing the same content. To begin with, participant will be asked about whether or not has Twitter account, and whether follows any brands on Twitter. In order to ensure all data are effective and representative, thus, if a participant choose “No” for these two questions, then his/her the questionnaire will end. After answering these filter-used questions, those “qualified” participants will be shown a mock tweet from a coffee brand (i.e. either Starbucks or Sabbarrio) and the tweet will include a manipulated hashtag differing in levels of relevance and length. It should be noted that each manipulated hashtag would be assigned to participants randomly. And then they will be asked how they would respond to such a tweet. In order to double-confirm the perceived
relevance and length of these manipulated hashtags work as intended, therefore, after reading the tweet, participants are first presented two sets of questions to test their perceived relevance and length for the shown hashtags.
In addition, beyond the manipulated variables, sometimes there are also some potential variables that can influence the dependent variable but they are not part of
the main experimental manipulation (Field, 2013). For this study, with the regard to the reasons why people engagement with brands on Twitter, Cargill (2013) takes the view that people on Twitter always look for the enjoyment, so they are most likely to engage with those tweets that they are interested in. Sometimes people are also willing to engage with their favorite brands just for the purpose of being loyal. As well as that some people engage with the brands just because they are used to interact with brands. Therefore, three variables are identified and should be controlled during analysis, namely coffee-drinking/jogging habits, brand preference, and daily Twitter behavior.
3.4 Measures
For the dependent variable, in order to do a comprehensive analysis of the user engagement on Twitter, user engagement will be tested from two perspectives: psychological engagement and behavioral engagement (Attfield el al., 2011). On the one hand, the psychological engagement includes three dimensions, namely focused attention, curiosity and intrinsic interest (Webster and Ho, 1997). On the other hand, the behavioral engagement consists of online behavior (Attfield el al., 2011), which is reflected by people’s usual actions on Twitter: clicking “like”, retweeting and using other people’s hashtags in their own tweets to join the conversation (Suh et al. 2010). The detailed questions for these four points:
Focused attention is measured using three scale items, which are taken from Webster and Ho (1997): 1. “ When exposed to this tweet, this hashtag holds my attention” 2. “ This hashtag keeps me totally absorbed in the tweet” 3. “Compared to
how long I normally look at a tweet, this hashtag leads me to look at this tweet for a longer time”. The answers are given on a 5-point Likert-type scale ranging from: “Strongly disagree” (1) to “Strongly agree”(5).
Curiosity is measured by two scale items: “This hashtag makes me curious about the tweet” and “This hashtag arouses my imagination for the tweet” (cf. Webster and Ho, 1997). The answers are given on a 5-point Likert-type scale ranging from: “Strongly disagree” (1) to “Strongly agree”(5).
Intrinsic interest is measured by two scale items: “ This hashtag is attractive for
me” and “ I like this hashtag” (cf. Webster and Ho, 1997). The answers are given on a 5-point Likert-type scale ranging from: “Strongly disagree” (1) to “Strongly
agree”(5).
Online behavior is measured by three questions: 1. “When you exposed to this hashtag, how likely are you to click “like” for the tweet?” 2. “When exposed to this hashtag, how likely are you to retweet the tweet?” 3. “How likely are you to embed this hashtag in your own post?” The answers are given on a 5-point Likert-type scale raging from: “Extremely unlikely” (1) to “Extremely likely” (5).
There are two independent variables, namely hashtag relevance and hashtag length. As mentioned, for those hashtags in the experiment the perceived relevance differs in two levels and the length differs in three levels. Before asking respondents about their attitude (i.e. engagement) towards the shown hashtag, they will be asked to indicate the perceived relevance and length for the hashtag. Firstly, respondents are asked to evaluate the perceived hashtag-brand fit of the shown hashtag by answering
three questions: 1. “The brand and the hashtag fit together well” 2. “The image of the brand and the image of the hashtag are similar” 3. “There is a logical connection between the brand and the hashtag”. This measure is taken from (Speed and Thompson, 2000) and the answers are given on 5-point Likert-type scale from
“Extremely unlikely” (1) to “Extremely likely” (5). Next, respondents will be asked to evaluate the perceived length of the hashtag, only one manipulation question is used (cf. Banker et al., 2006): "Please evaluate your perceived length of this hashtag based on the number of word”, the answer is given on 5-point Likert-type scale: “Very short”, “Short”, “Moderate”, “Long” and “Very long”.
Brand familiarity, the moderating variable, will be measured by two scale items, asking respondents to indicate their perceived familiarity of the involved brand (i.e. Starbucks or Sabbarrio): the first one is “Overall, I am familiar with this brand”. The second one is “I am familiar with the process of purchasing at its coffee stores.” All the answers are also given on a 5-point Likert-type scale from “Strongly disagree” (1) to “Strongly agree”(5).
In addition, for this study, there are three control variables should be included in the data analysis that are brand preference, coffee or jogging habits and the people’s daily Twitter behavior. Firstly for the brand preference, respondents will be asked by two questions to indicate their perceived preference for the presented brand (cf. Sen et al. 2001), one is “How much would you say you like or dislike this brand (Starbucks/ Sabbarrio)?” another is “When you go to a cafe, to what extent do you go to
“Strongly disagree” (1) to “Strongly agree”(5). Next, turning to the coffee
drinking/jogging habits, respondents should also answer two questions (cf. Mogilner et al. 2008): “How often do you drink coffee/ jog?” with the answers given from “Never” (1) to “Always” (5). And “To what extent do you think "a coffee/ jogging lover" can describe you?” the answers are given from “Does not describe me” (1) to “Describes me extremely well” (5). Moreover, to identify respondents’ daily Twitter behavior, they will be asked, “In general, how often do you interact with brands on Twitter (e.g. reply, retweet, or click "like")?” the answer are also given on a 5-point Likert-type scale from “Never” (1) to “Always” (5).
Furthermore, some demographic information regarding age, gender will be included in the survey. And respondents will be asked about their general perception about hashtag such as, “Do you think the hashtag is useful on Twitter?” with the answers given from “Not at all useful” (1) to “Extremely useful” (5), if “Not at all useful” is selected, then the survey will end. However, for those respondents who consider the hashtag is useful to different extent, the survey will be continued and asks them “What do you think is the most useful function of the hashtag?” and the answers are four different functions mentioned by Yang et al. (2012). And besides, respondents will be asked “Have you used hashtags for your own posts?” if they select “No”, then the survey will end. For those respondents selecting “Yes”, they will be asked, “In general, how many hashtags do you use for your each tweet?” the answers are “Only one”, “Two”, and “More than three (including three)” (see Appendix II for the complete questionnaire).
3.5 Data analysis
Before the main study, a pilot study at small scale will be conducted first because it enables to pre-test whether all manipulations and measurements are working as intended and to double check all questions are clear without understanding from participants (Van Teijlingen and Hundley, 2002). There are twelve experimental conditions for this study, for each condition I need two respondents to do the pretest because the questionnaire pretest requires at least 20 respondents (Czaja, 1998). The data for the pilot study were collected via both online and offline means. Specifically, ten UvA students engaged in the offline study, through which I can talk with them face to face and see whether the questions are clear, get the feedbacks regarding the survey from them immediately as well. The remaining fourteen questionnaires were distributed by email, and in total, twenty-four questionnaires are collected from May 18 to 20, 2016. After ensuring the measurements and manipulations are effective via the pretest, the survey will be distributed to a larger extent to collect data for the main study.
The data analyses, overall, will be conducted with these steps via SPSS 23 for Mac: manipulation check, descriptive analysis and hypothesis testing. For
manipulation check, the reliability will be run first to see whether the main
measurements of the study are reliable and internally consistent. And then one sample t-test will be run for each manipulation to ensure they work as intended. Before
hypotheses testing, some descriptive statistics will be provided to introduce the overview of the sample and the results. For hypothesis testing, each hypothesized
effect in this study will be tested by ANCOVA separately, whilst controlling for some variables namely brand preference; coffee-drinking/ jogging habits and Twitter behavior. If the result reveals the main effects are significant, then the LSD post-hoc test is conducted to further probe the effect. This analyzing method is also used for the interaction effect of hashtag relevance and length on engagement, and the interaction effect of brand familiarity as well. Furthermore, after the ANCOVA analyses for the overall engagement, the MANCOVA will be conducted to have a closer look at the effects on individual dimensions of engagement.
4. Results and Analysis
4.1 Reliability and the manipulation check
To verify that the manipulated hashtags and brand familiarity were perceived by participants as intended, the manipulation check was performed. Firstly, reliability enables to examine the consistency of measurements (Kimberlin and Winterstein, 2008). Thus, for this study, the reliability checks were run for all continuous variables, namely familiarity, relevance, and engagement including four dimensions, brand preference and coffee-drinking/ jogging habits as well.The Cronbach’s alpha, which represents the estimator of the internal consistency, has been tested to verify if all the items in one scale measure the same, or if some questions should not be used for analysis (Kimberlin and Winterstein, 2008). As exhibited in table 3, all variables have a Cronbach’s alpha > 0.7, which indicates high level of internal consistency.
Table 3. Cranach’s alpha
Variable N of items Cronbach’s Alpha Familiarity Relevance Preference Habits Attention Curiosity Interest Action Engagement 2 3 2 2 3 2 2 3 10 .921 .945 .811 .826 .742 .740 .733 .885 .878
Next, turning to the manipulation checks, one sample t-test is run to check the manipulations respectively, namely familiarity, relevance and length (Table 4). Because one sample t-test enables to determine “whether there is sufficient evidence to conclude that the mean of the population from which the sample is taken is different from the specified value” (Elliott and Woodward, 2007, p48).
First, for the manipulated brand familiarity there are two questions requiring respondents to indicate whether they perceive that the presented brand is familiar on a 5-point scale, raging from 1 (Strongly disagree) to 5 (Strongly agree). The result shows that people in the familiar brand condition (i.e. Starbucks Coffee) rated the brand significantly familiar (M = 4.125, SD = 0.607, t (11) = 6.413, p < .05). In contrast, the Sabbarrio Coffee (M= 1.625, SD=0.607) is perceived significantly unfamiliar, t (11) = -7.838, p < .05. Thus, the manipulated brand familiarity works as intended.
Second, respondents are asked to evaluate the perceived length of the hashtag as “Very long”, “Long”, “Moderate”, “Short” or “Very short”. The result shows that the
perceived length is significantly short (M=1.75, SD= 0.707) for the short hashtag condition, t (7)= -5.00, p < .05, and besides, the perceived length is significantly long (M= 4.62, SD= 0.517) for the long hashtag condition, t (7)= 8.881, p < .05. Moreover, for the moderate hashtag condition, the mean hashtag length (M = 2.625, SD = 0.916) was not significantly different from the hypothesized value of 3 (3 = Moderate), t (7) = –1.158, p = 0.285. Thus, all lengths have been effectively manipulated.
Lastly, respondents are required to indicate whether they perceive the shown hashtag is relevant to the brand on a 5-point scale ranging from 1 (Strongly disagree) to 5 (Strongly agree). The result shows that the perceived fit between the hashtag and brand is significantly high (M= 4.194, SD = 0.264) for the high-fit hashtag condition, t (11) = 15.654, p < .05, and the perceived fit is significantly low (M=1.638,
SD=0.558) for the low-fit hashtag condition, t (11)= -8.437, p < .05. Therefore, this variable has been effectively manipulated.
Table 4. Results of One-sample t-test
Variable N Mean Std. Deviation t df p Familiar Unfamiliar Short Moderate Long High fit Low fit 12 12 8 8 8 12 12 4.125 1.625 1.750 2.625 4.625 4.194 1.638 .607 .607 .707 .916 .517 .264 .558 6.413 -7.838 -5.00 -1.158 8.881 15.654 -8.437 11 11 7 7 7 11 11 .000 .000 .002 .285 .000 .000 .000
Note: Significant at the p< 0.05 level
4.2 Descriptive analysis
be calculated before the hypotheses testing. Firstly, it is important that readers have a good understanding of the sample on which the study was focused, so the frequency table (Table 5) is presented for the demographic statistics of the sample (Gender, age, twitter behavior and daily Twitter time). As shown, the sample for this study consists of 255 participants in total. Specifically, 165 participants are women accounting for 64.7%, and the others are men. The majority of the participants are between the ages 18 and 24 with 68.3%, and the least of them are over 35 with 0.8%. Turning to the Twitter behavior, when participants were asked about how often do they interact with brands on Twitter, around half of the participants (48.6%) considered they sometimes interact with brands, and only 15 participants never interact with brands on Twitter accounting for 5.9%. Furthermore, the daily Twitter time shows that the most participants (81.6%) tend to spend less than 2 hours on Twitter per day, and only 3 participants from 255 spends more than 6 hours per day on Twitter with 1.2%.
Table 5. Demographic statistics
Variable Classification Frequency Percentage
Gender Male Female 90 165 35.3 64.7 Age Under 18 18-24 25-34 35-44 20 174 59 2 7.8 68.3 23.1 0.8 Twitter behavior Never Rarely Sometimes Often Always 15 59 124 46 11 5.9 23.1 48.6 18.1 4.3
Daily Twitter time (per day)
0-2 hours 3-6 hours More than 6 hours
208 44 3 81.6 17.2 1.2 Total 255 100.0
Next, the descriptive statistics of the dependent variable with different dimensions are provided for per condition (see Table 6 for the detailed statistics). It is shown that for the familiar brand, the highest level of overall engagement is found in the condition of incongruent and moderate length hashtag (M = 3.35, SD = .93). Beyond the overall engagement, a closer look at each dimension shows that the level of people’s curiosity is the highest among four dimensions (M = 3.50, SD = .98). The levels of interest and behavioral engagement are relatively lower (M = 3.48, SD = .97; M = 3.44, SD = 1.10 respectively). However, in this condition the level of people’s focused attention is the lowest (M = 3.07, SD = 1.01). In contrast, for the familiar brand the lowest level of overall engagement is found in the condition of incongruent and long hashtag (M = 2.88, SD = .54). In particular, in this condition the behavioral engagement has the lowest level (M = 2.57, SD = .73), meaning people are less willing to behaviorally engage with the incongruent and long hashtag initiated by the familiar brand. Furthermore, for the unfamiliar brand the highest level of overall engagement is found in the condition of congruent and moderate length hashtag (M = 3.76, SD = .75). In particular, the level of people’s interest (M = 4.28, SD = .72) in this condition is the highest among four dimensions and even higher than the level of overall engagement. The behavioral engagement and curiosity also have relatively high levels (M = 3.83, SD = .96; M = 3.75, SD = .90 respectively), while the level of
people’s attention is also the lowest (M = 3.33, SD = .86). The lowest level of overall engagement for unfamiliar brand is found in the condition of incongruent and short hashtag (M = 2.43, SD = .60), meaning that the incongruent and short hashtag posted by the unfamiliar brand will be highly possible to be ignored by the public.
Table 6. Descriptive statistics of variables for per condition
Condition
Variable N Minimum Maximum Mean
Std. Deviation Relevance Length Familiarity
High-fit Short Familiar
Engagement 19 2.10 4.00 3.14 .56 Attention 19 1.00 4.00 3.21 .73 Curiosity 19 1.50 4.00 2.79 .77 Interest 19 2.00 5.00 3.60 .86 Action 19 1.00 4.00 3.00 .81
High -fit Moderate Familiar
Engagement 21 1.20 4.50 3.10 1.01 Attention 21 1.00 5.00 2.76 1.10 Curiosity 21 1.00 5.00 3.21 1.23 Interest 21 1.50 5.00 3.64 1.10 Action 21 1.00 4.67 2.98 1.13
High -fit Long Familiar
Engagement 20 1.20 3.90 2.82 .80 Attention 20 1.00 5.00 2.82 1.02 Curiosity 20 1.50 5.00 3.35 1.15 Interest 20 1.00 3.50 2.50 .81 Action 20 1.00 4.33 2.68 .81
Low-fit Short Familiar
Engagement 20 1.10 4.20 3.06 .91 Attention 20 1.00 4.67 3.13 1.13 Curiosity 20 1.00 4.50 2.55 .92 Interest 20 1.00 4.00 2.55 .92 Action 20 1.33 5.00 3.65 1.13
Low -fit Moderate Familiar
Engagement 24 1.20 4.80 3.35 .93 Attention 24 1.00 5.00 3.07 1.01 Curiosity 24 1.50 5.00 3.50 .98 Interest 24 1.00 5.00 3.48 .97 Action 24 1.33 5.00 3.44 1.10
Low -fit Long Familiar
Engagement Attention Curiosity 24 1.90 3.90 2.88 .54 24 2.00 4.33 3.12 .70 24 2.00 4.00 3.08 .73
Interest 24 24 1.00 1.67 4.50 4.00 2.77 2.57 .82 .73 Action
High-fit Short Unfamiliar
Engagement 20 2.20 4.80 3.70 .79 Attention 20 2.00 5.00 3.71 .80 Curiosity 20 2.00 5.00 3.85 .95 Interest 20 2.00 4.50 3.33 .77 Action 20 1.33 5.00 3.82 1.06
High -fit Moderate Unfamiliar
Engagement 20 2.40 4.90 3.76 .75 Attention 20 2.00 5.00 3.33 .86 Curiosity 20 2.00 5.00 3.75 .90 Interest 20 3.00 5.00 4.28 .72 Action 20 1.33 5.00 3.83 .96
High-fit Long Unfamiliar
Engagement 20 1.30 4.40 2.87 .83 Attention 20 1.00 4.33 2.92 .82 Curiosity 20 1.00 5.00 3.58 1.13 Interest 20 1.00 5.00 2.88 1.04 Action 20 1.00 4.67 2.35 1.02
Low-fit Short Unfamiliar
Engagement 19 1.70 3.60 2.43 .60 Attention 19 1.00 4.00 2.37 .90 Curiosity 19 1.50 3.50 2.26 .69 Interest 19 1.00 3.50 2.42 .75 Action 19 1.33 4.00 2.60 .86
Low -fit Moderate Unfamiliar
Engagement 25 1.70 5.00 2.96 .82 Attention 25 1.00 5.00 2.75 1.12 Curiosity 25 1.50 5.00 3.20 1.06 Interest 25 1.50 5.00 3.32 .79 Action 25 1.00 5.00 2.77 1.07
Low -fit Long Unfamiliar
Engagement 23 1.10 4.40 2.81 .86 Attention 23 1.00 5.00 2.77 1.00 Curiosity 23 1.00 4.50 2.80 .95 Interest 23 1.00 5.00 3.04 1.20 Action 23 1.33 4.33 2.70 1.03 In addition, there are two sets of questions asking participants about their perception of hashtag usefulness and daily application of the hashtag, which will also be analyzed by the frequency table and charts. For the first set question regarding the perception of hashtag usefulness, most of the participants (43.1%) perceive the
hashtag is very useful on Twitter, while only 24 participants from 255 consider the hashtag is not at all useful accounting for 9.4%. Additionally, those participants who consider the hashtag is useful to different extent were asked to indicate the most useful function of hashtag claimed by Yang et al. (2012). The frequency result shows that the top 3 most useful functions sharing the similar percentage are: (1) “The hashtag can make users easier to search the information”, accounting for 29.3%, (2) “The hashtag can make the tagged tweet to be found and shared to a larger extent” with 27.2%, and (3) “The hashtag defines a virtual community of users with the same background, the same interest, or involved in the same conversation or task” with 26.3%. This finding will be discussed in the next chapter. Furthermore, the second set of questions ask participant about their daily application of hashtag on Twitter. The results show that the majority of participants use hashtags on their own tweets, accounting for 79.7%. And in general, most of participants also indicate that they usually embed just one hashtag for per tweet (64.0%) and only 9.1% of them use more than three hashtags for per tweet. Additionally, three control variables, namely brand preference (M= 2.92, SD= .98), coffee-drinking/ jogging habits (M= 2.93, SD= 1.06) and Twitter behavior (M= 2.92, SD= 0.90) will be included in the hypotheses testing.
4.3 Hypotheses testing
ANCOVA enables to “exert stricter experimental control by taking account of confounding variables to give us a ‘purer’ measure of effect of the experimental manipulation” (Field, 2013, p496). It is noted that the assumption should be checked before performing the ANCOVA. Firstly, it is required that if there are more than one covariates in the model then these covariates should not be highly correlated (Field, 2013). Therefore, the correlation of all covariates should be checked first. As shown in Table 7 the correlation test results indicate that none of the covariates have significant correlation values, thus, this assumption is met. As well as that, another assumption requires for the equal variances (Field, 2013). To test this assumption, the Levene’s test will be conducted before looking at the results from the ANCOVA. If this assumption is also met then the ANCOVA can be conducted to test the main effects of hashtag relevance and length and their interaction effect on user engagement, besides, the moderating effect of brand familiarity on engagement, whilst controlling brand preference (M=2.920, SD=0.977), coffee-drinking/ jogging habits (M=2.931, SD=1.063) Twitter behavior (M=2.918, SD= 0.903) as well. If the main effect is significant, Fishers’s LSD post hoc test will be carried out to make direct comparisons between two means from two individual groups to see which levels within each independent variable are significant. After finish the ANCOVA analysis, the MANCOVA analysis continues to figure out the effect on which specific dimensions contributes to the level of overall engagement.
Table 7. Correlations between main variables 1 2 3 4 5 6 7 8 9 10 11 1. Engagement 1 2. Attention 3. Curiosity 4. Interest 5. Action 6.Relevance .840** .778** .803** .864** .176** 1 .600** .528** .587** .124** 1 .549** .515** .232** 1 .625** .198** 1 .075 1 7. Length -.119 -.083 .114 -.078 -.264** -.032 1 8. Familiarity -.011 .028 -.066 -.060 .025 -.004 .005 1 9. Preference .168** .173** .059 .063 .209** -.015 -.079 .662** 1 10. Habits .313** .219** .182** .272** .336** .094 -.061 -.039 .033 1 11. Behavior .248** .201** .164** .176** .254** -.027 -.028 -.030 .026 .113 1
**. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
Hypothesis 1 states that hashtag relevance has an effect on engagement. Firstly, the result of Levene’s test has a significant level greater than 0.05 (p= .06), indicating that the assumption of equal variances has not been violated (Field, 2013). Thus, a one-way ANCOVA is conducted to test the main effect of hashtag relevance on engagement and compare the effectiveness of two levels of relevance whilst controlling three variables. As shown, there is a significant main effect of relevance on engagement, F (1,250)= 7.70, p< .05, but the effect size is low (η2
= .03) (Field, 2013) meaning that only 3% of the variance in engagement is explained by the hashtag relevance. Based on the results, H1 is supported. Since the main effect of relevance is validated, then the LSD post-hoc test is conducted. It is shown that there is a significant difference between low-fit condition and high-fit condition (p<.05). Comparing the estimated marginal means shows that the highest level of engagement is found in high-fit hashtags (M = 3.21) compared to low-fit hashtags (M = 2.94),
supporting H1a but rejecting H1b. Furthermore, a closer look at each dimension of engagement, the MANCOVA results indicate that there is a significant effect of hashtag relevance on people’s curiosity (F = 13.88, p < .05) and intrinsic interest (F = 9.27, p < .05), but the effect on attention (p= .057) and action (p= .329) is not significant. The estimated marginal means show that both the higher levels of curiosity and intrinsic interest are found in high-fit hashtag condition (M = 3.42, 3.35 respectively) compared to low-fit condition (M = 2.95, 2.98 respectively). The results are summarized in Table 8.
Table 8. ANCOVA, MANCOVA summary and Mean values for relevance effect
Note: All measures were given on Likert-type scales ranging from 1 (= lower) to 5 (=higher) Note: Significant at the p< .05 level
Hypothesis 2 suggests that hashtag length has an effect on engagement. Based on the equal variances (p= .31), a one-way ANCOVA is conducted to test the main effect of hashtag length on engagement and compare the effectiveness of three lengths whilst controlling three variables (Table 10). There is a significant main effect of length on engagement, F (2,249)= 6.08, p< .05, η2
= .047, indicating that there is a significant difference in mean level of engagement between the lengths, H2 is
ANCOVA M congruent Mincongruent F df p η2 Engagement 7.70 1 .006 .03 3.21 2.94 MANCOVA F df p η2 Attention 3.66 1 .057 .014 3.11 2.89 Curiosity 13.88 1 .000 .053 3.42 2.95 Interest 9.27 1 .003 .036 3.35 2.98 Action .96 1 .329 .004 3.10 2.97
supported. The LSD post-hoc test shows that a significant difference exists between moderate-length condition and short condition (p< .05) and moderate-length condition and long condition (p< .05). Comparing the estimated marginal means shows that the highest level of engagement is found in moderate-length hashtags (M = 3.28) compared to short hashtags and long hashtags (M = 3.04, 2.88 respectively), supporting H2a. MANCOVA further reveals that the hashtag length has significant effect on three dimensions of engagement, namely curiosity (F = 6.76, p < .05), intrinsic interest (F = 22.08, p < .05) and online action (F = 12.14, p < .05). The LSD post hoc tests reveal that moderate length hashtags perform the best in these three dimensions, while the long hashtags perform the worst in people’s interest and online action. Surprisingly, with regard to people’s curiosity the long hashtags (M = 3.20) perform better than the short hashtags (M = 2.84). It is worth stressing that hashtag length is influential for people’s behavioral engagement, which will be discussed in next chapter. The results for the length effect are summarized in Table 9.
Table 9. ANCOVA, MANCOVA summary and Mean values for length effect
Note: All measures were given on Likert-type scales ranging from 1 (= lower) to 5 (=higher) Note: Significant at the p< .05 level
ANCOVA
M short M moderate Mlong
F df p η2 Engagement 6.08 2 .003 .047 3.04 3.28 2.88 MANCOVA F df p η2 Attention .47 2 .628 .004 3.08 2.97 2.95 Curiosity 6.76 2 .001 .051 2.84 3.41 3.20 Interest 22.08 2 .000 .151 2.93 3.67 2.82 Action 12.14 2 .000 .089 3.21 3.25 2.62
