Real-time Interaction-Modification Strategy
SUBMITTED IN PARTIAL FULLFILLMENT FOR THE DEGREE OF MASTER OF SCIENCE
ALRICK
MADURO
10795863
M
ASTER
I
NFORMATION
S
TUDIES
HUMAN-CENTERED MULTIMEDIA
F
ACULTY OF
S
CIENCE
UNIVERSITY OF AMSTERDAM
July 28
th, 2016
1st Supervisor 2nd SupervisorDr. Frank Nack Dr. Gui Liberali
Real-time Interaction-Modification Strategy
Alrick Maduro
University of Amsterdam Graduate School of Informatics
Science Park 904 Amsterdam info@alrickmaduro.com
ABSTRACT
Predicting what a user is trying to do or how a user prefers to browse or see the data provided by a commercial web application may bring in more revenue for the company if performed correctly. The real-time interaction-modification strategy experiment is a real-world experiment trying to calculate a users cognitive style by implementing the Bayes rule1 using outsourced data, and present the user with a design created by a designers’ perspective on cognitive styling to best suit a users’ cognitive style. The experiment was performed on a service oriented web application. According to the results of this experiment, although generating more click-through rates and sales conversions with the modification-experiment group, a statistical significance between the modification-experiment group and the control group could not be established.
General Terms
Algorithms, Management, Measurement, Documentation, Performance, Design, Economics, Experimentation, Security, Human Factors, Standardization, Languages, Theory, Legal Aspects.
Keywords
Interaction strategy; cognitive styles; conversion rates; clickstream analysis; Bayesian theorem; real-time analysis
1. INTRODUCTION
On the World Wide Web there are many commercial web applications created by web developers whether independently or employed by companies. Web applications are usually created for specific purposes e.g. to gather information or to provide a service or to sell one or more products. For most of these commercial web applications, the more traffic it receives, the more revenue it can generate from advertisements or sales. Many believe that a method to increase the chances of sales, once users visit a web application, is to know the preferences and goals of each user and act accordingly to that user’s preference [1]. In other words, this can be seen as an interaction strategy, whether reactive or proactive, by building a model of the preferences and knowledge of each user, and use this model throughout the user’s interaction with the web application in order for the application to adapt to the needs of that individual user. Such adaptive hypermedia system can influence sales, create leads, and be beneficial to a party or company. The more the user interacts, the more information is gathered and can be used as input to algorithmic recommendations, targeting the user with relevant advertisement or upsell targeting.
A problem or limitation a company or organization may have is that these companies provide mostly “static” content. This means the content or the manner in which the content is displayed on the
1https://en.wikipedia.org/wiki/Bayes%27_rule 2
https://en.wikipedia.org/wiki/A/B_testing
web application does not change based on the information gathered from the user. As stated by (Brusilovsky, 2001), a limitation of traditional “static” hypermedia applications is that they provide the same page content and same set of links to all users. If the user population is relatively diverse, a traditional system will suffer from the inability to serve all things to all people. Therefore by not implementing an interaction strategy based on the user’s information, a loss of sales or lack of user interest when interacting with the web application is probable. As specified before, a web application is build for specific purposes and can contain various sections, paths and goals. These sections and goals can range e.g. from service, sales and product information. An approach to suit the relatively diverse population, which can include new customers, existing customers or users looking for new products to purchase, is to construct an adaptive hypermedia web application based on a real-time interaction-modification strategy, hereinafter named ‘RIMS’.
Interaction-modification strategy uses the Bayes theorem. A theorem also used in ‘Website Morphing’, which is an adaptive system to infer cognitive styles and identify optimal morphs (Hauser et al. 2009). Cognitive styles can be described as individually preferred and habitual approach to organizing and representing information. In other words the way an individual thinks, perceives and remembers information [3].
Albeit that various existing interaction strategies may be applied on web applications like A/B Testing2 and WUM (Spiliopoulou et al. 1999); which means improving the layout and structure of the web application based on the navigation history, these methods are mostly trial and error.
This research aims to answer questions regarding modifying elements on sections within a web application based on user cognitive styles and if this approach is either beneficial or unrewarding to the users and the web application selected goals. The result of this research can be interesting for other applications or areas such as user targeting, sales and service. To elaborate on the extent of this RIMS research; a stand-alone web application tool - prototype was created and could be implemented as an add-on to an already existing applicatiadd-on i.e. the cadd-onsumer Ziggo3 website.
The article is structured as follows. Firstly the related work is outlined. Next, the research question and the resulting experiment are outlined, including a description of the used RIMS technology. The results of the experiment performed are then presented. The paper ends with conclusions and a view on future work.
2
https://en.wikipedia.org/wiki/A/B_testing 3https://www.ziggo.nl
2. RELATED WORK
There is a lot of literature and functioning tools with regards to adaptive hypermedia applications. This chapter sums up the most important and relevant findings. These findings were used to construct a model, which formed the base for building the RIMS tool to perform the modification experiment that will be explained later on in this research.
2.1 Gathering of literature
Literature articles related to personalizing hypermedia applications are widely available on the Internet. These articles are diverse in the sense that they focus on different aspects of personalizing an application e.g. one-to-one marketing, where the customer need to feel they have a unique personal relationship with the business or company (Peppers and Rogers, 1997). When a sale or conversion has been achieved, a post-sales phase or technique can be taken to reassure customers of their purchase decision by delivering additional values through service, support and loyalty programs, creating opportunities for long-term customer retention and eventual cross- and upselling (Kobsa et al, 2001).
Those are some of the techniques that are applicable in realizing or constructing an adaptive hypermedia application. Methods describing adaptive hypermedia e.g. the AHA model (De Bra and Calvi, 1998) or AHAM model (De Bra et al. 1999) have shown that general purpose adaptive hypermedia systems can be designed and implemented but tend to be too complicated for non-technical users.
According to Brusilovsky [1], there are two types of adaptation in adaptive hypermedia systems i.e. adaptive presentation and adaptive navigation. Each page visited by a user, updates a model, which in turn determines how the next page the user visits should be altered. While sequences of pages and possibly behavior of user groups can be taken into account, the AHA model works on a page-by-page basis. Each page has a set of rules that determine how the model is updated for the visiting user.
AHA’s adaptive presentation technique is based on the conditional inclusion of fragments within a page whilst using different colors for link anchors indicating the probability or desirability of users preference for it’s adaptive navigation. The AHAM proposes a richer model, based on AHA, which consists of more attributes based on conditional action rules.
This research took note of techniques discussed in articles referenced above. By using this algorithm based on real time events triggered by the user, the RIMS experiment will add on or better compliment the above-mentioned models in the assumption that it would benefit the users.
2.2 Gathering of existing tools
As stated in the introduction, there are existing interaction-strategies that may already be applied to current web applications e.g. A/B Testing, which may or may not be based on current or real-time gathered user information. There are tools or applications that provide what are called ‘persuasion or predictive conversion optimizations’. This means the tool will try and predict what the user will do based on e.g. navigational paths.
One of these applications is called sagent.io4, which monitors real-time navigation behavior and product-data and apply different persuasion tactics in order to persuade the users to
4
https://www.sagent.io
purchase a product. Another product available on the consumer market focusing on personalization is Adobe Test and Target5. This is an application that is adjacent to its own Content Management System, Adobe Experience Manager (AEM). AEM can track its users and relay the user’s information to the adjacent Adobe Test and Target module, which by way of A/B testing can calculate the conversion ratio and significance per specific testing scenario. Based on the above-mentioned literature research and current functional tools examined, a model was built which will be explained in the next chapter for which a real-time interaction-modification strategy application was created to not only perform the experiment on any given site with minimal configuration settings but to also view the results per experiment group.
2.3 Model
Creating a model for this research relied on finding a method to identify a users’ cognitive style after each click and assign a corresponding pre-defined design to the user. As described by Hauser et al. (2009), identifying optimal morphs for each user relies heavily on the cognitive style from the clickstream data with Bayesian updating. Their website morphing model classified its model into two loops i.e. the cognitive inference loop and the dynamic programming loop. Although neither of the loops is used in this experiment, a similar model to the cognitive inference loop was constructed.
The model used for this RIMS experiment would allow the measurement of essential attributes such as e.g. conversion ratio and significance between the control and experiment group. The next chapter describes how the data is collected, what decisions were made and the design and setup of the RIMS application.
Figure 1: RIMS model
The modification-assignment rule illustrated in figure 1 is the assignment of a design based on designers’ perspective of the chosen cognitive style.
5
3. METHOD
Performing this research took communication and cooperation between individuals and departments of the companies involved i.e. Ziggo and Erasmus RSM University. Interviews and meetings were conducted with both parties to extract the information needed. The Erasmus RSM students of the ‘Learning from Big Data course’ provided the following data e.g. the calibration study, analyzing the data from the calibration survey, analyzing the data from the panel of judges and computing the necessary calculations providing the link probabilities. Further details about these deliverables are described in the following chapters including which page of the web application was chosen to perform the RIMS experiment on and which cognitive styles to analyze.
To summarize; getting the information needed to perform the experiment was a qualitative process arranging meetings and plans with the departments of Ziggo e.g. communicating with design, development and targeting department to not only create design based on chosen styles but to also develop, deploy and temporarily suspend current targeting or A/B tests to not influence the validity of the RIMS experiment results.
Since the experiment took place on a public web domain i.e. Ziggo.nl, gathering the results from unique visitors traffic of approximately 200.000 a day6, would require a quantitative approach.
3.1 Problem statement
Offering everything by means of content and information to a relatively diverse population will limit the ability to facilitate all things to all people. The use of the interaction-modification strategy to personalize the presentation of content and information for individual users may diminish this limitation and provide better service, click-through rates or conversions.
Based on the problems just mentioned, the following research question was formed:
Can interaction modification improve the adaptability for a service oriented website?
The adaptability was measured with click-through rates. This means whether a user clicked on a modified fragment based on the cognitive style of that user.
The following hypothesis was tested by this research experiment:
Matching the designers’ cognitive style design to a users’ cognitive style can increase the webpage’s click-through rates.
3.2 Addressing the problem
Before addressing the problem stated in the previous chapter and using the model created in chapter 2.3, the cognitive styles have to be chosen with which to perform the experiment. Hauser et al. (2009) describes a list of possible cognitive styles but the most likely to affect respondents’ preferences for website characteristics are chosen as options. A maximum of no more than six styles were proposed to keep the scope of the experiment feasible in the allocated time planned. After meeting, proposing, explaining and discussing the RIMS experiment with the company’s management board and other professionals involved, a decision was made how to select the amount of cognitive styles to analyze.
6
Daily traffic data on a specific page is gathered from already existing and implemented analytical tools i.e. Google Analytics
A calibration study had to be done to prime data for the automated inference loop [4], which was elected to be executed on the Ziggo.nl website in order to choose the most relevant cognitive styles out of a minimum of 700 respondents. Prior to the calibration, a pre-calibration panel took place at RSM by the RSM students. As part of the pre-calibration, a panel of judges assessed the website’s links according to the cognitive dimensions. During the panel, judges were shown slides containing pictures of relevant links taken from the website (appendix II) and were asked to rate each link according to its characteristics. The assessor evaluated every link as well according to their apparent characteristics. A link can take a value of 1 or 0, according to whether it can be recognized by a certain click-alternative
characteristic.
For example, a link that sends a visitor to a certain page in which news are presented is to be characterized as a ‘news’ related link and is attached with the number 1 for this characteristic. In practice, a link can have more than one characteristic (C); hence it can contain the value of 1 for the characteristics it entails and 0 for the ones it does not entail. These link characteristics are concerned with the expected interpretation of each of the available click-alternatives that a website visitor is confronted with while navigating through a website. Hauser et al. (2009) referred to these characteristics as click-alternative characteristics. Click-alternative characteristics are also referred to as Ckjns. Ckjns can be
defined as the characteristics of the jth click-alternative of the kth click decision made by the nth visitor (Hauser et al. 2009). After produced, Ckjns are used to create a linkage between a visitor’s
cognitive style and the characteristics of the click alternatives that were chosen by the user during an online session [11].
The calibration study was done by way of a survey (appendix 1). The steps taken to design and how contents of this calibration survey were created will be explained in the next chapter. The results of this calibration are given in chapter 4. This sampling strategy is also an attempt to obtain representative sample of (potential) customers.
Once the calibration results were known and the calculations performed by the RSM students were received, it was up to the RIMS application to serve the correct cognitive style modification to the users, based on their cognitive style preference.
The RIMS application will use adaptive presentation when rendering the modifications on the client-side of the website, adapting the presentation of information within a page providing prerequisite, additional or comparative explanation [9]. This is a technique also used in the AHA model (De Bra and Calvi, 1997)(De Bra and Calvi, 1998).
To summarize; in deliberation with all parties involved the following requirements were set:
• RIMS calculates and updates the cognitive style of current visitor using the Bayes theorem and link choices
• A total of ten pages are monitored for clicks and up to ten links per page.
• Current visitor only sees a modification after the visitor has made more than three clicks on monitored links. The user also stays with the modification seen and that modification will not be switched or changed even if the cognitive style of the user changes for any particular reason.
3.3 RIMS Design and Setup
Building a tool or prototype to perform the experiment based on the model created in chapter 2.3, required some research on how to create an application that was not only implementable on one website but on various websites if needed and easily setup for production use. To build a quick but production stable robust prototype to perform this RIMS experiment, a quick look was taken into hackathons. These are events where e.g. software engineers and designers collaborate to build fast reliable prototypes within a set period that range from a day to usually a week.
3.3.1 Framework
After examining the top listed or most used web frameworks7 in hackathon events, a choice had to be made on which was the most relevant for the RIMS experiment. The requirements were: • The web framework had to be relatively easy and quick to
setup with the available and current knowledge of the developer.
• Able to be executed on the web-application – a client side script – JavaScript
• Able to work as a stand-alone Applications Programming Interface (API) collecting data from the client side script • A portal with a HTML5 graphical user interface to configure
the dimension and modification settings.
• Able to handle asynchronous events – for data requests and responses.
Based on the above requirements the decision was made to build an application on a nodeJS8 framework named SailsJS9 that not only had the potential to work as an API but to be it’s own stand alone application where a configurations platform could be build, including a login and sign up portal. This enables the possibility to create accounts and make this experiment available for multiple websites and other domains with minor configurations.
3.3.2 Client side script
3.3.2.1 Monitoring
One of the main tasks for the client side script is to monitor user click behavior. On initialization, the configured monitored pages and links are retrieved from the API. These pages and links are then monitored using JavaScript event listeners and triggers an event that will send the click information e.g. user-id, clicked monitored element and current user cognitive style of the user, depicted as an array of intervals, as a whole in JSON10 format, to the stand-alone API.
Furthermore, each user is initially put into a cell. This can be seen as a grouping. The amount of groupings or cells may depend on the goal of the experiment. For the Ziggo RIMS experiment the decision was taken to use three cells. One cell for the control group, one cell for the modification group, and one cell for randomized modification group that can also be used as reference.
7 https://techcrunch.com/2015/07/28/which-programming- languages-get-used-most-at-hackathons/ 8 https://nodejs.org/en/ 9 http://sailsjs.org/ 10 http://www.w3schools.com/json/
3.3.2.2 Modifications
The API returns the new cognitive style array and the modification options to the client side script as response from the above mentioned click trigger requests. Should a user’s dominant cognitive style match one of the modification style options, and if the user clicked more than three times (chapter 3.2 requirements) throughout the website on monitored links, then the client side script will change the default elements on the assigned experiment modification page e.g. an image or text, to the new configured modified elements.
3.3.2.3 Calibration
To present the calibration survey, an html template engine, DustJS
by LinkedIn11 was used to facilitate and display a survey from the client side script to show to the users. To not be a nuisance to all approximate 200.000 unique visitors a day with this survey (figure 2), which only a minimum of 700 respondents were required, a math mod ruling was used to only target 1 out of each 10 visitors at random. The likert scale was the method chosen for a visitor to answer a calibration question with a required value it represents i.e. 1 to 5, as done in Hauser et al (2009).
All the data including questions and answers for the calibration survey, monitored pages and links, dimensions (cognitive styles) and modification configurations is received from the API. Details regarding data from the API will be explained in the next sub-chapter, but to summarize; the administrator of the API account can insert their monitored pages and links based on their uniform resource locator (url) and element class names or id’s, calibration questions and available answers, regardless of cognitive style or dimensions.
The following figure illustrates how the visitors got to see the calibration survey.
Figure 2: Calibration survey on client website
3.3.3 Stand-alone API
The API application contains all data provided to and received, from the client side script described in the previous sub-chapter. This data includes for example, user behavior, cognitive styles configurations and modification settings per webpage experiment i.e. defining which images or text to change per dimension.
11
3.3.3.1 Database model
Before starting development of the application, a database model for the API had to be designed and build to secure the data gathered. Based on development experience and to speed up development, an ORM12 in combination with MySQL13 was the preferred database choice to use with the API. A database model was created based on the following Enhanced Entity-Relationship diagram (figure 3).
It may be noticed that there are no relationship lines depicted in the diagram. This was done by choice to be able to use an object-relational mapping (ORM) in case future releases demand new databases or new type systems that may be incompatible. In other words, this mapping or ORM is a virtual object database that can be used within the programming language.
Figure 3: Database EER diagram
From the figure 3 diagram a portal was constructed for an administrator to configure and setup the modification settings. This portal was made available through a Ziggo subdomain.
3.3.3.2 Security setup
An SSL (Hypertext Transfer Protocol Secure - https14) protocol was configured and used due to higher security restrictions and to not cause any browser security notices and erroneous asynchronous requests from an SSL (port 443) secured server to a standard http server (port 80), or in other words, from the client side script to the API. Furthermore the API application endpoints, which mean the uniform resource locators the client side script requests and sends information to, are allowed to receive calls from any whitelisted server IP. This is also called CORS or
12 https://en.wikipedia.org/wiki/Object-relational_mapping 13 https://www.mysql.com/ 14 https://en.wikipedia.org/wiki/HTTPS
Origin Resource Sharing15, which is an http access control method.
3.3.3.3 Legal aspects
Due to Dutch laws and the sensitive subject of handling customer data of cable and telecom companies, no customer can be traced back from clickstream data gathered e.g. the RIMS visitor user-id is a hash of combined data from domain name, a randomized number and time of date. The information in the database is only accessible by Ziggo and used for research purposes. This data will not be shared to third parties without a written consent from Ziggo and its legal services. A non-disclosure agreement (NDA) was written for and signed by collaborators, from the RSM Erasmus University, regarding the RIMS experiment.
3.3.3.4 Design
The portal for the API was build solely for the purpose of professionals who work in targeting, sales or service departments looking to better the adaptability to their website’s visitors. This portal for the API can be seen as a task-based application for all or most characteristics or requirements fall under the Task-Based Applications Design Goals (Unger and Chandler, 2012), which are:
• Enabling users to do something they couldn’t do elsewhere or if they can, to do it better or more efficiently.
• Supporting users with easy instructions and visual prioritization of key tasks.
• Reducing load on the user by making optimal usage of system resources e.g. reusing data versus requiring duplicate entries.
• Design that facilitates learning and a plan that demonstrates the value to the user or administrator in this case.
Based on the task the application goals, and with cooperation and feedback of the Ziggo design department, a recommended color palette (figure 4) was chosen for the look and feel of this application. How the palette was used in building the RIMS application can be seen in figure 5.
Figure 4: Color palette design RIMS
As described in chapter 4, the data collected from the calibration survey resulted in identification of the dimensions representing the four cognitive styles to analyze i.e. holistic - analytic, visual - verbal. New designs or modifications needed to be created for these dimensions i.e. analytic versus visual, analytic versus verbal, holistic versus visual and holistic versus verbal. The Ziggo design department took this task and created four designs based on cognitive style research and constant feedback from the stakeholders.
The designs showing the differences for the same website page between cognitive styles, from the designers’ point of view, can be seen in the following illustrations (figure 6, figure 7, figure 8, figure 9).
15
Figure 5: RIMS API Portal - Calibration overview
3.4 Gathering of test subjects
The resources to perform this experiment were made available by granting access to the Ziggo.nl domain. This meant that the experiment could be exercised in the real world environment to current users and visitors or possible new customers browsing Ziggo’s website. Based on Ziggo’s current analytical data, an average of 13.000 unique visitors a day, for that specific page, that could be experimented on. The limitations and other risks were other A/B tests and targeting tasks that were currently running on the to-be-modified page i.e. ‘Alles-in-1 vergelijken’ page. Therefore a strict time period was planned to carry this experiment out disabling any other tests the moment RIMS was set active. The experiment ran from July 6th from 18.53h to July 7th at 10.23h on the Ziggo.nl domain.
As described in section 3.3.2.1, the user groups are divided into three cells where each cell group represents a part in the experiment i.e. control-group and the experimental groups. A random algorithm assigns a cell for each user evenly on initial visit, which means the groups are unpaired. The reason therefore is because of the real-world scenarios used, e.g. a user usually will not perform the same sale multiple times on the same day and it is not ethical for a company to request a visitor or customer to buy multiple times via different scenario’s for experimental purposes. Once a visitor was assigned a cell group, he or she could no longer be switched from one cell group to another. The visitors were unaware that they were taking part in an experiment. The initial monitored links settings i.e. cognitive preferences per link were configured based on calculations of the pre-calibration panel (chapter 3.2) as described and performed by Segev (2016).
4. ANALYSIS AND RESULTS
4.1 Preliminary analysis of sample population
The data collected from the calibration survey resulted in the identification of two factors representing the four cognitive styles that were chosen and analyzed (i.e. Holistic-Analytic and Verbal-Visual) [11]. The amount of users calibrated or visitors that filled in the survey was a total of 797 visitors, which was sufficient to provide results of the segment distribution (i.e. Holistic-Visual (29%), Holistic-Verbal (22%), Analytic-Visual (28%) and Analytic-Verbal (21%)), using link-alternative characteristics and the clickstream of Ziggo website visitors and posterior probabilities to identify a visitor’s cognitive style, calculated after making one click on the website [11].The modification experiment itself, excluding the calibration, obtained 27.649 users throughout the website, monitoring their behavior by each click made. Cell group 1 was assigned as the control, while cell group 2 was the modification experiment group. Cell 3 was assigned as the random modification group. The group of cell 3 was created to further examine the results if time allowed it, to see what would happen if users with a certain cognitive style would be presented a random style, and not only their own, on the modified page. It is important to note that the numbers shown, as results, are figures extracted only from the experiment groups. There are more points of entries to make a sale and there were more sales made than shown during the period the RIMS experiment took place. The CTR’s and sales conversion figures are based on the visitors that came from the to-be modified page i.e. ‘Alles-in-1 vergelijken’.
Figure 7: Analytic-verbal design Figure 6: Analytic-visual design
Figure 8: Holistic-verbal design
4.2 Analysis of CTR’s and conversion rates
Analyzing the Click Through Rates (CTR’s) of the page being modified, which in this experiment was the ‘Alles-in-1vergelijken’ page, a combination of tools were used i.e. Google
Analytics and the RIMS click measurements / database information. Furthermore, based on the amount of unique visitors a day, quantitative data was used to accept or reject the hypothesis with-in the period of the experiment.
Given that the sample of visitors in the experiment are representative of a larger population, hypothesis testing can be used to understand whether any differences or effects discovered in the experiment exist in the population. The variables used in this experiment were defined as the following; dependent variables were the click-through rates and the optional sales conversions. The independent variable would be the modifications. The confidence level aimed for was 95%.
4.2.1 Testing the hypotheses
From the data (figure 11) it is clear that there is an increase of CTR for the modification experiment group compared to the control group, but to find out if this is not by chance, a hypothesis test must be done. To test the hypothesis, a similar test to A/B-testing statistical analysis16 was chosen, meaning the following steps were taken.
A null-hypothesis was formulated for the research hypothesis i.e. the click-through rate of the control group is not less than the click-through rate of the experimental group. Mathematically formulated as H0: m-mc ≤ 0 where m equals the click-through rate
of the experiment group (either cell 2 or cell 3) and mc is the
click-trough rate of the control group (cell 1). The alternative hypothesis is therefore that the experimental modification group has a significantly higher click-through rate (CTR) than the control group.
Cell Visitors CTR Sales
1 9300 72 (0.77%) 10 (0.11%) 13.9% (of CTR) 2 9199 75 (0.88%) 12 (0.13%) 16.0% (of CTR) 3 9149 70 (0.76%) 10 (0.10%) 14.3% (of CTR)
Figure 11: Results of experiment visitors, click-through on modified page and sales per cell group
The sampled click-through rates are all normally distributed random variables, which could be compared to a coin flip experiment, replacing the ‘head’ or ‘tails’ with ‘converted’ or ‘did not convert’ as result. Instead of seeing whether the experiment group deviates too far from a fixed percentage, a measurement is done whether it deviates too far from the control group.
From the table shown above (figure 11), the mean and standard deviations of CTR’s for each group were calculated (figure 12).
Cell Mean(µ) Standard Deviation(σ) Z-score
1 0,0077 0,0877 N.A.
2 0,0082 0,0899 0.876
3 0,0077 0,0871 - 0.080
Figure 12: Mean, Standard Deviation and Z-score of CTR 16
http://20bits.com/article/statistical-analysis-and-ab-testing
To find out if the difference between the modification experiment group (cell 2) and the control group (cell 1) is large enough to conclude that the modification experiment did in fact increase click-through rates, the probability distribution of m – mc must be
known. As stated by similar A/B test statistical analysis15, which uses the Z-scores and One-tailed [16] test, the sum or difference of two normally distributed random variables is itself normally distributed which gives way to calculate a 95% confidence interval. In this case the null hypothesis can be rejected with 95% confidence level if the Z-score is higher than 1.645 for α=0.05 as illustrated in Figure 13, or in other words, if the modification experimental group click-through rate (cell 2) is significantly higher than the control click-through rate.
Figure 13: Normal Distribution Z-score
Based on the Z-score of 0.876 for the modification experiment group, which is cell 2, the null-hypothesis is accepted. Even though there was uplift for the CTR of the modification experiment group cell 2 of 4% against the control group and uplift of 7% against the cell 3 random modification experiment group. Performing the one-way ANOVA [15] test on these independent groups by giving the means, standard deviations and number of subjects concluded that there were no statistically significant differences between group means (F (2,214)=0.001, p = .999). Although from a business perspective the RIMS experiment result is significant because of the higher rates achieved. The following details are also known from the experiment; as can be seen in the results table (figure 11), cell group 1 had 9300 users; cell group 2 had 9199 users and cell group 3 had 9149 users. From all of those users combined, only 7487 users managed to get to the ‘Alles-in-1
vergelijken’ page to be able to see a modification, which is about
27% of the 27.649 total visitors. From those 27% of visitors, only 1744 (22.3%) did not bounce (44.5%) or exit (33.2%) upon arrival.
5. CONCLUSION, EVALUATION AND
DISCUSSION
A conclusion was formulated based on the results of the modification experiment, in comparison to a control group to test if real-time interaction-modification can improve adaptability for a service-oriented web-application. Although the results show an increase in click-through rates and sales conversion, there is no statistical significance discovered between the groups.
The time-period of the RIMS experiment was chosen at a particular time, as specified in chapter 3.4, to make sure no other campaigns or factors would influence user behavior other than the
modifications themselves. After the experiment was completed, a meeting was held with the stakeholders i.e. managers, to present and assess the outcome variable of the experiment.
The RIMS experiment utilized feedback while gathering qualitative and quantitative data simultaneously [13].
The level of statistical significance based on the statistical test chosen showed that the null hypothesis is true. It may be possible to consider the probability that a difference in mean scores could have arisen based on the assumption that there really is no difference. Whilst there is relatively little justification why a significance level of 0.05 was used rather than 0.01 or 0.10, it is widely used in academic research.
As stated before, even though the CTR was higher for the RIMS experiment-modification group (cell 2), there was no statistical significance that the modification experiment can increase click-through rates. Although from a business perspective the modification group (cell 2) did convert more clicks and more sales than the control group, which could be perceived as significant. The limitation for this experiment was mostly time. Time to further develop the RIMS tool to not only make visualizing data results in real time available but to also expand the scope of the modification experiment. The various modifications where designed by designers’ perspective on how a design should look like for a specific cognitive style. This of course is disputable since a design is an opinion of one or more individuals or would be sensible as seen in the eye of the beholder. Due to the fact that the RIMS experiment was only active for about 15 hours, better results may or may not be achieved if the experiment could be ran longer. Another option or discussion point that can be considered is to suggest another power analysis to determine the sample size required to detect an effect and evaluate the significance value. The RIMS experiment or tool can used as an add-on or base even for future research and collaborations regarding interaction-modifications and website adaptability.
6. ACKNOWLEDGMENTS
This project would not have been made possible if not by the help, advice and with the collaboration of the following individuals: Dr. Guillherme (Gui) Liberali, Dr. Frank Nack, Ivo Krooswijk, Ido Segev, Pravesh Mahabali, Frits Lammerts and the students from RSM University that helped out with this project.
7. REFERENCES
[1] Brusilovsky, P. (2001). Adaptive Hypermedia. User Modeling and User-Adapted Interaction, 11(1-2), 87-110. [2] M. Spiliopoulou, L. C. Faulstich, K. Winkler, A Data Miner
analyzing the Navigational Behaviour of Web Users, Proceedings of workshop on Machine Learning in User Modeling of the ACAI’99, Creta, Greece, July, 1999. [3] Riding, R., Rayner, S.: Cognitive Styles and Learning
Strategies: Understanding Style Differences in Learning and Behavior. David Fulton Publisher, London (1998).
[4] Hauser, J. R., Urban, G. L., Liberali, G., and Braun, M. 2009. “Website Morphing,” Marketing Science (28:2), pp. 202-223.
[5] Peppers, D. and Rogers, M. (1997). The One to One Future. Building Relationships One Customer at a Time. New York: Currency Doubleday.
[6] Kobsa, A., Koenemann, J., & Pohl, W. (2001). Personalised hypermedia presentation techniques for improving online customer relationships. The knowledge engineering
review, 16(02), 111-155.
[7] De Bra, P., & Calvi, L. (1998, June). AHA: a generic adaptive hypermedia system. In Proceedings of the 2nd
Workshop on Adaptive Hypertext and Hypermedia (pp.
5-12).
[8] De Bra, P., Houben, G. J., & Wu, H. (1999, February). AHAM: a Dexter-based reference model for adaptive hypermedia. In Proceedings of the tenth ACM Conference on Hypertext and hypermedia: returning to our diverse roots: returning to our diverse roots (pp. 147-156).
[9] De Bra, P., Brusilovsky, P., & Houben, G. J. (1999). Adaptive hypermedia: from systems to framework. ACM
Computing Surveys (CSUR), 31(4es), 12.
[10] De Bra, P., & Calvi, L. (1997, October). Creating Adaptive Hyperdocuments for and on the Web. In WebNet.
[11] Ido Segev. 2016. Towards a personalized website:
Identifying Cognitive Styles and click-alternative preferences of Online Visitors at Ziggo.nl. Master’s thesis. Rotterdam
School of Management (RSM), Rotterdam, Netherlands. [12] Soni, D., Nord, R. L., & Hofmeister, C. (1995, April).
Software architecture in industrial applications. In Software
Engineering, 1995. ICSE 1995. 17th International Conference on (pp. 196-196). IEEE.
[13] Dennis, M. L., Fetterman, D. M., & Sechrest, L. (1994). Integrating qualitative and quantitative evaluation methods in substance abuse research. Evaluation and Program
Planning, 17(4), 419-427
[14] Unger, R., & Chandler, C. (2012). A Project Guide to UX Design: For user experience designers in the field or in the making. New Riders.
[15] Christensen, R. (1996). One-way ANOVA. In Plane Answers
to Complex Questions (pp. 79-93). Springer New York.
[16] Meng, X. L., Rosenthal, R., & Rubin, D. B. (1992).
Comparing correlated correlation coefficients. Psychological
APPENDIX I.
Survey
Will be scaled from 1 (strongly agree) to 5 (strongly disagree)
1. I usually prefer to read an overview about a certain topic rather than focusing on a detailed explanation of its parts. (holistic-analytic)
2. I prefer planning before acting (deliberative-impulsive)
3. I rather read an explanation of charts, figures or graphs before trying to understand their meaning myself. (verbal-visual) 4. I usually consider meaningful decisions extensively before making a final decision. (deliberative-impulsive)
5. When making a decision, I tend to look for the specific details of the offering rather than at its overview. (analytic-holistic) 6. I prefer gaining insights from looking at a picture rather than reading a text. (visual-verbal)
7. I am detail oriented, and start with the details in order to build a complete picture. (analytic-holistic)
8. When making a decision, I take my time and thoroughly consider all relevant factors. (deliberative-impulsive) 9. I see what I read in mental pictures. (visual-verbal)
10. Even when time allows to consider every situation from all angles, I usually make quick decisions (impulsive-deliberative). 11. I enjoy deciphering graphs, charts and figures more than reading text. (visual-verbal)
12. I find that to adopt a careful, analytic approach to making decisions takes too long (holistic-analytic) *There are three questions for each of the bipolar dimensions
APPENDIX II.
APPENDIX III.
Glossary of terms
The following notations were taken from Hauser et al. (2009):
Ckjns = Characteristics of the jth click-alternative of the kth click decision by visitor n.
Jkn = number of click-alternatives at the kth click by visitor n.
Kn = number of clicks made by visitor n.
qrn = f(𝑟n|𝑦n,𝑐kjn𝑠, 𝛺) ; inferred probability that visitor n is in cognitive style segment r. rn = indexes cognitive style segments.
