The Collaborative Maze
The effect of visualization tools on computer
supported collaboration
Frank Houweling
10199969
Thesis Bachelor Information Science
Thesis Supervisor: Dick Heinhuis Date: 07-07-2014
University of Amsterdam Faculty of Science
Abstract
The information age greatly influenced the way we work today. Work is more and more about complex problem solving. To cope with these problems, collaboration is of great importance. Visualization tools pose opportunities to support collaboration. The research question is thus: “To what extent can supporting the participants’ ability to visualize their insights into the collaborative problem using a CSC system improve the effectiveness of their collaboration?”. To find an answer to this question, a laboratory experiment is
designed in which collaboration is of great importance. The effectiveness of the collaboration of couples in this setting is then measured under two conditions: a condition with visualization tools, and a condition without these tools but with video- and audio conferencing. In the experiment was found that visualization tools can greatly improve the successfulness of the collaboration. This result creates opportunities for researchers to design visualization tools to support collaboration.
Table of Contents
Abstract ... 2
1. Introduction ... 5
Research Question ... 6
2. Scope of Definitions ... 7
Collaboration ... 7
CSC Systems ... 7
Visualization ... 8
3. Literature Review ... 9
Insights in the field of Collaboration ... 9
Collaboration from an economic point of view ... 9
Collaboration and Group Dynamics ... 9
Collaboration and Communication ... 11
Shared mental model and Visualization ... 11
Visualization and Awareness ... 12
Enabling visualization in a CSC system ... 13
Hypotheses ... 14
4. Experiment Design ... 16
Choosing a workspace and task suitable for the experiment ... 16
Measuring the successfulness of collaboration ... 17
Defining factors which should be supported by both systems ... 19
Dörnyei: Contact and Interaction ... 19
Carstensen & Schmidt: Complex Information needs, Task Interdependencies, Common Information Spaces and Semantic Meaning ... 20
Identified factors in applied CSC System research ... 20
Dourish, Carstensen & Schmidt; Flexibility and adaption ... 21
Prinz et al.; Non-‐intrusive social exchange ... 21
Daft et al, Wolff et al, Rice; The role of media richness ... 21
Proposing two CSC systems for this task ... 23
5. Method ... 24
Groups and Conditions ... 24
Measurements ... 24
Subjects ... 24
Tasks ... 25
Software and Technology ... 25
Physical set-‐up and Interface ... 27
Procedure ... 28
6. Results & Discussion ... 30
Results ... 30
Discussion ... 35
References ... 38
Appendices ... 42
Appendix A: Questions of post survey ... 42
Appendix B: Participant demographics ... 44
Appendix C: Explanation before experiment ... 46
Appendix D: Descriptive statistics of the average playing time ... 49
Appendix E: Mann Whitney U Test on Question Two ... 50
Appendix F: One Sample Chi-‐Square Test of Question One ... 51
Appendix G: Independent samples T-‐Test for Playing Time ... 52
Appendix H: Mann Whitney U Test on Question Four ... 53
1. Introduction
In the information age, the way people work and their working environment changes rapidly. Because of automation, people focus on problems that cannot be solved by computers. Work is therefore characterized by complex (cooperative) problem solving and decision making activities. People need to seek ways to solve the complex problems they face daily (Carstensen & Schmidt, 1999). To do this, they have to work together with other people over growing distances (Alberts, 2001). Because of this, Ross (2011) states that collaboration is of growing importance.
With the growing importance and complexity of collaboration grows the need to improve and support it. Computer Supported Collaboration (CSC) seeks ways to support collaboration by developing digital systems. To do this, different types of systems are available. These systems support one or more of the different factors needed for successful collaboration. To discuss and evaluate the properties of all available factors in collaboration (and with that CSC systems) is outside the scope of a bachelor thesis. Therefore, a brief tour of the research in this area is given, after which only one type of systems, visualization systems, is discussed in further detail and evaluated for it’s ability to support collaboration.
With the great importance of collaboration, the field of Computer Supported Collaboration gets a lot of attention within the academic world. A big part of this research is done in the field of CSC systems and visualization systems, for example by Swaab et al. (2002). This poses the question if new research is needed. There are two main reasons why new research can be important.
First, existing research is not generalizable enough. For example Swaab et al. (2002) discussed the opportunities visualization techniques pose in supporting group negotiations. From their research can be concluded that visualization techniques have a positive effect on the collaboration of these groups. This poses the expectation that visualization can be a tool in group work, but this can not yet be generalized to all forms of collaboration, because negotiation is a very specific form of collaboration. New research might find empirical evidence in a more general setting, which can be better applied to other collaborative settings.
The second reason why new research is needed is that existing research is outdated. CSC technologies that support visualization have undergone tremendous improvements in the last years. New technologies like WebRTC enable us to use real-time communication tools over the web, and enable people to use visualization techniques when they are not near each other. Because of these improvements, these technologies might now be ready to be
used in real-life collaborative settings, and the influence of such technologies on the effectiveness of the collaboration might have changed.
Research Question
In this research the opportunities of improving collaboration through visualization techniques are discussed, in a way that is generalizable to many forms of collaboration and using the newest technologies available. The research question is: “To what extent
can supporting the participants’ ability to visualize their insights into the collaborative problem using a CSC system improve the effectiveness of their collaboration?”
To answer this question several sub questions are answered in advance: ● Which theories can be used to describe visualization?
● In what way can visualization be supported in a CSC setting?
● Can we find empirical evidence of the role of visualization in collaboration?
To answer the main question, an experiment is conducted to measure the effect of visualization on the successfulness of the collaborative work.
2. Scope of Definitions
Before we can find theories to describe visualization and collaboration, further clarification on the scope of these terms is needed. In the field of (computer supported) collaboration, no consensus on terminology has yet been achieved. The definition of some widely used terms differs greatly in the available literature. In this chapter the meaning of these terms is defined in more detail, to avoid ambiguities in the remainder of this thesis.
Collaboration
Collaboration is a broad term with a wide range of definitions. Many definitions view collaboration as a structured form of communication. John-Steiner et al. (1998) discussed different definitions, and proposed a more modern definition that focuses less on communication (dialogue) and highlights both the group task and the manner in which the members approach their collaborative efforts. It includes all the group processes that are important for collaboration in the information age.
“The principals in a true collaboration represent complementary domains of expertise. As collaborators, they not only plan, decide, and act jointly, they also think together, combining independent conceptual schemes to create original frameworks. Also, in a true collaboration, there is a commitment to shared resources, power, and talent: no individual's point of view dominates, authority for decisions and actions resides in the group, and work products reflect a blending of all participants' contributions. We recognize that collaborative groups differ in their conformance to this profile and that any single group may exhibit some of the features only episodically or only after long association.“ (Minnis et al., 1994 in John-Steiner et al., 1998 ) In this research, collaboration is seen as such a collaborative effort to solve a complex problem.
CSC Systems
There are as many definitions for computer supported collaboration as there are definitions for collaboration. Difficulties lie in the thin line between CSC systems, and other types of systems. Grudin (1991) considered different definitions of which systems are CSC systems. In Grudin’s review (1991), some say technologies providing access to shared
files could be seen as CSC systems (Crowley in Ensor: 1990), but others argue that CSC systems must be based upon an understanding of the collaborative aspects of the work to be supported (Schmidt and Bannon: 1991). It is clear that there is no consensus on the the types of systems which are CSC systems. But it is reasonable that not the type of system but the fact that it supports collaboration is what makes a system a CSC system.
In this research, we will mostly factor on the last category of definitions, where, as discussed in the literature review, visualization systems based on an understanding of the collaborative aspects (the collaborative problem) often are more successful.
Visualization
Visualization is an important concept in this research. There is a lot of agreement on the meaning of the term ‘visualization’, and because of that many similar definitions are available. The definition of Fitzberg et al. (2009): ‘visualizing is the activity of representing abstract data or knowledge in a way which reinforces human cognition.’ is short but complete and fits the purpose of visualization in this thesis very well.
3. Literature Review
As discussed in the introduction, the effect of enabling visualization through a CSC system on the successfulness of collaboration is researched. This focus on visualization did not fall from the sky. To design a system to support collaboration, it is important to find factors that are important for the successfulness of collaboration, and then choose a factor that has great potential to be improved. Firstly, a general overview of where these factors can be found is given. The concept of the shared mental model, as one of these factors, will be introduced. Consequently, the opportunities of visualization tools in CSC systems will be discussed in more detail using the shared mental model. After the shared mental model, awareness, another factor, will be used to do the same. To conclude our literature review, literature about visualization is used to construct a theory about how to build a successful CSC visualization tool.
Insights in the field of Collaboration
Because of the interests participants have in collaborative activities, an economic view is very useful in finding factors for effective collaboration, and a good starting point in our overview.
Collaboration from an economic point of view
Collaboration is often discussed as a game of risk and profit. Good examples of theories in collaboration and cooperation are the Prisoners Dilemma that forms a theoretical background for ineffective collaborations from an economical perspective (Bixenstine et al., 1966), and the Dictators Games experiment that proves people tend to be ‘fair’ when sharing and collaborating. (Engel, 2011) The small group framework of Lewin (Forsyth, 1998) states that working together and collaborate to achieve a certain goal can benefit all stakeholders. “The whole is greater than the sum”.
Presumably, these theories give a good image of collaboration on a macroeconomic scale, but are not well used in other cases. This is because these theories and models explain how people react to collaborative situations in general, but do not support reasoning about how people will react in a specific case.
Collaboration and Group Dynamics
Literature from the social sciences often tackles group dynamics and organizational structures. Because people collaborate in groups these dynamics might pose an explanation for the effectiveness of visualization.
Factors for the way a person performs in a collaborative group is the person’s age, intelligence, social sensitivity or emotional stability. (Shaw, 1971) It is not at all obvious how these personal factors could be supported by a CSC system, if it is possible at all. What remains is the way a group organizes itself. The dynamics of organizations are changing, and this might pose opportunities for CSC systems to support these changes. There are multiple trends that can be seen in the way people collaborate in an organization.
Firstly, Carstensen & Schmidt (1999) argue that the strategies organizations use to manage complex problems are changing. The traditional strategy is to cut complex problems into smaller sub problems. These sub problems are then divided in a hierarchical organization, using a top-down manner, and solved sequentially. For this method of solving complex problems, different levels of management are required which makes problem solving via hierarchical organizations too slow for the modern information society, where fast decision-making is of big importance. Instead of the hierarchical design, organizations are more often shifting towards novel organizational concepts like the collaborative working environment.
Secondly, Prinz et al. (2006) describes this collaborative working environment (CWE). In a CWE are, just like in most modern working environments, different professionals who work together by exchanging information and knowledge in order to reach a common understanding of the working field. In CWE the environment in which they do so is not geographically bound. To achieve this, different tools are important to enable professionals to collaborate with each other. Professionals who can no longer achieve their tasks without these technologies are called e-professionals. CWE enables people to stop working in chain production models but more in dynamic and adaptive teams where social capital and communication are of great importance.
However, Carstensen & Schmidt (1999) state that in a CWE “the coordination and integration of the myriads of interdependent and yet distributed and concurrent activities becomes enormously complex”.
Many of the factors resulting from these organizational changes relate to planning and decision-making. These factors are quite interesting, and there is a substantial body of literature about supporting decision making and planning with CSC systems. CSC systems supporting these group dynamics often exist of groupware systems focused on making planning of research more effective. A lot of research is done on this field, and new research might not add new insights because of the lack of the effects of new technologies
on these technology-undemanding problems. Other, more technology demanding systems might be more interesting, for example a system which uses multimedia technology.
Collaboration and Communication
Factors that could be supported with more media might be found in communication, where a high information richness is often important. The role of communication in successful collaboration is often explained with the shared mental model. For people the participate well, a good shared mental model between them needs to be achieved.
Shared mental model and Visualization
The concept of the shared mental model states that to solve a shared problem, the participants in collaboration should have a shared mental representation of the problem area.
Exact definitions vary between literature, but Jonker et al. (2011) defined the shared mental model very well by dividing the definition in smaller definitions. The model in the context of collaboration is the to-be-solved problem and all the relevant information about this problem. The internal representation of this model in the mind of the human is the mental model. The shared mental model is the degree in which people in a collaboration have the same mental model. Jonker et al. (2011) then concluded with the definition of Converse (1993), who described a shared mental model, focusing on the role of the shared mental model in collaboration:
“Knowledge structures held by members of a team that enable them to form accurate explanations and expectations for the task, and, in turn, coordinate their actions and adapt their behavior to demands of the task and other team members.”
This shared mental model is of great importance for collaboration. Converse (1993) describes how a person organizes his or her knowledge (and therewith his or her mental model) into structured patterns. This helps participants of a collaboration to quickly comprehend and respond to new information in the collaborative setting. Next to that, it helps them to describe, explain and predict behavior, and with that coordinate their efforts. A shared mental model is a context in which the team members can successfully communicate. It will enable a person to predict which actions he needs to perform, and which actions other participants of the collaboration will perform. When the members of a
team have the same mental representation of the to-be-solved problem, they can thus collaborate better.
It is important to note the difference between a good mental model, and a good shared mental model. One participant can have a perfect mental representation of the problem, but when all the other participants have a completely opposite mental representation of the problem the shared mental representation is bad, and thus the shared mental model too. Previous research found evidence for visualization to support the shared mental model of a team. Swaab et al. (2002) discussed the opportunities visualization techniques pose in supporting the shared mental model of a group. In their experiment, groups would collaborate in a negotiation setting. Collaborative teams, which worked with visualization tools, improved three aspects1 of negotiations. The first positively influenced factor is the
participants’ convergence; the extent to which the participants’ mental image grow together. This resulted in a better shared mental model. Secondly, the participants formed a more cohesive group. Lastly, the groups’ entitativity is improved. Entitativity is a measure for the effectiveness of working as a group: “the extent to which an assemblage of individuals is perceived to be a group rather than an aggregation of independent, unrelated individuals; the quality of being an entity”. (Campbell; 1958).
Visualization tools aid the focusing of groups in such a way that they develop shared mental models. These shared mental models result in a better collaboration result.
Visualization and Awareness
But, the shared mental model is not the only theory that could be used to describe the effects of visualization on collaboration. Awareness is a broad term, but in the context of collaboration described as the following by Wolff et al. (2007): one person in a collaboration should be aware of the actions of other participants, to be able to provide context for it’s own activities. In spoken language, this is often described as: “to know what’s going on” (Endsley, 1995). To achieve awareness between participants they should inform each other about their activities in real-time (Lanza et al., 2010).
The effect of visualization on awareness is discussed by Storey et al. (1995). They describe several effective software visualization tools that provide awareness of human activities in software development.
But, this research is just like the previously discussed research of Swaab et al. (2002) not generalizable, because of the specific collaborative environment in which the tools are used. As Storey et al. (1995) remark themselves, are the activities in software
1 Swaab et al. (2002) uses the term aspects, but in the context of our article this could better be
development specifically suited for visualization tools because of the lack of a physical product.
Both Awareness and the Shared Mental Model describe the effects of visualization on collaboration. A CSC visualization tool could in theory support the shared mental model and awareness, and with that improve the effectiveness of that collaboration. Before we can design such a CSC tool, we need to further investigate visualization. Not all visualizations might support the shared mental model even well, and the CSC tool needs to take in account effective visualization techniques.
Enabling visualization in a CSC system
A lot of research is available on how good visualizations are designed. These theories can be used to design a CSC system to support visualizations. There are three theories about visualization on which the CSC system in this thesis is focused.
Firstly, Carpendale (2003) gives examples of how the usage of different visual variables can greatly influence the clarity of a visualization. The user should be able to create an effective visualization using a wide range visual variables, for example position, size, color and texture, especially when his mental representation of the problem is complex.
Secondly, and slightly in contrary with the first theory, the naturalness of the conversation should not be at risk.
Naturalness, as discussed by Wolff et al. (2007), is an important property that should be taken in account while designing a CSC system. A natural interface is an interface where people can concentrate more on the task, not on the interface. Finding the most natural interface is quite a challenge. Poston and Serra (1996) illustrate why: “In the abstract, a mouse cursor seems far better than a finger, pointing more precisely at a point in the monitor screen. In practice, every screen has fingermarks.”
In the case of a visualization system, it might be important to not clutter the interface with too many visual variables.
Figure 1
Lastly, according to Munzner’s Nested Model for Visualization Design and Validation (2009), the design of a visualization support tool is highly dependent of the domain context, which he in his model (see Figure 1) calls the problem characterization. The problem characterization is in our case a collaborative problem.
Figure 2
Visual variables found by Mackinlay (1986)2
A CSC system which supports visualization should support participants with the proper tools to create a visualization suitable to the to be solved problem, and the visual variables available are specific for the collaborative environment. Mackinlay’s (1986) comprehensive overview of all visual variables (see Figure 2) clearly shows the differences of the visual variables. These differences should be used to choose the right visual variables, before applying any. It is thus not possible to make a perfect CSC visualization system, which works in all collaborative settings.
Hypotheses
From the discussed literature can be concluded that it is important for the different participants in a collaboration to have a comparable mental model, and to convey awareness between the participants. To improve the shared mental model and to convey awareness, earlier research suggests visualization as a support tool. A visualization tool should enable users to make a clear visualization of their mental model, but have an easy
2
Mackinlay’s (1986) table of visual variables, edited by Joe Perry.
interface to achieve a sufficient naturalness. The visualization tool itself should fit in well with the collaborative setting.
Based on these facts, the following hypothesis is set: “A CSC system which enables the participants to visualize the to be solved problems will provide a better collaboration than a CSC system that does not.”
4. Experiment Design
A theoretical basis of what is a good visualization system is found. Also, we found properties of a system that supports visualization. It is now the goal of this research to find empirical evidence of the role of visualization in collaboration.
In this chapter, available theories and previous work are used to design an experiment. To design the experiment, the setting and task to be used will be discussed first. This setting should be well suited for a collaborative activity. After that, ways to measure the effectiveness of the collaboration in solving the task are discussed.
Finally, the two groups needed in the experiment, a group where visualization is well supported, and a group without visualization tools, are described in more detail.
Choosing a workspace and task suitable for the experiment
The collaborative setting in which the experiment takes place can influence results and generalizability. It is therefore important to carefully choose our collaborative environment. While choosing a setting to evaluate the to-be-designed system many options are available. Two general options were considered.
The first option is implementing the system for a real-life task and testing the system on e-professionals. The advantage of this is that the result will probably be justifiable on most situations. The downside of this option is the fact that much other factors are in play, and the results of this system are hard to measure. This is why many previous experiments choose for a qualitative approach in such a real-life setting. Pinelle et al. (2000) wrote a review of 45 CSC system evaluations. From the literature discussed around 72% chose for a purely qualitative research method.
Another, more modeled system, could however be used in a quantitative evaluation. This is important to be able to gather statistical proof for our hypotheses. Eventually, such a modeled task was chosen because of this reason.
The challenge lies now in modeling the experiment in such a way that it depends heavily on collaboration.
To do this, one of the properties of the modern collaborative environment discussed in the introduction should be used. Like in the real world, the different participants should have a complex problem they need to solve together. Secondly, it is important that there is an difference in the knowledge available between the participants. This is because only then will the participants need to actively collaborate to form a shared mental model.
Next to the collaborative setting of the experiment, the setting in space and time needs to be considered. In the CSC space/time matrix in table 1, four options are defined.
Face to face interactions and ongoing tasks are not interesting, as we seek a solution to the problems posed in the introduction of this paper. The problems included the greater distances over which needed to be communicated.
The two options to be considered in different places were remote interactions and communication and coordination. Previously discussed tools like WebRTC are very well suited for remote interactions in real time. Because these new technologies are interesting to research, remote interactions were chosen.
Table 1
Computer Supported Collaboration Space/Time Matrix by Johansen, 1988
Same place Different place
Same time Face to Face Interactions Remote Interactions
Different time Ongoing Tasks Communication and
Coordination
Based on the properties discussed the following task and workspace are chosen.
The modeled task proposed exists of a simple multiplayer maze game that should simultaneously be played by two users. A maze-game is controlled enough to be seen as a laboratory experiment, and such an online simultaneous game is a perfect example of remote interactions.
The goal of the maze game is to move a character to the end of the maze. However, this is not as simple as it looks. In this maze game, the users get randomly assigned a task. The first user sees the maze, but isn’t able to move the character. The second user is able to move the character, but isn’t able to see the maze.
The maze is very well suited as a workspace for our experiment. Because of the difference in knowledge and abilities of the two participants, the participants have to collaborate to be able to reach the end of the maze as fast as possible. We can then design CSC systems to help them in doing this.
Measuring the successfulness of collaboration
Next to the focus on the workspace and task of the experiment, it is also important to focus on choosing the right measurement for an experiment. This, of course, to achieve a valid result. A good start to look for these measures is by looking at existing research on CSC system evaluations.
In existing research, different measurements for collaborative effectiveness are used frequently.
In the review of CSC system evaluations of Pinelle et al. (2000), most researchers used observation to gather information about the effectiveness of their CSC system.
Many other researchers used a questionnaire to gather information about how well the effectiveness of the collaboration was perceived by the participants. For example Gutwin et al. (1996), Brothers et al. (1990) and Olson et al. (1992) who used scales (mostly likert scales from a negative to a positive response) with multiple questions about how positively parts of the collaboration were perceived.
But, both these measurements are quite generic for many fields of research and might yield les results than measurements specific for collaboration.
Hwang et al. (2005) discussed such a collaboration-specific measurement. An important part of collaboration is the fact that collaboration is self-explanation; individuals articulate concepts by communicating them to a group. Hwang et al. (2005) therefore measured the amount of shared information to measure collaborative effectiveness.
Stevens (2013) used another collaboration-specific measurement. In his thesis, the development of new ideas (brainstorming) in individual and group sessions is evaluated. These group sessions could be seen as a collaborative effort. For measuring the effectiveness of the brainstorming sessions the quality and quantity of the ideas is measured. In a way, the effectiveness of the collaborative session is measured by measuring the effectiveness by which the collaborative goal is achieved.
Hymes et al. (1992) used a measure comparable to the one of Steven (2013) in their groupware evaluation. They designed an application that should support brainstorming, and evaluated it by comparing different groups using different systems and used the amount of unique ideas generated as a score of successfulness in collaboration.
Measuring the result of a collaborative activity as the result of the collaboration functions as an aggregate of the collaboration itself. This is because collaboration is of course only focused on generating an as good as possible result.
While keeping all influences on the result of the collaboration except the visualization CSC system identical throughout the experiment, measuring the result gives a generalizable picture of the effect of visualization.
In this experiment the total time needed to complete the maze was used as the primary measurement of successfulness of the collaboration. The lower the time is, the more successful the couple was in achieving good collaboration.
In combination to the time, a questionnaire is used to check if the participants also recognized the results found with the total time.
Defining factors which should be supported by both systems
When designing the experiment, it is important to make sure we have sufficient internal validity. To do this, nothing else but the effect of visualization should be measured. This is not as simple as it looks.
For example comparing a collaboration with visualization tools to a collaboration with no communication possible at all, which is the first thing that will come to mind, will always result in a positive result for visualization. This is because a system with (limited) communication tools is compared to a system with no communication tools at all, and because of that the possibility to visualize the participants’ mental model is not the only positively influencing difference between the two conditions.
It is thus important to create two CSC systems with tools which both equally support the most important factors in collaboration, with the exception of the visualization tool which supports the shared mental model and awareness.
In the following paragraphs a framework of relevant factors will be build, and the two conditions used in the designed experiment will both comply to the majority of these factors; they are both good CSC systems, but one system has visualization enabled and the other one does not. By doing so, only the effects of the addition of visualization are tested.
In favor of clarity the following requirement is used while choosing factors we will describe: only factors in the successfulness of collaboration that are somehow relevant on the design of CSC systems are included.
Dörnyei: Contact and Interaction
Dörnyei (1997) investigates the role of group dynamics, motivation and communication for collaboration in language learning.
He first discusses the fact that multiple past studies found group cohesiveness as an important factor in collaboration. There are multiple aspects in achieving group cohesiveness. The first two are the amount of time spent together and the shared group history. Other identified aspects are: proximity, contact, interaction, cooperation, the rewarding nature of group experience, successful completion of whole group tasks, joint hardship, intergroup competition, common threat, group legends, investing in the group, public commitment and defining the group against another.
From the aspects above only contact and interaction are obvious to be supported by a CSC system.
Shaw (1971) defines interaction as: “By interaction is meant that they emit behavior in each other’s presence, they create products for each other, or they communicate with each other. In every case that would identify as an instance of interaction there is at least the possibility that the actions of each person affects the other.”
Interaction should be supported by a CSC system by making sure people can share knowledge quickly and naturally.
Carstensen & Schmidt: Complex Information needs, Task Interdependencies,
Common Information Spaces and Semantic Meaning
Carstensen & Schmidt (1999) argue that CSC systems should support complex information needs. To do so they should manage task interdependencies, common information spaces and semantic meaning.
Managing task interdependencies and common information spaces sound comparable, but they are completely different. While managing task dependencies, the system should inform other stakeholders if a stakeholder changes the state of the work. It should monitor the state of affairs, and with that facilitates mutual awareness. While managing common information spaces, the system keeps track of changes in information objects in which the collaborative team is working.
Carstensen & Schmidt (1999) argue that in the collaborative setting (and then specifically in the CSC system), flexibility is needed for users to work and collaborate in their own ways. The challenge here is to ensure this flexibility on the one hand, while maintaining semantic meaning for the system to adapt itself to cooperation and coordination activities. Carstensen & Schmidt (1999) also give an example of how these factors could be supported. They argue that a CSC system should exist of different tools a user could use to support his or her collaborative activities. In a way, the CSC system provided should consist of basic building blocks and platforms, and the user should be able to combine the building blocks he wants.
Identified factors in applied CSC System research
These theories provide a solid representation on which the effectiveness of a system could be tested, but using factors specific for CSC system use cases might even give a better image of how collaboration is best supported with CSC systems. Because of that, factors in more applied literature are now discussed.
Dourish, Carstensen & Schmidt; Flexibility and adaption
Dourish (1990) indicates that system appropriation (or tailorability) poses great challenges to CSC systems. Users may use the technology in a way completely different than intended while designing the system. In their design, systems should be adjusted to the users’ attitudes towards new systems, and be suitable for their workplace practices. The systems should also take into account the way people adopt new technology. A lack of flexibility in a system can have negative consequences on the collaboration between stakeholders.
Prinz et al.; Non-intrusive social exchange
Prinz et al. (2006) describes how CSC systems should support the e-professionals’3 social exchange of knowledge in a non-intrusive manner.
For supporting this factor, it is important to design a system where the technology is on the background, and the professionals and the task on the foreground.
Daft et al, Wolff et al, Rice; The role of media richness
Wolff et al. (2007) states that, for enhancing communication between the stakeholders, and with that enhancing togetherness, media richness is an important factor. Media richness is “the medium’s capacity for immediate feedback, the number of cues and senses involved, personalization, and language variety” (Rice, 1993). Daft et al. (1983) presents the Information richness-framework which states that more media richness improves communication over a form of media which oversimplifies information (like text based communication). On the other hand, the framework identifies the fact that a communication system could also be too media rich, and because of that can have a negative effect on the communication. A CSC system will need to find a balance in it’s media richness.
3 Prinz talks about e-professionals simply because these are a topic in his paper, but it is assumed
Table 2
Important factors in collaboration
Author Factor / Requirement: Page
Dörnyei (1997) Improve contact
Improve interaction
Carstensen & Schmidt (1999)
Support complex information needs
Manage task interdependencies
Manage common information spaces
Maintain a semantic meaning for the system to adapt itself to cooperation.
Dourish (1990), Carstensen & Schmidt (1999)
Flexible enough to fulfill a user's needs, should be designed in a way which takes in account the way in which people adopt new systems
Prinz et al. (2006) Support social exchange of knowledge in a non-intrusive manner.
Daft et al. (1983), Wolff et al. (2007), Rice (1993)
Have enough - but not too much - media richness to achieve optimal communication.
Proposing two CSC systems for this task
Based on the factors defined in the previous chapter, and summarized in Table 2, two systems are designed for use in the experiment. The two conditions in the experiment will both use one of these CSC systems. Both system support all factors in the table, but only one has visualization tools.
Both systems have support for video- and audio communication. This form of communication is often used in CSC systems and would be approved by the majority of factors in the framework.
One system also enables visualization. The problem to be solved requires both users to have a good representation of the problem. But, the information available about the problem is not shared equally over the participants: just like in a real-life setting. One participant can see the maze, and the other cannot. In this system, the participant who can see the maze can use not only the video- and audio communication tools to share his knowledge. He can also use the provided drawing tool to draw the maze on the other screen, and visualize the problem description.
There is a limited range of visualization tools available, because this is a relatively non-complex problem, and according to the literature discussed not too many options should be given to achieve a natural interface. The participants could draw a line in only one color ant thickness. They can however very well use length, shape and position to visualize their insights.
5. Method
Groups and Conditions
For this experiments couples of two participants were used. The couples were formed out of people who wish to participate in the experiment at the same time. The software divides the pairs randomly but equally over two conditions. The following two conditions were examined:
1. The CSC system with visualization support
2. The CSC system without visualization support (but with video- and audio communication)
Measurements
The variables are as following. The independent variable is the implementation of visualization tools (see Groups and Conditions). The dependent variable is the successfulness of the collaboration. The dependent variable is measured by the time needed to complete the experiment, and the perceived successfulness of the collaboration measured in Question 2 of our questionnaire. The time needed to complete the experiment is the main measurement, because a fast time is the only goal of this task. Time can also easily be quantified and is thus useful for empirical analysis.
Next to the total time, another measurement is the perceived effectiveness. This is measured by a likert scale-based survey after the experiment. The most important question in this survey is Question 2, “How well do you think you collaborated with the other player?”. With the likert scale-based survey are also some open questions and multiple-choice questions to gather demographics and further insights into the results that are found. The questions in the questionnaire can be found in Appendix A.
Subjects
There were 42 pairs (84 participants) in this study. Subjects were mostly recruited from different sources; on the Internet via websites like facebook, reddit and Internet forums. But also in real-life situations like with friends and family, and with random samples at a local school. Subjects from different age groups and computer proficiencies where selected, by reaching out to people using a selection of different websites. The average demographics of the participants can be found in Appendix B. There was no control on the pairing of participants: people were paired together if they were available for participation at the same moment in time.
Tasks
A pair of two persons is first assigned to one of the two conditions, after which the two roles in the conditions are randomly assigned to the two participants. The total four rules can be found in table 3.
Table 3.
The different conditions and roles in the experiment
Participant who can move the character, but can’t see the maze
Participant who can see the maze, but can’t move the character. (and can, when available, use the visualization tool to draw)
System with visualization tools
System without visualization tools
The participants got a global task they should achieve together. The task was simply to “guide” the character on the screen to reach the end of the maze in as little time as possible. Further strategies on how to achieve this are not given to the couple. It is however explained which tools are available for them to collaborate. Also, the fact that not all information is available to both participants in a couple, and which information is available to which participant is given before the experiment begins.
For each condition, there are two different explanatory texts. The one displayed to the participants is dependent on his role. All different options and their explanatory texts are available in Appendix C.
Software and Technology
As stated in the introduction, there are two primary reasons for the need of new research in the field of visualization CSC systems. New research should focus on being more generalizable, and use state-of-the art technology also found in the real working environment. In the Experiment Design, we already discussed in depth why this experiment is better suited to be generalized. Now, the used technologies are discussed, with a focus on being state-of-the-art and useful in the real working environment.
The maze game built for this experiment uses HTML5, CSS and JavaScript with the jQuery4 and PeerJS5 libraries on the client side, and NodeJS with PeerJS and ExpressJS
on the server side.
Some of these technologies were used for animations, were others were important for real-time connectivity with the other participant.
For the animations, jQuery was used in combination with an important HTML5 feature called the canvas element, which enables us to quickly draw 2d graphics on the screen. This was used to draw the maze and made the animations when the player ‘walked’ through it.
Figure 3
PeerJS Peer Discovery process with Ajax and WebRTC
To make the application connect to the other participants’ application, different technologies are needed.
To build the two-step peer-discovery system as shown in Figure 3, two client side libraries were used: jQuery Ajax (which is part of the jQuery library mentioned earlier) and PeerJS. The jQuery Ajax library was used in combination with a custom built backend for the
4
jQuery Javascript Library. Visited at http://jquery.com/ on July 4, 2014
5PeerJS, simplifies peer-to-peer data, video and audio calls. Visited at http://peerjs.com/ on July 4,
automated peer discovery process between participants. It acts thus as a traffic controller which communicates with the server to find another participant, and then instructs PeerJS. After these instructions, PeerJS helps concur the implementation differences of WebRTC between the different browsers by running as an adapter on the client platform. These implementation differences are there because WebRTC is another new technology of HTML5 that is not completely implemented in any browser yet, but enough features are6 to
run the experiment.
On the server side, a custom WebRTC server is implemented using NodeJS and the server side version of the PeerJS library, in combination with the ExpressJS web framework. This server side application only helps the two clients into finding each other, after which WebRTC technology setup a direct connection between the two pears (pear-to-pear). ExpressJS was almost only used for routing purposes.
Both the client and server side technologies used are relatively new (the first drafts of WebRTC originate from 20117) and are very likely to be available to all modern computer
systems in years, because of the support of some of the biggest web browsers (at this moment: Google Chrome, Mozilla Firefox and Opera). The combination of these browsers accounts at this moment for more than 70% of all Internet users.8
Physical set-up and Interface
The used software is web based, which enables anyone with a modern web browser to participate from home. Some participant’s might use provided laptops for this experiment, but on these laptops the experiment would be conducted just like it would have on their own device.
At the moment a participant opens the game’s webpage, their browser is checked for WebRTC compatibility. Many modern web browsers have WebRTC enabled. If WebRTC is not enabled in the browser of the participant, the participant is told he or her can, unfortunately, not participate in the experiment. The participant needs a microphone and a webcam to participate. Fortunately, almost any modern laptop has both build in.9
The software interface is shown in figure 4. During playing the participants of a pair can see the character, the drawings, the video screen and a stopwatch. There is also a button
6Is WebRTC ready yet? http://www.iswebrtcreadyyet.com on July 3, 2014
7WebRTC 1.0: Real-time Communication Between Browsers. First draft. Visited at
http://dev.w3.org/2011/webrtc/editor/webrtc-20110823.html on July 3, 2014
8 StatCounter global statistics. Visited at http://gs.statcounter.com/ on July 3, 2014
9 On April 15, 2014 at least 1362 of 1485 laptops (92%) on the tweakers.net price watch has a
that they can use to ‘give up’ and precede to the survey. For one of the two participants, the maze is also displayed.
Figure 4
The software interface used in the experiment
Procedure
All participants of the experiment followed the same procedure when the experiment was conducted. Important is to note that participants were able to quit at any time, simply by closing the browser window. During the experiment, participants would follow the following steps:
1. Introduction
Potential participants get, over the Internet or verbally, a short introduction in the experiment. The main goal of the game is explained, and people are asked if they want to participate in the experiment. If they agree to do so, they can open the experiment webpage.
The experiment automatically checks if the participant’s computer is capable of the new technologies used in this experiment. If the participant’s misses these technologies, he is thanked but declined.
3. Wait for second player
The participant is asked to ‘find’ a second participant to collaborate with. He/she has a link to be shared with another person to collaborate with. When another participant arrives, both participants go to the next screen.
4. Task division and explanation
The system both decides if the current team is in the visualization or non-visualization conditions and divides the two tasks over the two participants. Both these are done randomly.
An explanation screen is then displayed tailored to the task and condition of the participant. The game starts when the players press the “Start the game” button. 5. The collaborative activity
The players now can work together (with- or without visualization tools) to reach the end of the maze as fast as possible. The game ends when they reach the end, or when one of the two players presses the “I give up” button.
6. Post-Survey
6. Results & Discussion
Results
In this result section the first measure to be analyzed is the primary measure; the total needed completion time of the different pairs (also: playing time). With this measure, the main research question will be answered. After this, secondary measures are analyzed to gather more insights into the found result.
Before the playing time was analyzed outliers were removed. No outstandingly long experiments were found. But, on the other hand, some participants had an extremely low time because they pressed the “I give up” button fairly quickly. Because of this reason people who gave up within 50 seconds are filtered out in the main measure test, because they didn’t even make an effort. In total, three couples of two participants were removed from the result set.
The non-visualization group (N = 18) was associated with an average playing time of M = 185.00 (SD = 102.22). By comparison, the group with visualization enabled was associated with a numerically smaller average playing time M = 102.24 (SD = 51.70). To test the hypotheses that the couples with visualization tools enabled are associated with a statistically significantly different mean playing time, an independent samples t-test was performed. More descriptive statistics can be found in Appendix D.
As can be seen in Figure 5, the result distributions were sufficiently normal for the purposes of conducting a t-test (skew < 2.0, kurtosis < 9.0; Schmider et al 2010).
The assumption of homogeneity of variances was tested via a Levene’s F test. From the results of this test, F(37) = 12.79, p = 0.001, can be concluded that the variances of the two conditions were heterogeneous. Equal variances were thus, in the analysis of the t-test, not assumed. As found in Appendix G, the independent samples t-test proved a statistically significant effect, t(24.28) = 3.11, p = .005. Thus, the couples with visualization enabled were associated with a statistically significant smaller time needed to complete the game.
Figure 5
Distribution of total time needed to complete the experiment under the two conditions
The found result is significant, and therewith we can conclude that couples with visualization tools performed better than couples without these tools.
Cohen’s d was calculated to find the effect size of the found result. It was estimated at d = 1.02, with effect-size r = 0.46. According to Cohen’s (1992) paper, this is considered to be a medium (r) to large (d) effect (r effects: small ≥ .10, medium ≥ .30, large ≥ .50, d effects: small ≥ .20, medium ≥ .50, large ≥ .80).
The post questionnaire is now used to gather more insights in why this is the case. A small part of the participants chose not to fill the questionnaire, and because of that there are another number of participants for the questionnaire than expected from the main measure descriptives. Next to that, questionnaires were filled individually, in comparison to the couples who have a shared final time.
To find if participants think collaboration in general is importance to solve this puzzle, question one (“How important do you think collaboration was to be able to get a high score (= fast time)?) was analyzed using a one-sample chi square test (a One-Sample Chi-Square Goodness-of-Fit test). We can see from the results in Appendix F that our test statistic is statistically significant: χ2(2) = 93.086,p < .0005. Therefore, we can conclude
that there are statistically significant differences in the answers to this question. They answered more often positively (Very Important, N = 52) and Important, N = 11) than negatively (Moderately Important, N = 5; Of little Importance, N = 0 and Unimportant, N = 2).
Participants felt that collaboration was important for being successful. But did the participants think they collaborated better when they were in the visualization group? To find this, another question of the questionnaire was analyzed. Question 2 (“How well do you think you collaborated with the other player?”) can be seen as a measure for perceived collaboration effectiveness. This question was answered using a likert scale (from one for very bad, to 5 for very well). As suggested by de Winter et al. (2010) a Mann Whitney U test was performed on this likert scale questions.
The group without visualization tools had a median of 4.00 and a mean rank of 33.48. The group with visualization tools also had a median of 4.00, but a mean rank of 37.01. A Mann-Whitney's U test evaluated the difference in the responses of our 5-Likert scale question between the two groups. We found no significant effect on the result (U = 539.500, Z = -.770, p = .448; more in Appendix E). The group with visualization tools enabled didn’t feel like they collaborated better, but they did get a faster playing time. The participants thus did not feel like they collaborated better.
But did they think digital tools were not important at all? To answer this question, Question 4 (“How important were the provided digital tools in achieving this collaboration?”) was analyzed. This question was also answered using a likert scale (from one for unimportant, to 5 for very important), and because of that a Mann Whitney U test was performed. The group were no visualization tools were available had a median of 4.00 and a mean rank of 30.20. The group with visualization tools had a median of 5.00, and a numerically higher mean rank of 39.48. A Mann-Whitney's U test was used to find a significant difference in the responses of our 5-point Likert scale question between the two groups. A significant difference was found (U = 441.000, Z = -2.022, p = .043; more in Appendix H).
Participants with visualization tools thus thought their tools were more important than participants without the visualization tools.
There were also open questions in the survey. One of these, Question 3 in which participants were asked to describe their strategy, was quantified by classifying the answers onto predefined classes.
The following unique answers were defined by analyzing the results: 1. Not Clear / None
2. Audio Only (One-Way) 3. Audio Only (Two-Way) 4. Audio & Video (One-Way) 5. Audio & Video (Two-Way)
6. First find path, then draw (One-Way)
7. Find path and draw simultaneously (One-Way) 8. Draw and Communicate (One-Way)
9. Draw and Communicate (Two-Way)
For all classes is, between the brackets, noted if the participants used one-way or two-way communication as defined by Lasswell (1948). This is done explicitly because this information might provide insights, and was often well described by the participants. After coding the textual answers to these predefined classes, the intercoder agreement was calculated. Different agreement statistics can be found in Appendix I. Krippendorff's Alpha was determined at 0.737 which according to Krippendorff’s (1980) strength of agreement (SOA) conventions (< .67 = to be discarded; .68 – .79 = shows tentative agreement; > .80 = definite agreement), shows a tentative agreement. The agreement was not higher because of the limited information given in the answers of the participants, which left a lot of room for the interpretation of the coder.
Figure 6
Strategies for the two experimental conditions
The distribution as shown in Figure 6 is as you would expect, with many participants using the visualization tools when able. The communication of the participants was in general however very one-sided; only one person communicated and the other listened.
Conclusion
In this paper, we investigated to what extent supporting the participants’ ability to visualize their insights resulted in a more effective collaboration.
The couples with visualization tools at their disposal collaborated significantly better, they were significantly faster than the couples who did not have visualization tools. In the post-survey, the participants indicated they thought collaboration was important to get a good result in the experiment. They did, however, not feel like they collaborated better when they were in a couple with visualization tools available. Maybe slightly in contrast to this,
Amount of participants
participants with visualization tools thought their tools were more important than the participants without visualization tools.
With these results in mind, we can accept our hypothesis and thus answer our main question. Supporting the participants’ ability to visualize their insights into the collaborative problem using a CSC system, greatly improves the result of the collaboration.
Discussion
In this thesis was chosen to perform an experiment in which the generalizability was improved by using a laboratory setting. Two small remarks have to be made about the internal validity
Firstly, in this research, the participants were allowed to “bring their own colleagues”. This is a great way to gather couples, and was because of that of importance to gather a sufficient sample in the short amount of time available, but could influence performance because these people might have adapted to each other in terms of communication. Secondly, the complete application and survey was translated to Dutch for the Dutch participants. These translations might have had slight differences in meaning, and therewith have influenced results. But this was also necessary to gather enough data in the short time available.
Apart from these two small remarks, this laboratory experiment had a notable better validity than existing research. It is arguable that, by performing such a modeled experiment, the external validity is at risk. When we simplify the collaborative setting, we are in danger of not enough representing a real-world scenario.
Based on the results of this thesis, two recommendations could be made for future research.
As found in the strategy section of the results, communication in this experiment was frequently one-way. One-way communication might not represent communication in a workspace environment, where information needs to be shared in multiple ways. Future research might choose for another setting where both participants have information they need to share.
Secondly, future research might focus on a collaborative setting that requires more domain-specific knowledge. According to Munzner’s Nested Model for Visualization Design and Validation (2009), the visualization support tool needed is highly dependent of the domain context. It is therefore important for the external validity to create a setting that well resembles the modern working environment, and thus requires domain specific knowledge.
Thirdly, real-life collaborative efforts often play in much longer settings (months to years) which brings in play different communicative factors and factors from the field of group dynamics. The setting in the experiment of this thesis was highly collaborative, but it did not comply to these two improvements. Future research might choose for a longer and more complex experiment; something which wasn’t possible in the time available for a bachelor thesis.
It is important to note for all three these recommendations that while improving the external validity, the internal validity should be preserved. By making the experiment more complex, it gets harder to only measure the effect of visualization, and not of external factors.
Next to these recommendations, some remarks on the scope of the results need to be made. The visualization tool used did support collaboration in the chosen setting, but might as well not be as successful in another setting. It is also not clear if, for every given collaborative setting, a successful visualization tool can be designed. It is however not possible to research all collaborative settings separately, and certainly not in a bachelor thesis. The chosen setting was strongly collaborative and simple enough to provide insights into the effects of visualization in other settings, yet more research is needed to prove the effects of visualization tools on collaborations in general.
The final times found in the experiment do not suggest any structural problems in the conducted experiment. They were as you would expect based on the theories found in the literature review. In short, couples with visualization tools performed significantly better on almost all measures.
One notable result is the fact that the distribution of the measured total times was a lot wider for the teams without the visualization tools than it was for the couples with visualization tools. A possible explanation is that people in general are equally skilled in visualizing their thoughts, but there are big differences in their verbal skills. Future research might pose us an answer to this question.
The questionnaire results show a similar picture as the total times. Only Question 2 (“How well do you think you collaborated with the other player?”) from the survey gave an unexpected result. Participants in the visualization-enabled group did not give a higher ranking to their collaboration.
This might be an interpretation issue; participants gave their collaborations fairly high grades, and this could be explained by the fact that some people might have thought: we have collaborated as well as we could with the given tools, and have not taken into account that they might have performed better with other tools. They did take this in
account while answering Question 3 (“How important were the provided digital tools in achieving this collaboration?”).
The results of this research expand the knowledge we have about the role of visualization in collaborative environments. The role of visualization has been proved, and insights have been gathered in how-to design a successful visualization CSC support tool. The conducted experiment is more generalizable than existing research, and conducted with state-of-the art technologies. This work can support future research in supporting collaboration with visualization systems.
