Facilitating Planning with Sensators: Tangible Objects Offering Multimodal
Feedback and Support for Off-loading.
Student: Werner de Valk, 5927145
Supervisor: Jouke Rypkema
UvA Representative: Wery van den Wildenberg
Co-assessor: Wery van den Wildenberg
Research institute: TNO Soesterberg
Master: MSc in Brain and Cognitive Sciences, University of Amsterdam, Cognitive Science Track
EC: 39
Abstract
Working memory plays an important role in planning, and its capacity is enhanced when using integrated multimodal stimuli. According to the theory of embodied cognition, using tangible objects might support complex cognitive processes, like planning. Because Sensators (tangible objects providing multimodal feedback) are both tangible and provide integrated multimodal feedback, we hypothesized that they support planning. In a virtual supermarket task participants tried to create shortest possible routes for two robots to pick up products, visualized on a touch-screen table. One robot moved twice as fast as the other, and some of the products could not be placed on top of others, adding complexity to the task. Contrary to our expectations, participants in the active Sensator condition (where feedback was provided by the Sensator) did not create better routes than those using passive Sensators (feedback originated from the touch-screen table) or those in the touch-screen condition (without Sensators). However, participants did create more routes in the active Sensator than in the passive Sensator and touch-screen conditions in the first six minutes of the planning task. We concluded that Sensators can provide support for the number of alternative plans that can be made during a planning task, but not necessarily result in better outcome when ample time is given for the task. We discuss improvements of the experiment that could potentially show support of other facets of planning, as well as future research that could provide a more direct measure of the supporting capacities of Sensators.
Introduction Background
Planning (a goal-directed sequence of behavior, Dehaene & Changeux, 1997) happens all the time, from air traffic control to grocery shopping. In order to successfully perform planning spatial and sequential relations have to be processed. In complex planning tasks these relations are high in number and/or complicated in nature – in this case human working memory may be too limited to compromise them all. Finding new means of support might be very helpful in cases where planning tasks are difficult and mistakes cause dangerous situations. Here we test the hypothesis that Sensators (interactive tangibles that provide multisensory feedback, Van Erp, Toet, & Janssen, 2013) facilitate planning.
In this section, we will first discuss how working memory plays an important role in planning and how multisensory feedback can support working memory.Then, we will discuss the embodiment of cognition, where means of support of complex thought processes like planning can be found as well.
Lastly, we introduce Sensators, which show both integrated multisensory feedback and support for embodied cognition, making them interesting candidates to provide cognitive support for planning.
Working memory. Multiple cognitive models of planning stress the importance of working
memory, the type of memory that actively holds information for a short period of time (Baddeley, 1983). One of them is the connectionist neural network model of sequence planning (Dehaene & Changeux, 1997). It relies on three neural systems: reward units that evaluate the status of the plan, plan units that generate the plan, and working memory. The latter maintains the information about the previous problem state, while the current plan is tested. It mimics behavior of healthy participants quite well, in solving the Tower of London (ToL) test. The ToL test is a puzzle in which beads sliding on pegs must be moved to match a designated goal configuration, which is often used to measure (deficits) in planning performance (Dehaene & Changeux; Newman, Carpenter, Varma, & Just, 2003). The model also simulates the behavior of patients showing working memory impairments because of frontal lesions, which is characterized by lower accuracy and increasing solution time.
Second, 4CAPS (cortical capacity–constrained collaborative activation-based production system) is a production system architecture with several connectionist features (Newman et al., 2003). The model differs from the neural network model of sequence planning described above in that it makes predictions about behavioral characteristics of participants performing the ToL test, but also for the amount of brain activity as measured by functional neuroimaging. The 4CAPS model distinguishes two cognitive processes. First, executive processes are represented in the dorsolateral prefrontal cortex (DLPFC), and include the formulation of a plan as well as controlling the execution of this plan. The second covers spatial processes, represented in the superior parietal cortex, which can be divided in perceptual control and, importantly, the visuo-spatial workspace. This visuo-spatial workspace maintains spatial representations or intermediate configurations that are produced in the left DLPFC, a description akin to working memory. Empirical evidence in support of this model comes from a strong correlation between reaction times and brain activations as predicted by the model, with human reaction times and brain activity as measured with fMRI, as a response to ToL tasks with increasing difficulty.
Last, the PELA (Planning, Execution, and Learning Architecture) model integrates a relational learning component with traditional planning and execution monitoring components (Jiménez, Fernández, & Borrajo, 2013). Planning the action is initially based on a classical (deterministic) planner, but is updated according to the probabilistic knowledge about the outcome of the execution of the plan, i.e. learning. At the start of the simulations, PELA performed not worse than
classical planning models. Later in the processes (when the model had ‘learned’) PELA addressed probabilistic planning problems better than classical models did. This indicates that learning processes play a significant role in planning. According to the cognitive overload theory (De Jong, 2009), these learning processes are, once again, linked with working memory: when its capacity is exceeded, learning will be hampered.
Evidently, working memory is tightly linked to planning. Indeed, when lesions in the frontal cortex result in decreased working memory capacities, planning performance is reduced as well (Owen, Downes, Sahakian, Polkey, & Robbings, 1990). Supporting working memory might thus indirectly enhance human planning performance. A means to enhance this capacity is to make use of the modality principle of working memory (De Jong, 2009). This principle is based on an essential feature of both Baddeley’s working memory model (Baddeley, 1983; 1992) and Paivio’s dual-coding theory (Paivio, 1991): the human information-processing system consists of two separate channels. These are the visuo-spatial sketchpad (for processing visual/pictorial information) and the phonological loop (for processing auditory/verbal information). Although both are coordinated by a ‘central executive’, on modality processing level they function largely independently; making use of multiple modalities might therefore expand the capacity of working memory.
Generally, providing multimodal feedback, as compared to unimodal feedback, leads to: faster reaction times (Hershenson, 1962), improved performance in perceptual discrimination tasks (Teder-Sälejärvi, McDonald, Di Russo, & Hillyard, 2002), and has overall benefits in perceptual and cognitive processing (Lee & Spence, 2008). Participants performed better in a complex visual car avoidance and combined mobile phone task, when feedback was multimodal (auditory and visual; tactile and visual; auditory, tactile and visual), as compared to unimodal (visual only) (Lee & Spence, 2008). A meta-analysis was performed on experiments in which baseline visual conditions were compared with vibrotactile cues providing additional information (Prewett, Elliott, Walvoord, & Coovert, 2012). Tasks included land navigations and virtual reality tasks. When added to baseline, vibrotactile cues enhanced task performance. Lastly, when planning tasks were increased in difficulty, users choose to interact multimodally more often (Oviatt & Lunsford, 2004). They argued that users spontaneously respond to dynamic changes in their own cognitive load by shifting to multimodal communication, in order to self-manage the limitations of their working memory.
Subjectively, multimodal feedback reduces perceived cognitive workload (Lee & Spence, 2008), which is tightly linked with working memory (De Jong, 2009). According to the cognitive load theory, cognitive capacity in working memory is limited; when too much capacity is required, overload occurs (De Jong, 2009). According to Mayer and Moreno (2003), cognitive overload originates in multiple domains. Importantly, the ‘domain of essential processing’ consists of selecting input,
organizing verbal or pictorial input, and integrating this information. Regarding this domain, the modality principle might offer an opportunity to reduce overload: visual information could be transferred to auditory input which might reduce the visual load.
More direct measures of working memory include the serial recall task, where a sequence of symbols is presented after which participants are asked to recall them. Penney (1989) showed multiple examples of higher recall after bisensory presentation of sequences of words (verbal and written) as compared to unimodal presentation (verbal). Specifically, Delogu, Raffone and Belardinelli (2009) measured participants response accuracy in a serial recall task including visual sequences of pictures versus bimodal sequences of pictures accompanied by sounds representing the same object. Participants recalled more symbols in the bimodal task than the unimodal condition, again suggesting that bimodal feedback results in enhanced memory performance due to increased working memory capacities. Furthermore, Mousavi, Low and Sweller (1995) showed that when presenting students with geometry problems, performance was better when the diagram was presented visually, and the problem statements verbally (auditory). Importantly, the researchers figured this task demanded holding diagrammatic information as well as information about the statements in working memory. Lastly, the n-back task (Kirchner, 1958) is often used to measure working memory. A sequence of symbols is presented; participants indicate whether the current symbol is equal to a symbol that was presented n symbols earlier. Response accuracy improved when symbols were shown bimodaly (auditory: hearing a number, and visually: seeing a mouth producing this sound) opposed to unimodaly (when the number was only presented verbally) (Frtusova, Winneke, & Phillips, 2013).
Additionally, integration of information is thought to reduce incidental processing; processes that are primed by the design of the task (Mayer & Moreno, 2003). By reducing the negative impact of unintegrated information representation (by presenting this information close to each other), working memory has to maintain less information. Ideally, information that is received during complex tasks, like planning, should thus be multisensory and spatially integrated.
Embodied cognition. The theory of embodied cognition is based on the idea that cognition is
deeply rooted in the body’s interactions with the world (Wilson, 2002). Instead of being abstract and centralized, human cognition is thought to be based on sensorimotor processing, because humans have evolved from species whose brains were used only for perceptual and motoric processing.
This idea is supported by the strong link between motor and perceptual processes (the same cells are involved in both action and perception); the fact that tactile and visual perception require movement of the body (moving hands to touch something; moving eyes and head in order to see);
and that actions can change the world that is being perceived (Hostetter & Alibali, 2008). Furthermore, thinking about a process results in the activation of perceptual and motor neurons involved in performing that process. Lastly, mirror neurons appear to fire both when perceiving an action and producing that action. Hostetter and Alibali (2008) hypothesize that planning movement is performed by the same perception and action mechanisms that are used for perceiving and moving through the world as well.
Wilson (2002) critically assessed some implications stemming from the embodiment of cognition. Off-loading is one of the few processes that passed his critique. During off-loading, the environment is used to reduce cognitive workload (Mayer & Moreno, 2003); information is left in the world until it is needed. Alternatively, epistemic actions (Kirsh & Maglio, 1994) can be used for off-loading as well. Here the environment is altered, in order to reduce cognitive work without coming closer to a solution directly. An example of these epistemic actions comes from the Tetris game, where players use the rotation and translation movements to simplify the task, instead of mentally store or manipulate the information (Kirsh & Maglio, 1994). Off-loading can happen spatially (drawing a blueprint), but also more symbolically (making a Venn diagram). According to this theory of embodied cognition, offline processes take place without connection with the environment, whereas online processes do interaction with the environment. Off-loading happens off-line when the simulation of physical events is reduced to sensorimotor resources in working memory. When working memory capacities are exceeded however, online interaction occurs, when information is off-loaded into the environment.
According to the same theory, gestures – instead of having a mere communicative function – might help speakers formulate their speech by supporting lexical retrieval (Krauss, 1998). Gestures reflect the spatio-dynamic features of that what one is trying to say, and by making these gestures the retrieval of words is supported. Additionally, according to Hostetter and Alibali (2008), who base their hypothesis on embodied cognition as well, gestures are the overt realization of mental images that are activated by speaking. They state that both language and mental imagery are based on the simulation of action and perception, instead of being derived from abstract symbols. The fact that reading words results in activation in brain areas responsible for the motoric implication of those words supports the action-perception simulation basis of language. Concerning mental images; these retain the same properties as the physical entities they represent, and motor imagery relies on neural mechanisms that are involved producing the actual movement. So, simulating an action activates neural areas responsible for planning the physical execution of this action (premotor cortex). When mental imagery, activated by speaking, results in such a simulation, the concurrent neural activation has the potential to spread to motor areas. This leads to an overt action; a gesture.
Supporting those gestures could potentially support the thought process they represent, which in our case would be thought processes concerning a planning task.
Another advantage of supporting gestures is, that according to the ‘image maintenance theory’, gestures can activate images in working memory (Wesp, Hesse, Keutmann, & Wheaton, 2001). The images that normally decay quickly, are reactivated by gesturing. As shown earlier, supporting working memory might indirectly support planning. Evidently, offering an environment where information can be off-loaded easily, and that stimulates bodily movement, might support complex mental processes, like planning tasks.
Sensators. Tangibles, and especially Sensators, have precisely the planning supporting properties
that are discussed: the information that is provided is integrated and multisensory, on media that are manipulatable and movable by hand. Traditionally, interaction between user and computer is based on graphical user interfaces (Fitzmaurice, Ishii, & Buxton, 1995). Graphical elements (like windows, menus and icons) are manipulated by haptic input devices (mouse and keyboard). Ishii introduced an alternative; the Tangible User Interface (TUI) (Ishii, 2008). Here digital information is controlled and represented by physical objects that are manipulatable and perceivable by hands. These ‘tangibles’ provide haptic (force based; experienced when lifting objects, feeling weight), and sometimes tactile (touch based; feeling texture, or vibration) feedback.
Most of the time, TUI are placed on a table, creating a table top tangible interface, or TTI (Ullmer & Ishii, 2000). An early example of a TTI is Urp (Underkoffler & Ishii, 1999), where tangibles representing optical artifacts (e.g. laser sources, mirrors) are placed on a rear-projected tabletop display, changing visual aspects depicted here (shadows, reflection). A more entertaining implementation is the reacTable (Jordà, Geiger, Alonso, & Kaltenbrunner, 2007) where music artists use acrylic square plates, representing sound controlling parameters, to create music; visual feedback about state and connectivity between the tangibles is again provided by rear-projection on the table. Alternatively, transparent tangibles have the advantage that information on the screen is not visually blocked, and can represent keyboards, knobs and sliders (Weiss, Hollan, & Borchers, 2010).
Contrary to the one-way direction of information in these passive tangibles, in the Planar Manipulator Display (Rosenfeld, Zawadzki, Sudol, & Perlin, 2004), tangibles are represented by mobile wireless robots, which can be moved by hand but also rearrange themselves according to demands of the software. Tangible Bots (Pedersen & Hornbaek, 2011) do the same and can be used to assist users by correcting errors and providing haptic feedback. Tangibles in the Pico
implementation (Patten & Ishii, 2007) represent and can be used to change parameters inside the software, but can also move by the aid of electromagnets in order to optimize those parameters.
Subjective advantages of TUI include a positive overall experience (Underkoffler & Ishii, 1999); the possibility to use both hands (Van Erp, Toet & Janssen, 2013), greater adaptability of control (Tuddenham, Kirk, & Izadi, 2010), usefulness for creating music, and lower perceived workload (Pedersen & Hornbæk, 2011). To compare TUI with a multi-touch surface, performance in manipulation and acquisition tasks was compared between the aid of a tangible ruler and a multi-touch screen solely (Tuddenham, Kirk & Izadi, 2010). Interface control objects were easier to acquire in the tangible condition, and, once acquired, were more accurately manipulated. When comparing active tangibles (tangibles that moved) with their passive state, users performed more accurately at rotation tasks but worse in coarse movements (Pederson & Hornbaek), and were better in solving complex spatial layout problems (Patten & Ishii).
The tangible that is used in the current experiment, the ‘Sensator’, is a TTI that provides multisensory (visual, auditory and vibrotactile) feedback (Erp, Toet & Janssen, 2013). It is used in combination with a multi-touch tabletop. A number of experiments have been carried out using it. For example, participants had to perform spatial alignment and rotation tasks with Sensators that either provided no feedback, or unimodal (visual, auditory or vibrotactile) feedback (A. Toet, personal communication, September 3, 2013). Participants performed better when receiving unimodal feedback, as compared to participants receiving no feedback. With no feedback, subjectively perceived performance was lowest and perceived mental workload was highest. In addition, when Sensators provided increased signal rate and decreased inter pulse interval length, perceived urgency increased (Erp, Toet & Janssen, 2013). These effects were universal across the three different modalities, with bimodal and trimodal feedback being perceived as more urgent than unimodal feedback.
Current Experiment
Because Sensators support working memory by offering feedback in an integrated multisensory way and because their haptic nature supports the embodied nature of cognition, they might facilitate planning. Here we investigated whether participants were able to generate more and better planning options when using active Sensators. A virtual supermarket was presented on a multi-touch computer screen, which could either be manipulated by hand, with the aid of active Sensators (providing feedback themselves) or passive ones (feedback originated from the screen). Active Sensators represented both embodied cognition support (because they had to be picked up
by hand) as well as multimodal integrated feedback, whereas passive Sensators provided embodied cognition support only, and no integrated multimodal feedback. We therefore hypothesized that the use of Sensators results in better planning performance than using the touch-screen only, and that using active Sensators results in better planning performance than using passive ones.
We predicted that this increase in planning performance between touch-screen, passive and active Sensators would be represented in three performance measurements: shorter routes, more alternative routes, and higher number of individual movements of products. Furthermore, we predicted workload would decrease when comparing passive Sensators with touch-screen, and active with passive Sensators. Lastly, we expected to find a learning effect: shorter routes, increased number of routes and increased number of movements when trial number increased. This effect is normal in tasks like these and is not very surprising. However, we predicted that there would be a stronger learning effect in the Sensator conditions than in the touch-screen condition.
Method Participants
Thirty-six participants (21 male, 15 female; age range 19-32 years, mean age = 26, SD = 3.3) participated in the experiment. They were randomly assigned to one of the conditions. All of them provided written informed consent to participate in this study, and they received €30 as compensation for their participation. Most participants were gathered trough the TNO-database, some were students at the University of Amsterdam. All participants had normal or corrected-to-normal vision and were naïve to the experimental hypothesis.
Material
Multi-touch table. With the Samsung Surface 40 (SUR 40) digital content could be manipulated
by multi-touch hand movements, or by using tangibles, see Figure 1 (Samsung, 2014). The horizontal positioned screen has a diameter of 102 cm and a resolution of 1920 x 1800 pixels. The SUR 40 includes an AMD Athlon X2 Dual-Core processor and an AMD Radeon HD 6750M graphics card, and runs on Windows 7. A LAN connection allowed the computer to send signals intended for the Sensators to two laptops, which then sent these signals to six Sensators each, through Bluetooth connections.
Figure 1. The Samsung Surface 40 (Samsung, 2014)
Sensators. The 12 Sensators used in this experiment were developed by TNO, within the ‘Intuitive
User Interfaces’ project, see Figure 2. Each has a width and length of 6.5 cm and height of 5 cm, and includes a custom Arduino TUI Printer Circuit Board, and an Amtel AVR processor. Two vibrating motors inside allow for vibrotactile feedback; an RGB LED enables the Sensator to show different colors; and an mp3 audio processing shield allows for the Sensators to play mp3 files.A Bluetooth communication shield enabled a connection with one of the two laptops.
During this experiment we labeled each Sensator with stickers, showing the name of the product it was presenting. Markers on the bottom of each Sensator enabled the touch-screen to track the location and number of each Sensator.
Figure 2. One of the Sensators.
The supermarket game. The planning game used in the experiment was based on the SPLITS
(Simulation for Planning In Time and Space) environment (McCann & Essens, 1991). A list of groceries had to be collected in a (virtual) super market. Participants had two robots, a fast robot with a small basket and a slow robot with a big cart that could be used to pick up products. Their task was to find out in which order the products must be collected so that the best combination of routes was created. This was based on the total time both robots took to finish their route (i.e. adding up the time of the two individual routes).
There were two complicating factors:
1) Some products were vulnerable: eggs, kiwi, toast and wineglass. These products would break when one of the heavy products were placed on top of them: water, orange soda, coca cola and milk. The remaining products were neutral (not vulnerable, not heavy) and could be placed everywhere: coffee, detergent, soap and spaghetti.
2) The robot with the basket was twice as fast as the other. However, the capacity was more limited than the cart: the cart could hold 10 products and the basket only 5.
Software application. The software running on the SUR 40 showed a map of a store (right), a
shopping cart and basket (middle left) and the score information (left), see Figure 3. Two lines were depicted on the map of the store, running from the entrance to the exit of the supermarket map; a red line for the basket and a blue line for the cart. At the beginning of the experimental trials 12 products were displayed on the map of the supermarket. The order in which these products were placed in the cart and basket determined the route of the two lines; changing the location of one of these products caused the route of the blue or red line to be updated instantly. Visual feedback corresponded to the location of a product: red when placed in supermarket, yellow when picked up, and green when placed in the basket or cart. The score information section showed (from top to bottom) the added up time both robots took to pick up all the products; the percentage of product that was uncrushed; and the number of products that were put in the basket and/or cart. The first time in the trial that the ‘Click me’ button was pushed, these numbers would show; consecutive button presses updated the information according to changes in the product order.
Three variables were automatically logged by the software application, after 6 minutes (at the middle of the trial), and after 12 minutes (at the end of the trial), resulting in six measures within each trial. The measure Score was calculated by taking the shortest of all the alternative routes that the participant had created, and subtracting the optimal score from this. This was done to be able to compare scores between different trials – the optimal score differed between trials because of different product placement configurations. This means a low number in the measure Score equals a good score; high numbers represent bad scores, that were far off the best score to beat. The measure Clicks was based on the number of times that people pushed ‘Click me’, but only in case this followed a route that covered all the products and conformed to the restrictions. Lastly, the amount of times a participant changed the location of a product was logged individually for every type of movement: from supermarket to cart, from supermarket to basket, from cart to supermarket, from cart to somewhere else in the cart, from cart to basket, from basket to supermarket, from basket to cart, and from basket to basket. These individual types of movements
were used during post-hoc analysis; the measure Movements was created by taking the total number of times people changed the location of a product.
Figure 3. Software application. The user has placed three products in basket, and three in cart;
spaghetti is just picked up.
Questionaires. We used NASA TLX software, based on the NASA Task Load Index, to measure
subjective workload (Cao, Chintamani, Ellis and Pandya (2009) through the six subscales: mental demand (MD), physical demand (PD), temporal demand (TD), performance (PF), effort (EF) and frustration level (FR). The participants marked their answer on a 20 point scale, corresponding to bipolar descriptions of the two endpoints.
Design
The supermarket task was performed with three different feedback types, touch-screen, passive Sensator and active Sensator. Participants were divided over these three groups according to a between-subjects design, covering 12 participants per condition. In the touch-screen condition, virtual icons on the screen represented the products in the supermarket. The touch-screen table provided feedback: when products were placed in the basket/cart, their color changed; when vulnerable products were broken, a sound was played from the speaker in the touch-screen table. In the passive Sensator condition, products were represented by the Sensators, which were placed on top of the corresponding icons on the screen. Feedback was presented the same way as in the touch-screen condition. Lastly, in the active Sensator condition, Sensators represented products and
provided the feedback themselves. Sensators changed color when picked up and again when placed in the basket/cart; they played a crushing sound and vibrated when a product was broken (both the smashing and smashed Sensator provided this feedback).
Procedure
When the participants arrived, they first read the task description and signed the informed consent. The test leader then verbally explained the task again and emphasized that they could hit ‘Click me’ as often as they wanted, and that they had enough time to try multiple routes and try to optimize those routes to get the shortest combination possible.
During the experiment the participants performed six tasks; each task had its own unique product placement configuration. There were three practice tasks. In task 1, only the basket was used. A list was shown of 5 products that had be to picked up. There were no constraints: crushing products was allowed and did not result in any feedback. In task 2 only the cart was used, to pick up 9 products. Constraints were introduced: heavy products should not be placed on top of the vulnerable products. In the third and last practice task both the cart and the basket were used, so that the goal was to figure out two routes showing the lowest total time when added up together. All 12 products had to be picked up. Now constraints were excluded: heavy products could be placed on vulnerable products.
After the participant finished the practice tasks, the three experimental trials were performed. In every trial both the cart and basket were used; a total of 12 products had to be picked up, taking into account the constraints that were introduced in task 2. Every trial took 12 minutes. During the three practice tasks, the test leader was present in the room; during the experimental task he was in the room next to it. After completing all tasks, the participants filled out the TLX.
Results Data-analysis
Three participants were excluded from further analysis, because their data showed extreme outliers for their score in almost all trials, which we interpreted as an indication that they did not understood the task well enough to perform it. This left 33 participants, 11 per condition.
We conducted six mixed between-within subjects analysis of variance to assess the impact of feedback type (touch-screen, passive Sensator and active Sensator) on Clicks, Movements and Score, across the three different trials. This was done for the two points of measurement (6 and 12 minutes
after start) independently. Since TLX was measured only once per participant, we conducted one-way between-groups analysis of variance to explore the effect of feedback. Because this results in seven different analysis, significance level was Bonferonni corrected: 0.05/7 = .0071.
A number of assumptions were not met. First of all, for Clicks at 6 minutes, Box’s test of equality of covariance matrices was significant at the .05 level, Box’s M = 28.961, F(12, 3289.77) = 2.01, p = .02. However, since it is advised to use a .001 significance level for this test (Field, 2009), further analyses was carried on. For Score at 6 minutes assumptions for homogeneity of variance was not met for trial 1, F(2, 28) = 5.18, p = 0.012; and 2, F(2, 28) = 6.84, p =.004, excluding the possibility to perform between-subjects analyses. Therefore, we transformed all the Score data according to the natural log procedure (log(X+1)), the only transformation that resulted in homogenic variance for all variables. However, transformed 6 minute Score data violated the assumption of equality of covariance, Box’s M = 40.64, F (12, 3439.12) = 2.84, p = .001, as well as the assumption of sphericity,
χ2(2) = .71, p .001. Additionally, transformed 12 minute Score data violated the assumption of covariance as well, Box’s M = 59.94, F(12, 1998.69) = 3.99, p < .001. Because transformed data resulted in more violations than untransformed data, initially we concluded not to use the transformation, and to drop further analysis of between-subject effects for Score after 6 minutes. Lastly, for Score at 12 minutes (untransformed), Mauchly’s test indicated that the assumption of sphericity was violated χ2(2) = .71, p<.05, therefore Greenhouse-Geisser was reported.
Instead of trying to optimize the current route by shuffling some products, participants sometimes used the strategy to place all of the products back into the supermarket and start over again. As a post-hoc analysis, we investigated the effect of feedback type on the number of times participants used this strategy. For this, we calculated the number of movements from basket to supermarket plus movements from cart to supermarket. When this number was higher than 10, this was taken as an indication that the participant had used the ‘put every product back into the supermarket’ strategy prior to pushing ‘Click me’. In this way, we gathered the total amount of Clicks that followed this strategy per participant per trial, which resulted in the variable ‘Replacements’. We conducted two mixed between-within subjects analysis of variance to assess the impact of feedback type on Replacements, across the three different trials. The first was based on measurements after 6, the other after 12 minutes. Box’s test of equality of covariance matrices was significant at the .05 level, for Replacement measured after both 6 minutes, Box’s M = 46.23, F(12, 4361.538)= 3.29, p < .001, as well as after 12 minutes, Box’s M = 50.77, F(12, 4361.54) = 3.61, p < .001. For Replacements measured after 6 minutes, the assumption of sphericity was violated, χ2(2) = .675, p = .003. Therefore we transformed Replacement score according to the natural log procedure (log(X+1)). After this transformation, every assumption was met.
We performed another post-hoc analysis, on the effect of feedback type on Score (measured after 6 minutes) which was initially excluded from further analysis because of violation of assumptions. For this, we averaged Score over the trials, which made it possible to perform a one-way between-groups analysis of variance. Here as well, homogeneity of variance was not met, F(2, 30) = 4.79, p = .016; therefore Brown-Forsythe was reported.
Score
Note that we initially excluded the between-subject effect of feedback type on Score, measured after 6 minutes, from analysis. We could still analyze differences between trials, which showed a large main effect, F(2, 46) = 10.36, p <.001, partial eta squared = .27. This is the learning effect that we predicted: participants performed better when they are becoming more experienced with the task. Contrasts revealed a linear relation (participants received higher scores in trial 3 than trial 2, and higher in trial 2 than trial 1), F(1, 28) = 20.54, p < .001. There was no significant interaction between feedback type and trial, F(4, 56) = .995, p = .418, indicating learning effects are equal in every condition, in contrast with the prediction.
Score, measured after 12 minutes, did not differ between feedback types, F(2, 23) = .87, p = .07, see Figure 4. Based on this, the prediction could not be confirmed. There was a large main effect for trial number, F(2,46) = 7.74, p = .001, partial eta squared = .25, as predicted. Contrasts revealed the same linear effect as the one that was found after 6 minutes, indicating that the learning effect persisted after 6 minutes, F(1, 23) = 9.18, p = .006. Lastly, there was no significant interaction between feedback type and trial, F(4, 46) = 1.42, p = .241.
Figure 4. Score (+/- SE) per condition. Low number means good score, or shorter route.
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
Touch-screen Passive Sensator Active Sensator
Sco re (m ill iseco n d s)
Clicks
For the number of Clicks, measured at 6 minutes, there was a large main effect of feedback type
F(2,27) = 6.19, p = .006, partial eta squared = .31, see Figure 5. Contrasts revealed a linear direction:
participants hit ‘Click me’ more often in the active Sensator condition than in the Passive Sensator condition, and more often in the Passive Sensator than in the Touch-Screen condition. This indicates that using Sensators results in more optimizations, and that active Sensators result in a higher number of optimizations than passive ones. This confirmed our prediction. There was no main effect of trial number, F (2, 54) = 3.92, p = .026, and no significant interaction between feedback type and trial, F (4, 54) = 1.0, p = .40. This indicates that the effect of feedback type on number of Clicks did not differ between trials.
For the number of Clicks, measured at 12 minutes, the differences between feedback types was not significant, F (2, 30) = 1.07, p = .36. On this basis, the prediction could not be confirmed. Furthermore, there was a large main effect of trial number, F(2, 60) = 6.84, p = .002, partial eta
squared = .19, see Figure 6. Contrasts revealed a linear relation (participants pushed Click me more
often in trial 3 than in trial 2, and more in trial 2 than trial 1), F(1, 30) = 13.79, p = .001. This indicates that participants created more routes as they got more experienced with the task, as predicted. There was no significant interaction between feedback type and trial, F (4, 60) = 1.69, p = .165, indicating that this increase number of routes did not differ between trials.
Figure 5. Number of Clicks (+/- SE) per condition.
0 2 4 6 8 10 12 14
Touch-screen Passive Sensator Active Sensator
N u m b e r o f Cl ic ks
Movements
There was no significant effect of feedback-type on the number of movements measuring at 6 minutes, F(2, 30) = .65, p = .528. This did not support the prediction. We found no effect for trial number either, F(2, 60) = 2.99, p = .057, and no significant interaction between feedback type and trial, F(4, 60) = .82, p = .517. See Figure 6.
For the number of movements that were measured at 12 minutes, there was no main effect for feedback type, F(2, 30) = .29, p = .753. There was no main effect for trial number, F(2, 60) = 3.43, p = .039, and no interaction between feedback type and trial, F(4, 60) = .6 5, p = .629.
Figure 6. Number of individual Movements (+/- SE) per condition.
TLX
The effect of feedback type on TLX score showed not to be significant, F(2, 30) = .26, p = .774.
0 10 20 30 40 50 60 70 80 90 100
Touch-screen Passive Sensator Active Sensator
N u m b e r o f M o ve m e n ts
After 6 minutes After 12 minutes
0 5 10 15 20 25 30 35 40 45 50
Touch-screen Passive Sensator Active Sensator
TLX
sco
Post-hoc
Replacement strategy. For the number of times participants used the replacement strategy when
measured after 6 minutes, there was no effect for feedback type, F(2, 30) = .84, p = .443. We found no effect for trial number either, F(2, 60) = 1.71, p = .189, and no significant interaction between feedback type and trial, F(4, 60) = .42, p = .796.
When measuring after 12 minutes, we found no effect of feedback type, F(2, 30) = .82, p=.452. There was a main effect for trial number, F(2, 60) = 3.67, p = .031, and no significant interaction between feedback type and trial, F(4, 60) = .60, p = .664.
Average Score. When Score was measured after 6 minutes and averaged over the 3 trials,
differences between the feedback types were not significant, F(2, 20.55) = .71, p = .502.
Discussion and Conclusion
Because Sensators provide both multisensory feedback and haptic, embodied interaction possibilities, they could support working memory. Since working memory plays a role in planning processes, these features could indirectly facilitate planning. To measure planning performance, efficiency as well as number of routes were gathered, together with the number of small modifications that subjects made in these routes. We hypothesized that planning performance would be best for active Sensators, intermediate for passive Sensators, and worst for the touch-screen condition. This pattern of results was present for the number of generated routes after 6 minutes but disappeared after 12 minutes. We found no effects on route efficiency and the number of small modifications. We also looked at the size of the learning effect, expecting a larger effect for the Sensator conditions compared to the touch-screen condition. Although for some measures a learning effect was indeed found, the size of the effect was equal for all conditions. Overall, we see that our predictions are partially confirmed while we don’t find either a negative nor positive effect on other performance measures.
An alternative explanation for the number of generated routes may be that the measure Clicks represents confusion instead of planning quality. After creating a new route, by changing the order of some products, participants had to push the button to see how short their new route was, which is why we took this as a measure of the number of routes that were created. However, one might argue that when a participant loses track of the current route, pushing ‘Click me’ gives an indication of how well he or she is doing. Following this interpretation, participants using active Sensators
would be the most confused, those using passive ones less, and those using touch-screen the least confused. However, this alternative explanation is not very likely, because participants pushing ‘Click me’ more often did not perform worse.
The question remains why the favorable effect of Sensators on the number of generated routes is only present in the first half of the trial. We argue that this is due to a ceiling effect. Participants did create an average of 3.8 routes in the first part of the trials. As a result, during the second half of the trials the pressure to create routes was decreased. A number of participants did indeed ask the test leader if they could go on to the next trial, because they felt they had created sufficient routes in the current trial. An interesting modified version of this experiment would therefore be a time pressured one, where participants are allowed only 6 minutes to complete the task, or even less.
Taken together, it can be concluded that evidence was found indicating that the usage of Sensators supports the creation of more alternative plans during a planning task without increasing experienced workload. However, the creation of more routes did not necessarily result in finding a more optimal routs. This may have been caused by the fact that the task was not complex enough; i.e. participants easily found an optimal route after a few alternatives and more tries could not improve their score further or only to a very limited (and non-significant) extent.
Another possibility is that the positive effects of the Sensators was partly nullified by the current apparatus. When active, Sensators provided visual, auditory and tactile feedback. The signal to change their color was based on whether the tag at the bottom of the touch-screen table was recognized. Because of limitations of the hardware of this system, sometimes when the Sensator was placed on the table the tag was sequentially detected and undetected. As a result, the touch-screen table sent a flashing command to the Sensators, which would then change colors again and again. The frequency of this color shift was about 3 Hz. This continued until the Sensator was picked up and placed again so that the touch-screen detected the tag correctly, i.e. continuously. The auditory and tactile feedback showed the same flashing pattern: the audio file playback was started over and over and the Sensator vibrated sequentially. Because the screen resonates slightly, in some cases the vibration caused by malfunctioning tag recognition resulted in the movement of other Sensators as well. When these Sensators impaired their own tag recognition, a new flashing pattern was evoked – a snow ball effect. Although before onset of the test the participants were told to simply move flashing Sensator(s) again, and to ignore some incidental flashes, many participants reported post-test that these flashing pattern distracted them from the task. This distraction could have seriously reduced the number of their routes and movements, and could have resulted in a score that is lower than what they could have achieved without this distractions. Without this small
hardware malfunction participants in the active Sensator condition could have produced higher scores, resulting in a significant effect of feedback type on Clicks, Movements and Scores.
Third and last, although the ecological validity of this experiment is high, there are many variables influencing the results. The potentially positive effects of the combination of multimodal feedback and haptic nature provided by the Sensators on working memory, were investigated indirectly by looking at planning. Therefore, a more direct measure of the link between embodied cognition and working memory (an aspect of planning) should be investigated directly in future research. Currently, this research is being conducted at TNO, Soesterberg, by using Sensators to perform a working memory task, in which the amount of embodiment varies. Here, the n-back task (Kirchner, 1958) is performed by playing a number of verbal letters out of the Sensators. The task is performed using one Sensator (baseline), a circle of Sensators that have to be touched by hand (adding bodily components), and the same circle where each Sensator playing the current letter has to be moved by hand (the most embodied condition). The last condition might offer the highest possibility of off-loading, followed by the middle condition. When off-loading indeed supports working memory, n-back memory scores will be better in these conditions, as compared to baseline, where no off-loading opportunities are provided.
The current experiment showed that using Sensators can increase the number of alternative plans that are made; an improved version of the experiment could show more possible benefits. By looking directly at the potential positive influence Sensators have on working memory, an important subcomponent of the main research question could be answered as well. The possibilities of new human-computer interactions like Sensators are vast, and they provide interesting ways of supporting demanding cognitive tasks.
References
Baddeley, A. (1992). Working Memory. Science, 255, 556–559.
Baddeley, A. D. (1983). Working memory. Philosophical Transactions of the Royal Society of
London, 302, 311–324.
Dehaene, S., & Changeux, J. P. (1997). A hierarchical neuronal network for planning behavior. In
Proceedings of the National Academy of Sciences of the United States of America (Vol. 94, pp.
13293–13298).
Delogu, F., Raffone, A., & Belardinelli, M. O. (2009). Semantic encoding in working memory: Is there a (multi)modality effect? Memory, 17(6), 655–663.
Fitzmaurice, G. W., Ishii, H., & Buxton, W. (1995). Bricks: Laying the foundations for graspable user interfaces. In ACM Proceedings of CHI’95 (pp. 442–449).
Frtusova, J. B., Winneke, A. H., & Phillips, N. A. (2013). ERP evidence that auditory-visual speech facilitates working memory in younger and older adults. Psychology and aging, 28(2), 481– 494.
Hershenson, M. (1962). Reaction time as a measure of intersensory facilitation. Journal of
Experimental Psychology, 63(3), 289–293.
Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: Gestures as simulated action.
Psychonomic Bulletin & Review, 15(3), 495–514.
Ishii, H. (2008). The tangible user interface and its evolution. Communications of the ACM, 51(6), 32–36.
Jiménez, S., Fernández, F., & Borrajo, D. (2013). Integrating planning, execution and learning to improve plan execution. Computational Intelligence, 29(1), 1–36.
Jordà, S., Geiger, G., Alonso, M., & Kaltenbrunner, M. (2007). The reacTable: Exploring the synergy between live music performance and tabletop tangible interfaces. In TEI ’07: Proceedings of
the 1st international conference on tangible and embedded interaction (pp. 39–146).
Kirchner, W. K. (1958). Age differences in short-term retention of rapidly changing information.
Journal of Experimental Psychology, 55(4), 352–358.
Kirsh, D., & Maglio, P. (1994). On distinguishing epistemic from pragmatic action. Cognitive
Science, 18(4), 513–549.
Krauss, R. M. (1998). Why do we gesture when we speak? Current Directions in Psychological
Science, 7, 54–59.
Lee, J., & Spence, C. (2008). Assessing the benefits of multimodal feedback on dual-task performance under demanding conditions. In Proceedings of the 22nd British HCI Group
Annual Conference on People and Computers: Culture, Creativity, Interaction (Vol. 1, pp. 185–
192).
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning.
Educational Psychologist, 38(1), 43–52.
Mousavi, S. Y., Low, R., & Sweller, J. (1995). Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 87(2), 319–334.
Newman, S. D., Carpenter, P. A., Varma, S., & Just, M. A. (2003). Frontal and parietal participation in problem solving in the Tower of London: fMRI and computational modeling of planning and high-level perception. Neuropsychologia, 41(12), 1668–1682.
Oviatt, S., & Lunsford, R. (2004). When do we interact multimodally? Cognitive load and multimodal communication patterns. In Proceedings of the 6th international conference on
Multimodal interfaces (pp. 1–8).
Owen, A. M., Downes, J. J., Sahakian, B. J., Polkey, C. E., & Robbings, T. W. (1990). Planning and spatial working memory following frontal lobe lesions in man. Neuropsychologia, 28(1), 1021–1034.
Paivio, A. (1991). Canadian Journal of Psychology. Canadian Journal of Psychology, 45(3), 255–287. Patten, J., & Ishii, H. (2007). Mechanical constraints as computational constraints in tabletop
tangible interfaces. Proceedings of the SIGCHI conference on Human factors in computing
systems - CHI ’07, 809.
Pedersen, E. W., & Hornbæk, K. (2011). Tangible bots: Interaction with active tangibles in tabletop interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2975–2984).
Penney, C. G. (1989). Modality effects and the structure of short-term verbal memory. Memory &
cognition, 17(4), 398–422.
Prewett, M. S., Elliott, L. R., Walvoord, A. G., & Coovert, M. D. (2012). A meta-analysis of vibrotactile and visual information displays for improving task performance. IEEE
Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(1),
123–132.
Rosenfeld, D., Zawadzki, M., Sudol, J., & Perlin, K. (2004). Planar manipulator display. IEEE
Computer Graphics and Applications, 24(1), 1–1.
Samsung. (2014). Samsung SUR40. Retrieved from http://www.samsung.com/nl/business/ business-products/large-format-display/specialized-display/LH40SFWTGC/EN-spec
Teder-Sälejärvi, W. A., McDonald, J. J., Di Russo, F., & Hillyard, S. A. (2002). An analysis of audio-visual crossmodal integration by means of event-related potential (ERP) recordings. Cognitive
Brain Research, 14(1), 106–114.
Tuddenham, P., Kirk, D., & Izadi, S. (2010). Graspables revisited : Multi-touch vs tangible input for tabletop displays in acquisition and manipulation tasks. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems (pp. 2223–2232).
Ullmer, B., & Ishii, H. (2000). Emerging frameworks for tangible user interfaces. IBM Systems
Journal, 39(3.4), 915–931.
Underkoffler, J., & Ishii, H. (1999). Urp: A luminous-tangible workbench for urban planning and design. In Proceedings of CHI ’99 (pp. 1–8).
Van Erp, J. B. F., Toet, A., & Janssen, J. B. (2013). Uni-, bi- and tri-modal warning signals: Effects of temporal parameters and sensory modality on perceived urgency. Unpublished manuscript.
TNO, Soesterberg, The Netherlands.
Weiss, M., Hollan, J. D., & Borchers, J. (2010). Augmenting interactive tabletops with translucent tangible control. In C. Müller-Tomfelde (Ed.), Tabletops - Horizontal Interactive Displays (pp. 149–170). London: Springer.
Wesp, R., Hesse, J., Keutmann, D., & Wheaton, K. (2001). Gestures maintain spatial imagery. The
American Journal of Psychologie, 114(4), 591–600.
