Ad hoc Color-Concept Mapping and Interpreting Visual Representations By Leslie Yunhsuan Lai B.A., Wesleyan University, 2014 Thesis Submitted in partial fulfillment of the requirements for the Degree of Master of Science in the Department of Cognitive, Linguistic, and Psychological Sciences at Brown University PROVIDENCE, RHODE ISLAND MAY 2017 ii This thesis by Leslie Yunhsuan Lai is accepted in its present form by the Department of Cognitive, Linguistic, and Psychological Sciences as satisfying the thesis requirement for the degree of Master of Science Date________________ _____________________________________ Dr. William Heindel, Advisor Approved by the Graduate Council Date________________ _______________________________________ Dr. Dima Amso Director of Graduate Studies, Department of Cognitive, Linguistic, and Psychological Sciences iii AUTHORIZATION TO LEND AND REPRODUCE THE THESIS As the sole author of this thesis, I authorize Brown University to lend it to other institution or individual for the purpose of scholarly research. Date___________________ ________________________________ Leslie Yunhsuan Lai, Author I further authorize Brown University to reproduce this thesis by photocopying or other means, in total or in part, at the request of other institution or individual for the purpose of scholarly research. Date___________________ ________________________________ Leslie Yunhsuan Lai, Author iv Table of Contents ___________________________________________________________________________ 1. Introduction .....................................................................................................................1–8 1.1 Interpreting colors in visualizations 1.2 Ad hoc color–concept mappings 2. Experiment 1...................................................................................................................9–13 2.1 Method and Design 2.1.1 Participants 2.1.2 Experiment Design 2.1.3 Procedure 2.2 Data analysis 2.3 Results and Discussion 3. Experiment 2.................................................................................................................14–16 3.1 Method and Design 3.1.1 Participants 3.1.2 Experiment Design 3.1.3 Procedure 3.2 Data analysis 3.3 Results and Discussion 4. Experiment 3.................................................................................................................17–21 4.1 Method and Design 4.1.1 Participants v 4.1.2 Experiment Design 4.1.3 Procedure 4.2 Data analysis 4.3 Results and Discussion 5. General Discussion.......................................................................................................22–24 References.........................................................................................................................25–27 vi Figures and Tables ___________________________________________________________________________ Figure 1: Proposed paradigm for structural mapping of color–concept mapping Figure 2: Example of ad hoc color–concept mapping paradigm Figure 3: Experiment 1 critical conditions Figure 4: Experiment 1 results reaction time as a function of mapped concept condition Figure 5: Experiment 2 results reaction time as a function of mapped concept condition Figure 6: Experiment 3 critical condition Figure 7: Experiment 3 results reaction time as a function of mapped concept condition Table 1: CIE xyY coordinates of a set of colors from experiment stimuli 1 1. Introduction 1.1 Interpreting colors in visualizations An important aspect of information visualization involves representing abstract concepts in visual representations such as graphs, maps, diagrams, and signs. Different types of visual features such as shapes, sizes, textures and colors are useful for differentiating information presented in visual representations (Bertin, 1983). Human visual perception possesses characteristics that have great influence on how information is processed. For example, certain visual features are more salient than the others as a result of visual hierarchy existed in the visual system. Previous studies showed that luminance and hue are perceptually strong visual features that can mask weaker features like curvature and shape (Klein, 2008; Lopes & Oden, 1991). On the other hand, interpretation of visual representation involves dynamic interplay between bottom–up and top–down processing, in which the mind must convert perceptual input into abstract concepts (Patterson, 2012). Therefore, an effective visualization design should concern with providing insight into abstract concepts through leveraging the functioning of the human visual system (Hearst, 2009) as well as facilitating problem solving and reasoning (Card, Mackinlay, & Schneiderman, 2010). Color is an essential visualization technique that has been characteristically used in different types of data visualization. There have been a handful of topics in the literature and general guidelines on selecting colormaps or color scale to produce distinguishable colors (see Bergman & Rogowitz, & Treinish, 1995; Penny, 1992; Robertson & Callaghan, 1986). For example, Ware (1988) found that the ability to read metric quantities from colormaps is greatly influenced by contrast effects when the color scale only utilizes one chromatic or achromatic channel in the visual system. He proposed spiral colormaps, in which color varies in both hue and intensity. The resulting colormaps alternate between chromatic channels in the visual system, thereby reducing the probability of contrast effect. In the context of 2 bivariate colormaps using separate axes for color and brightness, Rogowitz, Treinish and Bryson (1996) demonstrated that controlling luminance can help maintain orthogonality between the visual representations of different data components. Also concerning with displaying discriminable colors in visualization, Healey (1996) conducted a visual search experiment and found that viewers can distinguish up to seven isoluminant colors simultaneously. A different line of visualization research investigates aspects of color associations and how these associations may influence observers’ ability to interpret visual representation. Observing a color is not merely a perceptual experience; observers form a rich network of associations that influence higher-level processing (Schloss & Palmer, in press). For example, it has been shown that people’s color preference is influenced by the extent to which they like the objects that are associated with the specific colors (ecological valence theory; Palmer & Schloss, 2010; Schloss & Palmer, 2014, Schloss, Poggesi, & Palmer, 2011; Strauss, Schloss, & Palmer, 2013; Yokosawa, Schloss, Asano, & Palmer, 2015). There has been evidence suggesting that colors might influence people’s judgments on attractiveness of the opposite gender (Elliot & Niesta, 2008) as well as performance on intellectual test (Elliot & Maier, 2007). Can color associations influence the ability to interpret visual representations? One would expect that a visualization of Crayola colors (Crayola LLC), for example, needs the color names of the crayons and the colors in the visual representation to match. To do otherwise would potentially create cognitive interference similar to the Stroop Effect (MacLeod, 1991; Stroop, 1935). Lin, Fortuna, Kulkarni, Stone and Heer (2013) found when data categories have strong color associations (e.g. fruits, vegetables, political parties, and brands), using colors that match the color-concept associations in data visualizations improves observers’ cognitive performance. For example, participants were faster at reading graphs depicting fruit sales when data for bananas were represented by yellow and 3 blueberries were represented by blue than when bananas were represented by orange and blueberries were represented by green. Their findings were consistent with the prediction that representing categorical data with colors associated with the category avoids the potential interference of Stroop-like effect (MacLeod, 1991; Stroop, 1935), and therefore reduces the need to access the legend. As an extension to the work from Lin et al. (2013), Setlur and Stone (2015) developed an automatic color palette generator that leverages semantic context for the data category in addition to explicit color–category associations. For example, ‘apple’ as a fruit is associated with red or green, but ‘apple’ as a brand is white or silver gray; therefore, the choice of colors based on color-concept associations is context–dependent. An implicit assumption underlying these types of ‘semantic coloring’ approach is that the extent to which colors influence interpretation depends on the strength of the associations between the colors and concepts. However, we argue that in addition to the color associations elicited by specific concepts (e.g. object with inherent characteristic color), colors can carry specific meanings within the context of visual representations. For example, colors can represent different amounts of amplitude, levels of brain activation or arbitrary categories in spectrograms, neuroimaging figures and bar graphs respectively. How do observers’ predictions about how colors might map onto concepts influence the ability to interpret visual representations? Are there other types of color–concept mappings that also have similar congruency effects on interpretation of visual representations? The present study investigated how colors are mapped onto abstract concepts that do not have obvious color associations (e.g. Virus M vs. Virus Q) and whether such mappings influence the ability to interpret visual representations. In approaching this problem, we propose that the process by which observers interpret visual representations is analogous to analogical reasoning. In analogical reasoning, the goal is to make inference about a new target domain (e.g. an atom) by using information from a 4 known base domain (e.g. the solar system) (Gentner, 1983; Gentner & Smith, 2012). According to structure mapping theory (Gentner, 1983), an analogy is more effective when the relation among the components in the base domain is consistent with that in the target domain. For example, the analogy of PLANET : SUN :: ELECTRON : NUCLUES is structurally consistent because the components are aligned in consistent ways (i.e. less massive is to more massive in the base domain as less massive is to more massive in the target domain). Conversely, the analogy of PLANET: SUN :: NUCLEUS : ELECTRONS is structurally inconsistent because they are misaligned (i.e. less massive is to more massive in the base domain is not as more massive is to less massive in the target). Furthermore, the goodness of an analogy depends on its transparency, which is based on two factors: surface similarity and common relations (Gentner & Toupin, 1986). The surface similarity factor refers to how similar the components are in the base and target domains. The common relations factor refers to relation among the components in the base and target domains. Analogies are highly transparent when the surface similarity and common relations are both strong. In their investigation of effects on transparency on children’s abilities to make analogical inferences between pairs of stories, Gentner and Toupin manipulated transparency by switching the characters (e.g. different animals) and the corresponding roles of the characters (e.g. hero vs. villains) in the target story. An analogy has high transparency when the characters and their roles are identical in the base and target stories (e.g. both stories contain a chipmunk-hero & a horse-villain). An analogy becomes less transparent when new characters resume the same roles in the target story (e.g. a bluebird-hero & a squirrel-villain). The researchers further introduced a cross-mapped condition, in which identical characters resume the opposite roles in the target story (e.g. a horse-hero & a chipmunk-villain). This is the least transparent condition. They found that children had more difficulty in the low transparency than in the high transparency conditions. 5 The cross-mapped condition was especially difficult, as it required reassigning the roles of mapped characters rather than learning new mappings. Further evidence from algebraic task performance supports the notion that analogical inference is easier when surface similarity is high (Ross, 1987; Reed, 1987) and when common relations between perceptual grouping and the order of operations are aligned (Landy & Goldstone, 2007). We hypothesized that the process of inferring the meaning of perceptual features (e.g. colors) in visual representation involves similar alignment process as in analogical reasoning. Applying the logic of structure mapping (Gentner, 1983) to the interpretation of visual representation, the base domain is defined as the internal mapping between percepts and concepts held by the observer, and the target domain is the stimulus mapping in the visual representation. An internal mapping may refer to observers’ color associations for objects with observable characteristic colors (e.g. yellow – banana) or sensory mappings between magnitude and lightness (e.g. darker color represents ‘more’) (Schloss, Gramazio, & Walmsley, 2015; Smith & Sera, 1992). In the present study, we explored the possibility of forming internal mapping between colors and concepts through learning (e.g. repeated exposure to consistent color-concept pairings in an experiment). On the other hand, the stimulus mapping is defined by notations in the stimulus (e.g. legends or labels) that specify which visual features are intended to depict which concepts. Figure 1 illustrates three kinds of relations between the internal and stimulus mappings with respect to structure mapping. In Fig. 1A – 1C, the internal mapping contains two concepts (c1 and c2) and two percepts (p1 and p2). For example, the concepts are banana and blueberry and the percepts are yellow and blue colors. They are linked such that c1 maps to p1 and c2 maps to p2 (c1’ – p1’; c2’ – p2’). In Fig. 1A, the stimulus mapping (c1’ – p1’; c2’ – p2’) matches the internal mapping, and therefore, they are structurally consistent (SC). In Fig. 1B, the stimulus mapping introduces two new percepts, p3’ and p4’, that are not in the internal mapping, and therefore, the internal 6 and the stimulus mappings (c’1 – p’3; c’2 – p’4) are structurally inconsistent (SI). In Fig. 1C, the stimulus mapping (c’1 – p’2; c’2 – p’1) and the internal mapping are reversed, and therefore, they are SI (Cross-mapped). The central prediction is that visual representations are easier to interpret when the internal color-concept mapping matches the stimulus mapping (i.e. structurally consistent). Figure 1. Proposed paradigm for characterizing the relation between internal mappings and stimuli mappings between percepts and concepts in visual representations. See text for details. 1.2 Ad hoc color–concept mapping Spence (2011) posits that cross-modal mappings can be learned through co- occurrence in natural environment and in statistical regularities within an experimental setting. For example, in natural environment, larger objects tend to produce lower frequency sounds when they are dropped (e.g. Carello, Anderson, & Kunkler-Peck, 1998), which can account for why larger shapes map to lower pitches (e.g. Evans & Treisman, 2010). In the laboratory, Ernst (2007) found that participants formed new mappings between luminance and haptic stiffness based on statistical concurrences, which influenced subsequent discriminability threshold. In the present study, we operationalized ad hoc color-concept mapping as learned pairings between percepts and concepts that viewers form from repeated exposure to consistent mappings. Three experiments were conducted to test the following 7 prediction: experience with new color-concept assignments during the experiment result in ad hoc mappings that bias the interpretation of subsequent visual representations involving the same colors or same concepts. Figure 2 illustrates the paradigm adopted in the three experiments in this study. In a series of three-trials, the first two trials contain graphs that depict data about two abstract concepts (c1’ and c2’), which are represented by two different bar colors (p1’ and p2’). In Trial 1, there is no a priori internal mapping between concepts and colors, but there is a stimulus mapping as specified by the legend (Fig. 2A). After experiencing the pairings in Trial 1, the stimulus mapping in Trial 1 is learned and is being defined as ad hoc internal mapping for Trial 2. In Trial 2 (Fig. 2B), the stimulus mapping is the same as the ad hoc internal mapping learned from Trial 1, and therefore, the internal and stimulus mappings in Trial 2 are structurally consistent. We define the concepts that are paired with the colors in Trial 1 and Trial 2 as the mapped concepts. Trial 3 is the test trial in which the graph represents the mapped concepts in ways that vary in structural consistency with respect to the ad hoc mapping established in Trial 1 and Trial 2 (see the Method section for details). 8 In experiment 1 and experiment 2, we tested how structural consistency between ad hoc mapping and stimulus mapping influenced the speed with which observers report information about subsequent graph involving the same colors or same concepts. We predicted that ad hoc internal mappings can influence the speed of interpreting proceeding graph, in which observers should be faster at responding to graphs containing color–concept mappings that are consistent with the ad hoc internal mappings. Experiment 3 investigated whether ad hoc color–concept mappings retain relational information (e.g. lightness difference) or they depend merely on surface matches of color appearance. If relational information is represented in the internal mapping, a stimulus mapping is structurally consistent when both surface colors and lightness relation are consistent with the ad hoc mapping. In contrast, a stimulus mapping is structurally inconsistent when either factor is inconsistent. Alternatively, if internal mapping depends solely on surface matches, changing lightness relation should not affect interpretation of visual representation as long as the surface colors remain the same. 9 2. Experiment 1 2.1 Method and Design 2.1.1 Participants 30 undergraduate students at Brown University (n=21) and Wesleyan University (n=9) participated in the experiment. We verified that participants have normal color vision using the HRR Color Vision Test – Standard Vision. Participants from Brown University were recruited from the Brown University subject pool and they received one course credit for participation. Participants from Wesleyan University voluntarily took part in this study and did not receive any compensation for participation. All participants were informed of the experiment procedure and consented prior to participating in the study and the Committee for the Protection of Human Subjects at Brown University approved the experimental protocol. 2.1.2 Experiment Design The experiment paradigm included a series of three trials. In both Trial 1 and Trial 2 (learning trials), two concepts were presented in the legend and were represented by two bar colors. In Trial 3, the graph varied along two factors: 2 concept correspondences x 2 color correspondences, which yielded to four critical conditions. Trial 1 and Trial 2 contained the identical stimulus-mappings between the bar colors (p1 and p2) and concepts (c1 and c2) in the legend, while the x-axis labels differed from Trial 1 to Trial 2. We defined repeated and consistent stimulus mappings involving the same concepts and colors in the two learning trials as ad hoc internal mapping. Trial 3 was the test trial. Figure 3 illustrates the four stimulus mapping conditions that varied in structural consistency with respect to ad hoc internal mapping: (A) color-same/concept-same (SC), (B) color-same/concept-different (SI: same bar colors/ new concepts in the legend and the mapped concepts were moved to the x- axis), (C) color-different/ concept-same (SI: new bar colors/ same mapped concepts in the legend) and (D) color-different/ concept-different (novel mapping: new bar colors/ new 10 concept in the legend and the mapped concepts were moved to the x-axis). The experiment had a within-subject design. Four critical conditions were assigned to 16 blocks of trials using a Latin square design to avoid carry-over effect, and each critical condition appeared in 4 blocks (replications). Block order was randomized across participants. A B C D Figure3. 2x2 design for the test trial (Trial 3), in which the colors and mapped concepts are the same (A), the colors are the same but the mapped concepts are different (B), the colors are different but the mapped concepts are the same (C), and both the colors and mapped concepts are different (D). The stimuli included forty-eight lecture slides that depict fictitious data about different domains such as public transportation, students’ academic performances, and product sales. Each domain contained two main concepts, which do not have obvious color associations (e.g. Virus M & Virus Q). All slides had the following format: one bar graph, brief description of the graph and one 2AFC question about the content of the graph. We constrained the graph on each slide to always show a two-way interaction effect to insure consistent difficulty of graph interpretation. The bar heights were otherwise randomly generated. 11 “Tableau 20” and “Tableau 10 medium” color palettes, whose coordinates are device dependent (vary depending on the monitor) (Lin, Fortuna, Kulkarni, Stone, & Heer, 2013), were used to generate bar colors. Tableau colors are designed for data visualization to insure that colors will be discriminable despite normal variations in monitor viewing conditions (Tableau Software, 2012) (see appendix for the CIE xyY coordinates of the colors). We selected eight hues at three lightness levels, corresponding to a total of twenty-four colors. In each trial, there were two bar colors that have two different hues, but are the same level of lightness. Different bar colors were assigned to trials based on the critical conditions described in experiment deign. We wrote our experiments in MATLAB, using the Psychophysics Toolbox extensions (Brainard, 1997; Pelli, 1997). 2.1.3 Procedure Each participant completed a total of 50 trials including 2 practice trials and 16 three- trial blocks. In each block, a participant was first presented with the two learning trials, one at a time, followed by one test trial. In each trial, a description about the graph appeared first, followed by a 2AFC question, and then after a 3.5 seconds delay, the bar graph appeared. The delay was implemented to insure that all participants had enough time to read the text before they responded to the graph. For all trials, the 2AFC question had the following structure: which concept from the legend (c1’ or c2’) has a larger value on the y-axis for a given x-axis category (e.g. c3’ or c4’). Participants were instructed to answer the question based on information provided in the graph as accurately and quickly as possible. Participant indicated their answer by pressing one of the response keys. Reaction times and responses were measured for all trials. Across trials, we randomized which x-value was probed (left/right), whether the upper concept in the legend mapped to answer A or B, and whether the upper concept in the legend had a larger or smaller y-axis value given the x-axis category. 12 2.2 Data analysis A mixed model factorial ANOVA showed no group differences in performance between participants recruited from Brown university and Wesleyan University (p > 0.9), and so the data from the two universities were merged for further data analysis. Reaction times (RTs) for the test trials were analyzed. We included trials with accurate responses (Accuracy = 97.3%) and were within 1.5 standard deviation of the mean RT for each subject in the analysis. Data from one subject was excluded due to missing data in one condition (n = 29; 90% of the dataset included in the final analysis). RTs were averaged within each critical condition from each participant. 2.3 Results and discussion A 2 x 2 repeated measures ANOVA revealed a significant interaction effect between the two factors: ‘color’ and ‘concept’ (F (1, 29) = 26.02, p < 0.001) (Fig. 4). When legend concepts in the test trial were the same as in the learning trials, participants were faster to respond to trials with the same bar colors (SC) compared to those with different bar colors (SI: old concepts mapped to new colors) (t (28) = -3.88, p < 0.005). Conversely, when the legend concepts in the test trials were different from that in the learning trials, participants were slower to respond to trials with the same bar colors (SI: new concepts mapped to old colors) compared to different bar colors (novel mapping: new concepts mapped to new colors) (t (28) = -3.53, p < 0.005). In addition, there was a main effect of concept (F (1, 29) = 66.27, p < 0.001). Participants were slower to respond when new concepts were presented in the legend while the old mapped concepts were moved to the x-axis. 13 Figure 4. RTs for Exp. 1. When colors are the same (circles) or different (squares) as a function of mapped concept kind. Consistent with our prediction, observers appeared to be sensitive to repeated and consistent color-concept mappings when viewing a series of visual representations involving the same concepts or colors. The results provided evidence that the speed of interpreting proceeding graphs can be influenced by ad hoc mappings established within minimal experience (i.e. two trials) with specific color-concept pairings. Further supported by our results, structurally consistency between internal and stimulus mappings appeared to predict how different stimulus mappings affect the speed to interpret subsequent visual representations. Viewers were faster at responding to proceeding graphs that are structurally consistent with the learned ad hoc mapping. Importantly, we found evidence suggesting that graphs with structurally inconsistent color-concept mappings may be more difficult to interpret than graphs with novel mappings. In addition, participants were overall slower in the concept-different conditions, in which new concepts were presented in the legend while the mapped old concepts were moved from to the x-axis. The overall slower responses might indicate an increase in task difficulty as a result of the inconsistent spatial mapping in addition to inconsistent color-concept mapping in the legend. To clarify the role of color- concept mapping in interpreting visual representation, we tested the same hypotheses in a follow up experiment containing the same critical conditions. However, the inconsistent spatial mapping aspect was removed from the concept-different conditions. 14 3. Experiment 2 3.1 Method and Design 3.1.1 Participants 30 subjects participated in experiment 2. Participants were recruited from the Brown University subject pool and received 5 Dollars for their participation. We verified that participants have normal color vision using the HRR Color Vision Test – Standard Vision. All participants were informed of the experiment procedure and consented prior to participating in the study and the Committee for the Protection of Human Subjects at Brown University approved the experimental protocol. 3.1.2 Experiment design The experiment has exactly the same 2-by-2 (color (same/different) x concept (same/different)) within-subjects design as in experiment 1. The only difference was that in concept-different conditions, new concepts were presented in the legend as well as on the x- axis. That is, the old mapped concepts were replaced completely instead of being moved from the legend to the x-axis like in experiment 1. As discussed, this should avoid creating inconsistent spatial mapping of the mapped concepts in the test graph and help clarify how color-concept mapping influence the interpretation of visual representation. 3.1.3 Procedure A set of new concepts corresponding to the fictitious domains was added to the original stimuli in experiment 1 to replace the x-axis labels in the concept-different conditions. As with the rest of the domain concepts, the new concepts were chosen such that they do not have obvious color associations. The stimuli were otherwise identical to that in experiment 1. Procedure is identical to experiment 1. 15 3.2 Data analysis Reaction times (RTs) for the test trials were analyzed. We included trials with accurate responses (Accuracy =96.2 %) and were within 1.5 standard deviation of the mean RT for each subject in the analysis (n = 30; 88% of the dataset were included in the analysis). RTs were averaged within each mapping conditions from each participant. 3.3 Results and discussion A 2 x 2 repeated measures ANOVA revealed a significant interaction effect between color and concept (F (1, 30) = 5.834, p < 0.05) (Fig. 5). Figure 5. RTs for Exp. 2. When colors are the same (circles) or different (squares) as a function of mapped concept kind. When legend concepts in the test trial were the same as in the learning trials, participants were faster at responding to trials with the same bar colors (SC) compared to different bar colors (SI) (t (28) = -3.37, p < 0.005). This result was consistent with what we found in experiment 1. However, unlike in Experiment 1, when legend concepts were different, participants did not show RT difference between color-same (SI) and color- different (novel stimulus mapping) conditions (t (28) = -.64 p > 0.5). Experiment 2 provided additional evidence supporting that observers are capable of forming ad hoc mappings from viewing a series of visual representations consisted of consistent color-concept pairings. Further, in both experiment 1 & 2, we found structurally consistent color-concept mappings facilitated the speed of interpreting proceeding visual representation involving the same 16 concepts (same-concept conditions were identical in experiment 1 & 2). In experiment 1, we found that changing bar colors facilitated RTs when new concepts were presented in the legend. However, in experiment 2, there was no RT difference regardless of whether the new concepts were mapped onto the old colors (SI) or new colors (novel mapping). The overall RTs were shorter in experiment 2, indicating that without the spatial switch (i.e. mapped concepts moved from the legend to x-axis) in the concept-different conditions, the task could potentially be easier compared to Experiment 1. As a result, observers might be able to resolve the structurally inconsistent mappings in visual representation more efficiently in experiment 2, and therefore, we did not observe the difference in RT between the SI and novel mapping conditions. Together, our findings suggest that the process of interpreting multiple visual representations involving the same concepts or colors is more effective when the mappings between colors and concepts are consistent. There is also evidence that inconsistent color-concept mappings can be more difficult to interpret than novel color- concept pairings in visual representation. 17 4. Experiment 3 A better understanding of the nature of ad hoc color–concept mapping would allow us to make further predictions about how these mappings influence interpretation. Therefore, in Experiment 3, we investigated the nature of color representations encoded in ad hoc color- concept mappings. Specifically, we tested whether the relation among colors (e.g. relative lightness) is represented in the internal mappings between colors and concepts or the mappings depend only on surface matches. 4.1 Method and Design 4.1.1 Participants 30 subjects participated in experiment 3. Participants were recruited from the Brown University subject pool and received 10 Dollars or 1 course credit for participation. We verified that participants have normal color vision using the HRR Color Vision Test – Standard Vision. All participants were informed of the experiment procedure and consented prior to participating in the study and the Committee for the Protection of Human Subjects at Brown University approved the experimental protocol. 4.1.2 Experiment design Experiment 3 adopted the same graph-reading paradigm in experiment 1 & 2. As illustrated in Fig. 6A, in Trial 1 and Trial 2 (learning trials), the two concepts (c1 and c2) in the legend were represented by two bar colors (p1 and p2), which have the same hue but differ in lightness (e.g. c1 mapped onto dark green & c2 mapped onto light green). Unlike in experiment 1 & 2, the legends in Trial 3 always represented the same mapped concepts ((c1 and c2). Further, as illustrated in Fig. 6B, the bar colors in Trial 3 varied along two factors: 2 hue correspondences x 2 lightness correspondences. In the hue-same conditions, the bar colors in Trial 3 have the same hue as in Trial 1 and Trial 2. Additionally, when the lightness relation was the same, the colors in the test trial were identical to that in the learning trials 18 (e.g. c1 mapped onto dark green & c2 mapped onto light green). In contrast, when the lightness relation was different, the colors were cross-mapped, such that the concept that was mapped to the lighter color (of the same hue) is now mapped to the darker color, and vice versa (e.g. c1 mapped onto light green & c2 mapped onto dark green). In the hue-different conditions, an analogous hue was chosen for the two colors in the test trial (e.g. learning trial hue: green; test trial hue: cyan). We implemented this constraint so that the difference between learning trial hue and test trial hue would be compatible across trials. The within subject design included four critical conditions: (1) hue-same/lightness-same (e.g. dark green mapped onto c1 & light green mapped onto c2), (2) hue-same/lightness-different (e.g. light green mapped onto c1/ dark green mapped onto c2), (3) hue-different/ lightness-same (dark cyan mapped onto c1/ light cyan mapped onto c2) and (4) hue-different/ lightness-different (light cyan mapped onto c1/ dark cyan mapped onto c2). As in the previous experiments, we randomized which x-value was probed (left/right), whether the upper concept in the legend mapped to answer A or B, and whether the upper concept in the legend had a larger or smaller y-axis value given the x-axis category across trials. In addition, we counterbalanced the position of the dark or light color bar (left/right), which yielded to 32 blocks (96 trials). The procedure was otherwise identical to experiment 1 & 2. 19 Figure 6. Color-concept stimulus mapping in the two learning trials within a domain block (A) and a 2 x 2 design for test trial (Trial 3) (C). See text for details. The stimuli consisted of ninety-six lecture slides that were in the same format as in experiment 1 & 2. Eight colors from the Tableau 20 color palette were utilized, including one dark and one light color from each of the following hues, red, orange, green and cyan. Colors were assigned to the mapped concepts with respect to the experimental conditions described in the experiment design section. 4.2 Data analysis Reaction times (RTs) for the test trials were analyzed. We included trials with accurate responses (Accuracy = 98 %) and were within 1.5 standard deviation of the mean RT for each subject in the analysis (n = 30; 91% of the dataset were included in the analysis). RTs were averaged within each mapping conditions from each participant. 20 4.3 Results and discussion A 2 x 2 repeated measures ANOVA revealed a main effect of lightness relation (F (1, 30) = 15.48, p < 0.001) (Fig. 7). Consistent with our hypothesis, participants were faster at responding to proceeding graph containing same-lightness relation than different-lightness relation regardless of whether the hue was the same (t (28) = -3.37, p < 0.005) or different (t (28) = -3.37, p < 0.005). Furthermore, there was no RT difference between the same-hue and different-hue conditions both when the lightness relation was the same and when it was different. Figure 7. RTs for Exp. 3. When hue is the same (circles) or different (squares) as a function of mapped lightness relation kind. These findings suggest that viewers are capable of forming ad hoc color-concept mappings based on not only surface matches (e.g. Exp. 1 & 2) but also abstract relational schemas. Interpreting cross-mappings requires reassigning the internal mappings between the same concepts and colors. Therefore, reversed lightness relation would be the most difficult to interpret. We also found that hue change did not affect reaction times, suggesting that relational information may be the critical dimension of viewers’ internal mapping established in this experiment. Further, as hue was not the critical dimension of the internal mapping, structural inconsistency caused by change in hue might not result in significant interference 21 with the internal mappings. In other words, viewers resolve the inconsistent surface matches more effectively because they were less salient in the mappings. 22 5. General Discussion The present study explored the question of whether observers’ predictions about how colors might be mapped onto concepts influence their ability to interpret visual representation. We established a new method to evaluate how the experience with color–concept pairings (i.e. ad hoc mappings) influences the speed of interpreting proceeding visual representations involving the same colors or concepts. The three experiments provided evidence that observers are capable of forming ad hoc mappings between colors and concepts within minimal exposure to specific color–concept pairings established in visual representations (e.g. graphs). In experiment 1, we found that observers were faster at responding to subsequent visual representations that were structurally consistent with the learned mappings. The same critical conditions in experiment 2 replicated this result. In experiment 1, we also found that observers were slower in responding to subsequent graphs that contain structural inconsistent mappings (e.g. new concepts mapped onto old colors) than graphs containing novel mappings (e.g. new concepts mapped onto new colors). However, in experiment 2, there was no RT difference between those two conditions. This discrepancy might be due to the possibility that observers were able to resolve structural inconsistent mappings more efficiently in experiment 2 because the task in experiment 2 was less difficult than Experiment 1, as indicated by overall shorter reaction times. Recall that in the concept–different conditions in experiment 2, there was no inconsistent spatial mappings because the old mapped concepts were replaced by new concepts instead of being moved to the x-axis like in experiment 1. Alternatively, there may be other factors in addition to color appearance that also contribute to ad hoc color-concept mappings and thus, have an influence on interpretation. To make further predictions about how ad hoc color–concept mappings might affect the ability to interpret visual representation requires a better understanding of how colors are represented in the ad hoc internal mapping. A number of psychophysiological studies have 23 highlighted the fact that information about the color of stimulus can be stored independently of other stimulus attribute, such as pattern and luminance (Cornelisson & Greenlee, 2000; Magnussen, Greenlee, & Thomas, 1996; Yoshizawa, Kubota, & Kawahara, 2011). Can observers retain relational information about stimulus colors in their internal mappings? Experiment 3 provided evidence that viewers are capable of forming ad hoc color-concept mappings based on not only surface matches as shown in Exp. 1 & 2, but also abstract relational schemas. Importantly, relational information encoded in the internal mappings appears to supplant surficial color appearance such that structural consistency of proceeding graphs depends whether the critical dimension of the mapping (e.g. lightness relation) is matched. A central fact about analogy is that it is selective. Not all commonalities are equally important in matching or prediction (Hall, 1988). Our finding is consistent with structure– mapping theory, in which Gentner (1980, 1983) proposed that people seek to map relation between objects rather than non-relational object attributes for interpretation of analogy. In both hue conditions, participants were faster at responding to graphs containing consistent lightness relation mappings, suggesting that relation is a selection constrain on the choice of information to form ad hoc color–concept mapping. There is prior evidence that structural consistency between the internal and stimulus color–concept mappings affects observers’ ability to interpret visual representation (Schloss, Gramazio, & Walmsley, 2015). However, to our knowledge, the present research is the first test of the idea that color-concept mappings can be formed online and can influence the interpretation of subsequent visual representation with respect to its structural consistency. This study provides evidence that ad hoc color concept mappings can be formed online; further, lightness relation can operate as a selection filter that supersedes surface matches in mapping. It is worth noting that our paradigm did not consider the potential effect of time on ad hoc mapping. For example, the same concept that is repeated in multiple visual presentations 24 oftentimes does not appear one after another over a course of a presentation. An open question is whether the effect of ad hoc mapping on interpretation persists in non-sequential viewing. Future studies are warranted to understand how different color information is retained in observers’ internal mappings and how they interact within the context of interpreting visual representation. 25 Reference Bergman, L. D., Rogowitz, B. E., & Treinish, L. A. (1995, October). A rule-based tool for assisting colormap selection. In Proceedings of the 6th conference on Visualization'95 (p. 118). IEEE Computer Society. Bertin, J. (1983). Semiology of graphics: diagrams, networks, maps. Brainard, D. H. (1997) The Psychophysics Toolbox, Spatial Vision , 10:443-446. Pelli, D. G. (1997) The VideoToolbox software for visual psychophysics: Transforming numbers into movies, Spatial Vision 10:437-442. Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999). Readings in information visualization: using vision to think. Morgan Kaufmann. Carello, C., Anderson, K. L., & Kunkler-Peck, A. J. (1998). Perception of object length by sound. Psychological science, 9(3), 211-214. Ware, C. (1988). Color sequences for univariate maps: Theory, experiments and principles. Computer Graphics and Applications, IEEE, 8(5), 41-49. Cornelisson, F. W., & Greenlee, M. W. (2000). Visual memory for random block patterns defined by luminance and color contrast. Vision Research, 40, 287–299. Egeth, H. E., & Yantis, S. (1997). Visual attention: Control, representation, and time course. Annual review of psychology, 48(1), 269-297. Elliot, A. J., & Maier, M. A. (2007). Color and psychological functioning. Current Directions in Psychological Science, 16(5), 250-254. Elliot, A. J., & Niesta, D. (2008). Romantic red: red enhances men's attraction to women. Journal of personality and social psychology, 95(5), 1150. Ernst, M. O. (2007). Learning to integrate arbitrary signals from vision and touch. Journal of Vision, 7(5), 7. Evans, K. K., & Treisman, A. (2010). Natural cross-modal mappings between visual and auditory features. Journal of vision, 10(1), 6. Gentner, D. (1980). The Structure of Analogical Models in Science (No. BBN-4451). BOLT BERANEK AND NEWMAN INC CAMBRIDGE MA. Gentner, D. (1983). Structure-mapping: A theoretical framework for analogy.Cognitive science, 7(2), 155-170. Gentner, D., & Smith, L. (2012). Analogical reasoning. Encyclopedia of human behavior, 130-136. Gentner, D., & Toupin, C. (1986). Systematicity and surface similarity in the development of analogy. Cognitive science, 10(3), 277-300. Hall, R. P. (1988). Computational approaches to analogical reasoning: A comparative analysis. Artificial, Intelligence, pp. 1-75. Healey, C. G. (1996, October). Choosing effective colours for data visualization. In Visualization'96. Proceedings. (pp. 263-270). IEEE. 26 Hearst, M. (2009). Search user interfaces. Cambridge University Press. Klein, G. (2008). Naturalistic decision making. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(3), 456-460. Landy, D., & Goldstone, R. L. (2007, August). The alignment of ordering and space in arithmetic computation. In Proceedings of the twenty-ninth annual meeting of the Cognitive Science Society (pp. 437-442). Lin, S., Fortuna, J., Kulkarni, C., Stone, M., & Heer, J. (2013, June). Selecting Semantically Resonant Colors for Data Visualization. In Computer Graphics Forum (Vol. 32, No 3pt4,pp. 401-410). Blackwell Publishing Ltd. Lopes, L. L., & Oden, G. C. (1991). The rationality of intelligence. Rationality and reasoning. Amsterdam: Rodopi, 199-223. MacLeod, C. M. (1991). Half a century of research on the Stroop effect: an integrative review. Psychological bulletin, 109(2), 163. Magnussen, S., Greenlee, M. W., & Thomas, J. P. (1996). Parallel processing in visual short-term memory. Journal of Experimental Psychology: Human Perception and Performance, 22, 202–212. Palmer, S. E. & Schloss, K. B. (2010). An ecological valence theory of color preferences. Proceedings of the National Academy of Sciences, 107, 8877-8882. Patterson, R. E. (2012). Cognitive engineering, cognitive augmentation, and information display. Journal of the Society for Information Display, 20(4), 208-213. Rheingans, P. (1992, October). Color, change, and control of quantitative data display. In Visualization, 1992. Visualization'92, Proceedings., IEEE Conference on (pp. 25 259). IEEE. Reed, S. K. (1987). A structure-mapping model for word problems. Journal of Experimental Psychology: Learning, memory, and cognition, 13(1), 124. Robertson, P. K., & O'Callaghan, J. F. (1986). The generation of color sequences fo univariate and bivariate mapping. Computer Graphics and Applications, IEEE, 6(2), 24-32. Rogowitz, B. E., Treinish, L. A., & Bryson, S. (1996). How not to lie with visualization. Computers in Physics, 10(3), 268-273. Ross, B. H. (1987). This is like that: The use of earlier problems and the separation of similarity effects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 13(4), 629. Schloss, K. B., & Palmer, S. E. (2014). The Politics of Color: Preference for Republican red versus Democratic blue. Psychonomic Bulletin & Review, 21,6, 1481-1488. Schloss, K. B., & Palmer, S. E. (in press). Ecological aspects of color preference. In A. J Elliot, M. D. Fairchild, & A. Franklin (Eds.),Handbook of colorpsychology. Cambridge, UK:Cambridge University Press. 27 Schloss, K. B., Gramazio, C., & Walmsley, C. S. “Which color means more? An investigation of color quantity mapping in data visualization.” Presented at the 15th Annual meeting of the Vision Sciences Society, St. Petersburg, FL. May 2015. Schloss, K. B., Poggesi, R. M., & Palmer, S. E. (2011). Effects of university affiliation and "school spirit" on color preferences: Berkeley vs. Stanford. Psychonomic Bulletin & Review, 18, 498-504. Setlur, V., & Stone, M. (2015). A Linguistic Approach to Categorical Color Assignment for Data Visualization. Smith, L. B., & Sera, M. D. (1992). A developmental analysis of the polar structure of dimensions. Cognitive psychology, 24(1), 99-142. Spence, C. (2011). Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics, 73(4), 971-995. Strauss, E. D., Schloss, K. B., & Palmer, S. E. (2013). Color preferences change after experience with liked/disliked color objects.Psychonomic Bulletin & Review, 20, 5 935-943. Stroop, J. R. (1935). Studies of interference in serial verbal reactions. Journal of experimental psychology, 18(6), 643. Yokosawa, K., Schloss, K. B., Asano, M., & Palmer, S. E. (2015). Cross-cultural studies o color preferences: US and Japan.Cognitive Science. Yoshizawa, T., Kubota, M., & Kawahara, T. (2011). Short-term memory of color sensation is robust against luminance distortion. Color Research and Application, 36, 47–54. 28 Table 1: CIE xyY coordinates for stimulus colors RGB CIE xyY Tableau 20 Tableau 20 31 119 180 0.2049 0.2116 13.78 255 127 14 0.5395 0.4136 32.58 44 160 44 0.3089 0.5969 23.75 214 39 40 0.6041 0.3426 14.97 148 103 189 0.2778 0.1973 16.22 227 119 194 0.36 0.2461 29.43 188 189 34 0.4085 0.5255 44.82 23 190 207 0.225 0.3089 37.45 174 199 232 0.2749 0.2995 52.04 255 187 120 0.4335 0.4269 55.34 152 223 138 0.3207 0.4805 58.34 255 152 150 0.4295 0.3496 42.06 197 176 213 0.3041 0.2911 44.04 247 182 210 0.3493 0.3108 54.06 219 219 141 0.3719 0.4494 64.79 158 218 229 0.2679 0.3282 59.37 Tableau 10 Tableau 10 114 158 206 0.2411 0.2587 28.42 255 158 74 0.4894 0.3212 42.53 103 191 92 0.3165 0.5354 39.1 237 102 93 0.5179 0.356 24.84 173 139 201 0.2945 0.2494 27.69 237 251 202 0.3534 0.2802 40.02 205 204 93 0.3963 0.4939 55.03 109 204 218 0.2458 0.323 49.04 Background Background 255 255 255 0.3146 0.339 93.64