Impact of sound on image-evoked emotions
René Van Egmond
Delft University of Technology, Landbergstraat 15, 2628 CE Delft, The Netherlands
ABSTRACTIn two experiments the effect of sound on visual information was investigated. In Experiment 1 the effect of the visual appearance of product types with an expensive deign and with an inexpensive design on the experience of the sound recordings of these products was investigated. Recordings and pictures were systematically interchanged. Thus, for example, the visual image of an expensive design was combined with a recording of the sound of an inexpensive and of an expensive design. It was found that product appearance did not affect the judgment on luxury, pleasantness, quality, and ease-of-use but that the experience of the sound dominated over the visual experience. In Experiment 2, pictures from the international affective pictures set were combined with frequency-modulated tones that varied in the amount of sensory pleasantness by manipulating the amount of roughness. The combination of sounds and pictures were rated on the valence and arousal dimensions of the circumplex model of core affect. It was found that the sounds only negatively affected the experience of the pictures on the valence dimension. The arousal level was not affected by the sounds. Both experiments show that sound can affect the perception and experience of pictures.
Keywords: sound, picture, IAPS, multimodal interaction, emotion, and products 1. Introduction
In the film industry, moviemakers are aware of the effect sound or music has on the emotional impact of the visual scene. In scary parts, one would not use a cheerful piece of music but music that enhances the scary experience. Several factors have been identified that can influence a person’s emotional experience of music1. In addition, it has been found
that people can relate specific sound tracks to specific movie scenes quite accurately2. In our daily life one is often
confronted with the sound of domestic appliances, feedback or alarms sounds. In our daily interaction with these products people are often not consciously aware that they use the information they derived from the sounds or that their emotional state changes because of the sound. This may lead to certain behavioral actions. Who does not use the auditory cue of water boiling in a water boiler and goes to the kitchen to make a cup of tea? Or who is not familiar with the experience of the level of quality when a car door closes3. In fact, the car industry employs acoustic engineers just to
optimalize the evoked experience of quality. Another example is the effect that alarms sounds have on medical staff working in the intensive care unit, this often leads to irritation such that they are more involved with coping with the irritation than attending to the actual problem4, 5. The experience of quality, for example, can be designed if one knows
which particular perceptual features one needs to manipulate in the sound. A number of studies have investigated and determined these features in the sounds of vacuum cleaners, coffeemakers, electrical toothbrushes 6, 7, 8, 9. Although
quality aspects of product sounds or of the visual appearance of products have been studied 10, 11, the interaction between
these perceptual modalities is hardly studied. Several multisensory studies have investigated the contribution of the different senses in the recognition of a product12 or cross-modal correspondences13, 14, and the importance of the different
senses in product usage. Although some studies have investigated the effect of the evoked emotions by sounds on the processing of visual images, the combined emotional experience of auditory and visual stimuli is not studied that frequently16. The sound employed are often emotionally spoken words in combination with pictures of faces. In this
paper, two studies are reported that investigate the effect of sound on the perceived quality and user-friendliness of the images of domestic appliances and the effect of frequency-modulated tones (varying in the amount of roughness) on the evoked valence and arousal by affective pictures.
1.1. Process Sound Perception
In product sound perception a distinction can be made between consequential and intentional sounds8. Consequential
sounds are the consequence of the appliance operating. Constructive parts that are moving but also a person interacting with the product will influence the perceptual features of a sound. Thus, gears, engines (at a certain rpm), the pressure applied by a person all result in specific sound characteristics. These consequential sounds are difficult to design because they involve many parts and even more difficult is to predict how the sound will be perceived before constructing the product. This is in contrast with the visual appearance of products of which one can generate already very liking computer images or even a 3D form can be generated with modern rapid prototyping techniques (3D “printing). Intentional sounds are the sounds intentionally added to a product (e.g., alarm and feedback sounds in user interfaces). These sounds are easier to manipulate or generate because they can be synthesized with computer programs (e.g., Logic Pro, Audition, Sound Studio) quite easily. Still the design of these sounds is often not adequate because of insufficient knowledge of the people who implement the sounds.
In the processing of sounds bottom-up and top-down processes play a role. In the attribution of meaning to a sound, but also in the recognition of structure or the emotional experience (appraisal) top-down or more cognitive aspects play an important role. However, certain perceptual features of sound (sensations) can be traced back to the analysis performed by the inner ear. These sensations can be captured by psychoacoustical measurements17. These measurements can then
be implemented in models that generate metrics. It has been proposed that sensory pleasantness is made up out of four psychoacoustical measures: roughness, sharpness, loudness, and tonalness17. An increase in the level of loudness,
roughness, and sharpness will result in a decrease of sensory pleasantness, whereas an increase in tonalness results in an increase of sensory pleasantness. Because product sounds are often sharp, rough, loud, and noisy these measures are important in describing the featural aspects of the timbre of these sounds. This is true for the consequential sounds but also for the intentional sounds (think of loud and sharp feedback or alarm sounds). Furthermore, the role of audition in the perception of tactile perception has been a topic of several studies18.
1.2. Roughness Perception
The perception of auditory roughness has been studied quite extensively for amplitude and frequency modulated tones17.
The unit of roughness is asper that is defined as: “To define the roughness of 1 asper, we have chosen the 60dB, 1-kHz tone that is 100% modulated in amplitude at a modulation frequency of 70 Hz”. For frequency modulated tones two parameters determine the roughness of a tone, these are, modulation frequency and the modulation index. The modulation index is dependent on the modulation frequency and the modulation depth. For a tone with a fixed carrier frequency, an increase of the modulation frequency for a certain modulation index will result in an increase of the roughness until a certain maximum is reached. If the modulation frequency increases after this maximum the tone will become less rough.
Several studies investigated the role of audition on the perception of perceived tactile roughness18. In these studies
sounds are presented via headphones to a listener who interacts with a product. It was for example found that increasing the loudness and sharpness of the sound of a tactile interaction with paper or sand paper changed the perceived tactile perception of roughness19, 20. Another study investigated the pleasantness and roughness of the vibrotactile stimulation
user’s felt on their teeth using an electrical toothbrush while they received manipulated sounds via headphones21. These
studies present evidence between the influence of the auditory modality and the tactile experience. However, the interaction between audition (e.g., roughness) and visual information has not been studied that extensively (see for an overview Schifferstein22).
1.3. Emotional Experience
Emotions have been studied extensively in relation to human behavior23, 24, 25. Some researchers suggest that all emotions
are evoked by a process of appraisal26. A distinction has been made between a complex process and a simple process of
appraisal. A simple appraisal will evoke emotions that find their origin in evolution and are called basic emotions. These basic emotions are essential in survival and are often related to action23, 25. For example, fear will be the result of an
appraised ‘physical threat’ and will result in the action to flee. Basic emotions exist in different cultures and are evoked in adults, children, and even in some animals 23, 25, 27, 28. Complex appraisals evoke ‘cognitive emotions’ that are more
dependent on meaning attribution29. For example, inspiration is a cognitive emotion that requires an appraised ‘mental
Emotions (or the words that reflect these emotions) are often represented on three underlying dimensions24:
valence/pleasantness, activation/arousal, dominance. The valence and activation dimension span up the circumplex model of core affect 32. On this circumplex, emotions can be mapped onto roughly four quadrants: high-pleasantness and
high arousal; low-pleasantness and high-arousal; low-pleasantness and low-arousal; high-pleasantness and low-arousal. Although physiological instruments are sometimes used to identify levels of arousal and pleasantness33, most emotion
measurement tools are verbal self-report scales. There have only been a limited number of tools or instruments developed that are non-verbal. The self-assessment Manikin graphically represents by cartoon drawings the amount of valence, arousal, and dominance34. A product emotion measurement tool (Premo) has been developed to measure more
complex/cognitive emotions that are relevant for the experience of product sounds30, 35.
1.4. Overview of the experiments
In two experiments the interaction between auditory and visual information will be studied. In the first experiment, the interaction between recordings and pictures of product types that vary on their design quality will be investigated. Especially, the influence of auditory and visual information on the perceived quality and pleasantness will be studied. In the second experiment, the influence of abstract (rough) sounds on a validated set of affective pictures will be studied. The affective pictures and sounds will be chosen on their position in the circumplex model of core affect in such a way that each quadrant of the circumplex will be represented.
2. EXPERIMENT 1
The perception of auditory quality of several products has been studied (e.g., cars, vacuum cleaners)3, 6, 7, 8, 10. In addition,
aesthetic aspects of visual forms have also been studied11. In this study, the interaction between the auditory and visual
modality is studied on perceived aspects of luxury, pleasantness, quality, and user-friendliness. It is investigated if products of an expensive design (e.g., vacuum cleaner) also generate higher quality sounds than a product of the same product type but of an inexpensive design. In addition, it is investigated if the visual representation of the product would influence the experience of the sound. In other words, would a product of an inexpensive design result in lower evaluation ratings than a product of an expensive design.
2.1. Method
A between-within subject design was employed with visual information (no image, text, inexpensive, expensive) as between subjects factor and product type (juicer, vacuum cleaner, water boiler) and sound quality (expensive,
inexpensive) as within subject factors. Four dependent variables (luxury, pleasantness, quality, user-friendliness) were measured on a 7-point scale.
2.1.1. Participants
Forty-eight students of the Delft University of Technology participated voluntarily. The age of the students was between 18 and 25 years. The students were randomly distributed over 4 groups for the between subject condition. The student reported normal hearing and had normal or corrected vision.
2.1.2. Stimuli
Three types of domestic appliances were selected: juicer, vacuum cleaner, and water boiler. For each product type an inexpensive and an expensive version was selected. Thus, this yielded six images of products. For the visual stimuli the products were photographed. The auditory stimuli were recorded from a distance of 40 cm. Thus, there were two recordings for each product type, i.e., an inexpensive and an expensive "sound".
2.1.3. Apparatus
The sounds were recorded in a silent room using a Sennheiser e 385 microphone and a BOSS BR 35 digital studio. The stimuli were presented using a specially written program (in QT) on a Compaq Evo W4000 personal computer. The sounds were presented via headphones (Sony MDR CD 550) in a silent room.
2.1.4. Procedure
After a participant entered the silent room, (s)he was instructed concerning the procedure. The participant was then asked to sit in front of the computer screen. After a practice trial the participant started the experiment. Four groups of participants were presented with different visual representations (between subjects variable). The four presentations were: no image, text, inexpensive image and expensive image. Each visual image was presented with the sound of a product type of an inexpensive or an expensive sound. A participant had to rate the combination of sound and image (on a 7-point scale, with 1 indicating very low and 7 indicating very high) on the following attributes: luxurious, pleasantness, quality, and user-friendliness. The image was presented together with the attributes in a window on the computer screen. When the image appeared the sound played and the participant could listen to sound as many times (s)he wanted. The order of the sound-image combinations was randomized over subjects. The experiment was self-paced. Thus, each participant rated 6 sound-image combinations.
2.2. Results
In Table 1, the mean rating values and their standard errors (between parentheses) are presented as a function of Visual
Information (left column) and Attributes (Luxury, Pleasantness, Quality, User-friendliness). As can be seen there
appears is hardly any differentiation over the different conditions for each of the attributes. The No image and the Inexpensive condition have the lowest mean rating values, whereas the Text and the Expensive conditions have somewhat higher mean rating values for the four attributes.
Table 1. Mean rating values as a function of Visual Information and Attributes. The standard error of the mean is presented between parentheses.
Condition Luxury Pleasantness Quality User-friendliness
No image 3.39 (.20) 3.08 (.21) 3.86 (.18) 3.54 (.18)
Text 3.63 (.19) 3.54 (.20) 4.13 (.18) 3.85 (.15)
Inexpensive 3.17 (.18) 3.28 (.18) 3.79 (.16) 3.53 (.18)
Expensive 3.42 (.17) 3.51 (.17) 4.03 (.18) 3.86 (.15)
In Figure 1 the mean rating values for the attributes (Luxury, Pleasantness, Quality, User-friendliness) as a function of product and design quality are presented. The error bars represent the standard error of the mean. Because there was no effect of the visual image on the attributes, the visual condition is not presented in this figure. In the Expensive sound condition, the mean rating for the juicer is the lowest and for the water boiler the highest for the attributes Luxury, Pleasantness, and User-friendliness. For the attribute Quality the rating for the vacuum cleaner and the water boiler are almost equal in the Expensive sound condition. In the Inexpensive sound condition the mean rating values for the juicer and the water boiler are higher than in the Expensive condition for all attributes (except for the water boiler in the User-friendliness condition. The rating for the vacuum cleaner is almost equal in the Inexpensive and Expensive conditions for all attributes. Note that inexpensive and expensive relate to the recorded sound of an inexpensive and expensive design, respectively.
An ANOVA using a General Linear Model for Repeated Measures was conducted on the rating values with Visual
Information (no image, text, inexpensive image and expensive image) as between-subjects factor and Product Type
(juicer, vacuum cleaner, water boiler) and Design Quality (inexpensive, expensive) as within-subjects factors. No significant main effects of Visual Information were found for the Luxury (F(3,44) = .67, MSE=2.53, ns), Pleasantness (F(3,44)=1.24, MSE=3.36, ns), Quality (F(3,44)=1.18, MSE=4.0, ns), and User-friendliness (F(3,44)=.88, MSE=2.45, ns) attributes. The main effects for Product type were significant for the attributes Luxury (F(2,88)=29.58, MSE=46.76,
p<.001), Pleasantness (F(2,88)=30.57, MSE=63.26, p<.001), Quality (F(2,88)=16.26, MSE=25.07, p<.001), and User-friendliness (F(2,88)=19.52, MSE=34.19, p<.001). The main effects for Design Quality were significant for the attributes Luxury (F(1,44)=12.11, MSE= 27.50, p<.005), Pleasantness (F(1,44)=33.34, MSE= 56.89, p<.001), Quality
(F(1,44)=23.54, MSE= 45.13, p<.001), and User-friendliness (F(1,44)=14.98, MSE= 20.06, p<.005). Only significant interaction effects between Product type and Design Quality were found for Luxury (F(2,88)=7.35, MSE=11.35, p<.005),
Pleasantness (F(2,88)=19.93, MSE=26.98, p<.001), Quality (F(2,88)=26.00, MSE=34.51, p<.001), and User-friendliness
tthI
p rca u 1—•-- nexpensive —•— !xpenslve
Juicer vacu uIncIe aner waterbol icr
tthI
p rca u 1 5. 5. 4. 2. p rca u 1 ProductI.
7-B-I
2-Figure 1. The mean ratings for the verbal attributes (Luxury, Pleasantness, Quality, User-friendliness) as a function of Product (juicer, vacuum cleaner, water boiler) and Quality (inexpensive, expensive). The error bars represent the standard error of the mean.
2.3. Discussion
The main finding is that the design quality of a product sound is negatively associated with aspects as luxury, pleasantness, quality, and user-friendliness when a listener is asked to rate a product-sound combination. In certain areas of industry (e.g., automotive industry) attention is paid to the quality of sound36. Furthermore, sound quality design and
evaluation is a growing discipline10. Therefore, it is apparent that for domestic appliance the sound quality is not
considered even for expensive brands. Problems with the sound of products often occur and there are feasible solutions8.
However, at this moment no tools are available to predict the sound that a product will radiate. Only, after the product is constructed or an operational prototype has been made the sound can be evaluated. Another interesting finding is that the sound of the vacuum cleaner was rated in most cases higher than the sound of the juicer. This may be contradictory of what one might expect. Furthermore, there was no differentiation between the inexpensive and expensive design for all attributes. This may indicate that the construction of the two vacuum cleaners was somewhat similar causing similar sounds.
No effect of visual information was found on aspects as luxury, pleasantness, quality, and user-friendliness. This is somewhat surprising because it has been found that for most products visual information appears to be more dominant15, 37. In addition, it was found that the type of visual context 38 influenced the response times to sounds and the recognition
of sounds. An explanation may be that because of the experimental set up listeners were more focused on evaluating the sound than on the combination of the visual appearance and the sound of the product. The use of headphones and of a single picture (instead of a small movie) may have caused this shift in attention.
This experiment used recorded sounds and images of existing products and evaluated the product-sound combination on one emotional dimension (i.e., pleasantness). In addition, the sounds and images were not tested independently. This may have introduced too much variability. In the next experiment, the selection of sound and images is more controlled. The images and sounds have been validated in previous studies on their experience on the valence and arousal dimensions.
3. Experiment 2
The interaction between the experience of affective pictures and affective sounds will be investigated. The affective pictures will be taken from the International Affective Picture System 39. It has been found that the valence evoked by
these pictures is dominant over the valence evoked by a human voice (singing)16. Listeners were more likely to ignore
the sounds when they were asked to attend the pictures than to ignore the pictures when they were attending the sound. In addition, it was found in the ERP study that congruent auditory information enhanced the processing of the visual information. However, this study employed only three levels of valence (sad, neutral, happy) and did incorporate the levels of arousal in combination with an affective sung human voice.
In the present study, the interaction between the visual and auditory modalities is studied using sounds and pictures of which the placement on the circumplex of core affect of pictures has been determined. In this study, the dimensions of valence and arousal will be used to select the pictures for each quadrant of the circumplex of core affect. The reason for this limitation is two-fold. First, dominance is often associated with one of the other dimensions. Second and more important, the position of sounds employed sounds is only known for the pleasantness and arousal dimension. Using the IAPS pictures can be chosen that would fit each quadrant of the circumplex of emotions.
As mentioned before, one can systematically vary roughness of frequency-modulated tones using modulation frequency and modulation depth. It has also been found that an increase of roughness results in an increase of sensory pleasantness. Van Egmond41 found that the modulation frequency was associated with the pleasantness dimension and modulation
depth with the arousal dimension. The advantage of the use of these sounds over a human voice is that these tones can be generated systematically. Therefore, it is possible just as with the pictures to select sounds that can be positioned onto each quadrant of the circumplex model of core affect. Note that the pleasantness dimension underlying emotion is different (but probably related) than the sensory pleasantness dimension of which roughness is one of the parameters. Sound and pictures that can be positioned in each of the four quadrants of the circumplex model of core affect will be chosen and systematically interchanged. Thus, a sound that evokes a high level of valence and arousal will, for example, be combined with a picture that evokes a low level of valence and a low level of arousal.
3.1. Method
A between-within subjects design was employed with sex (male, female) as between subjects factor and sound (6 levels) and picture (6 levels) as within subject factors. In addition, to the selection of sounds and pictures that can be positioned on the circumplex of emotions, a sound and a picture were chosen that had a neutral position on the pleasantness and arousal dimension. Furthermore, a no-sound and a no-picture condition were added. The sounds and pictures were evaluated on the pleasantness and arousal dimensions.
3.1.1. Participants
Twenty-one students of the Delft University of Technology participated of which eleven were male (age M=22.7 years,
SD=2.20 years) and ten were female (age M=21.6 years, SD=1.51 years). All participants had normal or corrected vision
and reported normal hearing. Initially, twenty-four participants started but for three of the participants the data got corrupted while writing them to the hard disk.
3.1.2. Stimuli
Visual stimuli were taken from the International Affective Picture System39 and the sounds were generated using the
The sounds were generated using the following formula: 0.5*sin(2*π*fc*t+(md/fm)*sin(2*π*fm*t))
In this formula the carrier frequency (fc), the modulation frequency (fm), and the modulation depth (md) were varied to generate the sounds. The sounds had a carrier frequency (fc) of 1000 Hz and had a duration of 500 ms. An offset and offset amplitude envelopes of 10ms were used to prevent the loudspeakers from clicking. In Table 2 the values for the modulation frequencies (fm), modulation depth, the nominal pleasure and arousal levels (stemming from Van Egmond41)
and the sound labels are presented. In total 5 sounds were generated that combine levels of high and low pleasantness and high and low arousal.
Modulation Frequency (fm) Modulation Depth (md) Pleasantness Arousal Label
1 600 High High hhs
1 30 High Low hls
35 300 Neutral Neutral nns
70 600 Low High lhs
70 30 Low Low lls
Table 2. Settings of frequency modulated sounds used in the experiment. The first two columns contain the modulation frequency and depth. Columns 3 and 4 contain the expected nominal pleasure and arousal levels (see Van Egmond, 2004). The column Label contains the coding used in the figures.
3.1.2.2. Pictures
Ten pictures were chosen from the International Affective Picture System (IAPS) 39. The pictures were chosen such that
pleasantness and arousal levels were systematically combined. For each combination of pleasantness and arousal 2 pictures were chosen. Because each combination pleasantness and arousal was combined with five sounds, two pictures were used to avoid overexposure of the same picture. In Table 1 the chosen pictures are presented with their reference number (Slide No. from IAPS). In addition, the mean valence and the arousal levels (from the IAPS) are presented. Columns five and six contain the nominal levels of pleasure and arousal the pictures should evoke. It can be seen that high and low levels of pleasantness and arousal have been systematically combined. Consequently, pictures are positioned on every quadrant of the circumplex model of core affect. In the last column the labels that will be used in the figures are presented.
Description Slide No. Valence Arousal Pleasantness Arousal Label
Erotic Female 4220 8.02 7.17 High High hhp
Erotic Female 4290 7.61 7.20 High High hhp
Flower 5010 7.14 3.00 High Low hlp
Clouds 5891 7.22 3.29 High Low hlp
Jail 2722 3.47 3.52 Low Low llp
Homeless Man 9331 2.87 3.85 Low Low llp
Nudists6 8466 4.86 4.92 Neutral Neutral nnp
Boxer 8232 5.07 5.10 Neutral Neutral nnp
Mutilation 3000 1.45 7.26 Low High lhp
DeadBody 9252 1.98 6.64 Low High lhp
Table 3. Chosen pictures from the International Affective system. The first column contains the description. The second, third, and fourth columns contain the reference, valence and arousal levels from the International Affective Picture Set. The fifth and the six columns contain the nominal pleasantness and arousal levels. The last column contains the labels of the sounds.
3.1.3. Apparatus
The experiment was performed in a soundproof booth. The stimuli were presented using a specially written program in QT (object C variant). The program was self-paced and ran on a Macintosh G4 computer. The sounds were presented through loudspeakers at a sound pressure level of 70 dB.
3.1.4. Procedure
After a participant entered the soundproof booth the procedure of the experiment was explained. The experiment consisted of two parts. In one part, a participant had to rate the picture-sound combinations. In the other part, the participant had to rate the sounds (a grey-background was used instead of a picture) and the pictures (no sound was used). The order of these two parts was randomly determined for each subject. The division in the experiment was used in order to avoid confusion. Two 7-point semantic differential scales were used to rate the pictures and the sounds. The number 1 indicated unpleasant and calm and the number 7 indicated pleasant and excited. There were 5 different levels of pleasantness-arousal combinations for the sounds and the pictures. For the pictures, two variants were chosen in order to avoid overexposure. This resulted in total in 25 picture-sound combinations. In addition, the five sounds (no-picture condition) and five pictures (no-sound condition) were rated separately on the semantic differential scales. Thus, a participant had to rate 35 stimuli in total. Note that the combination of no-sound and no-picture was not presented because participants then had to rate 5 times nothing. This yielded in an incomplete design. The two variants of the pictures were randomly assigned to a specific sound condition. The order of the stimuli was randomized for each participant. The window of the computer program consisted of a canvas (in which a picture or a grey surface was shown), above the canvas a button to play the sounds, and on the right side the semantic differential (consisting of 7 radio-buttons). On the bottom right corner a button was present that was used to start the experiment or to proceed to the next stimulus. A participant had to listen to the sound and rate the stimulus on the two scales before (s)he could proceed. Between the two parts of the experiment there was a small break in which the new stimuli were loaded. The entire procedure was self-paced. A trial version preceded the actual experiment using 3 picture-sound combinations, one sound without picture, and one picture without sound. These five stimuli were different from those of the actual experiment.
3.2. Results
The data were analyzed using a linear mixed model (SPSS version 16.0) with subject as subject variable, picture and sound as repeated variables. The model was a full factorial model with sex, sound and picture as fixed factors. This model was used because of the missing data for the no sound and no picture condition. This incomplete design could not be analyzed with the General Linear Model for Repeated Measures employed in Experiment 1.
For Pleasantness, there were significant main effect for the variables Sex (F(1, 500.67) = 38.71, p<.001), Picture (F(5, 233.95) = 102.07, p<.001), and Sound (F(5, 175.52)= 29.73, p<.001). In addition, there were interaction effects for
Pleasantness between Sex and Picture (F(5, 179.10)= 6.73, p<.001) and between Sound and Picture (F(24, 87.13)= 4.88, p<.001). For Arousal, there were main effects for Sex (F(1, 557.91) = 12.21, p<.005), Picture (F(5, 179.10)= 41.86,
p<.001) and an interaction effect between Sex and Picture (F(5, 179.10) = 6.02, p<.001). No significant effect for Sound
on Arousal was found, F(5, 170.10)= 1.35, ns.
Overall the female participants gave lower ratings than male participants for the pleasantness (M=2.67, SD=1.60 and
M=3.23, SD=1.40, respectively) and arousal (M=3.89, SD=1.60 and M=3.56, SD=1.45, respectively). In Figure 2 the
mean ratings for Pleasantness and Arousal are presented as a function of Picture and Sex for the no-sound condition. It can be seen that the pictures received comparable rating values to those of the IAPS values. However, the ratings for pleasantness for the pictures in conditions hhp (high pleasure and high arousal) and hlp (high pleasantness and low arousal) differ between male and female participants. The pleasantness for the female participants is lower than for the male participants in the high pleasantness and high arousal condition, whereas for the high pleasantness and low arousal condition they are somewhat higher. For the arousal there seems to be a general trend that the pictures obtained a lower arousal rating from the female than from the male participants. However, the difference for the high pleasantness and high arousal picture condition appear to be larger than for the other conditions. This may explain the interaction effect between Sex and Picture for both pleasantness and arousal.
F
l
— A(]
I —--
I I-U
— = =Figure 2. Mean rating values for Pleasantness (left figure) and for Arousal (right figure) as a function of picture (x-axis) and sex (legend) for the no-sound condition is presented. Error Bars represent standard error of the mean.
In Figure 3 the mean ratings for pleasantness and arousal are represented as a function of picture (x-axis) and sound (legend). Note that there is no rating for the no-picture (nop) and no-sound combination. The levels of the pleasantness and arousal ratings for the different picture conditions seem to be comparable with the values of the IAPS (cf. Table 3). If the left and the right graphs are compared it can be readily seen that sound has an effect on the pleasantness dimension but not for the arousal dimension. In general, it can be seen that sound negatively influences the pleasantness ratings, because the no-sound (nos) condition is higher than for the conditions with sound. The effect appears to be the largest for the high pleasantness low arousal picture condition. If the non-picture condition (sound only) is compared, the effect of picture appears to be that the ratings increase, except for the high pleasure low arousal sound condition that appears to determine the maximum rating (open square). The high-pleasantness/low-arousal sound condition is higher than for the no-sound condition for the low-pleasantness/low-arousal, low-pleasantness/high-arousal, and neutral picture conditions. In addition, sound improved the ratings to the most negative picture condition (lhp) for the lls, lhs, nns, hls sound conditions slightly.
Figure 3. The mean ratings for pleasantness and arousal ratings as a function of picture and sound. The error bars
3.3. Discussion
One of the main findings is that sound affected the pleasantness ratings to pictures but did not affect the arousal ratings. The effect on the pleasantness ratings is mostly negative, but in case of the sound with a high pleasantness and low
arousal levels there is a slight improvement in the pleasantness ratings for most of the picture conditions. For the low-pleasure/high arousal picture condition most sounds had a slight positive effect on the picture ratings. These differences and the larger effect of sound on the high-pleasure/low-arousal condition may explain the interaction effect between sound and picture (although there are so many degrees of freedom that it will be difficult to indicate the exact cause of this interaction effect). In other words they differentiate over the arousal dimension, although this differentiation is less than of the sounds on the pleasantness dimension. An explanation may be that the sounds are synthesized sounds having hardly any ecological validity (the sounds can be considered as alarm sounds). The combination of these sounds with highly ecological valid pictures may result in a combination that is incompatible to a subject. Therefore, the effect on the pleasantness ratings may always result in a more negative judgment. However, it should be noted that there is a different influence in the level that the sounds affect the pleasantness ratings. Furthermore, it has also been found that even with sung tones (more ecologically valid) the effect for pictures on the perceived valence of the tones is larger than the effect of sung tones on pictures. In addition, in negative picture condition the no-sound ratings are often higher than in the picture condition. This may indicate that the incompatibility is not the major cause of the difference. If the sounds are considered as alarm sounds (and evoke as sense of sensory unpleasantness) this may explain the negative effect on the pictures in the high pleasure condition.
The finding that sound affects the pleasantness ratings and not the arousal ratings are more difficult to explain because an earlier study showed that these sounds could be mapped onto the pleasantness and arousal dimension. It may be that the arousal level induced by the realistic pictures is dominant (or suppresses) the effect induced by the sounds. This may be explained by considering that the sounds do not have any meaning for the subjects, whereas the pictures do have a meaning. This may mean that evoking a sense of arousal always involves the attribution of meaning or an underlying cognitive process when seeing pictures and hearing sounds. Van Egmond8 already suggested that an alarm sound might
evoke a sense of unpleasantness or urgency because of the featural aspects of the sounds. A bottom-up process will determine these. The need to act if one hears an alarm sound may be not evoked if the sound is not meaningful. Thus, the level of arousal when hearing the sound may then not increase. Another reason may be that the pleasure-arousal values of the frequency-modulated tones were indirectly derived. In Van Egmond 14, people had to choose a sound for a
specific cognitive emotion. The values of pleasure-arousal of these emotions were then used in the present study as the pleasure-arousal values for the frequency modulated tones. Thus, in a next study it would be better to first have people rate the frequency-modulated on these dimensions. In general, female participants have given lower ratings for the pleasantness and the arousal dimension. However, this affect is especially apparent (and expected) for the high-pleasure/high arousal pictures that contained female nudity. This probably is the cause between the relatively big difference in male and female ratings for the high-pleasure/high arousal pictures (and thus explaining the interaction effect), but this does not explain why female participants have given a lower rating overall.
4. General Discussion
In two experiments it has been shown that sound and pictures can affect our experience of the sound. In Experiment 1, it has been shown that for ecological valid sounds and ecological valid pictures, the sound is the dominant modality for certain product. The effect of seeing a more expensive or an inexpensive design did not affect the judgment of the sound on aspects of luxury, pleasantness, quality, and ease of use. That luxury and quality aspects were not affected by seeing a more expensive design may have implication for the industry. They need to incorporate the design of the sound in order not to affect the positive experience of the visual appearance of the product. However, this means that in the prototype phase of the design process sound needs to be incorporate. This will need the development of tools and methods that allow to “prototype” a sound. In Experiment 2, it has been shown that ecological valid pictures affect the pleasantness and arousal more than abstract sounds do. Especially, the arousal dimension seems to be unaffected by sound. The sounds affect the pictures in the high-pleasantness picture condition negatively, whereas a slight increase in pleasantness has been observed for low-pleasantness picture condition. However, the method of using frequency-modulated tones seems to be promising, because they allow a systematic variation in pleasure and arousal levels. In future research, the interaction between the visual and auditory systems needs to be more systematically investigated. Therefore, it would be interesting to find an aspect in the visual modality that can be manipulated just as systematically as roughness in the auditory modality. Especially, it may be useful to manipulate perceptual aspects that two domains have in common (e.g., an aspect as roughness).
I gratefully Phil van den Eerenbeemt, Albert Kannemans, Arno Scheepens, and Alex Zakkas for setting up and conducting the experiments.
REFERENCES
1. Hevner, K. (1935). Expression in music: A discussion of experimental studies and theories. Psychological Review, 186-204.
2. Lipscomb, S. D., & Kendall, R. A. (1994). Perceptual judgment of the relationship between musical and visual components in film. Psychomusicology, 13, 60-98.
3. Kuwano, S., Fastl, H., Namba, S., Nakamura, S., &Uchida, H. (2002). Subjective evaluation of car door sound.
Proceedings of the Sound-Quality Symposium, Deadborn MI, 2002.
4. Block, F. E., Nuutinen, L., & Ballast, B. (1999). Optimization of alarms: a study on alarm limits, alarm sounds, and false alarms, intended to reduce annoyance. Journal of Clinical Monitoring and Computing, 15, 75-83.
5. Edworthy, J., & Adams, A. (1996). Warning design: a research prospective. London: Taylor & Francis.
6. Bodden, M., & Igsleder, H. (2002). Active sound design: Vacuum cleaner. Revista de Acustica (special issue Froum Acusticum, Sevilla, Spain), 33.
7. Guyot, F., Christine, M., Castellengo, M., & Fabre, B. (1995). Characterization of the sound quality of vacuum cleaners. Journal of the Acoustical Society of America, 91, 3048.
8. Van Egmond, R. (2008). The experience of product sounds. In H.N.J. Schifferstein and P. Hekkert (eds.), Product
Experience (Chapter 3). London: Elsevier.
9. Zampini, M., Guest, S., & Spence, C. (2003). The role of auditory cues in modulating the perception of electric toothbrushes. Journal Of Dental Research, 82(11), 929-932.
10. Blauert, J., & Jekosch, U. (1997). Sound-Quality Evaluation - a Multi-Layered Problem. Acustica, 83(5), 747-753. 11. Hekkert, P., & Leder, H. (2008). Product aesthetics. In H.N.J. Schifferstein and P. Hekkert (eds.), Product
Experience (Chapter 10). London: Elsevier.
12. Schifferstein, H.N.J., & Cleiren, M.P.H.D. (2005). Capturing product experiences: a split-modality approach. Acta
Psychologica, 118, 293-318.
13. Schifferstein, H.N.J., Tanudjaja, I. (2004). Visualising fragrances through colours: the mediating role of emotions.
Perception, 33, 1249-1266.
14. Van Egmond, R. (2004). Compact disc cover design: transfer of musical style and emotional contents into graphical design. In D. McDonagh, P. Hekkert, J. van Erp, and D. Gyi (Eds.) Design and emotion: The experience of everyday
things, pp. 238-242. London: Taylor and Francis.
15. Schifferstein, H.N.J. (2006). The perceived importance of sensory modalities in product usage: A study of self-reports. Acta Psychologica, 121, 41-64.
16. Spreckelmeyer, K. N., Kutas, M., Urbach, T. P., Altenmuller, E., & Munte, T. F. (2006). Combined perception of emotion in pictures and musical sounds. Brain Research, 1070(1), 160-170.
17. Zwicker, E., & Fastl, H. (1990). Psychoacoustics. Facts and Models. Berlin: Springer Verlag.
18. Spence, C., & Zampini, M. (2006). Auditory contributions to multisensory product perception. Acta Acustica united
with Acustica, 92, 1009-1025.
19. Jousmäki, V., Hari, R. (1998). Parchment-skin illusion: Sound-biased touch. Current Biology, 8, 869-872.
20. Guest, S., Catmur, C., Lloyd, D., Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental
Brain Research, 146, 161-171.
21. Zampini, M., Guest, S., & Spence, C. (2003). The role of auditory cues in modulating the perception of electric toothbrushes. Journal Of Dental Research, 82(11), 929-932.
22. Schifferstein, H.N.J., & Spence, C. (2008). Multisensory product experience. In H.N.J. Schifferstein and P. Hekkert (eds.), Product Experience (Chapter 5). London: Elsevier.
23. Izard, C. E. (1977). Human Emotions. New York: Plenum.
24. Mehrabian, A. & Russell, J. A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in
Personality, 11, 273-294.
25. Plutchik, R. (1980). Emotion: A psychoevolutionary synthesis. New York: Harper & Row.
26. Roseman, I.J. (1991). Appraisal determinants of discrete emotions. Cognition and Emotion, 5, 161-200. 27. Ekman, P. (1971). Universals and cultural differences in facial expressions of emotions. In J.K. Cole (Ed.),
28. Tomkins, S. S, (1984). Affect theory. In K. R. Scherer & P. Ekman (Eds.), Approaches to emotions (pp.162-196). Hillsdale NJ: Erlbaum.
29. Ortony, A. Clore, G.L., & Collins A. 1988. The cognitive structure of emotions. Cambridge: Cambridge University Press.
30. Desmet, P.M.A. 2002. Designing Emotions. Doctoral Dissertation, Delft University of Technology, Delft, The Netherlands.
31. Van Egmond, R., Desmet, P., Van Der Helm, A. (2004). Basic and cognitive emotions in sound perception. In A. Kurtgözü (Ed.). Proceedings of the Fourth International Conference on Design and Emotion, pp. 1-11. Ankara: METU.
32. Russell, J.A. 1980. A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161-1178. 33. Picard, R.W. (2000). Toward computers that recognize and respond to user emotion. IBM Systems Journal,
39(3&4).
34. Bradley, M.M., Lang, P.J. (1994). Measuring emotion – the self-assessment mannequin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25, 49-59.
35. Desmet, P.M.A. (2008). Product emotion. In H.N.J. Schifferstein and P. Hekkert (eds.), Product Experience (Chapter 15). London: Elsevier.
36. Bijsterveld, K. (2008). Mechanical Sound. Technology, Culture and Public Problems of Noise in the 20th Century (accepted MIT Press/Inside Technology Series).
37. Posner, M.I., Nissen, M.J., & Klein, R.M. (1976). Visual dominance: an information-processing account of its origins and significance. Psychological Review, 83, 157-171.
38. Özcan, E. & Van Egmond, R. (submitted). The Effect of Visual Context on the Identification of Ambiguous Environmental Sounds.
39. Lang, P.J., Bradley, M.M., & Cuthbert, B.N. (2005). International affective picture system (IAPS): Digitized photographs, instruction manual and affective ratings. Technical Report A-6. University of Florida, Gainesville, FL. 40. Van Egmond, R. (2004). Emotional experience of frequency modulated sounds: Implications for the design of alarm
sounds. In: D. de Waard, K.A. Brookhuis and C.M. Weikert (Eds.). Human factors in design, pp. 345-356. Maastricht, The Netherlands: Shaker Publishing.
41. Boersma, P., & Weenink, D. (2006). Praat doing phonetics by computer (Version 4..5.08) [Computer Program]. Retrieved December 20, 2006, from http://www.praat.org/
