Surprise - How is it Processed

Written for U of M Psych 5063 - Introduction to Functional Magnetic Resonance Imaging

Abstract

We have found that activations in the amygdala, and the anterior cingulate cortex are correlated with surprising visual, audio and mixed visual and audio stimuli in our individual participant with one session.  The ventromedial prefrontal cortex (vmPFC) was not activated, although we expected it to be activated.  We cannot generalize this to any other population with any significant degree of confidence.  We used a particular set of options in FSL tools to achieve this result and this paper will cover how we arrived at the choices we made.

Introduction

There are six basic emotions: happiness, sadness, anger, fear, disgust, and surprise.  Surprise is the most understudied of Ekman’s six basic emotions. It is defined as a transient emotional state experienced/resulting from an unexpected event.  It can have any intensity and valence.  It is divided into:

“Wonder”- which is generated when there is something unexpected and/or rare.
“Startle”- which is generated by an unexpected stimulus.

Surprise is a stimulus that is hard to explain.  The more cognitive work is needed to explain a stimulus, the more intense the surprise reaction is.  Early theories indicate it as a basic emotion with a universally recognized facial expression (from Paul Ekman’s work).  Recent theories categorize it as a cognitive state - the process of making sense of unexpected events.  Surprise can be positively or negatively valenced, as opposed to many emotions which have a definite and particular direction or valence.  However, surprise is rarely neutral.  It tends to be a low intensity negative if it doesn’t evoke a specific direction. 

The theory of surprise

The literature contains two groups of theories on surprise, probability and error prediction.

Probability

Meyer, Reisenzei, & Schützwohl (Meyer, Reisenzei, & Schützwohl, 1997), said that surprise is a low probability event or a disconfirmed expectation.

Error prediction

Alternatively, surprise can be seen as a failure to make sense of an event (Kahneman & Miller, 1986; Maguire et al., 2011).  Available knowledge in long-term memory is used to resolve surprise, by understanding and integrating the surprising event.

Our Approach

We tended to believe in the probability approach.  We thought that investigating surprise might help with understanding some behavioral implications.  Surprise induced attention leads to rapid learning.  We chose to investigate surprise because it is easier to produce surprise in a class setting, since our consent form was vague, and we hoped to be given broader acceptance of surprising behavior as we were more likely to make mistakes.

Problem Statement

Surprising visual, audio and mixed visual and audio stimuli during a decoy task with unexpected visual, audio, and audiovisual surprising stimuli will activate specific areas of the brain.

Neurological Response to Surprise

We think that the central nucleus of amygdala, the anterior cingulate cortex, and the ventromedial prefrontal cortex (vmPFC) are likely to be activated by surprise.

Physiological Response to Surprise

Many physiological responses to surprise are common in the literature: skin conductance, and heart rate increase and blood volume, pulse and pulse transit time decrease.  These may lead to BOLD changes we might be able to see.

Materials and Methods

Participant

We used Alexander J. Bratch as our participant (or subject).  He is a member of our class, and a graduate student in Psychology.  He suffers from tinnitus, which may affect the experimental results.  He was selected on his willingness to participate, experience in staying still, and our knowledge of his likes and dislikes, etc.  Bratch was an ideal participant for the aforementioned reasons.  Alex is ambidextrous, perhaps brought on by being left-handed in a right handed world.  He also has an unusual configuration of brain blood vessels.

Main Experiment

We, for the main experiment, have chosen to present 30 surprise pictures, videos and sounds in a random approach within the 200 decoy stimuli using an event-related design. We have defined a localization procedure to identify the amygdala, the anterior cingulate cortex, and the ventromedial prefrontal cortex (vmPFC) in the subject’s brain by presenting emotional pictures.  This is followed by a post experiment survey of the subject, presenting the surprises once again and getting a rating by the subject of the emotions associated with the stimulus.

Decoy Task

The dominant activity during the study runs is watching a large number of stimuli, intended to be boring.  Using an event-related design we presented: 200 stimuli, each for 383 ms:

·         40 auditory-only /ba/ and /ga/ syllables

·         40 visual-only /ba/ and /ga/ syllables

·         40 congruent audiovisual (AV) /ba/ and /ga/ syllables

·         40 McGurk incongruent AV syllables (i.e., visual /ga/ with auditory /ba/)

·         40 non-McGurk incongruent AV syllables (i.e., visual /ba/ with auditory /ga/)

Inter-stimulus interval was jittered with one of 2 s, 4 s, or 6 s.

A single male speaker was used for maximum prediction, and to remove confounds.  A demonstration of the kind of stimuli was shown to the participant before scanning to establish his expectation of the decoy material.

Experimental Task

There were thirty surprise stimuli presented during the decoy tasks in a random (jittered) fashion lasting 0.5 seconds each.  There were ten each of audiovisual stimuli, visual stimuli, and audio stimuli.  These were very unexpected and hard to explain (surprising) compared to the stream of pictures and audio of the single male speaker, since the participant had no knowledge that they would be happening, and especially since some stimuli included pictures of him.

  

We have 40 * 5 = 200 decoy stimuli with 30 target trials. These trials were presented with five runs with 3 min 52 seconds each.  During the experimental scans, the participant was instructed to watch and listen carefully to the stimuli as he would be tested on the experiment after it (which did not happen, just to make him focus during scans).  We only were able to conduct one session with one subject, so predicting effects of surprise on a larger population are impossible.

Functional Localization Task

The localization procedure is a set of 50 highly emotional pictures, with stimulus duration of one second and the interstimulus interval (ISI) of one second.  This gave us fifty two-second-long trials for localization with the expectation of activating the amygdala, the anterior cingulate cortex, and the ventromedial prefrontal cortex (vmPFC).

 

Three Likert scales about each of the thirty experimental stimuli: arousal, valence, and liking

The survey of emotional content of the thirty stimuli asked the subject to review each stimulus and rate it on a scale of emotional content.  It was presented to the participant after the scanning session.

Purpose: to co-vary the activation by the responses to these three scales to rule out their effects, since the emotionality of any stimulus may mask the impact of surprise alone (cognitive subtraction approach).

The values were scored from 1 to 5 (left to right, respectively)

You can see the survey and the stimuli at Surprise Qualtrics Link.  They may not be available after a short while.  We ran some statistics for the results to see how they are distributed.

	Valence	Arousal	Liking
Stand. Dev	1.1055416	1.33166562	1.1055416
Average (mean)	3.33333333	2.6	3.33333333

Data Collection tools:

The fMRI data were collected on November 24, 2015 with the 3T-B Siemens MAGNETOM Prisma with Tim 4G Technology in Room 131-A at the Center for Magnetic Resonance Research, University of Minnesota.   This 3 Tesla scanner uses Total Imaging Matrix technology.  The Tim matrix concept allows for whole-body examinations with a field of view of up to 205 cm without repositioning the patient.

Scans for this experiment

For this session, the participant went through eight scans:

Scan 1 (Structural imaging): A T₁-weighted MP-RAGE (magnetization-prepared rapid acquisition with gradient echo) anatomical scan was acquired for registration purposes.  We acquired the MP-RAGE structural volume (TR = 2300, TE = 2.93, flip angle = 9°) with 200 transverse slices, each 1 mm thick and 1mm × 1 mm in-plane resolution, with a 208 x 226 matrix.  The results from this were labeled SE002. 

Scan 2-6 (Main experiment): Five T₂*-weighted gradient-echo-planar imaging (EPI) scans for detecting surprise related effect in the hemodynamic response to the stimulus.  Each functional run involved the acquisition of 116 EPI volumes (gradient-echo, TR = 2000, TE = 25, flip angle = 80°), each with 34 transverse slices, 3 mm thick, and a 64 × 64 matrix yielding an in-plane resolution of 3 mm × 3 mm. A functional run lasted 3 min and 52 s, and the subject completed 5 functional runs.  The results from this were labeled SE003 though SE007.

Scan 7 (Independent functional localizer): T₂*-weighted EPI scan for identification of Regions of Interest (ROIs).  We conducted one T₂*-weighted gradient-echo-planar imaging (EPI) scan for localizing the brain regions expected to be activated during the functional runs.  This scan measured the hemodynamic response to strong emotional stimuli.  The run involved the acquisition of 75 EPI volumes (gradient-echo, TR = 2000, TE = 25, flip angle = 80°), each with 34 transverse slices, 3 mm thick, and a 64 × 64 matrix yielding an in-plane resolution of 3 mm × 3 mm.  The results from this were labeled SE008.  However, the MRI machine was not properly set up and it scanned an incomplete image of the brain.  So, we did nothing with this scan.

Scan 8 (Fieldmap): We performed reversed-phase EPI scans to account for distortion and motion. The first of these scans was to capture the magnitude of the magnetic distortion.   The second was to capture the phase of the distortion.  Each functional run involved the acquisition of one volume, each with 34 transverse slices, 3 mm thick, and a 64 × 64 matrix yielding an in-plane resolution of 3 mm × 3 mm. The results from these were labeled SE011 and SE012.

Software tools used during the experiment

E-Prime Studio, E-Prime Run Version 2.0 was used to drive the experimental tasks and log the time and actions during the experiment.

The stimulus rating survey was presented through Qualtrics: Online Survey Software & Insight Platform.

Preprocessing

We examined the raw data, motion correction, slice timing correction, distortion correction, spatial smoothing, and spatial normalization (which isn't necessary for just one participant, but was helpful so we could better label active regions).

Data Analysis Tools

We used FSL from the University of Oxford to do the analysis.

Brain Extraction

First, we needed to adjust the anatomical scan to extract the brain from the rest of the anatomy (brain extraction).  We used FSL’s BET tool to do this.  We used the echo times from the mag and phase headers.  For our example, echo time 1 was 5.76, and echo time 2 was 8.22.  The difference was 8.22-5.76=2.46 and echo spacing was 0.49 from the scanner.  We ran Fsl_prepare_fieldmap, using fieldmap.radpersec.nii.gz and then BET.

Motion correction, slice timing correction, distortion correction, spatial smoothing, and spatial normalization

Our functional scan output (five experimental runs and the functional localizer) each needed several kinds of correction to get the data ready for analysis. 

First, we needed to run MCFLIRT motion correction (Motion Correction in FMRIB's Linear Image Registration Tool) to identify and adjust the data to align structures after head motion between volumes.

Second, we looked at slice timing correction.  Our instructor told us we did not need to correct for any differences due to the timing of the scans taking slices because the 3T scanners do a good job of capturing the images in two seconds, and thus the inter-slice motion is not very much.  Also, the Siemens machine does the slice acquisition in a pattern not supported by the native slice timing correction in FSL.  Thus, we made no slice timing corrections.

Third, we needed to adjust for the magnetic field distortion in the scanner.  This is called B0 Unwrapping in the Feat tool, and takes the images, magnitude and phase, we captured for distortion correction (labeled SE011 and SE012).  It also takes the effective EPI echo spacing (28 ms, in our case) and EPI TE (2.46 ms, in our case).  We also had to use an unwrap direction of -y.  The FSL tool for distortion correction introduced an artifact, and because of this, we dropped it.

Fourth, we did spatial smoothing to reduce the likelihood of false positives.  Due to the nature of our 3 mm dimensions, we chose 5 mm smoothing, to bring in all surrounding voxels. 

Fifth, we did spatial normalization to MNI using a 2MM image.

Sixth, we set high bandpass 90 ms cut-off and BET brain extraction on the fMRI data (as we did on the structural image above) to further localize the data and remove artifacts of respiration, etc.

We also used Boundary-Based Registration (BBR) to register the fMRI images to the structural images.  And, we used Normal Search because all of our images are in the same orientation.

Seventh, we used FILM prewhitening and the full model setup.  FILM (FMRIB's Improved Linear Model) in a robust and accurate nonparametric estimation of time series autocorrelation to prewhiten each voxel's time series; we did this for improved estimation efficiency compared with methods that do not pre-whiten.  We used the Gamma convolution setting.

Eighth, we used a Z threshold of 2.3 and a Cluster P threshold of 0.05.  The default is better than .05, which is too liberal.  We used actual z min/max and transparent blobs for rendering. Time series plots were created as well.

Feat

We used the FSL tool Feat, giving it the parameters, and letting it do this set of steps.

We ran the following first-level analyses:

●        Run1: input to the run 1 functional scan (SE003), and contrast run1decoy with run1surp txt files in the design, and the structural scan to co-register

●        Run2: input to the run 2 functional scan (SE004), and contrast run2decoy with run2surp txt files in the design, and the structural scan to co-register

●        Run3 input to the run 3 functional scan (SE005), and contrast run3decoy with run3surp txt files in the design, and the structural scan to co-register

●        Run4 :input to the run 4 functional scan (SE006), and contrast run4decoy with run4surp txt files in the design, and the structural scan to co-register

●        Run5: input to the run 5 functional scan (SE007), and contrast run5decoy with run5surp txt files in the design, and the structural scan to co-register

High-level Analysis

We fed the five outputs from the low-level analysis into a high level analysis to contrast the decoy tasks with the surprising stimuli.  We did not separate out the audio vs. the visual vs. the audio-video stimuli.

We used the default settings in FSL for Post-Stats.

We set up contrasts for each condition in each run:

EV1 value 1 and EV2 value 0 (beta-weight for EV1, the surprises) – giving cope1.
EV1 value 0 and EV2 value 1 (beta-weight for EV2, the decoys) – giving cope2.
EV1 value 1 and EV2 value -1 (EV1 > EV2, the surprise effect) – giving cope3.
EV1 value -1 and EV2 value 1 (EV2 > EV1, the decoy effect) – giving cope4.

Likert Scales

We decided not to run any analysis based on the Likert scales looking at the range of values and the standard deviation of those values.

Results

We anticipated greater activity during the surprise task in the amygdala, anterior cingulate cortex, and ventromedial prefrontal cortex. 

Activated areas

We found activation in the amygdala that was very intense.  The anterior cingulate cortex was activated but with less intensity.  And, the ventromedial prefrontal cortex was not activated significantly.  The cope3 contrast (Surprise – Decoy) shows these activations:

Discussion 

FMRI analysis is a fragile science, in my view, because it is based on very small changes in large systems that are monitored and reported in a complex process that is error prone.  In general, there are many wrong ways to analyze and set of fMRI data, and few right ways.  Generally missing from our class were methods for identifying analysis errors. 

Activation areas

We do not know why we did not quite find what we expected.  With our limited data set, we don’t have sufficient data to take it much further. To speculate, we could believe that the vmPFC was not activated because the emotions evoked were not strong enough or long-lasting enough to require invoking control from the vmPFC.

We anticipated seeing some impact of the physiological responses to surprise in the BOLD response, but we did not notice any effect of the physiological changes.

Looking through the rendered data for the 5 runs, run 4 seems to have much more activation than run 2 or 3 and runs1 and 5 show very little activation. 

We made errors in the process of collecting the functional localizer scan, which left us without the functional localizer input to identify regions of interest, so we performed whole brain analysis, which may have reduced the accuracy of our results.

As a group, we had a discussion about how to run the analysis – across all stimulus types or separately for Audio-only, Visual-only and Audio-Video.  The issues were around how much data we had and how much is needed for significant results based on degrees of freedom versus the possibility of different results based on those stimulus types.  We decided to go across all stimulus types, with an option to later compare the stimulus types.  We realized this was a violation of the usual research methodology of doing the smallest grouping first and combining later if they look similar.

Likert Scales

There are two possible methods of analyzing the relationship among the subjective ratings, emotionality dimensions, and the activity levels.

1. The effects of subjective ratings and emotionality dimensions are random

Subjective ratings and emotionality dimensions treated as co-variates.

2. The effects are fixed (ex. negative stimuli are more surprising)

Subjective ratings and emotionality dimensions treated as independent variables

As we said above, there are a sufficient range of values for each of these ratings, and we don’t think it is useful to analyze the impact of these ratings on the activation we found.

Future Studies

This study was inadequate to generalize, so more study is called for to gather more and better data on areas activated by surprise and potential relationships to emotional valence, long-term memory, short-term memory and many other possible independent variables and co-variants.

References

Ekman, P. and Friesen, W. 1971 Ekman, P., & Friesen, W. (1971).

Constants across cultures in the face and emotion.

Journal of Personality and Social Psychology, 17(2), 124-129.

Jenkinson, M., Bannister, P., Brady, J. M. and Smith, S. M. (2002) Jenkinson, M., Bannister, P., Brady, J. M. and Smith, S. M.

Improved Optimisation for the Robust and Accurate Linear Registration and Motion Correction of Brain Images

 NeuroImage, 17(2), 825-841, 2002.

Kahneman, D., and Miller, D. T. (1986) Kahneman, D., and Miller, D. T.

Norm theory: Comparing reality to its alternatives.

Psychological Review, 93(2), 136-153.

Meyer, W. U., Reisenzein, R., and Schützwohl, A. (1997) Meyer, W. U., Reisenzein, R., and Schützwohl, A.

Toward a process analysis of emotions: The case of surprise.

Motivation and Emotion, 21(3), 251-274.

Maguire, R., Maguire, P., and Kaene, M. T. (2011) Maguire, R., Maguire, P., and Kaene, M. T.

Making sense of surprise: An investigation of the factors influencing surprise judgements.

Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(1), 176-186.

Acknowledgments

Thanks to Alexander J. Bratch for being cooperative and (hopefully!) surprised.

My fellow students on our team, Sara J.M. Arnold, Sori Baek, Erin Begnel, Sekine Ozturk, and Luodi Yu have been great at cooperating and teaching me the basics of psychology.

I want to thank Amanda Rueter for help early in the class, and Brittany Bostrom for her help as well.

Thanks to Dr. Philip Burton for teaching and overseeing.

This research was made possible by the State of Minnesota and the University of Minnesota funding for the Center for Magnetic Resonance Research and the Biomedical Science Research Facilities Authority, among others.