1 cite as: J Gerontol B Psychol Sci Soc Sci, 2020, Vol. XX, No. XX, 1–9
doi:10.1093/geronb/gbaa137 Advance Access publication August 29, 2020
© The Author(s) 2020. Published by Oxford University Press on behalf of The Gerontological Society of America. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.
Research Article
Age-Related Changes in Audiovisual Simultaneity
Perception and Their Relationship With Working Memory
Yi-Chuan Chen, DPhil,
1,Su-Ling Yeh, PhD,
2,3and Pei-Fang Tang, PhD
4,5,*
1
Department of Medicine, Mackay Medical College, New Taipei City, Taiwan.
2Department of Psychology, National Taiwan
University, Taipei, Taiwan.
3Center for Artificial Intelligence and Robotics, National Taiwan University, Taipei, Taiwan.
4School and Graduate Institute of Physical Therapy, National Taiwan University, Taipei, Taiwan.
5Department of Physical
Medicine and Rehabilitation, National Taiwan University Hospital, Taipei, Taiwan.
*Address correspondence to: Pei-Fang Tang, PhD, School and Graduate Institute of Physical Therapy, National Taiwan University, Rm 325, Floor 3, No. 17, Xu Zhou Road, Zhong Zheng District, Taipei, Taiwan, ROC. E-mail: pftang@ntu.edu.tw
Received: January 23, 2020; Editorial Decision Date: August 19, 2020 Decision Editor: Angela Gutchess, PhD
Abstract
Objectives: Perceiving simultaneity of a visual and an auditory signal is critical for humans to integrate these multisensory
inputs effectively and respond properly. We examined age-related changes in audiovisual simultaneity perception, and the relationships between this perception and working memory performances with aging.
Methods: Audiovisual simultaneity perception of young, middle-aged, and older adults was measured using a simultaneity
judgment (SJ) task, in which a flash and a beep were presented at 1 of 11 stimulus-onset asynchronies (SOAs). Participants judged whether these two stimuli were perceived simultaneously. Precision of simultaneity perception, the SOA corre-sponding to the point of subjective simultaneity (PSS), and response errors at each SOA were estimated using model fitting. The precision and PSS are associated with multisensory perception per se, whereas the response error reflects executive ability when performing the SJ task. Visual working memory of the same middle-aged and older adults was measured using the Cambridge Neuropsychological Test Automated Battery (CANTAB) beforehand.
Results: Compared to young adults’ performances, middle-aged and older adults showed a decreased precision, a shift of
PSS toward the visual-leading SOAs, and increased response errors at the visual-leading SOAs. Among these changes, only the increased response errors correlated with worse spatial recognition memory in middle-aged and older adults.
Discussion: Age-related decrements in audiovisual simultaneity perception start from middle age and are manifested in
both perceptual and executive parameters. Furthermore, higher-order executive ability is plausibly a common cause for age-related degenerations in the audiovisual simultaneity perception and visual working memory.
Keywords: Audiovisual simultaneity window, Cross-modal temporal sensitivity, Simultaneity judgments, Working memory
When crossing a street, for example, seeing a moving car and hearing its engine’s revving noise can help people no-tice the car, and accordingly, decide to keep walking or stop. Such daily experiences support the idea that integrating multisensory information gives rise to a more accurate and precise perception, as well as a faster response regarding an object (e.g., Alais & Burr, 2004; van der Burg et al., 2008;
Chen & Spence, 2011; Ernst & Banks, 2002; Miller, 1991;
Vroomen & de Gelder, 2000). For older adults, it has been demonstrated that some cognitive performances, such as recognition of facial expressions and auditory objects, and the speed of visual object detection or discrimination, were improved following audiovisual integration as compared to unisensory conditions (e.g., Chaby et al., 2015; Heikkilä et al., 2018; Laurienti et al., 2006; Peiffer et al., 2007; see
de Dieuleveult et al., 2017; Freiherr et al., 2013, for
views). On the contrary, impaired audiovisual integration abilities have been found in age-related cognitive degen-eration and disorders, such as mild cognitive impairment and Alzheimer’s disease (Chan et al., 2015; Murray et al., 2018; Wu et al., 2012). Roberts and Allen (2016) proposed three hypotheses regarding the relationships between the declines in perception and cognition with aging: impover-ished sensory information increases cognitive load, poor cognitive resources narrow down the bandwidth of percep-tual processing, or there is a third factor associated with aging as a common cause for perception and cognition. The relations between multisensory perception and cognition therefore provide critical understanding regarding the sys-tematic changes in the aging brain, which has been rarely studied to date (e.g., Baum & Stevenson, 2017). Here we aimed to examine this issue by characterizing the detailed changes of audiovisual simultaneity perception with aging and relating it to older adults’ working memory perform-ances, given the fact that working memory has been shown to be declined with aging (Hedden & Gabrieli, 2004).
In order to accurately integrate sight and sound originating from the same source, perceiving their simul-taneous onset provides a critical cue (Stein & Meredith, 1993; Welch & Warren, 1980). Nevertheless, inconsistent results of age-related changes in audiovisual simultaneity perception have been reported (see Brooks et al., 2018, for a review). Some studies demonstrated that older adults tolerated a larger range of stimulus-onset asynchronies (SOAs) between visual and auditory signals when per-ceiving them as simultaneous compared to young adults; that is, the precision of audiovisual simultaneity perception decreases with age (Bedard & Barnett-Cowan, 2016; Chan et al., 2014a; Noel et al., 2016; Setti et al., 2011; Stevenson et al., 2018; Virsu et al., 2003). In contrast, other studies reported that the precision of audiovisual simultaneity per-ception, as well as the condition which the participants most likely judged the two signals being simultaneous (i.e., the point of subjective simultaneity, PSS), remained constant until around 80 years old (Basharat et al., 2018;
Bedard & Barnett-Cowan, 2016; Fiacconi et al., 2013). These contrasting results therefore motivated us to improve the methodology when measuring audiovisual simultaneity perception in different age groups.
A method commonly used to measure adults’ audio-visual simultaneity perception is the simultaneity judgment (SJ) task. On each trial, a visual flash and an auditory beep are presented at predesignated SOAs (auditory-leading, si-multaneous, or visual-leading), and participants have to judge whether the two stimuli were perceived at the same or different times. The proportion of simultaneous re-sponses increases when the two stimuli are presented closer in time, forming a Gaussian distribution as a function of SOA. In order to estimate the asymptote of the Gaussian distribution (i.e., nonsimultaneous responses were made reliably), we used a larger range of SOAs in the current study (±600 ms) as compared to previous studies (±300 ms,
e.g., Bedard & Barnett-Cowan, 2016; Noel et al., 2016;
Stevenson et al., 2018).
When fitting individual data and estimating critical parameters that determine a participant’s audiovisual simul-taneity perception, we used a model specifically designed for the SJ task developed by García-Pérez and Alcalá-Quintana (2012; see also Alcalá-Quintana & García-Pérez, 2013) rather than general mathematical functions used in previous studies (Basharat et al., 2018; Bedard & Barnett-Cowan, 2016; Noel et al., 2016; Stevenson et al., 2018). García-Pérez and Alcalá-Quintana’s model assumes that sensory signals are processed independently in each sensory pathway, and then reach a central comparator that decodes their arrival times. Hence, the perceived asynchrony be-tween the two signals is determined by the processing time and variance in each sensory pathway, giving rise to a prob-ability distribution rather than a constant. A person’s pre-cision of audiovisual simultaneity perception corresponds to a threshold that, if the perceived asynchrony in a trial is smaller than the threshold, then a simultaneous response would be made. Note that this model also assumes that a person may sometime misreport his or her judgment be-cause of eye blinks, lapse of attention, and errors of motor execution during the testing period. Hence, this model gen-erates eight parameters that account for a person’s perfor-mance in the audiovisual SJ task. The processing variability of visual and auditory signals (λV and λA, respectively) and their processing time difference (τ = τA − τV) are three parameters associated with unisensory processing. The pre-cision (or threshold) of simultaneity perception (labeled as δ) and the PSS correspond to half of the width and the mid-point of the distribution at the 50% simultaneous response, respectively. These two parameters are associated with au-diovisual perceptual processing. Finally, the participants’ misreporting “simultaneous” in the auditory-leading trials (εAF) and in the visual-leading trials (εVF), or misreporting “not simultaneous” in the 0-ms trials (εS) are three param-eters associated with the participant’s executive functions when performing the SJ task.
In the current study, the goal was to understand the link between audiovisual simultaneity perception and working memory with aging. To do so, we measured the age-related change of audiovisual simultaneity perception by testing three age groups: young, middle-aged, and older adults. Using the SJ task and García-Pérez and Alcalá-Quintana’s (2012) model, three unisensory parameters (λV, λA, and τ), two perceptual parameters (δ and PSS), and three executive parameters (εAF, εVF, and εS) determining the performance in the audiovisual SJ task were estimated. We then exam-ined the relations between audiovisual simultaneity percep-tion and working memory performances measured by the Cambridge Neuropsychological Test Automated Battery (CANTAB). Given that each parameter generated from the SJ task corresponds to a particular level of informa-tion processing, its correlainforma-tion with the working memory performances would provide critical insights regarding
the link between multisensory perception and cognition (Roberts & Allen, 2016). This correlation analysis mainly focused on the middle-aged and older adults in order to avoid being misrepresented by young adults’ age range and ceiling performance in the CANTAB (see below).
Method
Participants
Three age groups of participants were tested: 40 young adults (21 females, mean age = 26.1 years, range = 20–34 years), 36 middle-aged adults (31 females, mean age = 60.5 years, range = 53–64 years), and 39 older adults (28 females, mean age = 69.4 years, range = 65–81 years). The lower boundary of age in the older group was defined in accordance with the suggestion of World Health Organization (n.d.) for de-veloped countries. Additional eight participants (two mid-dle-aged and six older adults) were tested but not included in the final analysis because their scores in the Montreal Cognitive Assessment (MoCA, see below) were lower than 26, suggesting that their cognitive functions were degraded. The participants were recruited from Taipei metropolitan areas. All of the participants went through a visual acuity test and had normal hearing by self-report. Written consent from all of the participants was obtained before starting the study. All of the participants were naïve regarding the pur-pose of the study. The study was approved by Institutional Review Board of National Taiwan University Hospital (IRB number: 201212161RIND, 201503037RINB).
An analysis using G*Power (version 3.1.9.2; Faul et al., 2007) for one-way analysis of variance (ANOVA) suggests that when testing 90 participants (30 in each age group) with α = 0.05 and power = 0.95, the main effect of age group would reach effect size = 0.42 (i.e., a large effect).
Stevenson et al. (2018) tested five age groups, including children (N = 24), adolescents (N = 22), younger adults (N = 26), middle-aged adults (N = 31), and older adults (N = 35), and the effect size of the main effect of age group was 0.39 (η p2 = 0.13 was reported). This therefore suggests that testing at least 30 participants in each age group in the current study would lead to an effect size comparable to those reported in previous studies.
Design and Procedure
All of the participants performed a measure of audio-visual simultaneity perception using the audioaudio-visual SJ task. Before starting the experiment, each participant tried whether he/she could better see the visual flash with or without his/her own optical glasses. The participant was then tested in the condition with clearer flash perception. After the participant completed the audiovisual SJ task, we tested the participant’s visual acuity in the same visual con-dition using a Landolt C Chart and asked each participant to stand 5 m away from the chart. The participants in the
middle-aged and older adults groups also took assessments of their cognitive functions using the MoCA and CANTAB, and gait performance (for the purpose of another study not reported here).
Audiovisual SJ task
Participants were seated in a dimly-lit room, facing a 19-inch laptop monitor (1440 * 900 pixels) with a distance of approximately 50 cm. A gray ring with 2° inner diam-eters and 0.6° thickness was displayed in the center of a black background on the monitor throughout the exper-iment. The visual stimulus was a 2° white disc presented in the gray ring for 17 ms (one frame at the 60 Hz refresh rate). The auditory stimulus (white noise with a loudness of 60 dB SPL) was presented from speakers placed on either side of the monitor (the room had a background noise level of 30 dB SPL). The duration of the beep was 17 ms with 2 ms on and off ramping. The presentation of the stimuli was controlled by Matlab (MathWorks Inc., Natick, MA) and Psychtoolbox extensions (Brainard, 1997; Pelli 1997).
Two factors were manipulated: age group (young, mid-dle-aged, and older) and SOA (−600, −400, −300, −200, −100, 0, 100, 200, 300, 400, and 600 ms), where negative values indicated that the auditory beep was presented first and positive values indicated that the visual flash was pre-sented first. The SOA between the flash and beep was con-firmed using an oscilloscope. Each SOA was tested twice in a block in a random order and all of the participants com-pleted 10 blocks, giving rise to 220 trials in total. Participants were allowed to take a short break between blocks and typi-cally took 30 min to complete the experiment.
During the experiment, participants were instructed to fixate on the ring. The participants’ task was to orally re-port “yes” if they perceived the flash and beep at the same time, or “no” if they perceived the flash and beep at dif-ferent times. An experimenter sat beside the participant and keyed the answers into a computer. The experimenter also ensured that the participants fixated on the ring, and if not, asked them to stop for a short break.
A practice session designed to ensure that the partici-pants understood how the task was conducted prior to the main experiment. There were eight trials, four with 0 ms SOA and the remaining four with longer SOAs (−600, −400, 400, or 600 ms). Participants needed to achieve an 85% accuracy (i.e., no more than one error) within three attempts in order to proceed.
Montreal Cognitive Assessment
MoCA is a well-developed quick assessment of cognitive functions that includes attention, visual spatial ability, short-term memory, working memory, language, executive functions, and knowledge of facts. The test scores of MoCA ranges from 0 to 30. Having a score of 26 or above is con-sidered cognitively normal (Nasreddine et al., 2005). The participants in the middle-aged and older group who were included in the final analyses had reached this criterion.
Cambridge Neuropsychological Test Automated Battery
Five visual working memory subtests that assess three im-portant aspects of visual working memory in the CANTAB were used (see Supplementary Appendix A). Two of them assess visual pattern working memory: Pattern Recognition Memory (PRM) and Delayed Matching to Sample (DMS), another two assess visual spatial working memory: Spatial Recognition Memory (SRM) and Spatial Working Memory (SWM), and the last one assesses visual spatiotemporal working memory: Spatial Span (SSP). Five behavioral in-dices associated with each test were measured and used to assess the relations between audiovisual simultaneity perception and working memory performances. Only middle-aged and older adults were tested because of two reasons: First, these tasks were too easy for healthy young adults, and their performances reached ceiling in our pre-liminary study; second, the age ranges of middle-aged and older adults were continuous (53–64 and 65–81 years, respectively), while there was an age gap between mid-dle-aged and young (20–34 years) adults. These two facts may bias the correlations between the performance in the audiovisual SJ and working memory tasks, and we there-fore only conducted the CANTAB working memory tests in middle-aged and older adults.
Results
Audiovisual Simultaneity Perception
Two criteria were set in order to exclude participants who did not follow instructions, did not attend to the task, or performed as outliers in each age group. First, if the simul-taneous response at the ±600 ms SOA was higher than the mean plus 2.5 standard deviation (SD) of that age group. Second, if the simultaneous response at the 0 ms SOA was lower than the mean minus 2.5 SD of that age group. Following these two criteria, five young adults, four mid-dle-aged adults, and five older adults were excluded. Hence, there were 35 young adults, 32 middle-aged adults, and 34 older adults remaining in the final analyses (see Table 1 for the summary of demographics).
Proportion of simultaneous responses
Mean proportion of simultaneous responses of each par-ticipant at each SOA was submitted to a two-way ANOVA with a between-subject factor of age group and a within-subject factor of SOA (see Figure 1). Both main effects were significant: age group (F(2,98) = 19.77, MSE = 0.11, p < .001, η p2 = 0.29), and SOA (F(10,980) = 602.80,
MSE = 0.02, p < .001, η p2 = 0.86). Critically, their
inter-action was also significant (F(20,980) = 8.00, MSE = 0.02, p < .001, η p2 = 0.14). Table 2 presents the results of a
one-way ANOVA on the factor of age group at each SOA, and the post hoc tests at the SOAs where the simple main ef-fect of age group was significant (paired t tests, two-tailed, Bonferroni corrected). The age group effect was significant
at 8 out of the 11 SOAs, with exceptions at the −400, −100, and 0 ms SOAs. The age group effect arose mainly because the older adults made more simultaneous responses than young adults when the sound led the flash by 600, 300, and 200 ms, as well as when the flash led the sound by 100 ms or greater. Middle-aged adults made more simultaneous re-sponses than young adults when the flash led the sound by 200, 300, and 400 ms (i.e., only in the visual-leading conditions).
Estimated parameters of SJ
Individual data were fitted with the Matlab routine for the SJ task (Alcalá-Quintana & García-Pérez, 2013). Eight parameters (λV, λA, τ, δ, PSS, εAF, εVF, and εS) were esti-mated, and each parameter was submitted to a one-way ANOVA on the factor of age group (see Table 3).
The precision of audiovisual simultaneity perception (δ) demonstrated a significant age group effect (p < .001). Post hoc tests revealed that δ was larger in both older and mid-dle-aged adults than young adults (both ps < .001). The PSS
Table 1. Mean Age, Education Years, and MoCA Score, and Median of Visual Acuity (Ranges in Brackets) of Demographics and Health Status in Three Age Groups Remaining in the Final Analysis
Age group Young Middle-aged Older
N 35 32 34 Female 21 27 25 Age (years) 25.6 [21–34] 60.6 [55–64] 69.2 [65–80] Education (years) — 13.5 [4–18] 14.8 [9–18] MoCA score — 28.0 [26–30] 28.3 [26–30] Visual acuity of
better eye (decimal)
1.0 [0.3–2.0] 0.9 [0.4–1.2] 0.8 [0.3–1.5]
Note: MoCA = Montreal Cognitive Assessment.
Figure 1. Mean percentage of simultaneous responses at each
stimulus-onset asynchrony (SOA) for the three age groups. Error bars indicate ±1 standard error (SE) of the mean.
was positive (i.e., located on the visual-leading side) and signif-icantly differed from 0 in the older (t(33) = 5.06, p < .001) and middle-aged (t(31) = 4.72, p < .001) groups, and had the same trend but not significant in the young adults (t(34) = 1.14, p = .26). Consistently, the age group effect was significant in the PSS (p < .005) that the PSS was located at more positive SOAs in the older and middle-aged adults as compared to the young adults (both ps < .05).
There was no age group effect for the three unisensory processing parameters: auditory processing variability (λA), visual processing variability (λV), and processing time dif-ference (τ) (all ps ≥ .05). The τ was negative (suggesting the auditory signal arrived first at the central comparator when the two stimuli were presented simultaneously) in both older (t(33) = −3.80, p < .005) and middle-aged groups (t(31) = −3.66, p < .005), and in the same direction but not significant in the young adults group (t(34) = −1.84, p = .07).
In contrast, there were significant age group effects for the executive parameters in the simultaneous condition (εS)
and in the visual-leading condition (εVF) (ps < .05). Post hoc tests revealed that the εS was higher for middle-aged than young adults (p < .05), and the εVF was higher for both older and middle-aged than young adults (both ps < .05).
Regarding the three parameters that reliably demon-strated significant differences across the three age groups, we verified that each of their correlation with age was sig-nificant when visual acuity was controlled (δ: r = 0.55, p < .001; PSS: r = 0.34, p < .005; εVF: r = 0.44, p < .001). Hence, the age-related changes of these three parameters cannot be simply explained by their current visual acuity (i.e., a peripheral sensory factor).
Correlations Between Audiovisual Simultaneity
Parameters and CANTAB Performance
The CANTAB performances in the middle-aged and older groups, and the results of the t test (two-tailed, equal vari-ance assumed) are reported in Supplementary Appendix B.
Table 2. Mean Percentage and Standard Error (SE, in Parentheses) of Simultaneous Responses in Each Age Group, and the Results of One-Way ANOVA and Post Hoc Tests for the Percentage of Simultaneous Responses (Bonferroni Corrected) at Each SOA
SOA (ms) Young Middle-aged Older F(2,98) p Post hoc tests
−600 0.3 (0.2) 1.4 (0.5) 2.1 (0.5) 5.29 < .01 Older > Young** −400 4.0 (1.3) 6.1 (1.8) 10.6 (2.5) 3.06 = .05 −300 16.6 (3.0) 23.6 (3.8) 29.7 (3.3) 3.89 < .05 Older > Young* −200 50.6 (5.2) 65.2 (4.3) 68.1 (3.5) 4.67 < .05 Older > Young* −100 87.1 (3.2) 92.7 (1.7) 93.8 (1.2) 2.58 = .08 0 97.0 (0.8) 94.4 (1.2) 95.0 (0.9) 1.87 = .16 100 85.4 (4.2) 91.6 (1.6) 95.4 (1.2) 3.48 < .05 Older > Young*
200 51.0 (5.6) 80.9 (3.5) 83.7 (2.2) 20.07 < .001 Older > Young**; Middle-aged > Young** 300 20.9 (4.2) 57.3 (4.6) 56.5 (4.2) 23.43 < .001 Older > Young**; Middle-aged > Young** 400 8.0 (2.9) 29.7 (4.1) 34.4 (4.5) 13.64 < .001 Older > Young**; Middle-aged > Young** 600 0.3 (0.2) 4.7 (1.2) 9.0 (2.0) 11.03 < .001 Older > Young**
Notes: ANOVA = analysis of variance; SOA = stimulus-onset asynchrony.
*p < .05. **p < .01.
Table 3. Mean and SE (in Parentheses) of Estimated Parameters of the Simultaneity Judgments Task in Each Age Group, and the Results of One-Way ANOVA as Well as Post Hoc Tests (Bonferroni Corrected)
Parameter Young Middle-aged Older F(2,98) p Post hoc tests
δ 213.1 (11.1) 283.4 (9.3) 290.3 (8.3) 19.87 < .001 Older > Young**; Middle-aged > Young** PSS 8.2 (7.2) 42.9 (9.1) 40.5 (8.0) 5.85 < .005 Older > Young*; Middle-aged > Young** λA 0.115 (0.047) 0.096 (0.041) 0.093 (0.041) 0.08 = .93
λV 0.207 (0.053) 0.171 (0.042) 0.183 (0.046) 0.16 = .86
τ −16.8 (9.1) −50.3 (13.7) −53.3 (14.0) 2.72 = .07
εAF 0.005 (0.002) 0.012 (0.004) 0.015 (0.005) 1.83 = .17
εS 0.015 (0.006) 0.049 (0.012) 0.023 (0.006) 4.44 < .05 Middle-aged > Young*
εVF 0.006 (0.003) 0.050 (0.011) 0.086 (0.018) 10.34 < .001 Older > Young**; Middle-aged > Young*
Notes: δ = precision (threshold) of audiovisual simultaneity perception; εAF = response errors in the auditory-leading trials; εS = response errors in the simul-taneous trials; εVF = response errors in the visual-leading trials; λA = processing variability of auditory stimulus; λV = processing variability of visual stimulus; PSS = point of subjective simultaneity; τ = processing time difference between visual and auditory stimulus (τA − τV).
*p < .05. **p < .01.
We then examined whether the three parameters of audio-visual simultaneity perception that demonstrated a significant effect of age group (i.e., δ, PSS, and εVF) were correlated with each other, and whether they were correlated with the five behavioral indices of working memory performances in the middle-aged and older adults (see Table 4). Partial cor-relations were conducted and the factors of age and visual acuity were used as covariates in order to control for their influences.
The results demonstrated that δ was correlated with both PSS and εVF (both ps < .01), while the latter two were not correlated. More critically, εVF was positively correlated with the response time (RT) in the SRM task (p < .05, uncorrected), and in the same trend but not signif-icant in the PRM (p = .08) and DMS tasks (p = .06). This result suggests that the middle-aged and older adults who made more response errors in the visual-leading condi-tions in the SJ task also took a longer time to recognize the locations of previously viewed objects (i.e., the SRM task).
Discussion
We measured the age-related changes of audiovisual si-multaneity perception, and examined their relations to visual working memory in middle-aged and older adults. The results demonstrate that, compared to young adults, the older adults (>65 years old) judged a flash and beep as simultaneous more often in both auditory-leading and visual-leading conditions, while the middle-aged adults (53–64 years old) mainly did so in the visual-leading condi-tions. Model-fitting results demonstrate that, starting from middle age, the precision of audiovisual simultaneity per-ception became worse (i.e., larger δ), the PSS shifted further toward the visual-leading condition, and response errors in the visual-leading condition increased (i.e., larger εVF). More critically, the middle-aged and older participants with higher εVF responded slower in the SRM task when the factors of age and visual acuity were controlled.
Three unique results are found in the current study: First, the poorer precision of audiovisual simultaneity per-ception with aging started from the middle age range (cf.
Basharat et al., 2018; Fiacconi et al., 2013; Stevenson et al.,
2018), which is earlier than the age reported in the litera-ture (see Baum & Stevenson, 2017; Brooks et al., 2018, for reviews). It is possible that the wider range of SOAs and a better model-fitting routine for the SJ task used in the current study leads to a more accurate estimation of the audiovisual simultaneity perception (cf. Bedard & Barnett-Cowan, 2016; Noel et al., 2016; Stevenson et al., 2018). Note that our middle-aged group (53–64 years) was older than the middle-aged group (40–59 years) in Stevenson et al. (2018). However, when we only analyzed the mid-dle-aged participants younger than 60 years old (N = 11), their precision remained worse than young adults (see
Supplementary Appendix C). Second, the PSS started to shift further toward the visual-leading side from the middle age range, contrasting the lack of age-related change of PSS reported in previous studies (Bedard & Barnett-Cowan, 2016; Fiacconi et al., 2013). The shift of PSS suggests that the precision became poorer more pronouncedly in the visual-leading than the auditory-leading condition with aging. Third, the εVF increased with age. This is an executive parameter associated with eye blinks, lapses of attention, and response errors in the visual-leading trials. Also, this factor correlated with the middle-aged and older adults’ working memory performance in terms of the RT measure in the SRM task.
It seems reasonable to suppose that the poorer preci-sion of audiovisual simultaneity perception with aging may be caused by sensory loss during aging (e.g., de Dieuleveult et al., 2017). Previous studies, however, have demonstrated that older adults’ poorer precision of audiovisual simultaneity perception cannot be accounted for by their degeneration of detection sensitivity and temporal acuity in vision and audi-tion (Chan et al., 2014a; Stevenson et al., 2018). Consistently, here we demonstrated that the correlation between partici-pants’ age and precision (δ) remained significant when the in-fluence of visual acuity was controlled. An alternative account suggests that the noise in each sensory system increases with age, leading to a higher variability in information processing (Mozolic et al., 2012). However, our model-fitting results dem-onstrated that the visual and auditory processing variability (λV and λA, respectively) did not reveal a significant effect of age group. Combining these results therefore suggests that the poorer precision of audiovisual simultaneity perception with
Table 4. Partial Correlations Between Three Parameters of Audiovisual Simultaneity Perception, and Their Correlations With the Five Behavioral Indices of the CANTAB Subtests When the Factors of Age and Visual Acuity Were Controlled
δ PSS εVF PRM RT DMS RT SRM RT SWM Total errors SSP Span length δ — 0.342** 0.416** 0.009 −0.039 −0.099 0.059 0.024 PSS — — 0.059 −0.009 −0.036 −0.111 −0.142 0.017 εVF — — — 0.218 0.236 0.262* −0.083 −0.051
Notes: δ = precision of audiovisual simultaneity perception; εVF = response errors in the visual-leading conditions; PSS = point of subjective simultaneity; RT = re-sponse time.
**p < .01. *p < .05, uncorrected.
aging is unlikely attributed to the degeneration of unisensory systems; instead, it is more plausible that the deteriorated pre-cision is attributed to degeneration occurring at the central comparator, which is responsible for decoding the timing of visual and auditory signals. Consistent neurophysiological ev-idence has revealed the reduction of the volume of superior temporal sulcus (STS) during aging (Peters, 2006), a cortical area that underpins people’s subjective perception of audio-visual simultaneity (Noesselt et al., 2012; see de Dieuleveult et al., 2017, for a review).
We observed a shift of PSS toward the visual-leading condition for the middle-aged and older adults further than for the young adults; more specifically, a more pronounced deteriorated of precision in the visual-leading than in the auditory-leading condition was observed in these adults. At least three possibilities may account for this age-related change. First, Stevenson et al. (2018) demonstrated that the decline of unisensory temporal acuity occurred earlier in vision than in audition during aging. Second, Murray et al. (2018) reported that the majority of older adults demon-strated an auditory dominance in terms of faster responses to auditory signals than visual signals, in contrast to young adults with similar RT to both signals. These two accounts would both predict a longer time for visual than auditory processing. That is, a visual signal needs to be presented earlier than an auditory signal in order for them to be per-ceived simultaneously, leading to a shift of the PSS toward the visual-leading condition. However, in the current study, the processing time difference between the visual and au-ditory signal (τ = τA − τV) did not significantly differ be-tween the three age groups. Furthermore, the window of audiovisual simultaneous perception was widened in both the visual-leading and auditory-leading conditions in the older adults, rather than an overall shift toward the visual-leading condition. Therefore, neither the account of earlier degeneration of vision than audition, nor the account of auditory dominance, provides a full explanation for the shift of PSS in the middle-aged and older adults.
The third account regards the malleability of the audi-ovisual mechanism. Perceiving audiaudi-ovisual simultaneity is challenging because of two temporal factors: In the phys-ical world, light travels faster than sound, and this travel time difference increases as the distance between stimulus source and perceiver increases. In addition, the neural proc-essing time from sensory receptors to the cortex is longer for vision than audition by around 30–40 ms (Vroomen & Keetels, 2010). Given that the travel time difference varies, whereas the neural processing time difference is constant, people’s window of audiovisual simultaneity perception is suggested to be malleable when taking the distance of stim-ulus source into consideration (Engel & Dougherty, 1971;
Sugita & Suzuki, 2003; see Vroomen & Keetels, 2010, for a review). Specifically, the farther the stimulus source, the earlier the arrival time of visual than auditory signal (i.e., the visual-leading condition). Previous studies have demon-strated that the visual-leading side is more malleable than
auditory-leading side after training (Donohue et al., 2010;
Powers et al., 2009), which is plausibly underpinned by the plasticity remaining in the audiovisual system in order to accommodate the travel time differences as a function of stimulus distances. Nevertheless, this plasticity decreases during aging (Chan et al., 2014b; Noel et al., 2016). It is possible that the middle-aged and older participants were not able to rapidly adjust their audiovisual simultaneity window to accommodate the setting in the laboratory (i.e., a close audiovisual source that the travel time difference was negligible). Hence, the pronounced degeneration of precision in the visual-leading condition may be attributed to the reduced plasticity in the aging brains, thus failing to accommodate the travel time differences of light and sound.
Regarding the three parameters of audiovisual simul-taneity perception that demonstrated an age group effect (i.e., δ, PSS, and εVF), εVF was the only one correlated with middle-aged and older adult’s performance in the spa-tial working memory task (SRM). The εVF parameter rep-resents a participant’s eye blinks, lapses of attention, and errors in responses when conducting the SJ task, which is a postperceptual factor that relates to the executive stage of information processing (see García-Pérez & Alcalá-Quintana, 2012). This executive ability should modulate people’s performance when a behavioral response is needed, including the SJ task and the SRM task in the CANTAB used here. Hence, we suggest that the degeneration of exec-utive function is a higher-order mechanism that leads to the degraded performance in both audiovisual SJ and working memory tasks. This is an evidence demonstrating that a third factor (i.e., the executive function) is the common cause determining older adults' perception and cognition, although they are not causally related (cf. Humes et al., 2013; see Roberts & Allen, 2016, for a review).
Our conclusions, nevertheless, should be reserved when considering the design of the current study due to the following reasons. First, we demonstrated that the middle-aged and older adults’ performance in the audi-ovisual SJ task was correlated with their age when con-trolled for their far-distance visual acuity (5 m). However, other early visual functions (such as near-distance visual acuity and contrast sensitivity at different spatial scales), as well as early auditory functions (such as audibility curve), were not measured. It therefore remains unclear whether these other unisensory processing factors would contribute to older adults’ audiovisual simultaneity per-ception. Second, although the working memory subtests chosen from the CANTAB are all well-developed tools, they are broad and exploratory. In future research, more specific working memory measures are required to better understand the relationships between various types of working memory functions and audiovisual simulta-neity perception. Finally, the current study is cross-sec-tional; a longitudinal study will be better characterizing the changes of audiovisual simultaneity perception during aging.
Overall, the current study assesses the changes of audi-ovisual simultaneity perception across young, middle-aged, and older adults. We demonstrated that precision starts to de-teriorate from middle age. More critically, such dede-teriorated precision was more pronounced in the visual-leading than auditory-leading condition, giving rise to a shift of the PSS further toward the visual-leading condition. We suggest that these two age-related changes are attributed to the degen-eration of central mechanisms taking charge of audiovisual perception, rather than the changes in the peripheral visual or auditory processing. Finally, we demonstrated that the de-clined performances in the audiovisual SJ tasks and working memory tasks have a common cause at the executive stage of information processing. Therefore, training older adults’ ex-ecutive functions should improve their behavioral perform-ances in general; however, individual training programs for audiovisual perception (e.g., Powers et al., 2009 in young adults) and working memory (e.g., Borella et al., 2010) are needed, and the training effect might not be transferable to each other. Our results are critical for understanding the aging brain and designing a training regime for older adults.
Supplementary Material
Supplementary data are available at The Journals of Gerontology, Series B: Psychological Sciences and Social Sciences online.
Funding
This study is supported by Ministry of Science and Technology in Taiwan: MOST 107-2410-H-715-001-MY2 to Y. Chen; MOST 106-2420-H-002-010-MY3 and MOST 108-2420-H-492-001-MY3 to S. Yeh; NSC 102-2410-H-002-213-MY2 and MOST 106-2314-B-002-086-MY3 to P. Tang.
Conflict of Interest
None declared.
Acknowledgments
We thank Meng-Tien Wu, Yen-Ling Chen, Chih-Yi Hsia, and Nai-Chi Chen for their great help in participant recruit-ment, data collection, and early stage of data processing. This is not a preregistered study. Further information and requests for resources should be directed to, and will be fulfilled by the Lead Contact, Pei-Fang Tang (pftang@ntu.edu.tw).
References
Alais, D., & Burr, D. (2004). The ventriloquist effect results from near-optimal bimodal integration. Current Biology, 14, 257– 262. doi:10.1016/j.cub.2004.01.029
Alcalá-Quintana, R., & García-Pérez, M. A. (2013). Fitting model-based psychometric functions to simultaneity and temporal-order judgment data: MATLAB and R routines. Behavior Research Methods, 45, 972–998. doi:10.3758/s13428-013-0325-2
Basharat, A., Adams, M. S., Staines, W. R., & Barnett-Cowan, M. (2018). Simultaneity and temporal order judgments are coded differently and change with age: An event-related potential study. Frontiers in Integrative Neuroscience, 12, 15. doi:10.3389/ fnint.2018.00015
Baum, S. H., & Stevenson, R. (2017). Shifts in audiovisual processing in healthy aging. Current Behavioral Neuroscience Reports, 4, 198–208. doi:10.1007/s40473-017-0124-7
Bedard, G., & Barnett-Cowan, M. (2016). Impaired timing of audi-ovisual events in the elderly. Experimental Brain Research, 234, 331–340. doi:10.1007/s00221-015-4466-7
Borella, E., Carretti, B., Riboldi, F., & De Beni, R. (2010). Working memory training in older adults: Evidence of transfer and maintenance effects. Psychology and Aging, 25, 767–778. doi:10.1037/a0020683
Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.
Brooks, C. J., Chan, Y. M., Anderson, A. J., & McKendrick, A. M. (2018). Audiovisual temporal perception in aging: The role of multisensory integration and age-related sensory loss. Frontiers in Human Neuroscience, 12, 192. doi:10.3389/fnhum.2018.00192
van der Burg, E., Olivers, C. N. L., Bronkhorst, A. W., & Theeuwes, J. (2008). Pip and pop: Non-spatial auditory sig-nals improve spatial visual search. Journal of Experimental Psychology: Human Perception and Performance, 34, 1053– 1065. doi:10.1037/0096-1523.34.5.1053
Chaby, L., Boullay, V. L., Chetouani, M., & Plaza, M. (2015). Compensating for age limits through emotional crossmodal integration. Frontiers in Psychology, 6, 691. doi:10.3389/ fpsyg.2015.00691
Chan, J. S., Kaiser, J., Brandl, M., Matura, S., Prvulovic, D., Hogan, M. J., & Naumer, M. J. (2015). Expanded temporal binding windows in people with mild cognitive impairment. Current Alzheimer Research, 12, 61–68. doi:10.2174/1567205 012666141218124744
Chan, Y. M., Pianta, M. J., & McKendrick, A. M. (2014a). Older age results in difficulties separating auditory and visual signals in time. Journal of Vision, 14(11):13, 1–11. doi:10.1167/14.11.13
Chan, Y. M., Pianta, M. J., & McKendrick, A. M. (2014b). Reduced audiovisual recalibration in the elderly. Frontiers in Aging Neuroscience, 6, 226. doi:10.3389/fnagi.2014.00226
Chen, Y. C., & Spence, C. (2011). The crossmodal facilitation of visual object representations by sound: Evidence from the back-ward masking paradigm. Journal of Experimental Psychology: Human Perception and Performance, 37, 1784–1802. doi:10.1037/a0025638
de Dieuleveult, A. L., Siemonsma, P. C., van Erp, J. B., & Brouwer, A. M. (2017). Effects of aging in multisensory integra-tion: A systematic review. Frontiers in Aging Neuroscience, 9, 80. doi:10.3389/fnagi.2017.00080
Donohue, S. E., Woldorff, M. G., & Mitroff, S. R. (2010). Video game players show more precise multisensory temporal proc-essing abilities. Attention, Perception & Psychophysics, 72, 1120–1129. doi:10.3758/APP.72.4.1120
Engel, G. R., & Dougherty, W. G. (1971). Visual-auditory distance constancy. Nature, 23, 308. doi:10.1038/234308a0
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion. Nature, 415, 429–433. doi:10.1038/415429a
Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, be-havioral, and biomedical sciences. Behavior Research Methods, 39, 175–191. doi:10.3758/bf03193146
Fiacconi, C. M., Harvey, E. C., Sekuler, A. B., & Bennett, P. J. (2013). The influence of aging on audiovisual temporal order judgments. Experimental Aging Research, 39, 179–193. doi:10.1080/0361 073X.2013.761896
Freiherr, J., Lundström, J. N., Habel, U., & Reetz, K. (2013). Multisensory integration mechanisms during aging. Frontiers in Human Neuroscience, 7, 863. doi:10.3389/ fnhum.2013.00863
García-Pérez, M. A., & Alcalá-Quintana, R. (2012). On the dis-crepant results in synchrony judgment and temporal-order judgment tasks: A quantitative model. Psychonomic Bulletin & Review, 19, 820–846. doi:10.3758/s13423-012-0278-y
Hedden, T., & Gabrieli, J. D. (2004). Insights into the ageing mind: A view from cognitive neuroscience. Nature Reviews Neuroscience, 5, 87–96. doi:10.1038/nrn1323
Heikkilä, J., Fagerlund, P., & Tiippana, K. (2018). Semantically con-gruent visual information can improve auditory recognition memory in older adults. Multisensory Research, 31, 213–225. doi:10.1163/22134808-00002602
Humes, L. E., Busey, T. A., Craig, J., & Kewley-Port, D. (2013). Are age-related changes in cognitive function driven by age-related changes in sensory processing? Attention, Perception & Psychophysics, 75, 508–524. doi:10.3758/ s13414-012-0406-9
Laurienti, P. J., Burdette, J. H., Maldjian, J. A., & Wallace, M. T. (2006). Enhanced multisensory integration in older adults. Neurobiology of Aging, 27, 1155–1163. doi:10.1016/j. neurobiolaging.2005.05.024
Miller, J. (1991). Channel interaction and the redundant-targets effect in bimodal divided attention. Journal of Experimental Psychology: Human Perception and Performance, 17, 160–169. doi:10.1037//0096-1523.17.1.160
Mozolic, J. L., Hugenschmidt, C. E., Peiffer, A. M., & Laurienti, P. J. (2012). Multisensory integration and aging. In M. M. Murray & M. T. Wallace (Eds.), The neural bases of multisensory processes (pp. 381–392). CRC Press.
Murray, M. M., Eardley, A. F., Edginton, T., Oyekan, R., Smyth, E., & Matusz, P. J. (2018). Sensory dominance and multisensory in-tegration as screening tools in aging. Scientific Reports, 8, 8901. doi:10.1038/s41598-018-27288-2
Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbonneau, S., Whitehead, V., Collin, I., Cummings, J. L., & Chertkow, H. (2005). The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society, 53, 695–699. doi:10.1111/j.1532-5415.2005.53221.x
Noel, J. P., De Niear, M., Van der Burg, E., & Wallace, M. T. (2016). Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan. PLoS One, 11, e0161698. doi:10.1371/ journal.pone.0161698
Noesselt, T., Bergmann, D., Heinze, H. J., Münte, T., & Spence, C. (2012). Coding of multisensory temporal patterns in human su-perior temporal sulcus. Frontiers in Integrative Neuroscience, 6, 64. doi:10.3389/fnint.2012.00064
Peiffer, A. M., Mozolic, J. L., Hugenschmidt, C. E., & Laurienti, P. J. (2007). Age-related multisensory enhancement in a simple audiovisual detection task. NeuroReport, 18, 1077–1081. doi:10.1097/WNR.0b013e3281e72ae7
Pelli, D. G. (1997). The VideoToolbox software for visual psycho-physics: Transforming numbers into movies. Spatial Vision, 10, 437–442. doi:10.1163/156856897X00366
Peters, R. (2006). Ageing and the brain. Postgraduate Medical Journal, 82, 84–88. doi:10.1136/pgmj.2005.036665
Powers, A. R., Hillock, A. R., & Wallace, M. T. (2009). Perceptual training narrows the temporal window of multisensory binding. The Journal of Neuroscience, 29, 12265–12274. doi:10.1523/ JNEUROSCI.3501-09.2009
Roberts, K. L., & Allen, H. A. (2016). Perception and cognition in the ageing brain: A brief review of the short- and long-term links between perceptual and cognitive decline. Frontiers in Aging Neuroscience, 8, 39. doi:10.3389/fnagi.2016.00039
Setti, A., Burke, K. E., Kenny, R. A., & Newell, F. N. (2011). Is inefficient multisensory processing associated with falls in older people? Experimental Brain Research, 209, 375–384. doi:10.1007/s00221-011-2560-z
Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. MIT Press.
Stevenson, R. A., Baum, S. H., Krueger, J., Newhouse, P. A., & Wallace, M. T. (2018). Links between temporal acuity and multisensory integration across life span. Journal of Experimental Psychology: Human Perception and Performance, 44, 106–116. doi:10.1037/xhp0000424
Sugita, Y., & Suzuki, Y. (2003). Audiovisual perception: Implicit estima-tion of sound-arrival time. Nature, 421, 911. doi:10.1038/421911a
Virsu, V., Lahti-Nuuttila, P., & Laasonen, M. (2003). Crossmodal temporal processing acuity impairment aggravates with age in developmental dyslexia. Neuroscience Letters, 336, 151–154. doi:10.1016/s0304-3940(02)01253-3
Vroomen, J., & de Gelder, B. (2000). Sound enhances visual perception: Crossmodal effects of auditory organization on vision. Journal of Experimental Psychology: Human Perception and Performance, 26, 1583–1590. doi:10.1037/0096-1523.26.5.1583
Vroomen, J., & Keetels, M. (2010). Perception of intersensory syn-chrony: A tutorial review. Attention, Perception & Psychophysics, 72, 871–884. doi:10.3758/APP.72.4.871
Welch, R. B., & Warren, D. H. (1980). Immediate perceptual re-sponse to intersensory discrepancy. Psychological Bulletin, 3, 638–667. doi:10.1037/0033-2909.88.3.638
World Health Organization. (n.d.). Health statistics and informa-tion systems. Retrieved May 14, 2020, from https://www.who. int/healthinfo/survey/ageingdefnolder/en/
Wu, J., Yang, J., Yu, Y., Li, Q., Nakamura, N., Shen, Y., Ohta, Y., Yu, S., & Abe, K. (2012). Delayed audiovisual integration of patients with mild cognitive impairment and Alzheimer’s disease com-pared with normal aged controls. Journal of Alzheimers Disease, 32, 317–328. doi:10.3233/JAD-2012–111070
