Associate Professor Kiwako Ito

The past decade has witnessed remarkable progress in speech synthesis research, to the point where synthetic voices can be hard to distinguish from natural ones, at least for utterances with neutral, declarative prosody. Neutral intonation often does not suffice, however, in interactive systems: instead it can sound disengaged or ¿dead,¿ and can be misleading as to the intended meaning. For concept-to-speech systems, especially interactive ones, natural language generation researchers have developed a variety of methods for making contextually appropriate prosodic choices, depending on discourse-related factors such as givenness, parallelism, or theme/rheme alternative sets, as well as information-theoretic considerations (Prevost, 1995; Hitzeman et al., 1998; Pan et al., 2002; Bulyko and Ostendorf, 2002; Theune, 2002; Kruijff-Korbayová et al., 2003; Nakatsu and White, 2006; Brenier et al., 2006; White et al., 2010). In this setting, it is possible to adapt limited-domain synthesis techniques to produce utterances with perceptually distinguishable, contextually varied intonation (see Black and Lenzo, 2000; Baker, 2003; van Santen et al., 2005; Clark et al., 2007, for example). To evaluate these utterances, listening tests have typically been employed, sometimes augmented with expert evaluations. For example, evaluating the limited domain voice used in the FLIGHTS concept-to-speech system (Moore et al., 2004; White et al., 2010) demonstrated that the prosodic specifications produced by the natural language generation component of the system yielded significantly more natural synthetic speech in listening tests and, in an expert evaluation, compared to two baseline voices.

This study examines the putative benefits of explicit phonetic instruction, high variability phonetic training, and their effects on adult nonnative speakers¿ Mandarin tone productions. Monolingual first language (L1) English speakers (n = 80), intermediate second language (L2) Mandarin learners (n = 40), and L1 Mandarin speakers (n = 40) took part in a multiday Mandarin-like artificial language learning task. Participants were asked to repeat a syllable¿tone combination immediately after hearing it. Half of all participants were exposed to speech from 1 talker (low variability) while the other half heard speech from 4 talkers (high variability). Half of the L1 English participants were given daily explicit instruction on Mandarin tone contours, while the other half were not. Tone accuracy was measured by L1 Mandarin raters (n = 104) who classified productions according to their perceived tonal category. Explicit instruction of tone contours facilitated L1 English participants¿ production of rising and falling tone contours. High variability input alone had no main effect on participants¿ productions but interacted with explicit instruction to improve participants¿ productions of high-level tone contours. These results motivate an L2 tone production training approach that consists of explicit tone instruction followed by gradual exposure to more variable speech.

Background: Past reports on the speech production of individuals with Williams syndrome (WS) suggest that their prosody is anomalous and may lead to challenges in spoken communication. While existing prosodic assessments confirm that individuals with WS fail to use prosodic emphasis to express contrast, those reports typically lack detailed phonetic analysis of speech data. The present study examines the acoustic properties of speech prosody, aiming for the future development of targeted speech interventions. Aims: The study examines the three primary acoustic correlates of prosodic emphasis (duration, intensity, F0) and determines whether individuals with WS have difficulty in producing all or a particular set of the three prosodic cues. Methods & Procedures: Speech produced by 12 individuals with WS and 12 chronological age (CA)-matched typically developing individuals were recorded. A sequential picture-naming task elicited production of target phrases in three contexts: (1) no contrast: gorilla with a racket ¿ rabbit with a balloon; (2) contrast on the animal: fox with a balloon ¿ rabbit with a balloon; and (3) contrast on the object: rabbit with a ball ¿ rabbit with a balloon. The three acoustic correlates of prosodic prominence (duration, intensity and F0) were compared across the three referential contexts. Outcomes & Results: The two groups exhibited striking similarities in their use of word duration and intensity for expressing contrast. Both groups showed the reduction and enhancement of final lengthening, and the enhancement and reduction of intensity difference for the animal contrast and for the object contrast conditions, respectively. The two groups differed in their use of F0: the CA group produced higher F0 for the animal than for the object regardless of the context, and this difference was enhanced when the animal noun was contrastive. In contrast, the WS group produced higher F0 for the object than for the animal when the object was contrastive. Conclusions & Implications: The present data contradict previous assessment results that report a lack of prosodic skills to mark contrast in individuals with WS. The methodological differences that may account for this variability are discussed. The present data suggest that individuals with WS produce appropriate prosodic cues to express contrast, although their use of pitch may be somewhat atypical. Additional data and future speech comprehension studies will determine whether pitch modulation can be targeted for speech intervention in individuals with WS.

The perception of prosodic prominence is thought to be influenced by multiple competing factors. Three experiments tested the effects of phonological salience, discourse context and listener¿s knowledge about the discourse on prosodic prominence judgements, using short adjective¿noun phrases extracted from a corpus of spontaneous speech. These phrases had either a prominent L + H* 0 or a less prominent H* !H* pitch accent contour. The phrases were presented in a discourse context which either supported or did not support a contrastive interpretation of the adjective. Effects of the contrastive context to increase the perception of prominence only emerged for the phrases with the phonologically prominent L + H* 0 pitch accent sequence. Additionally, the magnitude of the contrast effect was correlated with the listener¿s awareness of the discourse context, suggesting an ample interplay between linguistic context, pragmatic context, and phonology in prominence perception.

Previous research on Mandarin spoken word recognition suggests that when processing lexical tone native listeners tune to various acoustic properties of the incoming signal such as f0 height, contour and change. These studies overlook the uneven distribution of tone across the Mandarin lexicon; given a particular string of segments, native listeners may be more likely to anticipate a certain tone due to prior experience with the language. The present study used the gating paradigm to investigate how much of the acoustic signal is needed for listeners to trigger such probability-based tone processing. Duration-blocked gates were generated from twelve low frequency and twelve high frequency mono-syllabic morphemes, with each syllable carrying either the most or least probable tone based on spoken corpora. Results from 22 mono-dialectal Mandarin speakers indicate that listeners immediately begin making use of tonal probability information after hearing only the onset and 40 ms of the vowel, primarily when hearing infrequent syllables. These findings demonstrate that the processing of suprasegmental information can be initiated with previously learned distributional knowledge until sufficient acoustic cues inform the listener of an incoming word.

Due to the language-specific prosodic architecture, a phrase-medial pitch expansion in Japanese may signal either a narrow contrast or the beginning of a new syntactic phrase. To examine how visual and discourse context affects the tug of war between these two interpretations, three eye-tracking experiments tested the interpretation of a pitch expansion in referential contexts that varied in the plausibility of contrastive interpretation. The results showed that the degree of structural interpretation was inversely related to the plausibility of contrastive interpretation. Although there was no direct evidence of contrastive processing of the pitch prominence, participants responded to the trials with the pitch expansion much slower than to those without the expansion only when the contrastive interpretation of the pitch prominence was contextually infelicitous. These results suggest that the phrase-medial pitch expansion may have been simultaneously processed for contrast and structural disambiguation. However, the task that demanded resolutions of standing referential ambiguities seemed to make the structural disambiguation the best use of the prosodic cue.

An eye-tracking study investigated how the interaction between syllable frequency and syllable-specific tonal probability guides online lexical access in speakers of mutually unintelligible Chinese dialects with three disparate tonal systems. Mono-dialectal Mandarin speakers, bi-dialectal Shanghai¿Mandarin speakers and bi-dialectal Cantonese¿Mandarin speakers searched for target Mandarin syllable¿tone combinations while their eye movements and mouse clicks were recorded. The results showed dialectal differences in online eye fixation patterns but not in offline mouse responses. For all groups, mouse clicks were fastest for infrequent syllables with most probable tones and slowest for infrequent syllables with least probable tones. In online eye movement responses, only mono-dialectal Mandarin speakers showed an interaction between syllable frequency and tonal probability. Mono-dialectal Mandarin speakers¿ fixations were fastest for infrequent syllables with probable tones and slowest for infrequent syllables with improbable tones. Mono-dialectal speakers also showed a greater amount of competition from the more probable segmental competitor when hearing improbable tones. Bi-dialectal speakers showed different timing in their integration of tonal probabilities. These findings suggest that highly bilingual speakers track and use Mandarin tonal probabilities, but their sensitivity to L2 tonal information may lag behind monolinguals for online word recognition.

We describe the mission and practices of the Language Sciences Research Lab, a fully functional research lab embedded within a science museum. Within this environment, we integrate cutting-edge research, formal instruction, informal learning, and outreach to the public so that our work in each domain interacts with and enriches the others. We are guided by core concepts from the field of informal science education, and we strive to inspire excitement and expand both scholarly and public knowledge about the language sciences.

Both off-line and on-line comprehension studies suggest not only toddlers and preschoolers, but also older school-age children have trouble interpreting contrast-marking pitch prominence. To test whether children achieve adult-like proficiency in processing contrast-marking prosody during school years, an eye-tracking experiment examined the effect of accent on referential resolution in six- to eleven-year-old children and adults. In all age groups, a prominent accent facilitated the detection of a target in contrastive discourse sequences (pink cat¿green cat), whereas it led to a garden path in non-contrastive sequences (pink rabbit¿green monkey: the initial fixations were on rabbits). While the data indicate that children as young as age six immediately interpret contrastive accent, even the oldest child group showed delayed fixations compared to adults. We argue that the children's slower recovery from the garden path reflects the gradual development in cognitive flexibility that matures independently of general oculomotor control. Copyright © Cambridge University Press 2012.

Two eye-tracking experiments tested how pitch prominence on a prenominal adjective affects contrast resolution in Japanese adult and 6-year old listeners. Participants located two animals in succession on displays with multiple colored animals. In Experiment 1, adults' fixations to the contrastive target (pink cat ¿ GREEN cat) were facilitated by a pitch expansion on the adjective while infelicitous pitch expansion (purple rabbit ¿ ORANGE monkey) led to a garden-path effect, i.e., frequent fixations to the incorrect target (orange rabbit). In 6-year olds, only the facilitation effect surfaced. Hypothesizing that the interval between the two questions may not have given enough time for children to overcome their tendency to perseverate on the first target, Experiment 2 used longer intervals and confirmed a garden-path effect in 6-year olds. These results demonstrate that Japanese 6-year olds can make use of contrast-marking pitch prominence when time allows an establishment of proper discourse representation. © 2011 Elsevier Inc.

Recent research on children's acquisition of prosody, or the rhythm and melody in language, demonstrates that young children use prosody in their comprehension and production of utterances to a greater extent than was previously documented. Spoken language, structured by prosodic form, is the primary input on which the mental representations and processes that comprise language use are built. Understanding how children acquire prosody and develop the mapping between prosody and other aspects of language is crucial to any effort to model the role of prosody in the processing system. We focus on two aspects of prosody that have been shown to play a primary role in its use as an organizational device in human languages, prosodic phrasal grouping, and intonational prominence. © 2009 Blackwell Publishing Ltd.

Three eye-tracking experiments investigated the role of pitch accents during online discourse comprehension. Participants faced a grid with ornaments, and followed prerecorded instructions such as "Next, hang the blue ball" to decorate holiday trees. Experiment 1 demonstrated a processing advantage for felicitous as compared to infelicitous uses of L + H* on the adjective noun pair (e.g., blue ball followed by GREEN ball vs. green BALL). Experiment 2 confirmed that L + H* on a contrastive adjective led to 'anticipatory' fixations, and demonstrated a "garden path" effect for infelicitous L+H* in sequences with no discourse contrast (e.g., blue angel followed by GREEN ball resulted in erroneous fixations to the cell of angels). Experiment 3 examined listeners' sensitivity to coherence between pitch accents assigned to discourse markers such as 'And then,' and those assigned to the target object noun phrase. © 2008.

Forced alignment would enable phonetic analyses of child directed speech (CDS) corpora which have existing transcriptions. But existing alignment systems are inaccurate due to the atypical phonetics of CDS. We adapt a Kaldi forced alignment system to CDS by extending the dictionary and providing it with heuristically-derived hints for vowel locations. Using this system, we present a new time-aligned CDS corpus with a million aligned segments. We manually correct a subset of the corpus and demonstrate that our system is 70% accurate. Both our automatic and manually corrected alignments are publically available at osf.io/ke44q.

The perception of prosodic prominence is known to be influenced by several distinct factors. In this study, we investigated the role of context, both global and local, in the prominence judgements of näi{dotless}ve listeners. Monolingual English listeners marked where they heard prominence on pairs of twoword phrases (e.g. blue ball, green drum). Stimuli varied in whether or not the first phrase implied a contrastive focus on the second phrase. We found clear evidence of a hierarchy of prominence across pitch accent types: L+H* > H* > !H* > unaccented. Additionally, we found that contrast status only affected prominence markings when the participants were made explicitly aware of the discourse context and were instructed to imagine themselves physically present to observe the conversation. This effect of global context suggests that information structure cannot be reliably interpreted in the absence of an established discourse context. Taken together, these results suggest that näi{dotless}ve listeners are sensitive to prominence differences at levels corresponding to categorical annotations. Perception of a word's relative prominence was consistently influenced by phonetic and phonological factors, while pragmatic factors (such as contrast-evoking context) required more elaborate plausibility manipulations in order to affect prominence perception.

This paper presents an eye-tracking experiment comparing the processing of different accent patterns in unit selection synthesis and human speech. The synthetic speech results failed to replicate the facilitative effect of contextually appropriate accent patterns found with human speech, while producing a more robust intonational garden-path effect with contextually inappropriate patterns, both of which could be due to processing delays seen with the synthetic speech. As the synthetic speech was of high quality, the results indicate that eye tracking holds promise as a highly sensitive and objective method for the online evaluation of prosody in speech synthesis. Copyright © 2009 ISCA.

L+H* is claimed to evoke contrast between lt whether this prominent accent projects a contrastive relation for the discourse foreground, spontaneous continuations of short stories were examined. For both subject and object positions, participants mentioned contrastive alternatives more often when the corresponding discourse entity was introduced with L+H* in the context. Also, the presence of L+H* on a discourse marker (e.g., And NEXT) that preceded the continuation facilitated the mention of the contrastive subject/object. These results do not merely confirm the function of L+H* on arguments in projecting contrastive relations, but also uncovers the facilitative mediation of the contrastive relation by L+H* on a discourse marker.

Previous eye tracking studies [1] demonstrated that a prominent L+H* accent in English instructions evoked a contrastive relationship between discourse referents, immediately guiding listeners' eye movements to contrastive candidates in the visual scene. However, the visual layout in the previous study conflated looks to contrastive candidates with those to previously fixated object groups, thus calling into question the cause of the very early fixations to contrastive targets. In order to eliminate the possibility that L+H* evokes contrast only when the visual layout allows easy detection of the target, the current experiment used layouts where contrastive candidates had separate visual domains. The results confirm the immediate effect of L+H*, demonstrating a faster increase in fixations to contrastive targets than with H*. In addition, results show an intonational 'garden-path effect' due to the immediate integration of L+H* as contrast marking accent.

We review psycholinguistic research on the use of intonation in dialogue, focusing on our own recent work. In experiments using complex real-world tasks and naïve speakers and listeners, we show that speakers reliably specific prosodic cues to signal their intensions, and that listeners use these cues to recognize syntactic and pragmatic aspects of discourse meaning.

Previous observation of spontaneous speech has shown consistent use of pitch accent by speakers to mark the contrastive status of words. To investigate how listeners process accentual prominence marking a contrast, eyemovements were monitored while participants listened to spoken directions and searched for ornaments to decorate holiday trees. Eye movement latencies to the target ornament cells were shorter when the intonation felicitously marked contrast on the color (e.g. First, hang the green drum ¿ Next, hang the ORANGE drum.) than when it did not (¿ orange DRUM). Felicitous pitch accent placement also induced earlier fixations to the target compared to trials that simply lacked the emphatic accent (¿ orange drum). In addition, the infelicitous use of the accent on the color modifier (e.g. green drum ¿ ORANGE ball) led to incorrect initial fixations to the preceding ornament cell (e.g. drum) before the noun itself was processed. These results demonstrate the immediate processing of accentual information on a modifier that leads to a strong expectation about the upcoming discourse entity.