Contributions of Linguistic Features to the Perception of Comprehensibility and

Yang Chunsheng University of Connecticut

Chen Kaidi University of Connecticut

Yang Han University of Chinese Academy of Social Sciences

Li Danwei University of Connecticut

Abstract In this study, we examined the contributions of various linguistic features in L2 Chinese read speech to the comprehensibility and accentedness ratings by native Chinese judges. By transcribing the segment errors, tone errors, prosody errors, lexicon errors and grammar errors in the reading passage produced by intermediate and advanced learners of Chinese, we were able to tease apart different roles of these various errors in the perception of L2 comprehensibility and accentedness. The data from the reading passage shows that tone error is the most robust predictor of L2 Chinese comprehensibility and accentedness which are strongly correlated with each other as well. While this study seems to support Yang’s (2016) speculation that the strong correlation between accentedness and comprehensibility is the product of mediating tone errors, it is expected that spontaneous speech should be used in future studies in order to further investigate the effects of lexicon and grammar, and that the distinctions of phonemic errors and phonetic deviations should be made in data analysis.

Key words segment errors; tone errors; comprehensibility; accentedness; L2 read speech

1. Introduction

Second language (L2) pronunciation has become an important research topic in the field of applied linguistics and second language acquisition over the past decade and there even exists an academic journal(Journal of Second Language Pronunciation) and an annual L2 pronunciation conference in Second Language Learning and Teaching Conference (PSLLT)exclusively devoted to L2 pronunciation teaching and research. In the field of Teaching Chinese as a Second/Foreign Language, tones and segments (i.e. vowels and consonants) have received most attention. While it is understandable that tones and segments are given most attention, other aspects of pronunciation, such as phrase/utterance-level prosody, rhythm and intonation,are also important in L2 pronunciation (李智强, 2018;Yang, 2016; Yang et al., 2021).

One common theme in L2 pronunciation research is how different aspects of L2 pronunciation affect intelligibility, comprehensibility and accentedness in L2 speech. Accentedness is easy to understand,which refers to the deviations of L2 pronunciation from “standard” norms (although it is often difficult to decide or justify such a standard). Intelligibility measures the extent to which speakers’ message is understood by listeners, and comprehensibility is listeners’ judgment regarding the effort required to process L2 speech (Munro & Derwing, 1995). Some studies have shown that prosody contributes more to L2 accentedness and comprehensibility than segments (Anderson-Hsieh et al., 1992; Derwing &Munro, 1997, 2015), whereas other studies (Sereno et al., 2015) found that segments contribute more to L2 perception than prosody. As can be expected,most studies along this thread of research focus on English as a second or foreign language (Derwing &Munro, 1997; Levis, 2005; Munro & Derwing, 2011;Sereno et al., 2015; Saito et al., 2015; Trofimovich& Isaacs, 2012, among many others). Therefore,typologically different languages, such as Chinese and Arabic, should be investigated to see whether the correlations can be found between different aspects of L2 pronunciation and the three pronunciation constructs hold for other languages and to see whether these relationships vary with languages. To this end,this study focuses on L2 Mandarin Chinese and looks into the effects of segmental and prosodic deviations on comprehensibility and L2 accentedness. We choose to focus on comprehensibility and accentedness in this study, following some recent studies, such as Trofimovich and Isaacs (2012), Kang et al. (2010),Saito et al. (2015), and Yang et al. (2021), in that both constructs are based on listeners’ perceptual ratings on the speech stimuli, which are more comparable to each other. Because Mandarin Chinese is a tone language in which tones are lexically specified, it is expected that the inclusion of L2 Mandarin Chinese will shed new lights on the relationship between different linguistic aspects and comprehensibility and accentedness, likely due to the mediating role of tones (Yang, 2016).

2. Background

2.1. Intelligibility, comprehensibility and accentedness

Starting from Munro and Derwing (1995), many studies have included two or three aspects of L2 pronunciation. As was mentioned above, intelligibility,comprehensibility and accentedness are different but related constructs measuring L2 speech. Due to the nature of foreign accent as a perceptual phenomenon(Thomson, 2017), previous studies (Munro & Derwing,1995; Trofimovich & Isaacs, 2012) have found that both L2 phonetic deviations (e.g. substitute a trilled/r/ for an English /r/) and phonemic deviations (e.g.substitute /l/ for /n/, cited from Thomson, 2017)lead to the perception of foreign accent; while only phonemic deviations affect L2 intelligibility and comprehensibility. Besides, syllable-level errors and non-native rhythmic patterns (Trofimovich & Isaacs,2012), lexical stress errors (Saito et al., 2015), slow speech rate and pausing (Kang et al., 2010) as well as reduced pitch range (Kang, 2010) are also found to affect L2 accent perception (cited from Thomson,2017). With respect to comprehensibility, besides phonemic divergences, lexical and grammatical errors(Trofimovich & Isaacs, 2012) and oral fluency (Saito et al., 2015) are proved to affect L2 comprehensibility as well (adapted from Thomson, 2017).

2.2. Studies on L2 Chinese

Most of the studies mentioned above are related to English as a second or foreign language. Although studies of other languages have started to emerge in recent years, more studies in this line, especially in those less researched languages, such as Chinese and Arabic, are in dire need. To the best of our knowledge, Yang (2016) and Yang et al. (2021) are the only two studies addressing similar issues in L2 Chinese. Yang (2016) used well-controlled sentences that are free from grammatical errors, and found that native Mandarin speakers were very sensitive to prosodic deviations in L2 Mandarin Chinese and that comprehensibility scores highly negatively correlate with accent ratings; while there is only weak correlation between intelligibility and accent ratings.Yang et al. (2021) attempted to tease apart the different effects of segments, rhythm/duration, and intonation on the L2 accentedness and comprehensibility by manipulating the rhythm and intonation patterns across the sentences produced by L1 and L2 speakers.It was found that Chinese listeners rely more on segments than rhythm and intonation in their L2 perception. And intonation is found to contribute more to L2 accentedness and comprehensibility ratings than rhythm.

2.3. Research gaps and research questions

While Yang (2016) and Yang et al. (2021) provide some preliminary understanding to L2 Chinese pronunciation, there are some research gaps. Firstly,Yang (2016) only used controlled short sentences, the findings may not apply to other speech data. Secondly,Yang et al. (2021) used speech manipulations to tease apart the effects of segments, rhythm and intonation.As pointed out in Yang et al. (2021), further studies using natural speech produced by L2 speakers should be conducted to test whether the findings from Yang(2016) and Yang et al. (2021) will be applicable to natural speech. Thirdly and lastly, besides phonetics and phonology, lexicon and grammar have not been included in previous studies on L2 Mandarin Chinese.The data reported in this paper is part of a larger study using both reading passage and spontaneous speech in the examination of the relationship between the various linguistic aspects and the ratings of comprehensibility and accentedness. By asking experienced language teachers to transcribe various linguistic errors in the reading passage and spontaneous speech, including segmental (consonant and vowel) errors, tone errors,prosody errors, lexicon errors and grammar errors,and by asking Mandarin speakers from northern China to rate the comprehensibility and accentedness of the sentences involved, we can tease apart the different effects of various linguistic errors on L2 speech perception. Since we are still working on the transcription and ratings of the spontaneous speech,we will only report the results based on the reading passage. Specifically, this paper attempts to answer the following questions:

1) How do segment errors, tone errors, prosody errors,lexicon errors and grammar errors from read speech affect comprehensibility and accentedness ratings in L2 Chinese?

2) How do comprehensibility and accentedness ratings relate to each other?

3. Methodology

3.1. Speech stimuli

The speech materials used in this study were adopted from a textbook for intermediate Chinese learners. The passage is cited as below:

星期天早上刚五点钟，王国明家里人都已经起来了。因为王国明要坐早上八点钟的火车到北京去，所以他家人都要到火车站去送他。 王先生帮着小王收拾行李，王太太特别给王国明做了很多吃的东西。 王太太想这两天天气热，火车上的东西恐怕不干净。他们把行李收拾好了的时候，已经七点钟了。王国明和家人坐出租车到火车站。他们到火车站的时候，别的同学都在那儿等着他呢。 王国明把他的同学介绍给他家人。 过了一会儿，王国明和他的同学开始上车。王家人和王国明和他的同学说再见。

Ten intermediate-level (2nd-year and 3rd-year) and ten advanced-level (4th-year or above) American L2 learners from a Midwestern public university in the US were recruited to record the reading of the passage.All subjects reported no speaking or hearing problems.They were each paid $10 for their participation in the recording. Table 1 presents the background information of the subjects in this study, including gender, age, duration (years) of Chinese learning, and duration (years) of their study in China.

Table 1 Demographic information of subjects in the two learner groups

Sentences from the 20 recordings were extracted. Some sentences had long hesitations and disfluency, which were, therefore, not used in the transcription and the rating of comprehensibility and accentedness. 212 sentences (118 sentences by the intermediate learners and 94 sentences by the advanced learners) were obtained. Although pauses are important prosodic aspects in L2 speech, this paper only focuses on segment, tone and prosody (temporal and intonational aspects) of L2 Chinese; therefore, pauses were removed in those sentences. Following the same transcription protocol of the whole study, lexicon errors and grammar errors were also included in this study,although there were very few numbers of such errors(mainly slips of tongues).

3.2. Transcriptions

All co-authors were involved in the transcriptions of the sentences. Prior to the respective transcription, all co-authors except for the first author were given ten sentences produced by an intermediate learner and required to transcribe the segment, tone, prosody,lexicon and grammar errors. Segment errors refer to substituting one segment for another one, such as substituting the mid vowel infengwith the low vowel infang. Tone errors are self-evident, namely that one tone was produced as another tone or as a non-tone.Prosody errors refer to temporal and intonational deviations, such as high starting pitch as reported in Yang (2016), or unusual syllable-lengthening. The lexicon and grammar errors are based on the native speaker’s intuition.

Then the four co-authors discussed their transcriptions together. Several issues were found in the testtranscriptions. 1). The distinction between segmental errors (phonemic errors, for example, q/tɕʰ/ produced as ch/ʈʂʰ/) and deviations (phonetic errors, for example,/r/ was produced with tongue tip/blade curled way back, as something like a velarized /r/); 2). prosodic deviations are hard to decide, because everyone, both L1 and L2 speakers, speaks differently. For the first issue, we decided to make the distinction between phonemic errors and phonetic deviations. For the second issue, we decided to rely on native speakers’intuition when deciding whether there is intonational or temporal deviations. (Note that we combined these phonemic and phonetic errors together in data analysis of this paper, due to the small number of both types of errors. Such differences will be made when the data from the spontaneous speech is available later.)

After reaching a consensus on transcriptions, the 2nd, 3rd and 4th authors each transcribed one third of the sentences. Then the first author checked all the transcriptions and discussed with the group whenever there were any disagreements. Then the number of errors in each type was calculated for each sentence.

3.3. Accentedness and comprehensibility ratings

For the accentedness and comprehensibility ratings,the 212 sentences were divided into four blocks, each of which contains either 50 or 51 sentences.

The accentedness and comprehensibility rating tasks were conducted online onwenjuan.com, a free online survey hosting site. Four questionnaires were created for the four blocks. Each questionnaire consists of two parts. The first part is to elicit the demographic information about the rater, such as dialects, age, gender, major and experience with L2 Chinese speakers, while the second part contains the recordings of the 50 or 51 sentences for accentedness and comprehensibility ratings. All sentences in each questionnaire were randomized, and instructions on how to rate each sentence were provided in each question to make sure that the raters are fully aware of the meaning of the rating scales of both comprehensibility and accentedness.

Isaacs and Thomson (2013) show that both untrained native raters and experienced language instructors perform similarly except for some minor qualitative differences in L2 speech rating tasks. Therefore,only four raters were recruited to rate using each questionnaire. In data analysis, the comprehensibility and accentedness ratings for each sentence were obtained by averaging the four raters’ ratings before being submitted to statistical analysis.

4. Results

In this part, we first report the results of the error transcriptions and then move to the analysis of the comprehensibility and accentedness ratings, followed by the analysis of the correlation of comprehensibility and accentedness.

4.1. Results of error transcriptions in the read speech

The different types of errors transcribed by the experienced Chinese instructors are summarized in Table 2.

Table 2 Numbers of different types of errors in the reading passage across groups

As can be seen in Table 2, the intermediate learners produced more errors across the board than the advanced learners, and tone errors are the most frequent type, followed by consonant/vowel errors.Considering the fact that these errors are from a reading passage, there should be no grammar or lexicon errors. Most L2 speakers in the test did not produce any lexicon or grammar errors, as shown in the small numbers of both types of errors, although a few made some errors, mainly slips of tongue in their reading.

4.2. Comprehensibility ratings

To analyze these comprehensibility ratings statistically,each sentence trial response was fitted to a linear mixed-effects model (LMM) using the lmer () function from the lme4 package in R (R core team, 2013).All reported test statistics and p-values represent calculations from the lmer () function. There were six fixed predictors. The five ordinal independent variables (amount of consonant/vowel errors, tone errors, prosody errors, grammar errors, and lexicon errors) were treated as continuous variables, and the one categorical variable (levels of proficiency) was contrast-coded (intermediate=-0.5, advanced=0.5).The dependent variable is comprehensibility rating score.We also included maximal random effects structures justified by the research design. Since each level of proficiency does not appear in every condition of error combination, it cannot be included in the random structure. We tried a maximal model with all five error types and their interactions, but it failed to converge. When the interactions in the random structure were dropped (Matuschek et al., 2017),the model converged. The maximal model ended up consisting of a random intercept for sentences, and random slopes for consonant/vowel errors, tone errors,prosody errors, lexical errors and grammar errors by sentences. It revealed a significant effect of tone errors(β=-2.266e+00, SE=7.185e-01, t=-3.154, p＜0.01★★). No other main effects or interactions were found.

4.3. Accentedness ratings

To analyze the accentedness ratings statistically,sentence trial responses were fitted to a linear mixedeffects model (LMM) using the lmer() function from the lme4 package in R. All reported test statistics and p-values represent calculations from the lmer()function. There were six fixed predictors: The five ordinal independent variables were treated as continuous variables, and the one categorical variable was contrast-coded (advanced=-0.5, intermediate=0.5).

The dependent variable is accentedness rating score.Similar to the setup of the comprehensibility model,the maximal approach was used in the accent model as well. The model converged when all five errors were included (but with their interactions excluded). The maximal model consisted of a random intercept for sentences, and random slopes for consonant/vowel errors, tone errors, prosody errors, lexical errors and grammar errors by sentences. It revealed a significant effect of tone errors (β=6.212, SE=1.048, t=5.928,p＜0.001★★★), a significant two-way interaction of proficiency level and tone errors (β=4.462, SE=2.096,t=2.129, p＜0.05★), and a significant three-way interaction of proficiency level, consonant/vowel errors and tone errors (β=-5.042, SE=2.221, t=-2.270,p＜0.05★).

The interactions of proficiency level and tone errors,and the interactions of proficiency level, tone errors and consonant/vowel errors were observed, although not that strong. The p-value of the two-way interaction is 0.0345, and the p-value of the three-way interactions is 0.0246. These interactions lend support to the information in Table 2, namely the advanced learners produced fewer errors than the intermediate learners across the board in the reading passage, especially with respect to consonant/vowel and tone productions.

4.4. Relationship of comprehensibility and accentedness ratings

Pearson’s correlation was conducted on comprehensibility and accentedness ratings. It shows that they are negatively correlated with each other significantly (r=-0.401, t=-16.072, p＜0.001). That is to say, with every unit increase of accentedness, the comprehensibility will drop by 0.401.

5. Summary and general discussion

In this study, we examined the effects of various types of errors on the rating of comprehensibility and accentedness in L2 Chinese read speech. By meticulously transcribing the segment errors, tone errors, prosody errors, grammar errors and lexicon errors in reading passages produced by intermediate and advanced learners of Chinese, we were able to tease apart the roles of these errors in L2 perception.Our data analyses show that tone errors are the only robust predictor of comprehensibility and accentedness ratings. Although no main effect of proficiency level was found, it does have effect when modulated by tone errors and consonant/vowel errors, as evidenced in more errors in the intermediate learner group than the advanced learner group.

That tone errors are the most robust predictor of accentedness and comprehensibility ratings and their strong correlation in L2 Chinese highlight the important role of tones in Chinese learning. As Mandarin is a tone language, the errors in tones will lead to the misunderstanding of message intended as well as the perception of foreign accent. The strong correlation between comprehensibility and accentedness ratings in this study lends support to Yang’s (2016) speculation. In Yang (2016), short sentences free from lexicon and grammar errors(adapted from reading scenarios) were used and it was found that comprehensibility ratings highly negatively correlate with the foreign accent ratings.Yang (2016) speculated that the correlation between comprehensibility and accentedness in L2 Chinese was due to the mediating roles of tone errors, which was borne out in this study, namely tone errors are the only robust predictor of both comprehensibility and accentedness ratings.

As pointed out in the introduction, there has been a debate on the different roles of segments and prosody in L2 speech perception (Anderson-Hsieh et al., 1992; Derwing & Munro, 1997, 2015; Munro &Derwing, 2011; Sereno et al., 2015). The role of tone errors in this study seems to suggest that prosody or suprasegmental aspects are more important than consonants and vowels when it comes to accentedness and comprehensibility in L2 Chinese read speech. Yang et al. (2021) found that segments are more important than prosody and intonation is more important than rhythm in L2 perception. One possible explanation between this study and Yang et al. (2021) might lie in the different research design. Yang et al. (2021)used speech manipulations to tease apart the different effects of segments and prosody, while this study adopted meticulous transcriptions by experienced language instructors. There is one caveat when the findings of this study are generalized or compared with other studies. As shown in Table 2, among the five types of errors in the reading passage, the tone errors are the most frequent, which definitely contributes to its robust role in predicting L2 accentedness and comprehensibility ratings. Therefore, it would be more reasonable to normalize the various types of errors in some way or to control the amount of different types of errors, which is, of course, not easy to achieve, even in manipulated speech.

One limitation of this study is that only read speech was used the effects of some linguistic features in spontaneous speech, such as lexical and grammatical errors, on comprehensibility and accentedness were not investigated. Actually, the important roles of tone errors in comprehensibility and accentedness ratings may be an artifact of the task itself. As mentioned earlier, we are working on the analysis of spontaneous speech; when completed, it will enable us to compare the differences between reading passage and spontaneous speech. Moreover, spontaneous speech will provide us the opportunity to investigate the effects of lexicon and grammar errors in L2 speech perception, as previous studies have shown that both lexicon and grammar errors (Trofimovich & Isaacs,2012) influence comprehensibility ratings.

In further studies, not only should spontaneous speech be used so that the effects of lexicon and grammar can be investigated, but also the distinctions of phonemic errors and phonetic deviations should be made, in that phonemic errors and phonetic deviations have been shown to have different effects on L2 accentedness and comprehensibility ratings (Munro & Derwing, 1995;Trofimovich & Isaacs, 2012; Thomson, 2017).

6. Conclusion

This paper is part of a larger project investigating the relationship between various linguistic errors in L2 Chinese and comprehensibility and accentedness perception. The data from the reading passage show that tone error is the most robust predictor of L2 Chinese comprehensibility and accentedness, and these two constructs are strongly correlated with each other. While this study seems to support Yang’s (2016)speculation that the strong correlation between accentedness and comprehensibility is a product of the mediating tone errors, this study does not address the roles of lexicon and grammar errors in spontaneous speech. It is expected that data from the spontaneous speech of the project will be able to shed some new light on this fascinating issue in L2 Chinese speech.