Authors: Chang Liu, Zhen-Hua Ling and Ling-Hui Chen
Abstract: This paper proposes a multilingual speech synthesis method using learned phonetic representations to avoid reliance on the pronunciation dictionaries of target languages. The learned phonetic representations consist of unsupervised phonetic representations (UPR) and supervised phonetic representations (SPR). In this method, a pre-trained wav2vec 2.0 model is adopted to extract UPRs and a language-independent automatic speech recognition (LI-ASR) model is built with a connectionist temporal classification (CTC) loss to extract segment-level SPRs from the audio data of target languages. Then, an acoustic model using UPRs and SPRs as intermediate representations is designed, which is composed of a UPR predictor, an SPR predictor, and a representation-to-mel-spectrogram (RTM) converter. The two predictors generate UPRs and SPRs from texts, respectively. The RTM converter first combines UPRs with SPRs using a Transformer-based encoder, and then feeds the merged representations into a decoder to generate mel-spectrograms. Considering the difficulty of collecting large training corpora for all languages in multilingual speech synthesis, the parameters of both the two predictors and the RTM converter can be pre-trained on non-target languages to further improve model performance. The results of our experiments on six target languages show that the proposed method outperformed the methods that directly predicted mel-spectrograms from character or phoneme sequences, and pre-training the acoustic model using a multilingual corpus further improved the performance of synthetic speech.Listen to this abstract speech synthesized by our Proposed_PT method trained on 5 hours data of LJSpeech:
Text |
Yet it must have been abundantly plain to the reader that the other evils existing were great and glaring. A brief summary of them will best prove this. | and "to dispose him, by meditation and seclusion, to return to an honest life." | A committee of aldermen was forthwith appointed to report upon the inspectors' report, | Men like Mobbs, the miscreant known as "General Haynau" on account of his blood-thirstiness, still murdered their wives; | he would be in some way provided for, he having been for some time previously in abject circumstances. |
| Ground Truth | |||||
| Taco_Char | |||||
| Taco_Phone | |||||
| Proposed | |||||
| Proposed_PT | |||||
| Proposed-UPR | |||||
| Proposed-SPR | |||||
| Proposed-UPR-Tr | |||||
| Proposed-SPR-Tr |
Text |
Comparando otros lugares o sitios que hemos visitado, nos decepcionó. | De la gente más linda y glamorosa, debería decir. | Su ex suegra, Tillie Wiecek, me dio su dirección. | La hija del director está enamorada de ti. | La vida es muy difícil para los indefensos en este mundo. |
| Ground Truth | |||||
| Taco_Char | |||||
| Taco_Phone | |||||
| Proposed | |||||
| Proposed_PT | |||||
| Proposed-UPR | |||||
| Proposed-SPR | |||||
| Proposed-UPR-Tr | |||||
| Proposed-SPR-Tr |
Text |
Аттила Батыс Римді айтқанына көндіріп алған соң, енді қайта айналып салық тапсырудан бас тартқан Византияға жаза жорығын жасауға қамданады. | Мүмкіндік болса, жұмыста ішетін түскі асты да үйден әзірлеп алып барыңыз. | Балаларға ұлттық ойындарды үйрету арқылы жас ұрпақтың денсаулығын реттеу, салауатты өмір сүруге жағдай жасау бүгінгі заман талабы. | Анамның қасына жатып ұйықтап едім, ол мені сол баяғы бала кездегідей құшақтап жатты. | Егер сізден жасы кіші адам сіздің қатеңізді айтып немесе сізге ескерту жасаса, оған ренжімейсіз бе? |
| Ground Truth | |||||
| Taco_Char | |||||
| Taco_Phone | |||||
| Proposed | |||||
| Proposed_PT | |||||
| Proposed-UPR | |||||
| Proposed-SPR | |||||
| Proposed-UPR-Tr | |||||
| Proposed-SPR-Tr |
Text |
मार्ग की स्थिति छत्रपति शिवाजी स्टेशन के पास नहीं मिल सकती है. | पांचवा चुना. यह जिस तरह से जुड़ा हुआ है | ताज महल पार्किंग स्थल मिला. क्या आप अब चलना चाहते हैं | हालांकि, शाम के शोज तक बिजनेस ठंडा पड़ने लगा था | लेकिन चहल ने वापसी की और ओवर की चौथी गेंद पर ही मिचेल सैंटनर को बोल्ड कर दिया. |
| Ground Truth | |||||
| Taco_Char | |||||
| Taco_Phone | |||||
| Proposed | |||||
| Proposed_PT | |||||
| Proposed-UPR | |||||
| Proposed-SPR | |||||
| Proposed-UPR-Tr | |||||
| Proposed-SPR-Tr |
Text |
Хора, които ние не познаваме, но познават българия и изпитват благодарност към нея. | Беше аранжирано и изложение на парфюмерийни продукти с българска роза. | Посоченото означава че тези растения не могат в никакъв случай да бъдат внесени. | Учениците представиха богата празнична програма която включваше много песни, стихотворения и танци. | Посетителите получиха за спомен и дребни сувенири от българия. |
| Ground Truth | |||||
| Taco_Char | |||||
| Taco_Phone | |||||
| Proposed | |||||
| Proposed_PT | |||||
| Proposed-UPR | |||||
| Proposed-SPR | |||||
| Proposed-UPR-Tr | |||||
| Proposed-SPR-Tr |
Text |
Elok lah tu jika dibandingkan dengan setahun. Mahu gemuk jadinya duduk restoran lama-lama. | Kehilangan Appa buat Sehun dan Hyuk dan diakhiri dengan keguguran InJong. Semuanya diceritakan buat lelaki tersebut. | Nor Azwa Ibrahim yang mewakili Ketua Jabatan Kimia dan Pegawai-Pegawai Sains dari Jabatan Biologi, Kimia dan Fizik. | Anak tengah lena tidur kat belakang, bapaknya pulak menggatal kat depanni. | Darleesa mengomel besar, Siapa yang nak heret dia ke sini? Boleh lah dia banting tulang bersihkan pelosok banglo besar itu. |
| Ground Truth | |||||
| Taco_Char | |||||
| Taco_Phone | |||||
| Proposed | |||||
| Proposed_PT | |||||
| Proposed-UPR | |||||
| Proposed-SPR | |||||
| Proposed-UPR-Tr | |||||
| Proposed-SPR-Tr |