Audio Samples from "Pronunciation Dictionary-Free Multilingual Speech Synthesis by Combining Unsupervised and Supervised Phonetic Representations"

Authors: Chang Liu, Zhen-Hua Ling and Ling-Hui Chen

Abstract: This paper proposes a multilingual speech synthesis method which combines unsupervised phonetic representations (UPR) and supervised phonetic representations (SPR) to avoid reliance on the pronunciation dictionaries of target languages. In this method, a pretrained wav2vec 2.0 model is adopted to extract UPRs and a language-independent automatic speech recognition (LI-ASR) model is built with a connectionist temporal classification (CTC) loss to extract segment-level SPRs from the audio data of target languages. Then, an acoustic model is designed, which first predicts UPRs and SPRs from texts separately and then combines the predicted UPRs and SPRs to generate mel-spectrograms. The results of our experiments on six languages show that the proposed method outperformed the methods that directly predicted mel-spectrograms from character or phoneme sequences and the ablated models that utilized only UPRs or SPRs.

Model comparison on English (LJSpeech)
Model comparison on Spanish
Model comparison on Kazakh
Model comparison on Hindi
Model comparison on Bulgarian
Model comparison on Malay
t-SNE visualization of SPRs and UPRs

Model comparison on English (LJSpeech)

Text	Yet it must have been abundantly plain to the reader that the other evils existing were great and glaring. A brief summary of them will best prove this.	He was in the transport side, when one of his fellows, in endeavoring to strike another prisoner with a large poker, missed his aim, and struck Watkins' knee	Another man was hired, himself a convict, whose fees for self and wife were twelve guineas.	he would be in some way provided for, he having been for some time previously in abject circumstances.	The murdered individual was thus identified. The next step was to ascertain where and with whom she had last been seen.
Ground Truth
Taco-Char
Taco-Phone
SPR
UPR
Proposed

Model comparison on Spanish

Text	Comparando otros lugares o sitios que hemos visitado, nos decepcionó.	De la gente más linda y glamorosa, debería decir.	Su ex suegra, Tillie Wiecek, me dio su dirección.	La hija del director está enamorada de ti.	La vida es muy difícil para los indefensos en este mundo.
Ground Truth
Taco-Char
Taco-Phone
SPR
UPR
Proposed

Model comparison on Kazakh

Text	Аттила Батыс Римді айтқанына көндіріп алған соң, енді қайта айналып салық тапсырудан бас тартқан Византияға жаза жорығын жасауға қамданады.	Мүмкіндік болса, жұмыста ішетін түскі асты да үйден әзірлеп алып барыңыз.	Балаларға ұлттық ойындарды үйрету арқылы жас ұрпақтың денсаулығын реттеу, салауатты өмір сүруге жағдай жасау бүгінгі заман талабы.	Анамның қасына жатып ұйықтап едім, ол мені сол баяғы бала кездегідей құшақтап жатты.	Егер сізден жасы кіші адам сіздің қатеңізді айтып немесе сізге ескерту жасаса, оған ренжімейсіз бе?
Ground Truth
Taco-Char
Taco-Phone
SPR
UPR
Proposed

Model comparison on Hindi

Text	मार्ग की स्थिति छत्रपति शिवाजी स्टेशन के पास नहीं मिल सकती है.	पांचवा चुना. यह जिस तरह से जुड़ा हुआ है	ताज महल पार्किंग स्थल मिला. क्या आप अब चलना चाहते हैं	हालांकि, शाम के शोज तक बिजनेस ठंडा पड़ने लगा था	लेकिन चहल ने वापसी की और ओवर की चौथी गेंद पर ही मिचेल सैंटनर को बोल्ड कर दिया.
Ground Truth
Taco-Char
Taco-Phone
SPR
UPR
Proposed

Model comparison on Bulgarian

Text	Хора, които ние не познаваме, но познават българия и изпитват благодарност към нея.	Беше аранжирано и изложение на парфюмерийни продукти с българска роза.	Посоченото означава че тези растения не могат в никакъв случай да бъдат внесени.	Учениците представиха богата празнична програма която включваше много песни, стихотворения и танци.	Посетителите получиха за спомен и дребни сувенири от българия.
Ground Truth
Taco-Char
Taco-Phone
SPR
UPR
Proposed

Model comparison on Malay

Text	Elok lah tu jika dibandingkan dengan setahun. Mahu gemuk jadinya duduk restoran lama-lama.	Kehilangan Appa buat Sehun dan Hyuk dan diakhiri dengan keguguran InJong. Semuanya diceritakan buat lelaki tersebut.	Nor Azwa Ibrahim yang mewakili Ketua Jabatan Kimia dan Pegawai-Pegawai Sains dari Jabatan Biologi, Kimia dan Fizik.	Anak tengah lena tidur kat belakang, bapaknya pulak menggatal kat depanni.	Darleesa mengomel besar, Siapa yang nak heret dia ke sini? Boleh lah dia banting tulang bersihkan pelosok banglo besar itu.
Ground Truth
Taco-Char
Taco-Phone
SPR
UPR
Proposed