Directory

Hsin-min Wang - Academia.edu

Skip to main content

Hsin-min Wang

Followers

15

Following

1

Co-author

1

Public Views

WARSE The World Academy of Research in Science and Engineering

Dr. Sujan K U M A R Roy

University of Rajshahi, Bangladesh

Badji Mokhtar - Annaba University

Nanyang Technological University

Cornelius Glackin

Odtü

Interests

Uploads

Papers by Hsin-min Wang

Specialized Speech Enhancement Model Selection Based on Learned Non-Intrusive Quality Assessment Metric

Interspeech 2019, 2019

Previous studies have shown that a specialized speech enhancement model can outperform a general ... more Previous studies have shown that a specialized speech enhancement model can outperform a general model when the test condition is matched to the training condition. Therefore, choosing the correct (matched) candidate model from a set of ensemble models is critical to achieve generalizability. Although the best decision criterion should be based directly on the evaluation metric, the need for a clean reference makes it impractical for employment. In this paper, we propose a novel specialized speech enhancement model selection (SSEMS) approach that applies a non-intrusive quality estimation model, termed Quality-Net, to solve this problem. Experimental results first confirm the effectiveness of the proposed SSEMS approach. Moreover, we observe that the correctness of Quality-Net in choosing the most suitable model increases as input noisy SNR increases, and thus the results of the proposed systems outperform another auto-encoder-based model selection and a general model, particularly under high SNR conditions.

Exploring the Encoder Layers of Discriminative Autoencoders for LVCSR

Interspeech 2019, 2019

Discriminative autoencoders (DcAEs) have been proven to improve generalization of the learned aco... more Discriminative autoencoders (DcAEs) have been proven to improve generalization of the learned acoustic models by increasing their reconstruction capacity of input features from the frame embeddings. In this paper, we integrate DcAEs into two models, namely TDNNs and LSTMs, which have been commonly adopted in the Kaldi recipes for LVCSR in recent years, using the modified nnet3 neural network library. We also explore two kinds of skip-connection mechanisms for DcAEs, namely concatenation and addition. The results of LVCSR experiments on the MATBN Mandarin Chinese corpus and the WSJ English corpus show that the proposed DcAE-TDNNbased system achieves relative word error rate reductions of 3% and 10% over the TDNN-based baseline system, respectively. The DcAE-TDNN-LSTM-based system also outperforms the TDNN-LSTM-based baseline system. The results imply the flexibility of DcAEs to be integrated with other existing or prospective neural network-based acoustic models.

運用序列到序列生成架構於重寫式自動摘要(Exploiting Sequence-to-Sequence Generation Framework for Automatic Abstractive Summarization)[In Chinese]

摘要自動摘要 (Automatic Summarization) 一直以來都是熱門的研究議題,過去多著重在節錄式 (Extractive) 摘要,而重寫式 (Abstractive) 摘要相... more 摘要自動摘要 (Automatic Summarization) 一直以來都是熱門的研究議題,過去多著重在節錄式 (Extractive) 摘要,而重寫式 (Abstractive) 摘要相當稀少。有鑑於近期深度學習被廣泛應用在自然語言處理,尤其是機器翻譯等領域的成功,讓重寫式摘要的研究又熱絡起來。近期文獻中已初步驗證了遞歸神經網路 (Recurrent Neural Network) 在文件的重寫式自動摘要之成效。因此本文欲探討加入注意力 (Attention) 機制的效果。注意力機制的特點是它能夠在生成文字的同時,對於關鍵片段增強注意力,藉此產生更佳的摘要。此外本文亦欲探究單向 (Uni-directional) 及雙向 (Bi-directional) 遞歸神經網路的差異。本文採用語料是大規模中文短文摘要集 (Large-scale Chinese Short Text Summarization Dataset, LCSTS)。結果顯示,本文所提出之改進對於摘要品質有明顯的助益。

THE ACADEMIA SINICA SYSTEM OF SOUND EVENT DETECTION AND SEPARATION FOR DCASE 2020 Technical Report

In this report, we present the system of sound event detection and separation in domestic environ... more In this report, we present the system of sound event detection and separation in domestic environments for DCASE 2020. The goal of the task aims to determine which sound events appear in a clip and detailed temporal ranges they occupy. The system is trained by using real data, which are either weakly-labeled or unlabeled, and synthesized data with a strongly annotated label. Our proposed model structure starts with a feature-level front-end based on convolution neural networks (CNN) followed by both embedding-level and instance-level back-end attention modules. To take full advantage of a large amount of unlabeled data, we jointly adopt guided learning mechanism and Mean Teacher, which averages model weights instead of label predictions, to carry out weakly-supervised and semi-supervised learning. A group of adaptive median windows for each sound event is also utilized in post-processing for smoothing frame-level predictions. In the public evaluation set of DCASE 2019, our best syst...

A margin-based discriminative modeling approach for extractive speech summarization

Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, 2014

The task of extractive speech summarization is to select a set of salient sentences from an origi... more The task of extractive speech summarization is to select a set of salient sentences from an original spoken document and concatenate them to form a summary, facilitating users to better browse through and understand the content of the document. In this paper we present an empirical study of leveraging various supervised discriminative methods for effectively ranking important sentences of a spoken document to be summarized. In addition, we propose a novel margin-based discriminative training (MBDT) algorithm that aims to penalize non-summary sentences in an inverse proportion to their summarization evaluation scores, leading to better discrimination from the desired summary sentences. By doing so, the summarization model can be trained with an objective function that is closely coupled with the ultimate evaluation metric of extractive speech summarization. Furthermore, sentences of spoken documents are embodied by a wide range of prosodic, lexical and relevance features, whose utilities are extensively compared and analyzed. Experiments conducted on a Mandarin broadcast news summarization task demonstrate the performance merits of our summarization method when compared to several well-studied state-of-the-art supervised and unsupervised methods. I.

Eigenspace-Based Linear Transformation Approach for Rapid Speaker Adaptation

This paper presents our recent effort on the development of the eigenspace-based linear transform... more This paper presents our recent effort on the development of the eigenspace-based linear transformation approach for rapid speaker adaptation. The proposed approach toward prior density selection for the MAPLR framework was developed by introducing a priori knowledge analysis on the training speakers via probabilistic principal component analysis (PPCA), so as to construct an eigenspace for speaker-specific full regression matrices as well as to derive a set of bases called eigen-transformations. The prior densities of MAPLR transformations for each outside speaker are then chosen in the space spanned by the first few eigen-transformations. By incorporating the PPCA model of transformation parameters into the MAPLR scheme, the number of free parameters can be significantly reduced, while the underlying structure of the acoustic space as well as the precise modeling of the inter-dimensional correlation among the model parameters can be well preserved. Rapid supervised adaptation experiments showed that the proposed approach not only is superior to the conventional MLLR approach using either diagonal or block-diagonal regression matrices, but also outperformed by a great amount the full-matrix MLLR with either a global transformation or multiple transformations corresponding to different phonetic classes.

Speech Enhancement-assisted Stargan Voice Conversion in Noisy Environments

Numerous voice conversion (VC) techniques have been proposed for the conversion of voices among d... more Numerous voice conversion (VC) techniques have been proposed for the conversion of voices among different speakers. Although the decent quality of converted speech can be observed when VC is applied in a clean environment, the quality will drop sharply when the system is running under noisy conditions. In order to address this issue, we propose a novel enhancement-based StarGAN (E-StarGAN) VC system, which leverages a speech enhancement (SE) technique for signal pre-processing. SE systems are generally used to reduce noise components in noisy speech and to generate enhanced speech for downstream application tasks. Therefore, we investigated the effectiveness of E-StarGAN, which combines VC and SE, and demonstrated the robustness of the proposed approach in various noisy environments. The results of VC experiments conducted on a Mandarin dataset show that when combined with SE, the proposed EStarGAN VC model is robust to unseen noises. In addition, the subjective listening test resul...

Tone Modeling for Speech Synthesis

Advances in Chinese Spoken Language Processing, 2006

Tone modeling for speech synthesis aims at providing proper pitch, duration, and energy informati... more Tone modeling for speech synthesis aims at providing proper pitch, duration, and energy information to generate natural synthetic speech from input text. As the speech processing technology progresses rapidly in recent years, some advanced tone modeling techniques for MTTS are proposed. In this chapter, two modern tone modeling approaches for Mandarin speech synthesis are discussed in detail.

The Academia Sinica Systems of Voice Conversion for VCC2020

Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020

This paper describes the Academia Sinica systems for the two tasks of Voice Conversion Challenge ... more This paper describes the Academia Sinica systems for the two tasks of Voice Conversion Challenge 2020, namely voice conversion within the same language (Task 1) and cross-lingual voice conversion (Task 2). For both tasks, we followed the cascaded ASR+TTS structure, using phonetic tokens as the TTS input instead of the text or characters. For Task 1, we used the international phonetic alphabet (IPA) as the input of the TTS model. For Task 2, we used unsupervised phonetic symbols extracted by the vector-quantized variational autoencoder (VQ-VAE). In the evaluation, the listening test showed that our systems performed well in the VCC2020 challenge.

Speech emotion recognition with skew-robust neural networks

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Ensemble of machine learning algorithms for cognitive and physical speaker load detection

Interspeech 2014

Noise Adaptive Speech Enhancement Using Domain Adversarial Training

Interspeech 2019

In this study, we propose a novel noise adaptive speech enhancement (SE) system, which employs a ... more In this study, we propose a novel noise adaptive speech enhancement (SE) system, which employs a domain adversarial training (DAT) approach to tackle the issue of a noise type mismatch between the training and testing conditions. Such a mismatch is a critical problem in deep-learning-based SE systems. A large mismatch may cause a serious performance degradation to the SE performance. Because we generally use a well-trained SE system to handle various unseen noise types, a noise type mismatch commonly occurs in real-world scenarios. The proposed noise adaptive SE system contains an encoder-decoderbased enhancement model and a domain discriminator model. During adaptation, the DAT approach encourages the encoder to produce noise-invariant features based on the information from the discriminator model and consequentially increases the robustness of the enhancement model to unseen noise types. Herein, we regard stationary noises as the source domain (with the ground truth of clean speech) and non-stationary noises as the target domain (without the ground truth). We evaluated the proposed system on TIMIT sentences. The experiment results show that the proposed noise adaptive SE system successfully provides significant improvements in PESQ (19.0%), SSNR (39.3%), and STOI (27.0%) over the SE system without an adaptation.

Extractive Broadcast News Summarization Leveraging Recurrent Neural Network Language Modeling Techniques

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015

Extractive text or speech summarization manages to select a set of salient sentences from an orig... more Extractive text or speech summarization manages to select a set of salient sentences from an original document and concatenate them to form a summary, enabling users to better browse through and understand the content of the document. A recent stream of research on extractive summarization is to employ the language modeling (LM) approach for important sentence selection, which has proven to be effective for performing speech summarization in an unsupervised fashion. However, one of the major challenges facing the LM approach is how to formulate the sentence models and accurately estimate their parameters for each sentence in the document to be summarized. In view of this, our work in this paper explores a novel use of recurrent neural network language modeling (RNNLM) framework for extractive broadcast news summarization. On top of such a framework, the deduced sentence models are able to render not only word usage cues but also long-span structural information of word co-occurrence relationships within broadcast news documents, getting around the need for the strict bag-of-words assumption. Furthermore, different model complexities and combinations are extensively analyzed and compared. Experimental results demonstrate the performance merits of our summarization methods when compared to several well-studied state-of-the-art unsupervised methods. Index Terms-speech summarization, language modeling, recurrent neural network, long-span structural information I. INTRODUCTION long with the growing popularity of Internet applications, ever-increasing volumes of multimedia, such as broadcast radio and television programs, lecture recordings, digital

A hierarchical tag-graph search scheme with layered grammar rules for spontaneous speech understanding

Pattern Recognition Letters, 2002

It has always been difficult for language understanding systems to handle spontaneous speech with... more It has always been difficult for language understanding systems to handle spontaneous speech with satisfactory robustness, primarily due to such problems as the fragments, disfluencies, out-of-vocabulary words, and ill-formed sentence structures. Also, the search schemes used are usually not flexible enough in accepting different input linguistic units, and great efforts are therefore required when they are used with different acoustic front ends in different tasks, specially in multi-modal and multilingual systems. In this paper, a new hierarchical tag-graph-based search scheme for spontaneous speech understanding is proposed. This scheme is based on a layered hierarchy of grammar rules, and therefore can integrate all the statistical and rule-based knowledge including acoustic scores, language model scores and grammar rules into the search process. More robust speech understanding is thus achievable. In addition, this scheme can accept graphs of different linguistic units such as phonemes, syllables, characters, words, spotted keywords, or phrases as the input, thus compatible to different acoustic front ends and multi-modal and multilingual applications can be easily developed. This search scheme has been successfully applied to a multi-domain, multi-modal dialogue system.

Homogeneous segmentation and classifier ensemble for audio tag annotation and retrieval

2010 IEEE International Conference on Multimedia and Expo, 2010

Audio tags describe different types of musical information such as genre, mood, and instrument. T... more Audio tags describe different types of musical information such as genre, mood, and instrument. This paper aims to automatically annotate audio clips with tags and retrieve relevant clips from a music database by tags. Given an audio clip, we divide it into several homogeneous segments by using an audio novelty curve, and then extract audio features from each segment with respect to various musical information, such as dynamics, rhythm, timbre, pitch, and tonality. The features in frame-based feature vector sequence format are further represented by their mean and standard deviation such that they can be combined with other segment-based features to form a fixed-dimensional feature vector for a segment. We train an ensemble classifier, which consists of SVM and AdaBoost classifiers, for each tag. For the audio annotation task, the individual classifier outputs are transformed into calibrated probability scores such that probability ensemble can be employed. For the audio retrieval task, we propose using ranking ensemble. We participated in the MIREX 2009 audio tag classification task and our system was ranked first in terms of F-measure and the area under the ROC curve given a tag.

Golden Mandarin(II)-an intelligent Mandarin dictation machine for Chinese character input with adaptation/learning functions

Proceedings of ICSIPNN '94. International Conference on Speech, Image Processing and Neural Networks, 1994

Golderr Mandarin (11) is an intelligent single-chip based real-time Mandarin dictation machine fo... more Golderr Mandarin (11) is an intelligent single-chip based real-time Mandarin dictation machine for Chinese language with v e q large vocabulary for the input of unlimited Chinese tezts into computers using voice. Thw dictation machine can be installed on any personal computer, in which only a single chip Motorola DSP 960020 is wed, with a preliminary character correct rate around 95% at a speed of 0.6 sec per character. Variow adaptation/ leaming function, have been developed for this machine, including fast adaptation to new speakers, on-line leaming the voice characteristics, tark domains, word patterm and noise environments of the wers, so the machine can be earilypersonalized for each wer. These adaptation/ leaming functions will be the major subjects of this pa?er.

Combining Hierarchical Classifiers with Video Semantic Indexing Systems

Lecture Notes in Computer Science, 2001

... Team USA Players Game Actions News Cindy Parlow Mia Hamm Team USA Game Actions News ... New Y... more

Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology Demonstrations - NAACL '03, 2003

This demonstration involves two-way automatic speechto-speech translation on a consumer off-the-s... more This demonstration involves two-way automatic speechto-speech translation on a consumer off-the-shelf PDA. This work was done as part of the DARPA-funded Babylon project, investigating better speech-to-speech translation systems for communication in the field. The development of the Speechalator software-based translation system required addressing a number of hard issues, including a new language for the team (Egyptian Arabic), close integration on a small device, computational efficiency on a limited platform, and scalable coverage for the domain.

Golden Mandarin (II)-an improved single-chip real-time Mandarin dictation machine for Chinese language with very large vocabulary

IEEE International Conference on Acoustics Speech and Signal Processing, 1993

Golden Mandarin (11) is an improved single-chip real-time Mandarin dictation machine for Chinese ... more Golden Mandarin (11) is an improved single-chip real-time Mandarin dictation machine for Chinese language with very large vocabulary for the input of unlimited Chinese sentences into computers uaing voice. In this dictation machine only a single chip Motorola D S P 96002D on an Ariel DSP-96 card is used, with a preliminary character correct rate around 95% in speaker dependent mode at a speed of 0.36 sec per character. This is achieved by many new techniquea, primarily a segmental probability modeling technique for syllable recognition specially considering the characteristics of Mandarin syllables, and a word-lattice-based Chinese character bigram for character identification specially conaidering the structure of Chinese language.

Automatic Construction of Regression Class Tree for MLLR Via Model-Based Hierarchical Clustering

Lecture Notes in Computer Science, 2006

In this paper, we propose a model-based hierarchical clustering algorithm that automatically buil... more In this paper, we propose a model-based hierarchical clustering algorithm that automatically builds a regression class tree for the well-known speaker adaptation technique-Maximum Likelihood Linear Regression (MLLR). When building a regression class tree, the mean vectors of the Gaussian components of the model set of a speaker independent CDHMMbased speech recognition system are collected as the input data for clustering. The proposed algorithm comprises two stages. First, the input data (i.e., all the Gaussian mean vectors of the CDHMMs) is iteratively partitioned by a divisive hierarchical clustering strategy, and the Bayesian Information Criterion (BIC) is applied to determine the number of clusters (i.e., the base classes of the regression class tree). Then, the regression class tree is built by iteratively merging these base clusters using an agglomerative hierarchical clustering strategy, which also uses BIC as the merging criterion. We evaluated the proposed regression class tree construction algorithm on a Mandarin Chinese continuous speech recognition task. Compared to the regression class tree implementation in HTK, the proposed algorithm is more effective in building the regression class tree and can determine the number of regression classes automatically.

Specialized Speech Enhancement Model Selection Based on Learned Non-Intrusive Quality Assessment Metric

Interspeech 2019, 2019

Previous studies have shown that a specialized speech enhancement model can outperform a general ... more Previous studies have shown that a specialized speech enhancement model can outperform a general model when the test condition is matched to the training condition. Therefore, choosing the correct (matched) candidate model from a set of ensemble models is critical to achieve generalizability. Although the best decision criterion should be based directly on the evaluation metric, the need for a clean reference makes it impractical for employment. In this paper, we propose a novel specialized speech enhancement model selection (SSEMS) approach that applies a non-intrusive quality estimation model, termed Quality-Net, to solve this problem. Experimental results first confirm the effectiveness of the proposed SSEMS approach. Moreover, we observe that the correctness of Quality-Net in choosing the most suitable model increases as input noisy SNR increases, and thus the results of the proposed systems outperform another auto-encoder-based model selection and a general model, particularly under high SNR conditions.

Exploring the Encoder Layers of Discriminative Autoencoders for LVCSR

Interspeech 2019, 2019

Discriminative autoencoders (DcAEs) have been proven to improve generalization of the learned aco... more Discriminative autoencoders (DcAEs) have been proven to improve generalization of the learned acoustic models by increasing their reconstruction capacity of input features from the frame embeddings. In this paper, we integrate DcAEs into two models, namely TDNNs and LSTMs, which have been commonly adopted in the Kaldi recipes for LVCSR in recent years, using the modified nnet3 neural network library. We also explore two kinds of skip-connection mechanisms for DcAEs, namely concatenation and addition. The results of LVCSR experiments on the MATBN Mandarin Chinese corpus and the WSJ English corpus show that the proposed DcAE-TDNNbased system achieves relative word error rate reductions of 3% and 10% over the TDNN-based baseline system, respectively. The DcAE-TDNN-LSTM-based system also outperforms the TDNN-LSTM-based baseline system. The results imply the flexibility of DcAEs to be integrated with other existing or prospective neural network-based acoustic models.

運用序列到序列生成架構於重寫式自動摘要(Exploiting Sequence-to-Sequence Generation Framework for Automatic Abstractive Summarization)[In Chinese]

摘要自動摘要 (Automatic Summarization) 一直以來都是熱門的研究議題,過去多著重在節錄式 (Extractive) 摘要,而重寫式 (Abstractive) 摘要相... more 摘要自動摘要 (Automatic Summarization) 一直以來都是熱門的研究議題,過去多著重在節錄式 (Extractive) 摘要,而重寫式 (Abstractive) 摘要相當稀少。有鑑於近期深度學習被廣泛應用在自然語言處理,尤其是機器翻譯等領域的成功,讓重寫式摘要的研究又熱絡起來。近期文獻中已初步驗證了遞歸神經網路 (Recurrent Neural Network) 在文件的重寫式自動摘要之成效。因此本文欲探討加入注意力 (Attention) 機制的效果。注意力機制的特點是它能夠在生成文字的同時,對於關鍵片段增強注意力,藉此產生更佳的摘要。此外本文亦欲探究單向 (Uni-directional) 及雙向 (Bi-directional) 遞歸神經網路的差異。本文採用語料是大規模中文短文摘要集 (Large-scale Chinese Short Text Summarization Dataset, LCSTS)。結果顯示,本文所提出之改進對於摘要品質有明顯的助益。

THE ACADEMIA SINICA SYSTEM OF SOUND EVENT DETECTION AND SEPARATION FOR DCASE 2020 Technical Report

In this report, we present the system of sound event detection and separation in domestic environ... more In this report, we present the system of sound event detection and separation in domestic environments for DCASE 2020. The goal of the task aims to determine which sound events appear in a clip and detailed temporal ranges they occupy. The system is trained by using real data, which are either weakly-labeled or unlabeled, and synthesized data with a strongly annotated label. Our proposed model structure starts with a feature-level front-end based on convolution neural networks (CNN) followed by both embedding-level and instance-level back-end attention modules. To take full advantage of a large amount of unlabeled data, we jointly adopt guided learning mechanism and Mean Teacher, which averages model weights instead of label predictions, to carry out weakly-supervised and semi-supervised learning. A group of adaptive median windows for each sound event is also utilized in post-processing for smoothing frame-level predictions. In the public evaluation set of DCASE 2019, our best syst...

A margin-based discriminative modeling approach for extractive speech summarization

Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, 2014

The task of extractive speech summarization is to select a set of salient sentences from an origi... more The task of extractive speech summarization is to select a set of salient sentences from an original spoken document and concatenate them to form a summary, facilitating users to better browse through and understand the content of the document. In this paper we present an empirical study of leveraging various supervised discriminative methods for effectively ranking important sentences of a spoken document to be summarized. In addition, we propose a novel margin-based discriminative training (MBDT) algorithm that aims to penalize non-summary sentences in an inverse proportion to their summarization evaluation scores, leading to better discrimination from the desired summary sentences. By doing so, the summarization model can be trained with an objective function that is closely coupled with the ultimate evaluation metric of extractive speech summarization. Furthermore, sentences of spoken documents are embodied by a wide range of prosodic, lexical and relevance features, whose utilities are extensively compared and analyzed. Experiments conducted on a Mandarin broadcast news summarization task demonstrate the performance merits of our summarization method when compared to several well-studied state-of-the-art supervised and unsupervised methods. I.

Eigenspace-Based Linear Transformation Approach for Rapid Speaker Adaptation

This paper presents our recent effort on the development of the eigenspace-based linear transform... more This paper presents our recent effort on the development of the eigenspace-based linear transformation approach for rapid speaker adaptation. The proposed approach toward prior density selection for the MAPLR framework was developed by introducing a priori knowledge analysis on the training speakers via probabilistic principal component analysis (PPCA), so as to construct an eigenspace for speaker-specific full regression matrices as well as to derive a set of bases called eigen-transformations. The prior densities of MAPLR transformations for each outside speaker are then chosen in the space spanned by the first few eigen-transformations. By incorporating the PPCA model of transformation parameters into the MAPLR scheme, the number of free parameters can be significantly reduced, while the underlying structure of the acoustic space as well as the precise modeling of the inter-dimensional correlation among the model parameters can be well preserved. Rapid supervised adaptation experiments showed that the proposed approach not only is superior to the conventional MLLR approach using either diagonal or block-diagonal regression matrices, but also outperformed by a great amount the full-matrix MLLR with either a global transformation or multiple transformations corresponding to different phonetic classes.

Speech Enhancement-assisted Stargan Voice Conversion in Noisy Environments

Numerous voice conversion (VC) techniques have been proposed for the conversion of voices among d... more Numerous voice conversion (VC) techniques have been proposed for the conversion of voices among different speakers. Although the decent quality of converted speech can be observed when VC is applied in a clean environment, the quality will drop sharply when the system is running under noisy conditions. In order to address this issue, we propose a novel enhancement-based StarGAN (E-StarGAN) VC system, which leverages a speech enhancement (SE) technique for signal pre-processing. SE systems are generally used to reduce noise components in noisy speech and to generate enhanced speech for downstream application tasks. Therefore, we investigated the effectiveness of E-StarGAN, which combines VC and SE, and demonstrated the robustness of the proposed approach in various noisy environments. The results of VC experiments conducted on a Mandarin dataset show that when combined with SE, the proposed EStarGAN VC model is robust to unseen noises. In addition, the subjective listening test resul...

Tone Modeling for Speech Synthesis

Advances in Chinese Spoken Language Processing, 2006

Tone modeling for speech synthesis aims at providing proper pitch, duration, and energy informati... more Tone modeling for speech synthesis aims at providing proper pitch, duration, and energy information to generate natural synthetic speech from input text. As the speech processing technology progresses rapidly in recent years, some advanced tone modeling techniques for MTTS are proposed. In this chapter, two modern tone modeling approaches for Mandarin speech synthesis are discussed in detail.

The Academia Sinica Systems of Voice Conversion for VCC2020

Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020

This paper describes the Academia Sinica systems for the two tasks of Voice Conversion Challenge ... more This paper describes the Academia Sinica systems for the two tasks of Voice Conversion Challenge 2020, namely voice conversion within the same language (Task 1) and cross-lingual voice conversion (Task 2). For both tasks, we followed the cascaded ASR+TTS structure, using phonetic tokens as the TTS input instead of the text or characters. For Task 1, we used the international phonetic alphabet (IPA) as the input of the TTS model. For Task 2, we used unsupervised phonetic symbols extracted by the vector-quantized variational autoencoder (VQ-VAE). In the evaluation, the listening test showed that our systems performed well in the VCC2020 challenge.

Speech emotion recognition with skew-robust neural networks

2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Ensemble of machine learning algorithms for cognitive and physical speaker load detection

Interspeech 2014

Noise Adaptive Speech Enhancement Using Domain Adversarial Training

Interspeech 2019

In this study, we propose a novel noise adaptive speech enhancement (SE) system, which employs a ... more In this study, we propose a novel noise adaptive speech enhancement (SE) system, which employs a domain adversarial training (DAT) approach to tackle the issue of a noise type mismatch between the training and testing conditions. Such a mismatch is a critical problem in deep-learning-based SE systems. A large mismatch may cause a serious performance degradation to the SE performance. Because we generally use a well-trained SE system to handle various unseen noise types, a noise type mismatch commonly occurs in real-world scenarios. The proposed noise adaptive SE system contains an encoder-decoderbased enhancement model and a domain discriminator model. During adaptation, the DAT approach encourages the encoder to produce noise-invariant features based on the information from the discriminator model and consequentially increases the robustness of the enhancement model to unseen noise types. Herein, we regard stationary noises as the source domain (with the ground truth of clean speech) and non-stationary noises as the target domain (without the ground truth). We evaluated the proposed system on TIMIT sentences. The experiment results show that the proposed noise adaptive SE system successfully provides significant improvements in PESQ (19.0%), SSNR (39.3%), and STOI (27.0%) over the SE system without an adaptation.

Extractive Broadcast News Summarization Leveraging Recurrent Neural Network Language Modeling Techniques

IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2015

Extractive text or speech summarization manages to select a set of salient sentences from an orig... more Extractive text or speech summarization manages to select a set of salient sentences from an original document and concatenate them to form a summary, enabling users to better browse through and understand the content of the document. A recent stream of research on extractive summarization is to employ the language modeling (LM) approach for important sentence selection, which has proven to be effective for performing speech summarization in an unsupervised fashion. However, one of the major challenges facing the LM approach is how to formulate the sentence models and accurately estimate their parameters for each sentence in the document to be summarized. In view of this, our work in this paper explores a novel use of recurrent neural network language modeling (RNNLM) framework for extractive broadcast news summarization. On top of such a framework, the deduced sentence models are able to render not only word usage cues but also long-span structural information of word co-occurrence relationships within broadcast news documents, getting around the need for the strict bag-of-words assumption. Furthermore, different model complexities and combinations are extensively analyzed and compared. Experimental results demonstrate the performance merits of our summarization methods when compared to several well-studied state-of-the-art unsupervised methods. Index Terms-speech summarization, language modeling, recurrent neural network, long-span structural information I. INTRODUCTION long with the growing popularity of Internet applications, ever-increasing volumes of multimedia, such as broadcast radio and television programs, lecture recordings, digital

A hierarchical tag-graph search scheme with layered grammar rules for spontaneous speech understanding

Pattern Recognition Letters, 2002

It has always been difficult for language understanding systems to handle spontaneous speech with... more It has always been difficult for language understanding systems to handle spontaneous speech with satisfactory robustness, primarily due to such problems as the fragments, disfluencies, out-of-vocabulary words, and ill-formed sentence structures. Also, the search schemes used are usually not flexible enough in accepting different input linguistic units, and great efforts are therefore required when they are used with different acoustic front ends in different tasks, specially in multi-modal and multilingual systems. In this paper, a new hierarchical tag-graph-based search scheme for spontaneous speech understanding is proposed. This scheme is based on a layered hierarchy of grammar rules, and therefore can integrate all the statistical and rule-based knowledge including acoustic scores, language model scores and grammar rules into the search process. More robust speech understanding is thus achievable. In addition, this scheme can accept graphs of different linguistic units such as phonemes, syllables, characters, words, spotted keywords, or phrases as the input, thus compatible to different acoustic front ends and multi-modal and multilingual applications can be easily developed. This search scheme has been successfully applied to a multi-domain, multi-modal dialogue system.

Homogeneous segmentation and classifier ensemble for audio tag annotation and retrieval

2010 IEEE International Conference on Multimedia and Expo, 2010

Audio tags describe different types of musical information such as genre, mood, and instrument. T... more Audio tags describe different types of musical information such as genre, mood, and instrument. This paper aims to automatically annotate audio clips with tags and retrieve relevant clips from a music database by tags. Given an audio clip, we divide it into several homogeneous segments by using an audio novelty curve, and then extract audio features from each segment with respect to various musical information, such as dynamics, rhythm, timbre, pitch, and tonality. The features in frame-based feature vector sequence format are further represented by their mean and standard deviation such that they can be combined with other segment-based features to form a fixed-dimensional feature vector for a segment. We train an ensemble classifier, which consists of SVM and AdaBoost classifiers, for each tag. For the audio annotation task, the individual classifier outputs are transformed into calibrated probability scores such that probability ensemble can be employed. For the audio retrieval task, we propose using ranking ensemble. We participated in the MIREX 2009 audio tag classification task and our system was ranked first in terms of F-measure and the area under the ROC curve given a tag.

Golden Mandarin(II)-an intelligent Mandarin dictation machine for Chinese character input with adaptation/learning functions

Proceedings of ICSIPNN '94. International Conference on Speech, Image Processing and Neural Networks, 1994

Golderr Mandarin (11) is an intelligent single-chip based real-time Mandarin dictation machine fo... more Golderr Mandarin (11) is an intelligent single-chip based real-time Mandarin dictation machine for Chinese language with v e q large vocabulary for the input of unlimited Chinese tezts into computers using voice. Thw dictation machine can be installed on any personal computer, in which only a single chip Motorola DSP 960020 is wed, with a preliminary character correct rate around 95% at a speed of 0.6 sec per character. Variow adaptation/ leaming function, have been developed for this machine, including fast adaptation to new speakers, on-line leaming the voice characteristics, tark domains, word patterm and noise environments of the wers, so the machine can be earilypersonalized for each wer. These adaptation/ leaming functions will be the major subjects of this pa?er.

Combining Hierarchical Classifiers with Video Semantic Indexing Systems

Lecture Notes in Computer Science, 2001

... Team USA Players Game Actions News Cindy Parlow Mia Hamm Team USA Game Actions News ... New Y... more

Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology Demonstrations - NAACL '03, 2003

This demonstration involves two-way automatic speechto-speech translation on a consumer off-the-s... more This demonstration involves two-way automatic speechto-speech translation on a consumer off-the-shelf PDA. This work was done as part of the DARPA-funded Babylon project, investigating better speech-to-speech translation systems for communication in the field. The development of the Speechalator software-based translation system required addressing a number of hard issues, including a new language for the team (Egyptian Arabic), close integration on a small device, computational efficiency on a limited platform, and scalable coverage for the domain.

Golden Mandarin (II)-an improved single-chip real-time Mandarin dictation machine for Chinese language with very large vocabulary

IEEE International Conference on Acoustics Speech and Signal Processing, 1993

Golden Mandarin (11) is an improved single-chip real-time Mandarin dictation machine for Chinese ... more Golden Mandarin (11) is an improved single-chip real-time Mandarin dictation machine for Chinese language with very large vocabulary for the input of unlimited Chinese sentences into computers uaing voice. In this dictation machine only a single chip Motorola D S P 96002D on an Ariel DSP-96 card is used, with a preliminary character correct rate around 95% in speaker dependent mode at a speed of 0.36 sec per character. This is achieved by many new techniquea, primarily a segmental probability modeling technique for syllable recognition specially considering the characteristics of Mandarin syllables, and a word-lattice-based Chinese character bigram for character identification specially conaidering the structure of Chinese language.

Automatic Construction of Regression Class Tree for MLLR Via Model-Based Hierarchical Clustering

Lecture Notes in Computer Science, 2006

In this paper, we propose a model-based hierarchical clustering algorithm that automatically buil... more In this paper, we propose a model-based hierarchical clustering algorithm that automatically builds a regression class tree for the well-known speaker adaptation technique-Maximum Likelihood Linear Regression (MLLR). When building a regression class tree, the mean vectors of the Gaussian components of the model set of a speaker independent CDHMMbased speech recognition system are collected as the input data for clustering. The proposed algorithm comprises two stages. First, the input data (i.e., all the Gaussian mean vectors of the CDHMMs) is iteratively partitioned by a divisive hierarchical clustering strategy, and the Bayesian Information Criterion (BIC) is applied to determine the number of clusters (i.e., the base classes of the regression class tree). Then, the regression class tree is built by iteratively merging these base clusters using an agglomerative hierarchical clustering strategy, which also uses BIC as the merging criterion. We evaluated the proposed regression class tree construction algorithm on a Mandarin Chinese continuous speech recognition task. Compared to the regression class tree implementation in HTK, the proposed algorithm is more effective in building the regression class tree and can determine the number of regression classes automatically.