Expressive Text-to-Speech using Style Tag

Minchan Kim Sung Jun Cheon Byoung Jin Choi Jong Jin Kim Nam Soo Kim

Abstract

As recent text-to-speech (TTS) systems have been rapidly improved in speech quality and generation speed, many researchers now focus on a more challenging issue: expressive TTS. To control speaking styles, existing expressive TTS models use categorical style index or reference speech as style input. In this work, we propose StyleTagging-TTS (ST-TTS), a novel expressive TTS model that utilizes a style tag written in natural language. Using a style-tagged TTS dataset and a pre-trained language model, we modeled the relationship between linguistic embedding and speaking style domain, which enables our model to work even with style tags unseen during training. As style tag is written in natural language, it can control speaking style in a more intuitive, interpretable, and scalable way compared with style index or reference speech. In addition, in terms of model architecture, we propose an efficient non-autoregressive (NAR) TTS architecture with single-stage training. The experimental result shows that ST-TTS outperforms the existing expressive TTS model, Tacotron2-GST in speech quality and expressiveness.


Style Tagging Examples

Generated samples using style tags. The style tags are originally written in Korean, and we additionally provide the translations with similar nuance to help understanding. The blue tags are used for training, and the red tags are not used.

Sentence 1

#평범하게 (#normally)

#다정하게 (#with affection)

#화가난 듯 (#seem angry)

#졸린 듯 (#sleepy)

#상쾌하게 (#refreshingly)

#참담한 (#disastrous)

Sentence 2

#평범하게 (#normally)

#다급하게 (#in a rush)

#쌀쌀맞게 (#coldly)

#속삭이듯 (#whispering)

#기분 좋은 (#agreeable)

#급박하게 (#urgently)

Sentence 3

#무기력한 (#lethargic)

#울먹이듯 (#feel like crying)

#숨이 차듯 (#out of breath)

#술 취한 듯 (#drunken)

#서글픈 (#doleful)

#당차게 (#confident)

Sentence 4

#무시하듯 (#looking down on)

#뾰로통하게 (#pouting)

#간절하게 (#desperately)

#징징대는 (#whining)

#침착하게 (#calmly)

#겁에 질린 (#frightened)

Sentence 5

#호탕하게 (#broad-minded)

#수줍은 듯 (#bashful)

#보채는 (#pushing)

#호통치듯 (#telling off)

#흥겨운 (#exciting)

#마음에 안드는 (#don't like)


Style Transfer Examples

Generated samples using reference speech. The 'reference's are used for extracting style embedding, and the 'synthesized's are synthesized speeches using that style embeddings.

Sentence 1

reference 1

synthesized 1

reference 2

synthesized 2

reference 3

synthesized 3

Sentence 2

reference 1

synthesized 1

reference 2

synthesized 2

reference 3

synthesized 3

Sentence 3

reference 1

synthesized 1

reference 2

synthesized 2

reference 3

synthesized 3

Sentence 4

reference 1

synthesized 1

reference 2

synthesized 2

reference 3

synthesized 3

Sentence 5

reference 1

synthesized 1

reference 2

synthesized 2

reference 3

synthesized 3