g2pM

A Neural Grapheme-to-Phoneme Conversion Package for Mandarin Chinese Based on a New Open Benchmark Dataset

Showing:

Popularity

Downloads/wk

0

GitHub Stars

177

Maintenance

Last Commit

1yr ago

Contributors

2

Package

Dependencies

0

License

Apache License 2.0

Categories

Readme

g2pM

Release Downloads license

This is the official repository of our paper A Neural Grapheme-to-Phoneme Conversion Package for MandarinChinese Based on a New Open Benchmark Dataset (Interspeech 2020).

Install

pip install g2pM

The CPP Dataset

In data folder, there are [train/dev/test].sent files and [train/dev/test].lb files. In .sent file, each lines corresponds to one sentence and a special symbol ▁ (U+2581) is added to the left and right of polyphonic character. The pronunciation of the corresponding character is at the same line from .lb file. For each sentence, there could be several polyphonic characters, but we randomly choose only one polyphonic character to annotate.

Requirements

  • python >= 3.6
  • numpy

Usage

If you want to remove all the digits which denote the tones, set tone=False. Default setting is tone=True.
If you want to split all the non Chinese characters (e.g. digit), set char_split=True. Default setting is char_split=False.

>>> from g2pM import G2pM
>>> model = G2pM()
>>> sentence = "然而,他红了20年以后,他竟退出了大家的视线。"
>>> model(sentence, tone=True, char_split=False)
['ran2', 'er2', ',', 'ta1', 'hong2', 'le5', '20', 'nian2', 'yi3', 'hou4', ',', 'ta1', 'jing4', 'tui4', 'chu1', 'le5', 'da4', 'jia1', 'de5', 'shi4', 'xian4', '。']
>>> model(sentence, tone=False, char_split=False)
['ran', 'er', ',', 'ta', 'hong', 'le', '2', '0', 'nian', 'yi', 'hou', ',', 'ta', 'jing', 'tui', 'chu', 'le', 'da', 'jia', 'de', 'shi', 'xian', '。']
>>> model(sentence, tone=True, char_split=True)
['ran2', 'er2', ',', 'ta1', 'hong2', 'le5', '2', '0', 'nian2', 'yi3', 'hou4', ',', 'ta1', 'jing4', 'tui4', 'chu1', 'le5', 'da4', 'jia1', 'de5', 'shi4', 'xian4', '。']

Model Size

LayerSize
Embedding64
LSTM x164
Fully-Connected x264
Total # of parameters477,228
Model size1.7MB
Package size2.1MB

Evaluation Result

ModelDev.Test
g2pC84.8484.45
xpinyin(0.5.6)78.7478.56
pypinyin(0.36.0)85.4486.13
Majority Vote92.1592.08
Chinese Bert97.9597.85
Ours97.3697.31

Reference

To cite the code/data/paper, please use this BibTex

@article{park2020g2pm,
 author={Park, Kyubyong and Lee, Seanie},
 title = {A Neural Grapheme-to-Phoneme Conversion Package for Mandarin Chinese Based on a New Open Benchmark Dataset
},
 journal={Proc. Interspeech 2020},
 url = {https://arxiv.org/abs/2004.03136},
 year = {2020}
}

Rate & Review

Great Documentation0
Easy to Use0
Performant0
Highly Customizable0
Bleeding Edge0
Responsive Maintainers0
Poor Documentation0
Hard to Use0
Slow0
Buggy0
Abandoned0
Unwelcoming Community0
100