KyTea Models

Return to KyTea

• Japanese Models
• Chinese Models

Here we provide a number of models for KyTea that you can download and use in your work. For all varieties of models, there are four models provided:

• Full SVM Model: These models are relatively large, but also achieve the highest accuracy. They are trained using the "-solver 1" option in train-kytea.
• Compact SVM Model: These models are smaller and faster to load, but are slightly less accurate ("-solver 5").
• Compact LR Model: These models are about the same size as the SVM model, and are able to produce probabilistic output ("-solver 6").

Models can be specified at run-time by specifying the -model option of kytea, or by using the environmental variable KYTEA_MODEL.

These models are for use in analyzing Japanese. They are mainly trained using the Balanced Corpus of Contemporary Written Japanese, and UniDic, with a few other additional resources to increase the coverage. The models may be used for research or commercial purposes, but may not be re-distributed without prior permission.

All of the models can be used for word (morpheme) segmentation, part of speech estimation, and pronunciation estimation. According to the segmentation standard (in Japanese), morphemes are used as the basic unit of segmentation, and conjugations are separated from their stems. All POS tags are in Japanese, but this script can change them into English. For reference, I show the word, word+pronuncation, and word+POS tag F-measure on some held-out data from BCCWJ.

These models work for KyTea version 0.4.0 and higher. Older models can be found here.

The following models are built from the Lancaster Corpus of Mandarin Chinese, and the CC-CEDICT dictionary. The models are in UTF-8 format with simplified characters, and can be used for both word segmentation and pronunciation estimation. Note that the pronunciation estimation model essentially chooses the most common pronunciation for each word, and cannot properly estimate the pronunciation of words with pronunciation variation such as "了", and thus should only be used as a reference. These models may be used for both research and commercial purposes.

In addition, we provide the CEDICT dictionary, and the subword dictionary that were used to create these models. These are licensed under the Creative Commons Attribution-Share Alike 3.0 License.

Word Segmentation Only

The following models are based on the Penn Chinese Treebank, and can be used for segmentation in accordance with the treebank.

The following models are based on the MSR and AS data provided for the Second International Chinese Word Segmentation Bakeoff, and can be used for word segmentation. Accuracies and comparison with other systems entered in the word segmentation bakeoff are provided for reference (except for the full LR model, which we have not had a chance to test yet). These models may be used for non-commercial purposes.

Return to KyTea
Last Modified: 2012-01-27 by neubig