Training Models for Kytea

KyTea Feature Files

Starting at KyTea 0.3.2, it is possible to train models using files containing the features used in KyTea training. The (rather large) file below is the file containing features used in training a standard KyTea model.

Available Data

Below is a list of language resources that are avaialable for training KyTea. If you know of any other resources, especially high-quality (99.9% accuracy or higher) ones or ones that have been annotated in KyTea's format, please tell us at kytea@.

Japanese

The main sources of data that go into KyTea's Japanese model are:

Using Dictionaries

KyTea's accuracy will greatly improve if you have a good dictionary or two that can be used along with the training corpus. Dictionaries can be specified by using the -dict option:

train-kytea -full train.txt -dict dict1.txt -dict dict2.txt -model train.mod

The format of the dictionary file is the same as that of the corpus, but with one word in one line.

コーパス/N/ko:pasu
の/P/no
文/N/buN
...

Note that it is possible to specify multiple dictionaries. If you have dictionaries that are from different sources (say one in-domain dictionary, and one out-of-domain dictionary), it is often better to separate them into different files and specify them one-by-one, as KyTea treats the words in different files differently.

Also, note that while all corpora (files specified by -full or -part) should be segmented according to a coherent standard, dictionaries may consist of entries that don't match the standard, such as noun phrases or compound words. Finally, do NOT make a dictionary simply using all the words in the training corpus. This will cause KyTea to over-fit the training corpus and over-segment all unknown words on your actual test data.

Local vs. Global Tagging Models

KyTea is able to train tagging models in two ways.

• Local models train a separate classifier for every word in the dictionary. For example, let's say we have three words A, B, and C, and tags X, Y, and Z. Our corpus looks like this:

  A/X A/Y B/Y B/Z C/Z
  

  The local model will train a classifier for A to decide between the two tags X and Y, train a classifier for B to decide between the two tags Y and Z, and always assign Z to C as it is the only tag that appears with C in the corpus. Local models are good for situations such as pronunciation estimation, where there is no clear relation between the tags of different words.
• Global models, on the other hand, train a single classifier to decide between all tags appearing in the corpus. In the previous example, this means that a single classifier decides between X, Y, and Z for all three words A, B, and C. Global models are better for situations where there is a clear relationship between tags assigned to different words such as tagging of parts of speech or named entities.

KyTea trains local tagging models by default. By specifying the -global n option, it is possible specify that global models should be used for the nth tag for each word.

train-kytea -full train.txt -global 1 -model train.mod

Note that the default model that comes with KyTea uses a local model for pronuncation estimation, and a global model for POS estimation.

Unknown Word Models

For the local models described in the previous section, words that don't appear in the corpus or dictionaries used in training become unknown words. KyTea contains functionality to predict pronunciations for unknown words by breaking words down into their component characters. In order to make this functionality active, you must add a subword dictionary in the form:

コー/ko:
パ/pa
文/buN
文/fumi

with one subword followed by a pronunciation on each line. Note that subwords can consist of multiple characters, and that a single character can have multiple pronunciations. This can be specified during training as follows.

train-kytea -full train.txt -subword subword.txt -model train.mod

Parameter Tuning

Training with Partially Annotated Data

It is possible to train KyTea with data that is not fully annotated with tags, or even word boundaries. If we have a file that is fully annotated with word boundaries, but some tags are unannotated, we create a file in the following form:

コーパス の/P 文//buN で す/TL 。//.

Note that the words "の" and "す" are annotated with only parts of speech, the words "文" and "。" are annotated with only pronunciations, and the words "コーパス" and "で" have no tag at all. For the pronunciations, notice that we leave the POS tag empty, which indicates no tag. These sorts of corpora must be specified with the -full tag during training.