基于字位信息的中文分词方法研究*

分析中文自动分词的现状，介绍和描述几种不同的分词思想和方法，提出一种基于字位的分词方法。此分词方法以字为最小单位，根据字的概率分布得到组合成词的概率分布，因此在未登录词识别方面比其它方法有更优秀的表现。使用最大熵的机器学习方法来进行实现并通过两个实验得出实验结果的比较分析。

The Research of Character-Position-Based Chinese Word Segmentation

This paper analyses the actuality and introduces several different representative approaches of Chinese word segmentation,then brings out a character-position-based segmentation method which takes the Chinese character as the least unit.It indicates the probability distribution of a word through the probability distribution of Chinese character,so it plays much better than other approaches in unknown word recognition.This idea takes a machine-learning method called maximum entropy for implementation and two experiments for comparing and analyzing the results.