统计语言模型中句子的语义连贯性判别

目前统计语言模型在语音识别、机器翻译和自动文摘等领域得到了广泛的应用.准确判别通过语言模型所得到的句子是否连贯、通顺,对于语言模型的评测和改进是个很重要的问题.本文采用基于词频统计的一组特征项,利用决策树算法自动对生成句的语义连贯性进行评测,在需要生成或识别连贯句的各自然语言处理领域具有广泛的实用价值.

Identifying Within-Sentence Semantic Coherence in Statistical Language Models

Nowdays statistical language model has been applied to many domain such as speech recognition, machine learning, and automatic summarization. It is extremely useful to distinguish coherent from non-coherent sentences , find aspects of language which are not adequately captured, and then incorporate them into the model to improve conventional statistical language models. We introduce a set of word-based statistical features which measure semantic coherence and can be used to enhance any language application where coherent sentences need to be generated or recognized. We train a decision tree using the constructed feature set to automatically classify sentences as coherent or not.