基于文本加权词共现的跨语言文本相似度分析

跨语言文本相似度计算在跨语言信息检索、数据挖掘、抄袭检测等领域有着重要应用，但是跨语言文本相似度计算因为不同语言文法、结构等问题，在空间映射、特征选择上与单语言文本相似度计算有很大差异。为解决上述问题，采用一种基于文本加权词共现关系的跨语言文本相似度计算方法，通过平行语料库构建跨语言词共现关系模型，使用该模型进行跨语言文本映射，对不同语言的文本进行相似度计算。该模型实际反映了某种语言中某些关键词共同出现时映射成另一种语言时的关键词概率分布。实验表明，该方法对跨语言文本排序的计算更接近人工评判标准。

Cross-linguistic Text Similarity Analysis Based on Co-occurrence of Text Weighted Words

Cross-language text similarity computation has important applications in cross-language information retrieval， data mining， plagiarism detection and other fields. However， cross-linguistic text similarity calculation differs greatly from single-language text similarity calculation in spatial mapping and feature selection due to the different grammar and structure of the languages. In order to solve the above problem， a cross-linguistic text similarity calculation method based on the co-occurrence relationship of text weighted words is adopted. This method constructs a cross-linguistic word co-occurrence relationship model by parallel corpus， and uses this model to map cross-linguistic texts， and calculates the similarity of texts in different languages. The model actually reflects the probability distribution of keywords in one language when some keywords appear together and map to another language. Experimental results show that the calculation of the cross language text sorting method is closer to the artificial evaluation standard.