基于专业术语提取的中文分词方法

针对在科技文献中,未登录词等相关专业术语其变化多端,在中文分词中难以识别,影响了专业领域文章的分词准确度,结合实际情况给出了一种基于专业术语提取的中文分词方法。通过大量特定领域的专业语料库,基于互信息和统计的方法,对文中的未登录词等专业术语进行提取,构造专业术语词典,并结合通用词词典,利用最大匹配方法进行中文分词。经实验证明,该分词方法可以较准确的抽取出相关专业术语,从而提高分词的精度,具有实际的应用价值。     

基于中文分词的专利挖掘分析方法研究

专利作为世界上最大的技术信息源,受到企业的日益重视。本文提出了一种基于中文分词的专利挖掘分析过程,首先进行专利信息的检索、提取和清洗,然后利用中文分词对专利名称进行关键词组的提取,细化专利名称、摘要等专利信息,最后在此基础上挖掘出专利的技术发展路线,不同技术之间的关联关系以及相似专利簇等。该过程方法在空调行业专利数据中得到了应用,有助于企业进行专利地图绘制、技术研发和专利战略实施。     

一种快速中文分词词典机制

通过研究目前中文分词领域各类分词机制,注意到中文快速分词机制的关键在于对单双字词的识别,在这一思想下,提出了一种快速中文分词机制:双字词-长词哈希机制,通过提高单双字词的查询效率来实现对中文分词机制的改进.实验证明,该机制提高了中文文本分词的效率.     

Local word vectors guiding keyphrase extraction

Automated keyphrase extraction is a fundamental textual information processing task concerned with the selection of representative phrases from a document that summarize its content. This work presents a novel unsupervised method for keyphrase extraction, whose main innovation is the use of local word embeddings (in particular GloVe vectors), i.e., embeddings trained from the single document under consideration. We argue that such local representation of words and keyphrases are able to accurately capture their semantics in the context of the document they are part of, and therefore can help in improving keyphrase extraction quality. Empirical results offer evidence that indeed local representations lead to better keyphrase extraction results compared to both embeddings trained on very large third corpora or larger corpora consisting of several documents of the same scientific field and to other state-of-the-art unsupervised keyphrase extraction methods.     

Word sense disambiguation based on context selection using knowledge-based word similarity

In this paper, we introduce a novel knowledge-based word-sense disambiguation (WSD) system. In particular, the main goal of our research is to find an effective way to filter out unnecessary information by using word similarity. For this, we adopt two methods in our WSD system. First, we propose a novel encoding method for word vector representation by considering the graphical semantic relationships from the lexical knowledge bases, and the word vector representation is utilized to determine the word similarity in our WSD system. Second, we present an effective method for extracting the contextual words from a text for analyzing an ambiguous word based on word similarity. The results demonstrate that the suggested methods significantly enhance the baseline WSD performance in all corpora. In particular, the performance on nouns is similar to those of the state-of-the-art knowledge-based WSD models, and the performance on verbs surpasses that of the existing knowledge-based WSD models.     

Word classification and hierarchy using co-occurrence word information

By the development of the computer in recent years, calculating a complex advanced processing at high speed has become possible. Moreover, a lot of linguistic knowledge is used in the natural language processing (NLP) system for improving the system. Therefore, the necessity of co-occurrence word information in the natural language processing system increases further and various researches using co-occurrence word information are done. Moreover, in the natural language processing, dictionary is necessary and indispensable because the ability of the entire system is controlled by the amount and the quality of the dictionary. In this paper, the importance of co-occurrence word information in the natural language processing system was described. The classification technique of the co-occurrence word (receiving word) and the co-occurrence frequency was described and the classified group was expressed hierarchically. Moreover, this paper proposes a technique for an automatic construction system and a complete thesaurus. Experimental test operation of this system and effectiveness of the proposal technique is verified.     

基于文本的留言本的设计

随着网络的发展和cn域名的普及,个人网站得到迅速发展。留言本是个人网站的一个重要的功能之一,本文利用JSP技术设计了基于文本的留言本,既可以满足个人网站的需要,又节省了数据库的费用。     

基于特征向量的中文文献相似度评价

科技文献抄袭现象伴随着科学技术的发展时有发生,这严重损害了文献原作者,也对科技文献的严肃性提出了挑战。本文利用分词技术提取文献特征向量,并结合动态规划算法对文献的相似度给出具体评价,针对不同抄袭的现象,发现其中存在的规律,具体问题具体分析。最后给出实际实验结果,为文献评审提供参考。     

基于特征词权值的渔业文本分类研究

渔业文本分类是充分利用渔业信息资源的有效途径。针对中文文献资料的结构特点,提出一种结合特征词权值和支持向量机(Support Vector Machine,SVM)的渔业文本分类方法,利用向量空间模型(Vector Space Model,VSM)构建文本向量空间,并结合特征词权值计算文本特征向量中的各特征项,将构建的文本向量送入SVM进行渔业文本分类。采用中国知网下载的标准文档进行了实验测试,并考察了准确率和召回率两个指标,实验结果表明,文章提出的渔业文本分类方法具有较好的分类效果。     

中文搜索引擎结构初探

随着Internet的进一步发展，网上中文信息的激增使中文搜索引擎日益受到人们的关注。本文对中文搜索引擎的结构做了初步分析，将其划分为四大模块：网页搜集模块、网页索引模块、查询模块和用户界面，并对各模块的工作原理、技术做了相应的说明。     

汉语信息抽取中事件的定位与分类

事件抽取是信息抽取的基本任务之一,而对文本中的事件准确定位和分类是保证事件抽取质量的前提.使用向量空间模型来表示事件描述片段的特征,并分类计算特征词的重要度,最后对文本中的事件片段进行定位和分类.试验结果表明该方法能够对文本中的事件片段进行较为准确的定位、分类,因此对于事件抽取任务的前期处理具有重要价值.     

基于改进的蚁群算法中文词语自动分类技术研究

研究了中文词自动分类问题。针对传统的蚁群算法中文词语分类精确度低等问题,提出了一种将蚁群算法应用到了中文词语自动分类中。方法建立在首先对大规模语料文本进行统计和计算的基础上,得到词的一元和二元信息,然后采用了蚁群算法对该信息进行词的分类。实验结果表明,提出的算法有效提高了词语分类的精确度。     

自然语言检索中的中文分词技术研究进展及应用

中文分词技术是实现自然语言检索的重要基础,是信息检索领域研究的关键课题,无论是专业信息检索系统还是搜索引擎都依赖于分词技术的研究成果。本文通过在国内外著名数据库中进行相关检索,分析了研究中文分词技术及其在著名搜索引擎中的应用。     

用偏最小二乘法提取石头口门水库水色信息

水体的高光谱数据在提供大量信息的同时，其波段间存在很高的相关性，常规的统计方法反演水质参数不但不能充分利用这些信息，并且也不能很好的去相关，而偏最小二乘回归分析可以较好的解决这一问题。因此本研究通过利用高光谱仪在石头口门水库进行反射光谱测量和同步水质采样分析，建立了叶绿素a和悬浮物含量的偏最小二乘回归模型。结果表明：该模型能较好的利用高光谱数据信息，各光谱波段自变量在最终模型中的系数大小在一定程度上较符合叶绿素a和悬浮物的光谱吸收、散射特性；通过与常规的比值模型、一阶微分模型进行对比，偏最小二乘回归模型明显优于前两者，其各决定系数均高于0.7，因此估测效果较理想，可用于内陆二类水体的水色信息提取。     

全文检索搜索引擎中文信息处理技术研究

本文深入分析了全文检索中文搜索引擎的关键技术,提出了一种适用于全文检索搜索引擎的中文分词方案,既提高了分词的准确性,又能识别文中的未登录词。针对向量空间信息检索模型,本文设计了一个综合考虑中文词在Web文本中的位置、长度以及频率等重要因素的词条权重计算函数,并且用量化的方法表示出其重要性,能够较准确地反映出词条在Web文档中的重要程度。最后对分词算法进行了测试,测试表明该方法能够提高分词准确度满足实用的要求。     

基于MODIS-EVI黄淮海平原冬小麦种植面积分带提取

黄淮海平原是中国最大的平原区,快速准确获取冬小麦的种植面积信息,对于黄淮海平原粮食估产和种植结构调整具有重要意义。本研究选用2009-2010年冬小麦生育期内MODIS-EVI数据,在利用HANTS算法重构基础上,根据冬小麦植被指数的季节节律性变化规律,构建模型提取黄淮海平原冬小麦面积信息。提取结果用统计数据进行验证,表明黄淮海平原各覆盖省份提取精度均值为62.9%,偏差为33.4,提取精度低,稳定性差。分析研究区域冬小麦生育期随纬度的变化规律,发现同一生育期随纬度增加明显线性推迟,据此对研究区进行了纬向水平分带。然后利用已建立模型分带提取冬小麦种植面积信息,提取结果中各省份精度均值为84.4%,偏差为4.9。可以看出,分带提取的各省份的精度均值明显高于未分带的提取结果,偏差也明显减小,提取方法更好,更稳定,可为以后大范围区域提取冬小麦面积信息提供方法借鉴。     

《中国分类主题词表》的自动扩充研究——从元数据中提取关键词并定位

词表是图书馆和信息检索领域最重要的知识组织工具,《中国分类主题词表》是传统词表的一种,它的更新和维护一直依靠手工进行,这制约了它在数字图书馆和网络信息环境下的应用。本文介绍了一项基于统计的、从元数据的标题中抽取关键词并定位在词表中的方法。大致包括三个步骤:从标题中提取关键词;确定抽取出的关键词的专指度;将专指度高的专业词汇定位在词表中。在《中国分类主题词表》和上海图书馆提供的计算机科技领域的元数据上所进行实验,结果证明该方法是可行的。这一方法可以应用到自动标引或编目中,有一定的实用性和广阔的应用前景。     

Examining the effect of whitening on static and contextualized word embeddings

Static word embeddings (SWE) and contextualized word embeddings (CWE) are the foundation of modern natural language processing. However, these embeddings suffer from spatial bias in the form of anisotropy, which has been demonstrated to reduce their performance. A method to alleviate the anisotropy is the “whitening” transformation. Whitening is a standard method in signal processing and other areas, however, its effect on SWE and CWE is not well understood. In this study, we conduct an experiment to elucidate the effect of whitening on SWE and CWE. The results indicate that whitening predominantly removes the word frequency bias in SWE, and biases other than the word frequency bias in CWE.     

基于统计的中文文本分类研究

本文对文本分类过程中关键的部分进行了改进,在分词阶段,对分词的速度和精度进行了改进,在特征选取阶段,把多种特征选取方法进行了融合,最后对分类器进行了优化,并给出了实验测试的结果,实验的结果表明,文本分类的效率的确有了提高.     

Automatic extraction of bilingual word pairs using inductive chain learning in various languages

In this paper, we propose a new learning method for extracting bilingual word pairs from parallel corpora in various languages. In cross-language information retrieval, the system must deal with various languages. Therefore, automatic extraction of bilingual word pairs from parallel corpora with various languages is important. However, previous works based on statistical methods are insufficient because of the sparse data problem. Our learning method automatically acquires rules, which are effective to solve the sparse data problem, only from parallel corpora without any prior preparation of a bilingual resource (e.g., a bilingual dictionary, a machine translation system). We call this learning method Inductive Chain Learning (ICL). Moreover, the system using ICL can extract bilingual word pairs even from bilingual sentence pairs for which the grammatical structures of the source language differ from the grammatical structures of the target language because the acquired rules have the information to cope with the different word orders of source language and target language in local parts of bilingual sentence pairs. Evaluation experiments demonstrated that the recalls of systems based on several statistical approaches were improved through the use of ICL.