Plan and generate: Explicit and implicit variational augmentation for multi-document summarization of scientific articles

Multi-Document Summarization of Scientific articles (MDSS) is a challenging task that aims to generate concise and informative summaries for multiple scientific articles on a particular topic. However, despite recent advances in abstractive models for MDSS, grammatical correctness and contextual coherence remain challenging issues. In this paper, we introduce EDITSum, a novel abstractive MDSS model that leverages sentence-level planning to guide summary generation. Our model incorporates neural topic model information as explicit guidance and sequential latent variables information as implicit guidance under a variational framework. We propose a hierarchical decoding strategy that generates the sentence-level planning by a sentence decoder and then generates the final summary conditioned on the planning by a word decoder. Experimental results show that our model outperforms previous state-of-the-art models by a significant margin on ROUGE-1 and ROUGE-L metrics. Ablation studies demonstrate the effectiveness of the individual modules proposed in our model, and human evaluations provide strong evidence that our model generates more coherent and error-free summaries. Our work highlights the importance of high-level planning in addressing intra-sentence errors and inter-sentence incoherence issues in MDSS.     

ATM: Adversarial-neural Topic Model

Topic models are widely used for thematic structure discovery in text. But traditional topic models often require dedicated inference procedures for specific tasks at hand. Also, they are not designed to generate word-level semantic representations. To address the limitations, we propose a neural topic modeling approach based on the Generative Adversarial Nets (GANs), called Adversarial-neural Topic Model (ATM) in this paper. To our best knowledge, this work is the first attempt to use adversarial training for topic modeling. The proposed ATM models topics with dirichlet prior and employs a generator network to capture the semantic patterns among latent topics. Meanwhile, the generator could also produce word-level semantic representations. Besides, to illustrate the feasibility of porting ATM to tasks other than topic modeling, we apply ATM for open domain event extraction. To validate the effectiveness of the proposed ATM, two topic modeling benchmark corpora and an event dataset are employed in the experiments. Our experimental results on benchmark corpora show that ATM generates more coherence topics (considering five topic coherence measures), outperforming a number of competitive baselines. Moreover, the experiments on event dataset also validate that the proposed approach is able to extract meaningful events from news articles.     

Comparing PSO-based clustering over contextual vector embeddings to modern topic modeling

Efficient topic modeling is needed to support applications that aim at identifying main themes from a collection of documents. In the present paper, a reduced vector embedding representation and particle swarm optimization (PSO) are combined to develop a topic modeling strategy that is able to identify representative themes from a large collection of documents. Documents are encoded using a reduced, contextual vector embedding from a general-purpose pre-trained language model (sBERT). A modified PSO algorithm (pPSO) that tracks particle fitness on a dimension-by-dimension basis is then applied to these embeddings to create clusters of related documents. The proposed methodology is demonstrated on two datasets. The first dataset consists of posts from the online health forum r/Cancer and the second dataset is a standard benchmark for topic modeling which consists of a collection of messages posted to 20 different news groups. When compared to the state-of-the-art generative document models (i.e., ETM and NVDM), pPSO is able to produce interpretable clusters. The results indicate that pPSO is able to capture both common topics as well as emergent topics. Moreover, the topic coherence of pPSO is comparable to that of ETM and its topic diversity is comparable to NVDM. The assignment parity of pPSO on a document completion task exceeded 90% for the 20NewsGroups dataset. This rate drops to approximately 30% when pPSO is applied to the same Skip-Gram embedding derived from a limited, corpus-specific vocabulary which is used by ETM and NVDM.     

Improving topic disentanglement via contrastive learning

With the emergence and development of deep generative models, such as the variational auto-encoders (VAEs), the research on topic modeling successfully extends to a new area: neural topic modeling, which aims to learn disentangled topics to understand the data better. However, the original VAE framework had been shown to be limited in disentanglement performance, bringing their inherent defects to a neural topic model (NTM). In this paper, we put forward that the optimization objectives of contrastive learning are consistent with two important goals (alignment and uniformity) of well-disentangled topic learning. Also, the optimization objectives of contrastive learning are consistent with two key evaluation measures for topic models, topic coherence and topic diversity. So, we come to the important conclusion that alignment and uniformity of disentangled topic learning can be quantified with topic coherence and topic diversity. Accordingly, we are inspired to propose the Contrastive Disentangled Neural Topic Model (CNTM). By representing both words and topics as low-dimensional vectors in the same embedding space, we apply contrastive learning to neural topic modeling to produce factorized and disentangled topics in an interpretable manner. We compare our proposed CNTM with strong baseline models on widely-used metrics. Our model achieves the best topic coherence scores under the most general evaluation setting (100% proportion topic selected) with 25.0%, 10.9%, 24.6%, and 51.3% improvements above the second-best models’ scores reported on four datasets of 20 Newsgroups, Web Snippets, Tag My News, and Reuters, respectively. Our method also gets the second-best topic diversity scores on the dataset of 20Newsgroups and Web Snippets. Our experimental results show that CNTM can effectively leverage the disentanglement ability from contrastive learning to solve the inherent defect of neural topic modeling and obtain better topic quality.     

Exploring coherent topics by topic modeling with term weighting

Topic models often produce unexplainable topics that are filled with noisy words. The reason is that words in topic modeling have equal weights. High frequency words dominate the top topic word lists, but most of them are meaningless words, e.g., domain-specific stopwords. To address this issue, in this paper we aim to investigate how to weight words, and then develop a straightforward but effective term weighting scheme, namely entropy weighting (EW). The proposed EW scheme is based on conditional entropy measured by word co-occurrences. Compared with existing term weighting schemes, the highlight of EW is that it can automatically reward informative words. For more robust word weight, we further suggest a combination form of EW (CEW) with two existing weighting schemes. Basically, our CEW assigns meaningless words lower weights and informative words higher weights, leading to more coherent topics during topic modeling inference. We apply CEW to Dirichlet multinomial mixture and latent Dirichlet allocation, and evaluate it by topic quality, document clustering and classification tasks on 8 real world data sets. Experimental results show that weighting words can effectively improve the topic modeling performance over both short texts and normal long texts. More importantly, the proposed CEW significantly outperforms the existing term weighting schemes, since it further considers which words are informative.     

TDAM: A topic-dependent attention model for sentiment analysis

We propose a topic-dependent attention model for sentiment classification and topic extraction. Our model assumes that a global topic embedding is shared across documents and employs an attention mechanism to derive local topic embedding for words and sentences. These are subsequently incorporated in a modified Gated Recurrent Unit (GRU) for sentiment classification and extraction of topics bearing different sentiment polarities. Those topics emerge from the words’ local topic embeddings learned by the internal attention of the GRU cells in the context of a multi-task learning framework. In this paper, we present the hierarchical architecture, the new GRU unit and the experiments conducted on users’ reviews which demonstrate classification performance on a par with the state-of-the-art methodologies for sentiment classification and topic coherence outperforming the current approaches for supervised topic extraction. In addition, our model is able to extract coherent aspect-sentiment clusters despite using no aspect-level annotations for training.     

A novel emerging topic detection method: A knowledge ecology perspective

Emerging topic detection has attracted considerable attention in recent times. While various detection approaches have been proposed in this field, designing a method for accurately detecting emerging topics remains challenging. This paper introduces the perspective of knowledge ecology to the detection of emerging topics and utilizes author-keywords to represent research topics. More precisely, we first improve the novelty metric and recalculate emergence capabilities based on the “ecostate” and “ecorole” attributes of ecological niches. Then, we take the perspective that keywords are analogous to living bodies and map them to the knowledge ecosystem to construct an emerging topics detection method based on ecological niches (ETDEN). Finally, we conduct in-depth comparative experiments to verify the effectiveness and feasibility of ETDEN using data extracted from scientific literature in the ACM Digital Library database. The results demonstrate that the improved novelty indicator helps to differentiate the novelty values of keywords in the same interval. More importantly, ETDEN performs significantly better performance on three terms: the emergence time point and the growth rate of pre-and post-emergence.     

Intelligent topic selection for low-cost information retrieval evaluation: A New perspective on deep vs. shallow judging

While test collections provide the cornerstone for Cranfield-based evaluation of information retrieval (IR) systems, it has become practically infeasible to rely on traditional pooling techniques to construct test collections at the scale of today’s massive document collections (e.g., ClueWeb12’s 700M+ Webpages). This has motivated a flurry of studies proposing more cost-effective yet reliable IR evaluation methods. In this paper, we propose a new intelligent topic selection method which reduces the number of search topics (and thereby costly human relevance judgments) needed for reliable IR evaluation. To rigorously assess our method, we integrate previously disparate lines of research on intelligent topic selection and deep vs. shallow judging (i.e., whether it is more cost-effective to collect many relevance judgments for a few topics or a few judgments for many topics). While prior work on intelligent topic selection has never been evaluated against shallow judging baselines, prior work on deep vs. shallow judging has largely argued for shallowed judging, but assuming random topic selection. We argue that for evaluating any topic selection method, ultimately one must ask whether it is actually useful to select topics, or should one simply perform shallow judging over many topics? In seeking a rigorous answer to this over-arching question, we conduct a comprehensive investigation over a set of relevant factors never previously studied together: 1) method of topic selection; 2) the effect of topic familiarity on human judging speed; and 3) how different topic generation processes (requiring varying human effort) impact (i) budget utilization and (ii) the resultant quality of judgments. Experiments on NIST TREC Robust 2003 and Robust 2004 test collections show that not only can we reliably evaluate IR systems with fewer topics, but also that: 1) when topics are intelligently selected, deep judging is often more cost-effective than shallow judging in evaluation reliability; and 2) topic familiarity and topic generation costs greatly impact the evaluation cost vs. reliability trade-off. Our findings challenge conventional wisdom in showing that deep judging is often preferable to shallow judging when topics are selected intelligently.     

Adapting information retrieval to query contexts

In current IR approaches documents are retrieved only according to the terms specified in the query. The same answers are returned for the same query whatever the user and the search goal are. In reality, many other contextual factors strongly influence document’s relevance and they should be taken into account in IR operations. This paper proposes a method, based on language modeling, to integrate several contextual factors so that document ranking will be adapted to the specific query contexts. We will consider three contextual factors in this paper: the topic domain of the query, the characteristics of the document collection, as well as context words within the query. Each contextual factor is used to generate a new query language model to specify some aspect of the information need. All these query models are then combined together to produce a more complete model for the underlying information need. Our experiments on TREC collections show that each contextual factor can positively influence the IR effectiveness and the combined model results in the highest effectiveness. This study shows that it is both beneficial and feasible to integrate more contextual factors in the current IR practice.     

Fuzzy topic modeling approach for text mining over short text

In this era, the proliferating role of social media in our lives has popularized the posting of the short text. The short texts contain limited context with unique characteristics which makes them difficult to handle. Every day billions of short texts are produced in the form of tags, keywords, tweets, phone messages, messenger conversations social network posts, etc. The analysis of these short texts is imperative in the field of text mining and content analysis. The extraction of precise topics from large-scale short text documents is a critical and challenging task. The conventional approaches fail to obtain word co-occurrence patterns in topics due to the sparsity problem in short texts, such as text over the web, social media like Twitter, and news headlines. Therefore, in this paper, the sparsity problem is ameliorated by presenting a novel fuzzy topic modeling (FTM) approach for short text through fuzzy perspective. In this research, the local and global term frequencies are computed through a bag-of-words (BOW) model. To remove the negative impact of high dimensionality on the global term weighting, the principal component analysis is adopted; thereafter the fuzzy c-means algorithm is employed to retrieve the semantically relevant topics from the documents. The experiments are conducted over the three real-world short text datasets: the snippets dataset is in the category of small dataset whereas the other two datasets, Twitter and questions, are the bigger datasets. Experimental results show that the proposed approach discovered the topics more precisely and performed better as compared to other state-of-the-art baseline topic models such as GLTM, CSTM, LTM, LDA, Mix-gram, BTM, SATM, and DREx+LDA. The performance of FTM is also demonstrated in classification, clustering, topic coherence and execution time. FTM classification accuracy is 0.95, 0.94, 0.91, 0.89 and 0.87 on snippets dataset with 50, 75, 100, 125 and 200 number of topics. The classification accuracy of FTM on questions dataset is 0.73, 0.74, 0.70, 0.68 and 0.78 with 50, 75, 100, 125 and 200 number of topics. The classification accuracies of FTM on snippets and questions datasets are higher than state-of-the-art baseline topic models.     

Application of automatic topic identification on Excite Web search engine data logs

The analysis of contextual information in search engine query logs enhances the understanding of Web users’ search patterns. Obtaining contextual information on Web search engine logs is a difficult task, since users submit few number of queries, and search multiple topics. Identification of topic changes within a search session is an important branch of search engine user behavior analysis. The purpose of this study is to investigate the properties of a specific topic identification methodology in detail, and to test its validity. The topic identification algorithm’s performance becomes doubtful in various cases. These cases are explored and the reasons underlying the inconsistent performance of automatic topic identification are investigated with statistical analysis and experimental design techniques.     

HVLM: Exploring Human-Like Visual Cognition and Language-Memory Network for Visual Dialog

Visual dialog, a visual-language task, enables an AI agent to engage in conversation with humans grounded in a given image. To generate appropriate answers for a series of questions in the dialog, the agent is required to understand the comprehensive visual content of an image and the fine-grained textual context of the dialog. However, previous studies typically utilized the object-level visual feature to represent a whole image, which only focuses on the local perspective of an image but ignores the importance of the global information in an image. In this paper, we proposed a novel model Human-Like Visual Cognitive and Language-Memory Network for Visual Dialog (HVLM), to simulate global and local dual-perspective cognitions in the human visual system and understand an image comprehensively. HVLM consists of two key modules, Local-to-Global Graph Convolutional Visual Cognition (LG-GCVC) and Question-guided Language Topic Memory (T-Mem). Specifically, in the LG-GCVC module, we design a question-guided dual-perspective reasoning to jointly learn visual contents from both local and global perspectives through a simple spectral graph convolution network. Furthermore, in the T-Mem module, we design an iterative learning strategy to gradually enhance fine-grained textual context details via an attention mechanism. Experimental results demonstrate the superiority of our proposed model, which obtains the comparable performance on benchmark datasets VisDial v1.0 and VisDial v0.9.     

创新生态系统:一个情境视角的系统性评述

新兴技术不确定性、组织模式网络化、市场的服务导向和价值共创等趋势下，传统创新生态系统所嵌入的技术情境、组织情境和市场情境发生转变。现有创新生态系统文献缺乏对研究情境全面的、系统的关注。研究以系统性评述为基础，整合应用文献计量分析与内容分析，识别出创新生态系统研究的焦点主题、理论发展、应用方法，以及由文献簇涌现的研究情境（包含技术情境、组织情境、市场情境）。研究从情境视角为创新生态系统理论与实践的发展提供启示。     

基于LDA模型的高校师德舆情演化及路径传导研究

【目的/意义】构建高校师德舆情微博用户评论LDA模型,可以更精准识别舆情演化特征和分析关键主题传 播路径,帮助高校和相关部门更为有效地进行舆情监管和舆情引导。【方法/过程】本文以“天津大学一教授学术造 假”事件为例,基于 LDA模型构建高校师德舆情下微博用户主题生成模型,采用困惑度评价指标确定 LDA模型最 优主题数,采用信息熵确定每一主题在不同日期的主题强度,通过关键词共现知识图谱、词云展现舆情话题的演 变,最后基于主题相似度确定主题传播路径。【结果/结论】LDA模型和信息熵可以解析出网络用户群体关注的重要 主题热点,精准识别舆情演化特征,识别主题最优传播路径进行舆论引导,对爆发的舆情实现预测和管制优化。【创 新/局限】文章创新性地构建高校学术道德舆情的LDA主题模型,有效确定微博用户群体主题、识别舆情演化特征、 分析主题间传播路径,具有普适性;进一步扩大高校师德其他舆情分析及结合网络舆情情感分析为下一步的研究 内容。     

Intelligent chatbot interaction system capable for sentimental analysis using hybrid machine learning algorithms

Chatbots actively help the human user through digital conversation through NLP based on artificial intelligence (AI). It can be pre-trained to understand the user's queries and produce an immediate response in NLP. The user input of the chatbot is any format like voice, text, sentiments, etc. Many research works have been implemented. The issues of existing works are that during the digital conversation, it does not accurately identify the user's requirement, it may go irrelevant to the user's query, and also, primarily, it is voice-based and faces hitches in the analysis of the user's intention, unable to track the context in long-conversation. Therefore, for understanding the context, sentimental calculations are essential. This paper proposed to make the immediate response of users in the chatbot by using Bi-directional Recurrent Neural Network with a Fuzzy Naïve Bayes classifier (BRNN-FNB). This paper aims to build chatbot models with AI-based sentimental analysis, which helps humans to perform accurate interactions. The working concept of chatbots is based on two forms of artificial intelligence domains: machine learning and natural language processing. It may be used in many applications like digital marketing, education, and online forums. The accuracy rate of the proposed work BRNN-FNB got 93% using the Seq-to-Seq technique. And also, the accuracy rate of the proposed work BRNN-FNB without using Seq-to-Seq got 92%.     

网络健康社区中健康信息传播网络及主题特征研究

【目的/意义】本文利用用户在健康信息问答过程中产生的真实数据,从网络结构和信息主题两个维度对网 络健康社区中的健康信息传播网络及主题特征进行研究,进而为社区的建设和维护提供建议。【方法/过程】首先, 利用社会网络分析法对不同时间阶段的健康信息传播网络进行指标测度和可视化呈现,探究其网络结构特征;然 后结合LDA和已有词表,对社区内健康信息进行主题识别和提取,分析健康信息主题的分布及其变化趋势。【结果/ 结论】本研究发现实验选取的网络健康社区内部的健康信息传播网络具有小世界效应,用户流动性大,关键节点变 更快;用户健康信息问答的内容集中在若干主题,且部分主题随时间呈现一定变化趋势。【创新/局限】网络健康信 息的生成和传播源于用户之间的信息交互。本文从网络结构和信息主题两个维度开展研究,更符合其内部机理, 研究更加深入和具象;网络健康社区种类众多,本文只选择其中一个社区进行研究,可能存在一定的局限性。     

HCA: Hierarchical Compare Aggregate model for question retrieval in community question answering

We address the problem of finding similar historical questions that are semantically equivalent or relevant to an input query question in community question-answering (CQA) sites. One of the main challenges for this task is that questions are usually too long and often contain peripheral information in addition to the main goals of the question. To address this problem, we propose an end-to-end Hierarchical Compare Aggregate (HCA) model that can handle this problem without using any task-specific features. We first split questions into sentences and compare every sentence pair of the two questions using a proposed Word-Level-Compare-Aggregate model called WLCA-model and then the comparison results are aggregated with a proposed Sentence-Level-Compare-Aggregate model to make the final decision. To handle the insufficient training data problem, we propose a sequential transfer learning approach to pre-train the WLCA-model on a large paraphrase detection dataset. Our experiments on two editions of the Semeval benchmark datasets and the domain-specific AskUbuntu dataset show that our model outperforms the state-of-the-art models.     

A Contrastive learning-based Task Adaptation model for few-shot intent recognition

Few-shot intent recognition aims to identify user’s intent from the utterance with limited training data. A considerable number of existing methods mainly rely on the generic knowledge acquired on the base classes to identify the novel classes. Such methods typically ignore the characteristics of each meta task itself, resulting in the inability to make full use of limited given samples when classifying unseen classes. To deal with such issues, we propose a Contrastive learning-based Task Adaptation model (CTA) for few-shot intent recognition. In detail, we leverage contrastive learning to help achieve task adaptation and make full use of the limited samples of novel classes. First, a self-attention layer is employed in the task adaptation module, which aims to establish interactions between samples of different categories so that new representations are task-specific rather than relying entirely on the base classes. Then, the contrastive-based loss functions and the semantics of the label name are respectively used for reducing the similarity between sample representations in different categories while increasing it in the same categories. Experimental results on a public dataset OOS verify the effectiveness of our proposal by beating the competitive baselines in terms of accuracy. Besides, we conduct the cross-domain experiments on three datasets, i.e., OOS, SNIPS as well as ATIS. We find that CTA gains obvious improvements in terms of accuracy in all cross-domain experiments, indicating that it has a better generalization ability than other competitive baselines in both cross-domain and single-domain settings.