An in-depth study of similarity predicate committee

In the last decades, many similarity measures are proposed, such as Jaccard coefficient, cosine similarity, BM25, language model, etc. Despite the effectiveness of the existing similarity measures, we observe that none of them can consistently outperform the others in most typical situations. Choosing which similarity predicate to use is usually treated as an empirical question by evaluating a particular task with a number of different similarity predicates, which is not computationally efficient and the obtained results are not portable. In this paper, we propose a novel approach to combine different similarity predicates together to form a committee so that we do not need to worry about choosing which of them to use. Empirically, we can obtain a better result than any individual similarity predicate, which is quite meaningful in practice. Specifically, our method models the problem of committee generation as a 0–1 integer programming problem based on the confidence of similarity predicates and the reliability of attributes. We demonstrate the effectiveness of our model by applying it on three datasets with controlled errors. Experimental results demonstrate that our similarity predicate committee is more robust and superior over existing individual similarity predicates.     

Rickety numbers: Volatility of university rankings and policy implications

The most popular world university rankings are routinely taken at face value by media and social actors. While these rankings are politically influential, they are sensitive to both the conceptual framework (the set of indicators) and the modelling choices made in their construction (e.g., weighting or type of aggregation). A robustness analysis, based on a multi-modelling approach, aims to test the validity of the inference about the rankings produced in the Academic Ranking of World Universities (ARWU) of Shanghai Jiao Tong University and those produced by the UK's Times Higher Education Supplement (THES). Conclusions are drawn on the reliability of individual university ranks and on relative country or macro regional performance (e.g., Europe versus USA versus China) in terms of the number of top performing institutions. We find that while university and country level statistical inferences are unsound, the inference on macro regions is more robust. The paper also aims to propose an alternative ranking which is more dependant on the framework than on the methodological choices.     

“全美最佳大学”排行榜的排行理念及其启示

《美国新闻与世界报道》每年9月所推出的"全美最佳大学"排行榜尽管是对美国国内的高等学校进行排名,但却在全球范围内享有盛誉。贯穿于"全美最佳大学"排行榜中的排行理念是其精髓所在,其分类排行、重视促进人才培养的评估指标、评估方法的客观性、为消费者服务的排行理念值得中国学习。     

中国武术散打王争霸赛积分排名挑战赛制的研究

针对解决中国武术散打王争霸赛与锦标赛赛制中存在的矛盾,既保证散打王参赛运动员的主体,又兼顾了锦标赛优秀运动员的参赛;既保证明星运动员的出场率,保证赛事水平、确保收视率,又能通过新赛制不断地推出新人.以散打王、锦标赛,以及相关项目的赛制为研究对象,通过积分排名的相应计算办法,确定参赛的运动员范围.进一步根据散王打竞赛市场特性的整体构思,设计新的赛制结构和进度,为探索符合市场要求的散打赛制体系提供理论依据.     

Ranking institutions within a discipline: The steep mountain of academic excellence

We present a novel algorithm to rank smaller academic entities such as university departments or research groups within a research discipline. The Weighted Top Candidate (WTC) algorithm is a generalisation of an expert identification method. The axiomatic characterisation of WTC shows why it is especially suitable for scientometric purposes. The key axiom is stability – the selected institutions support each other's membership. The WTC algorithm, upon receiving an institution citation matrix, produces a list of institutions that can be deemed experts of the field. With a parameter we can adjust how exclusive our list should be. By completely relaxing the parameter, we obtain the largest stable set – academic entities that can qualify as experts under the mildest conditions. With a strict setup, we obtain a short list of the absolute elite. We demonstrate the algorithm on a citation database compiled from game theoretic literature published between 2008–2017. By plotting the size of the stable sets with respect to exclusiveness, we can obtain an overview of the competitiveness of the field. The diagram hints at how difficult it is for an institution to improve its position.     

论循环比赛的名次排列

本文以离散数学中的图论方法,就循环比赛中的名次排列进行了研究与论证,有一定的借鉴意义。     

Dealing with textual noise for robust and effective BERT re-ranking

The pre-trained language models (PLMs), such as BERT, have been successfully employed in two-phases ranking pipeline for information retrieval (IR). Meanwhile, recent studies have reported that BERT model is vulnerable to imperceptible textual perturbations on quite a few natural language processing (NLP) tasks. As for IR tasks, current established BERT re-ranker is mainly trained on large-scale and relatively clean dataset, such as MS MARCO, but actually noisy text is more common in real-world scenarios, such as web search. In addition, the impact of within-document textual noises (perturbations) on retrieval effectiveness remains to be investigated, especially on the ranking quality of BERT re-ranker, considering its contextualized nature. To mitigate this gap, we carry out exploratory experiments on the MS MARCO dataset in this work to examine whether BERT re-ranker can still perform well when ranking text with noise. Unfortunately, we observe non-negligible effectiveness degradation of BERT re-ranker over a total of ten different types of synthetic within-document textual noise. Furthermore, to address the effectiveness losses over textual noise, we propose a novel noise-tolerant model, De-Ranker, which is learned by minimizing the distance between the noisy text and its original clean version. Our evaluation on the MS MARCO and TREC 2019–2020 DL datasets demonstrates that De-Ranker can deal with synthetic textual noise more effectively, with 3%–4% performance improvement over vanilla BERT re-ranker. Meanwhile, extensive zero-shot transfer experiments on a total of 18 widely-used IR datasets show that De-Ranker can not only tackle natural noise in real-world text, but also achieve 1.32% improvement on average in terms of cross-domain generalization ability on the BEIR benchmark.