Efficient and effective spam filtering and re-ranking for large web datasets |
| |
Authors: | Gordon V Cormack Mark D Smucker Charles L A Clarke |
| |
Institution: | (1) University of Waterloo, 200 University Avenue West, Waterloo, ON, N2L 3G1, Canada;; |
| |
Abstract: | The TREC 2009 web ad hoc and relevance feedback tasks used a new document collection, the ClueWeb09 dataset, which was crawled
from the general web in early 2009. This dataset contains 1 billion web pages, a substantial fraction of which are spam—pages
designed to deceive search engines so as to deliver an unwanted payload. We examine the effect of spam on the results of the
TREC 2009 web ad hoc and relevance feedback tasks, which used the ClueWeb09 dataset. We show that a simple content-based classifier
with minimal training is efficient enough to rank the “spamminess” of every page in the dataset using a standard personal
computer in 48 hours, and effective enough to yield significant and substantive improvements in the fixed-cutoff precision
(estP10) as well as rank measures (estR-Precision, StatMAP, MAP) of nearly all submitted runs. Moreover, using a set of “honeypot”
queries the labeling of training data may be reduced to an entirely automatic process. The results of classical information
retrieval methods are particularly enhanced by filtering—from among the worst to among the best. |
| |
Keywords: | |
本文献已被 SpringerLink 等数据库收录! |
|