Mining the Web for Lexical Knowledge to Improve Keyphrase Extraction:
  Learning from Labeled and Unlabeled Data
  Keyphrases are useful for a variety of purposes, including summarizing,
indexing, labeling, categorizing, clustering, highlighting, browsing, and
searching. The task of automatic keyphrase extraction is to select keyphrases
from within the text of a given document. Automatic keyphrase extraction makes
it feasible to generate keyphrases for the huge number of documents that do not
have manually assigned keyphrases. Good performance on this task has been
obtained by approaching it as a supervised learning problem. An input document
is treated as a set of candidate phrases that must be classified as either
keyphrases or non-keyphrases. To classify a candidate phrase as a keyphrase,
the most important features (attributes) appear to be the frequency and
location of the candidate phrase in the document. Recent work has demonstrated
that it is also useful to know the frequency of the candidate phrase as a
manually assigned keyphrase for other documents in the same domain as the given
document (e.g., the domain of computer science). Unfortunately, this
keyphrase-frequency feature is domain-specific (the learning process must be
repeated for each new domain) and training-intensive (good performance requires
a relatively large number of training documents in the given domain, with
manually assigned keyphrases). The aim of the work described here is to remove
these limitations. In this paper, I introduce new features that are derived by
mining lexical knowledge from a very large collection of unlabeled data,
consisting of approximately 350 million Web pages without manually assigned
keyphrases. I present experiments that show that the new features result in
improved keyphrase extraction, although they are neither domain-specific nor
training-intensive.
