Learning to Extract Keyphrases from Text
  Many academic journals ask their authors to provide a list of about five to
fifteen key words, to appear on the first page of each article. Since these key
words are often phrases of two or more words, we prefer to call them
keyphrases. There is a surprisingly wide variety of tasks for which keyphrases
are useful, as we discuss in this paper. Recent commercial software, such as
Microsoft's Word 97 and Verity's Search 97, includes algorithms that
automatically extract keyphrases from documents. In this paper, we approach the
problem of automatically extracting keyphrases from text as a supervised
learning task. We treat a document as a set of phrases, which the learning
algorithm must learn to classify as positive or negative examples of
keyphrases. Our first set of experiments applies the C4.5 decision tree
induction algorithm to this learning task. The second set of experiments
applies the GenEx algorithm to the task. We developed the GenEx algorithm
specifically for this task. The third set of experiments examines the
performance of GenEx on the task of metadata generation, relative to the
performance of Microsoft's Word 97. The fourth and final set of experiments
investigates the performance of GenEx on the task of highlighting, relative to
Verity's Search 97. The experimental results support the claim that a
specialized learning algorithm (GenEx) can generate better keyphrases than a
general-purpose learning algorithm (C4.5) and the non-learning algorithms that
are used in commercial software (Word 97 and Search 97).
