Dropping diversity of products of large US firms: Models and measures
  It is widely assumed that in our lifetimes the products available in the
global economy have become more diverse. This assumption is difficult to
investigate directly, however, because it is difficult to collect the necessary
data about every product in an economy each year. We solve this problem by
mining publicly available textual descriptions of the products of every large
US firms each year from 1997 to 2017. Although many aspects of economic
productivity have been steadily rising during this period, our text-based
measurements show that the diversity of the products of at least large US firms
has steadily declined. This downward trend is visible using a variety of
product diversity metrics, including some that depend on a measurement of the
similarity of the products of every single pair of firms. The current state of
the art in comprehensive and detailed firm-similarity measurements is a Boolean
word vector model due to Hoberg and Phillips. We measure diversity using
firm-similarities from this Boolean model and two more sophisticated variants,
and we consistently observe a significant dropping trend in product diversity.
These results make it possible to frame and start to test specific hypotheses
for explaining the dropping product diversity trend.
