K-Core Decomposition on Super Large Graphs with Limited Resources
  K-core decomposition is a commonly used metric to analyze graph structure or
study the relative importance of nodes in complex graphs. Recent years have
seen rapid growth in the scale of the graph, especially in industrial settings.
For example, our industrial partner runs popular social applications with
billions of users and is able to gather a rich set of user data. As a result,
applying K-core decomposition on large graphs has attracted more and more
attention from academics and the industry. A simple but effective method to
deal with large graphs is to train them in the distributed settings, and some
distributed K-core decomposition algorithms are also proposed. Despite their
effectiveness, we experimentally and theoretically observe that these
algorithms consume too many resources and become unstable on super-large-scale
graphs, especially when the given resources are limited. In this paper, we deal
with those super-large-scale graphs and propose a divide-and-conquer strategy
on top of the distributed K-core decomposition algorithm. We evaluate our
approach on three large graphs. The experimental results show that the
consumption of resources can be significantly reduced, and the calculation on
large-scale graphs becomes more stable than the existing methods. For example,
the distributed K-core decomposition algorithm can scale to a large graph with
136 billion edges without losing correctness with our divide-and-conquer
technique.
