Fast Botnet Detection From Streaming Logs Using Online Lanczos Method
  Botnet, a group of coordinated bots, is becoming the main platform of
malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor
distribution, etc. This paper focuses on design and experiment of a new
approach for botnet detection from streaming web server logs, motivated by its
wide applicability, real-time protection capability, ease of use and better
security of sensitive data. Our algorithm is inspired by a Principal Component
Analysis (PCA) to capture correlation in data, and we are first to recognize
and adapt Lanczos method to improve the time complexity of PCA-based botnet
detection from cubic to sub-cubic, which enables us to more accurately and
sensitively detect botnets with sliding time windows rather than fixed time
windows. We contribute a generalized online correlation matrix update formula,
and a new termination condition for Lanczos iteration for our purpose based on
error bound and non-decreasing eigenvalues of symmetric matrices. On our
dataset of an ecommerce website logs, experiments show the time cost of Lanczos
method with different time windows are consistently only 20% to 25% of PCA.
