Peek Inside the Closed World: Evaluating Autoencoder-Based Detection of
  DDoS to Cloud
  Machine-learning-based anomaly detection (ML-based AD) has been successful at
detecting DDoS events in the lab. However published evaluations of ML-based AD
have used only limited data and provided minimal insight into why it works. To
address limited evaluation against real-world data, we apply autoencoder, an
existing ML-AD model, to 57 DDoS attack events captured at 5 cloud IPs from a
major cloud provider. We show that our models detect nearly all malicious flows
for 2 of the 4 cloud IPs under attack (at least 99.99%) and detect most
malicious flows (94.75% and 91.37%) for the remaining 2 IPs. Our models also
maintain near-zero false positives on benign flows to all 5 IPs. Our primary
contribution is to improve our understanding for why ML-based AD works on some
malicious flows but not others. We interpret our detection results with feature
attribution and counterfactual explanation. We show that our models are better
at detecting malicious flows with anomalies on allow-listed features (those
with only a few benign values) than flows with anomalies on deny-listed
features (those with mostly benign values) because our models are more likely
to learn correct normality for allow-listed features. We then show that our
models are better at detecting malicious flows with anomalies on unordered
features (that have no ordering among their values) than flows with anomalies
on ordered features because even with incomplete normality, our models could
still detect anomalies on unordered feature with high recall. Lastly, we
summarize the implications of what we learn on applying autoencoder-based AD in
production: training with noisy real-world data is possible, autoencoder can
reliably detect real-world anomalies on well-represented unordered features and
combinations of autoencoder-based AD and heuristic-based filters can help both.
