Combining Visible and Infrared Spectrum Imagery using Machine Learning
  for Small Unmanned Aerial System Detection
  Advances in machine learning and deep neural networks for object detection,
coupled with lower cost and power requirements of cameras, led to promising
vision-based solutions for sUAS detection. However, solely relying on the
visible spectrum has previously led to reliability issues in low contrast
scenarios such as sUAS flying below the treeline and against bright sources of
light. Alternatively, due to the relatively high heat signatures emitted from
sUAS during flight, a long-wave infrared (LWIR) sensor is able to produce
images that clearly contrast the sUAS from its background. However, compared to
widely available visible spectrum sensors, LWIR sensors have lower resolution
and may produce more false positives when exposed to birds or other heat
sources. This research work proposes combining the advantages of the LWIR and
visible spectrum sensors using machine learning for vision-based detection of
sUAS. Utilizing the heightened background contrast from the LWIR sensor
combined and synchronized with the relatively increased resolution of the
visible spectrum sensor, a deep learning model was trained to detect the sUAS
through previously difficult environments. More specifically, the approach
demonstrated effective detection of multiple sUAS flying above and below the
treeline, in the presence of heat sources, and glare from the sun. Our approach
achieved a detection rate of 71.2 +- 8.3%, improving by 69% when compared to
LWIR and by 30.4% when visible spectrum alone, and achieved false alarm rate of
2.7 +- 2.6%, decreasing by 74.1% and by 47.1% when compared to LWIR and visible
spectrum alone, respectively, on average, for single and multiple drone
scenarios, controlled for the same confidence metric of the machine learning
object detector of at least 50%. Videos of the solution's performance can be
seen at https://sites.google.com/view/tamudrone-spie2020/.
