Generalized Bilinear Deep Convolutional Neural Networks for Multimodal
  Biometric Identification
  In this paper, we propose to employ a bank of modality-dedicated
Convolutional Neural Networks (CNNs), fuse, train, and optimize them together
for person classification tasks. A modality-dedicated CNN is used for each
modality to extract modality-specific features. We demonstrate that, rather
than spatial fusion at the convolutional layers, the fusion can be performed on
the outputs of the fully-connected layers of the modality-specific CNNs without
any loss of performance and with significant reduction in the number of
parameters. We show that, using multiple CNNs with multimodal fusion at the
feature-level, we significantly outperform systems that use unimodal
representation. We study weighted feature, bilinear, and compact bilinear
feature-level fusion algorithms for multimodal biometric person identification.
Finally, We propose generalized compact bilinear fusion algorithm to deploy
both the weighted feature fusion and compact bilinear schemes. We provide the
results for the proposed algorithms on three challenging databases: CMU
Multi-PIE, BioCop, and BIOMDATA.
