Compatibility-aware Heterogeneous Visual Search
  We tackle the problem of visual search under resource constraints. Existing
systems use the same embedding model to compute representations (embeddings)
for the query and gallery images. Such systems inherently face a hard
accuracy-efficiency trade-off: the embedding model needs to be large enough to
ensure high accuracy, yet small enough to enable query-embedding computation on
resource-constrained platforms. This trade-off could be mitigated if gallery
embeddings are generated from a large model and query embeddings are extracted
using a compact model. The key to building such a system is to ensure
representation compatibility between the query and gallery models. In this
paper, we address two forms of compatibility: One enforced by modifying the
parameters of each model that computes the embeddings. The other by modifying
the architectures that compute the embeddings, leading to compatibility-aware
neural architecture search (CMP-NAS). We test CMP-NAS on challenging retrieval
tasks for fashion images (DeepFashion2), and face images (IJB-C). Compared to
ordinary (homogeneous) visual search using the largest embedding model
(paragon), CMP-NAS achieves 80-fold and 23-fold cost reduction while
maintaining accuracy within 0.3% and 1.6% of the paragon on DeepFashion2 and
IJB-C respectively.
