Learning deep representation of multityped objects and tasks
  We introduce a deep multitask architecture to integrate multityped
representations of multimodal objects. This multitype exposition is less
abstract than the multimodal characterization, but more machine-friendly, and
thus is more precise to model. For example, an image can be described by
multiple visual views, which can be in the forms of bag-of-words (counts) or
color/texture histograms (real-valued). At the same time, the image may have
several social tags, which are best described using a sparse binary vector. Our
deep model takes as input multiple type-specific features, narrows the
cross-modality semantic gaps, learns cross-type correlation, and produces a
high-level homogeneous representation. At the same time, the model supports
heterogeneously typed tasks. We demonstrate the capacity of the model on two
applications: social image retrieval and multiple concept prediction. The deep
architecture produces more compact representation, naturally integrates
multiviews and multimodalities, exploits better side information, and most
importantly, performs competitively against baselines.
