End-to-End Entity Classification on Multimodal Knowledge Graphs
  End-to-end multimodal learning on knowledge graphs has been left largely
unaddressed. Instead, most end-to-end models such as message passing networks
learn solely from the relational information encoded in graphs' structure: raw
values, or literals, are either omitted completely or are stripped from their
values and treated as regular nodes. In either case we lose potentially
relevant information which could have otherwise been exploited by our learning
methods. To avoid this, we must treat literals and non-literals as separate
cases. We must also address each modality separately and accordingly: numbers,
texts, images, geometries, et cetera. We propose a multimodal message passing
network which not only learns end-to-end from the structure of graphs, but also
from their possibly divers set of multimodal node features. Our model uses
dedicated (neural) encoders to naturally learn embeddings for node features
belonging to five different types of modalities, including images and
geometries, which are projected into a joint representation space together with
their relational information. We demonstrate our model on a node classification
task, and evaluate the effect that each modality has on the overall
performance. Our result supports our hypothesis that including information from
multiple modalities can help our models obtain a better overall performance.
