Learning Neighborhood Representation from Multi-Modal Multi-Graph:
  Image, Text, Mobility Graph and Beyond
  Recent urbanization has coincided with the enrichment of geotagged data, such
as street view and point-of-interest (POI). Region embedding enhanced by the
richer data modalities has enabled researchers and city administrators to
understand the built environment, socioeconomics, and the dynamics of cities
better. While some efforts have been made to simultaneously use multi-modal
inputs, existing methods can be improved by incorporating different measures of
'proximity' in the same embedding space - leveraging not only the data that
characterizes the regions (e.g., street view, local businesses pattern) but
also those that depict the relationship between regions (e.g., trips, road
network). To this end, we propose a novel approach to integrate multi-modal
geotagged inputs as either node or edge features of a multi-graph based on
their relations with the neighborhood region (e.g., tiles, census block, ZIP
code region, etc.). We then learn the neighborhood representation based on a
contrastive-sampling scheme from the multi-graph. Specifically, we use street
view images and POI features to characterize neighborhoods (nodes) and use
human mobility to characterize the relationship between neighborhoods (directed
edges). We show the effectiveness of the proposed methods with quantitative
downstream tasks as well as qualitative analysis of the embedding space: The
embedding we trained outperforms the ones using only unimodal data as regional
inputs.
