Power Law Graph Transformer for Machine Translation and Representation
  Learning
  We present the Power Law Graph Transformer, a transformer model with well
defined deductive and inductive tasks for prediction and representation
learning. The deductive task learns the dataset level (global) and instance
level (local) graph structures in terms of learnable power law distribution
parameters. The inductive task outputs the prediction probabilities using the
deductive task output, similar to a transductive model. We trained our model
with Turkish-English and Portuguese-English datasets from TED talk transcripts
for machine translation and compared the model performance and characteristics
to a transformer model with scaled dot product attention trained on the same
experimental setup. We report BLEU scores of $17.79$ and $28.33$ on the
Turkish-English and Portuguese-English translation tasks with our model,
respectively. We also show how a duality between a quantization set and
N-dimensional manifold representation can be leveraged to transform between
local and global deductive-inductive outputs using successive application of
linear and non-linear transformations end-to-end.
