title: Why are embeddings necessary in Pytorch?
body: <p>I (think) I understand the basic principle behind embeddings: they're a shortcut to quickly perform the operation <code>(one hot encoded vector) * matrix</code> without <em>actually</em> performing that operation (by utilizing the fact that this operation is equivalent to indexing into the matrix) while still maintaining the same gradient as if that operation was actually performed.</p>
  <p>I know in the following example:</p>
  <pre class="lang-py prettyprint-override"><code>e = Embedding(3, 2)
  n = e(torch.LongTensor([0, 2]))
  </code></pre>
  <p><code>n</code> will be a tensor of shape <code>2, 2</code>.</p>
  <p>But, we could also do:</p>
  <pre class="lang-py prettyprint-override"><code>p = nn.Parameter(torch.zeros([3, 2]).normal_(0, 0.01))
  p[tensor([0, 2])]
  </code></pre>
  <p>and get the same result without an embedding.</p>
  <p>This in and of itself wouldn't be confusing since in the first example <code>n</code> has a <code>grad_fn</code> called <code>EmbeddingBackward</code> whereas in the 2nd example <code>p</code> has a <code>grad_fn</code> called <code>IndexBackward</code>, which is what we expect since we know embeddings simulate a different derivative.</p>
  <p>The confusing part is in chapter 8 of the fastbook they use embeddings to compute movie recommendations. But then, they do it without embeddings in basically the same manner &amp; the model still works. I would expect the version without embeddings to fail because the derivative would be incorrect.</p>
  <p>Version with embeddings:</p>
  <pre class="lang-py prettyprint-override"><code>class DotProductBias(Module):
      def __init__(self, n_users, n_movies, n_factors, y_range=(0,5.5)):
          self.user_factors = Embedding(n_users, n_factors)
          self.user_bias = Embedding(n_users, 1)
          self.movie_factors = Embedding(n_movies, n_factors)
          self.movie_bias = Embedding(n_movies, 1)
          self.y_range = y_range
          
      def forward(self, x):
          users = self.user_factors(x[:,0])
          movies = self.movie_factors(x[:,1])
          res = (users * movies).sum(dim=1, keepdim=True)
          res += self.user_bias(x[:,0]) + self.movie_bias(x[:,1])
          return sigmoid_range(res, *self.y_range)
  </code></pre>
  <p>Version without:</p>
  <pre class="lang-py prettyprint-override"><code>def create_params(size):
      return nn.Parameter(torch.zeros(*size).normal_(0, 0.01))
  
  class DotProductBias(Module):
      def __init__(self, n_users, n_movies, n_factors, y_range=(0,5.5)):
          self.user_factors = create_params([n_users, n_factors])
          self.user_bias = create_params([n_users])
          self.movie_factors = create_params([n_movies, n_factors])
          self.movie_bias = create_params([n_movies])
          self.y_range = y_range
          
      def forward(self, x):
          users = self.user_factors[x[:,0]]
          movies = self.movie_factors[x[:,1]]
          res = (users*movies).sum(dim=1)
          res += self.user_bias[x[:,0]] + self.movie_bias[x[:,1]]
          return sigmoid_range(res, *self.y_range)
  </code></pre>
  <p>Does anyone know what is going on?</p>