title: In PyTorch, what exactly does the grad_fn attribute store and how is it used?
body: <p>In PyTorch, the <code>Tensor</code> class has a <code>grad_fn</code> attribute. This <em>references</em> the operation used to obtain the tensor: for instance, if <code>a = b + 2</code>, <code>a.grad_fn</code> will be <code>AddBackward0</code>. But what does &quot;reference&quot; mean exactly?</p>
  <p>Inspecting <code>AddBackward0</code> using <code>inspect.getmro(type(a.grad_fn))</code> will state that the only base class of <code>AddBackward0</code> is <code>object</code>. Additionally, the source code for this class (and in fact, any other class which might be encountered in <code>grad_fn</code>) is nowhere to be found in the <a href="https://github.com/pytorch/pytorch" rel="noreferrer">source code</a>!</p>
  <p>All of this leads me to the following questions:</p>
  <ol>
  <li>What precisely is stored in <code>grad_fn</code> and how is it called during back-propagation?</li>
  <li>How come the objects that get stored in <code>grad_fn</code> do not have some sort of common super class, and why is there no source code for them on GitHub?</li>
  </ol>