title: Spark Driver does not release memory when using spark.sql.autoBroadcastJoinThreshold
body: <p>I have came across abnormal behaviour,</p>
  
  <p>I have a query (inside loop) in which I have inner joins over 5 tables one with around 200MB and all other are under 10MB (All persisted at the start of loop, and unpersisted at the end of loop). </p>
  
  <p>Whenever I use spark.sql.autoBroadcastJoinThreshold (tried default, 5MB, 1MB and 100KB), after running same query multiple times it keeps on adding driver memory and eventually fails because of out of memory ( WARN TaskMemoryManager: Failed to allocate a page (16777216 bytes), try again.)</p>
  
  <p><strong>But,</strong> If I try same thing with <strong>spark.sql.autoBroadcastJoinThreshold=-1</strong>, <strong>it works</strong> without any issues. </p>
  
  <p>My Spark(2.0.0) config is :</p>
  
  <p>driver memory : 10g
  Executor memory : 20g
  cores : 3
  Nodes : 5</p>
  
  <p>( I guess I'm giving more resources than needed, but it doesn't work even if I reduce executor memory to 4g. 
  It processes same number of times irrespective of memory configuration.
  )</p>
  
  <p>PS: I am not creating any broadcast variables manually.<br>
  and I am new to Spark. </p>