title: How to use os.walk() in Databricks to calculate directory size in Azure datalake
body: <p>How to use <code>os.walk</code> in Databricks to calculate directory size in Azure datalake.
  Python version i'm using is 3.</p>
  
  <p>I was Using Recursive method first to get directory size which failed when the file path is deeper inside directory with <code>OOM</code> error.</p>
  
  <p>Now I was curious if <code>os.Walk()</code> will work or not.</p>
  
  <p>Any snippet would help.</p>
  
  <p>Recursive function code is below: <strong>[This fails when its deeper path, So, I need different solution]</strong></p>
  
  <pre><code>from dbutils import FileInfo
  from typing import List
  
  root_path = "/mnt/ADLS/...../..../"
  
  def discover_size(path: str, verbose: bool = True):
    def loop_path(paths: List[FileInfo], accum_size: float):
      if not paths:
        return accum_size
      else:
        head, tail = paths[0], paths[1:]
        if head.size &gt; 0:
          if verbose:
              &lt;some code&gt;
        else:
              &lt;some code&gt;
    return loop_path(dbutils.fs.ls(path), 0.0)
  
  discover_size(root_path, verbose=True)
  </code></pre>