title: Combining data of multiple folders
body: <p>I have two folders (Q &amp; P) of files (&gt;10) with data. I want to automatically analyse all matching files within the folders. This means P1 &amp; Q1 , P5 &amp; Q5 etc. I have no errors, but I do not get the results I want. I think this is because my loop does not memorise the intermediate steps (only does operations for the last file in the folders), is this correct? How can I analyse all matching files after my initial loops for the folders? Help is much appriciated!</p>
  <pre><code>p = [] #empty list for results precipiation loop
  q = [] #empty list for results discharge loop
  
  #Preciptation folder read 
  for root, dirs, files in os.walk(r&quot;C:TSA_P&quot;):
      for filename in files:
          file_path = os.path.join(root, filename)
          with open(file_path) as file:
              file = pd.read_csv(file, delimiter=' ', header=None, skiprows=1, names=['year','month','day','precipitation_cpc'])
              file.columns = file.columns.str.replace(' ', ',')
              file.loc[:,'dt'] = pd.to_datetime(file[['year', 'month', 'day']])
              file.index = file['dt']
              file.columns = ['year', 'month', 'day','Rain','dt']
              p.append(file)
              
  #file.head()
  
  #Discharge folder read
  for root, dirs, files in os.walk(r&quot;C:\TSA_Q&quot;):
      for filename in files:
          file_path = os.path.join(root, filename)
          with open(file_path) as f:
              f = pd.read_csv(f, delimiter=' ', header=None, skiprows=1, names=['year','month','day','streamflow_m3s', 'qual_control_by_ana', 'qual_flag'])
              f.columns = f.columns.str.replace(' ', ',')
              f.loc[:,'dt'] = pd.to_datetime(f[['year', 'month', 'day']])
              f.index = f['dt']
              f.columns = ['year', 'month', 'day','Discharge','Quality','Flag','dt']
              f = f.loc[(f.index.year&gt;=1980)]
              q.append(file)
  #f.head()
  
  #Part C
  C = pd.DataFrame() #Data P &amp; Q per catchment
  Format = (24*3600) #mm/d to mm/s
  C['Psum'] = file['Rain'].resample('Y').sum()/Format #Daily to yearly
  C['Qsum'] = f['Discharge'].resample('Y').sum() #Daily to yearly
  #C.head()
  </code></pre>