title: HadoopDataSource: Skipping Partition {} as no new files detected @ s3:
body: <p>So, I have an S3 folder with several subfolders acting as partitions (based on the date of creation). I have a Glue Table for those partitions and can see the data using Athena.</p>
  <p>Running a Glue Job and trying to access the Catalog I get the following error:</p>
  <pre><code>HadoopDataSource: Skipping Partition {} as no new files detected @ s3:...
  </code></pre>
  <p>The line that gives me problems is the following:</p>
  <pre><code>glueContext.getCatalogSource(database = &quot;DB_NAME&quot;, tableName = &quot;TABLE_NAME&quot;, redshiftTmpDir = &quot;&quot;, transformationContext = &quot;datasource0&quot;).getDynamicFrame().toDF()
  </code></pre>
  <p>I'll want at every point to access all the data in those S3 subfolders as it is updated regularly.</p>
  <p>I'm thinking the problem is the Glue Job Bookmark not detecting new files, but this is not running directly as part of a Job but as part of a library used by a Job.</p>
  <p>Removing &quot;transformationContext&quot; or changing its value to empty hasn't worked.</p>