title: How to parse data in a variety of data formats/structures?
body: <p>I'm terribly unfamiliar with the data engineering space, but here goes:</p>
  <p>I have users that upload data in a variety of formats, that I want to convert to a single standard format. For example:</p>
  <p>Source Format #1</p>
  <pre><code>{
    &quot;firstName&quot;: &quot;Bob&quot;,
    &quot;lastName&quot;: &quot;Smith&quot;,
    &quot;birthdate&quot;: &quot;1990-01-01&quot;,
    &quot;favoriteFood&quot;: &quot;pizza&quot;,
    &quot;favoriteDrink&quot;: &quot;beer&quot;
  }
  </code></pre>
  <p>Source Format #2</p>
  <pre><code>{
    &quot;name&quot;: {
      &quot;first&quot;: &quot;Bob&quot;,
      &quot;last&quot;: &quot;Smith&quot;
    }
    &quot;birthdate&quot;: {
      &quot;year&quot;: &quot;1990&quot;,
      &quot;month&quot;: &quot;01&quot;,
      &quot;day&quot;: &quot;01&quot;
    }
    &quot;favorites&quot;: {
      &quot;food&quot;: &quot;pizza&quot;,
      &quot;drink&quot;: &quot;beer&quot;
    }
  }
  </code></pre>
  <p>Target format:</p>
  <pre><code>{
    &quot;name&quot;: {
      &quot;first&quot;: &quot;Bob&quot;,
      &quot;last&quot;: &quot;Smith&quot;
    }
    &quot;birthdate&quot;: &quot;1990-01-01&quot;,
    &quot;favorites&quot;: {
      &quot;food&quot;: &quot;pizza&quot;,
      &quot;drink&quot;: &quot;beer&quot;
    }
  }
  
  </code></pre>
  <p>And so on. There could be dozens or hundreds of different source formats that I want to convert from into some target format.</p>
  <p>My first thought was to build a type of &quot;format catalog&quot; that I test each format against, and apply the transformation/map to the target format if there's a match. This sounds to me like a common enough problem that it's probably been addressed in some way, but I'm uncertain of what to even google for.</p>
  <p>The only thing I've run into so far that <em>kind of</em> sounds right is <a href="https://aws.amazon.com/glue/features/" rel="nofollow noreferrer">AWS Glue</a> (it talks about &quot;automatic schema discovery&quot;), but even that seems more directed at yanking data out of a variety of databases and ETLing them into a single database rather than reading raw data (from either data sent to an API endpoint or from a flat file on disk).</p>
  <p>Features I'm interested in, in order of desirability:</p>
  <ol>
  <li>If source data matches a known (or automatically discoverable) format, convert it to a target format.</li>
  <li>Define required and optional fields. Example: if <code>favorite</code>s aren't present, still extract the other &quot;known&quot; fields</li>
  <li>Be able to recognize different types of formats (such as JSON vs XML).</li>
  </ol>
  <p>Edit: in case it matters, most of my existing code is written in Python, but this is a big enough use case that I could build a separate service in another language to handle.</p>