title: Import large data (json) into Firebase periodically
body: <p>We are in the situation that we will have to update large amounts of data (ca. 5 Mio Records) in firebase periodically. At the moment we have a few json files that are around ~1 GB in size.</p>
  
  <p>As existing third party solutions (<a href="https://github.com/firebase/firebase-import" rel="nofollow noreferrer">here</a> and <a href="https://github.com/firebase/firebase-streaming-import" rel="nofollow noreferrer">here</a>) have some reliability issues (import object per object; or need for open connection) and are quite disconnected to the google cloud platform ecosystem. I wonder if there is now an "official" way using i.e. the new <em>google cloud functions</em>? Or a combination with <em>app engine</em> / <em>google cloud storage</em> / <em>google cloud datastore</em>.</p>
  
  <p>I really like not to deal with authentication — something that cloud functions seems to handle well, but I assume the function would time out (?)</p>
  
  <h3>With the new firebase tooling available, how to:</h3>
  
  <ol>
  <li>Have long running cloud functions to do data fetching / inserts? (does it make sense?)</li>
  <li>Get the json files into &amp; from somewhere inside the google cloud platform?</li>
  <li>Does it make sense to first throw large data into google-cloud-datastore (i.e. too $$$ expensive to store in firebase) or can the firebase real-time database be reliably treaded as a large data storage.</li>
  </ol>