title: NodeJS Async Programming
body: <p>Totally new to programming with async functions. Also new to node.js which could be adding to my issue. I've read a lot and keep running into the similar problems and it seems like I've been randomly getting some portions of the async code to work, while others doesn't. Here is a simplified version of what I have:</p>
  <p>Essentially I'm searching a site for music, scrapping all the results (scraper_start.js) and then it is sent to scrape_individual.js to gather data. It is currently able to get all the data, but when it downloads the album art it comes in &quot;too late&quot;.</p>
  <p>The image does get logged to the console, but only after info gets returned. Also if you have any good resources to learn async programming please share them - I haven't been able to find a website that is nice and clean and gets into examples big enough that they become realistic (such as multiple async functions working at once and sometimes relying upon each other). Please critic my code as well - I am trying to learn!</p>
  <p>File scraper_start.js:</p>
  <pre><code>const rp = require('request-promise');
  const cheerio = require('cheerio');
  const scrape = require('./scrape_individual.js');
  const base_url = 'https://www.test.ca';
  const url = 'https://www.test.ca/search?mysearchstring';
  
  rp(url)
      .then(function(html)
      {
          const $ = cheerio.load(html);
          var results = []
          
          var hits = $('h3 &gt; a').length;
          console.log(&quot;TOTAL HITS: &quot; + hits);
  
          results = $('h3 &gt; a').map(function(i,v){ return $(v).attr('href'); }).get()
          
          return Promise.all(
              results.map(function(url) 
              {
                  return scrape(base_url + url);
              })
          );
      })
      .then(function(my_data) 
      {
          console.log(my_data);
  
      });
  </code></pre>
  <p>File scrape_individual.js:</p>
  <pre><code>const rp = require('request-promise');
  const cheerio = require('cheerio');
  var info = {}
  
  
  const scrape = function(url)
      {
          return rp(url)
          .then(function(html)
          {
              const $ = cheerio.load(html);
              if (!html.includes('contentType = &quot;Podcast&quot;'))
              {
                  info = {
                      title: $('h2.bc-heading:first').text(),
                      img3: null};
                  
                  img_data($('.bc-image-inset-border').attr('src'))
                      .then(function(v) 
                      { 
                          console.log(v);
                          info.img3 = v; // Log the value once it is resolved
                      })
                      .catch(function(v) {
                      
                      });
                  
                  return info;
              }
          })
      };
      
  function img_data(src) 
  {
      return new Promise(function(resolve, reject)
      {
          const { createCanvas, loadImage } = require('canvas');
          
          loadImage(src).then((image) =&gt; 
          {
              const canvas = createCanvas(image.width, image.height);
              const ctx = canvas.getContext('2d');
              ctx.drawImage(image, 0, 0);
  
              resolve(canvas.toDataURL());
          });
      });
  }
  
  module.exports = scrape;
  </code></pre>
  <p><strong>UPDATE: New Code with ASYNC / AWAIT</strong></p>
  <p>scraper_start.js:</p>
  <pre><code>const rp = require('request-promise');
  const cheerio = require('cheerio');
  const scrape = require('./scrape_individual.js');
  const base_url = 'https://www.test.ca';
  const url = 'https://www.test.ca/search?mysearchstring';
  var data = [];
  
  async function get_links(url)
  {
      let html = await rp(url);
      const $ = cheerio.load(html);
      var results = [];
      
      var hits = $('h3 &gt; a').length;
      console.log(&quot;TOTAL HITS: &quot; + hits);
      
      hrefs = $('h3 &gt; a').map(function(i,v){ return $(v).attr('href'); }).get()
      
      await Promise.all(hrefs.map(async (href) =&gt;
          {
              let data_single = await scrape.scrape_book3(base_url + href);
              data.push(data_single);
          }));
      
      //QUESTION AREA 1: This data works great with all info.
      console.log(data);
      return data
  }
  
  get_links(url);
  //QUESTION AREA 2: This data gets printed before getting the actual data returned.
  console.log(data);
  </code></pre>
  <p>scrape_individual.js:</p>
  <pre><code>const rp = require('request-promise');
  const cheerio = require('cheerio');
  var info = {}
  
  //scrape2(url)
  module.exports.scrape_individual = scrape2;
  
  async function scrape2(url)
  {
      let html = await rp(url);
      const $ = cheerio.load(html);
      
      if (!html.includes('contentType = &quot;Podcast&quot;'))
      {
          let my_image = await img_data($('.bc-image-inset-border').attr('src'));
          
          info = {title: $('h2.bc-heading:first').text(),
                  img3: my_image};
                  
          //console.log(info);
          return info;
      }
  }
  
  async function img_data(src) 
  {
      const { createCanvas, loadImage } = require('canvas');
      let image = await loadImage(src);
      const canvas = createCanvas(image.width, image.height);
      const ctx = canvas.getContext('2d');
      ctx.drawImage(image, 0, 0);
      //console.log(canvas.toDataURL());
      return canvas.toDataURL();
  
  }
  </code></pre>
  <p>This code works great now. Also easier to understand. Please feel free to critic as I am trying to master this. My question now is more of a general coding question.</p>
  <p>Within scraper_start.js where the end result ends up (data), I marked two comments with &quot;QUESTION AREA 1&quot; and &quot;QUESTION AREA 2&quot;</p>
  <p>QUESTION AREA 1: works entirely fine, which I would assume because it is in the async function
  QUESTION AREA 2: outside of async function, does not have the object returned yet as there is nothing to say await. Is there a way to make it wait?</p>
  <p>My question is pretty loaded. I can't use await since it's not in an async function from my understanding. Does this mean all my code needs to be in functions if I want to maintain an important order? What is best practice? Why not call every function as async?</p>
  <p>edit: Fixing typos
  edit2: Added ASYNC / AWAIT modifications</p>