title: I am attempting to move a bunch of images in a google spreadsheet over to Cloudinary using Node JS
body: <p>I have a csv file, with google drive images in them that are public. I want to upload them to cloudinary, this was working at first, but I think because I requested so many times google drive is blocking my automated requests from cloudinary, I thought I could bypass this by using a follow-redirects. The reason is because when I first land on a link like</p>
  <p><code>https://drive.google.com/uc?export=view&amp;id=1JZ1epWdE0TMehu_DxLPlxWS9Z7n-OlB8</code></p>
  <p>it redirects to</p>
  <p><code>https://doc-08-38-docs.googleusercontent.com/docs/securesc/randomids&amp;user=randomids&amp;hash=randomids</code></p>
  <p>I have an array of google drive links which follows this url pattern params being my id in my function</p>
  <pre><code>let imgSrc = `https://drive.google.com/uc?export=view&amp;id=${params[1]}`
  </code></pre>
  <p>I then use this to visit the redirected url to try and see the image so I can upload it to cloudinary</p>
  <pre><code>const { http, https } = require(&quot;follow-redirects&quot;);
  
         https
            .get(imgSrc, (response) =&gt; {
              response.on(&quot;data&quot;, (chunk) =&gt; {
                // console.log(chunk);
                var buf = chunk.toString();
                console.log(buf);
              });
              // console.log(response);
            })
            .on(&quot;error&quot;, (err) =&gt; {
              console.error(err);
            })
  </code></pre>
  <p>However the response I am getting from google is the following</p>
  <pre><code>&lt;html&gt;&lt;head&gt;&lt;meta http-equiv=&quot;content-type&quot; content=&quot;text/html; charset=utf-8&quot;/&gt;&lt;title&gt;Sorry...&lt;/title&gt;&lt;style&gt; body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }&lt;/style&gt;&lt;/head&gt;&lt;body&gt;&lt;div&gt;&lt;table&gt;&lt;tr&gt;&lt;td&gt;&lt;b&gt;&lt;font face=sans-serif size=10&gt;&lt;font color=#4285f4&gt;G&lt;/font&gt;&lt;font color=#ea4335&gt;o&lt;/font&gt;&lt;font color=#fbbc05&gt;o&lt;/font&gt;&lt;font color=#4285f4&gt;g&lt;/font&gt;&lt;font color=#34a853&gt;l&lt;/font&gt;&lt;font color=#ea4335&gt;e&lt;/font&gt;&lt;/font&gt;&lt;/b&gt;&lt;/td&gt;&lt;td style=&quot;text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%&quot;&gt;&lt;div style=&quot;border-bottom: 1px solid #dfdfdf;&quot;&gt;Sorry...&lt;/div&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/div&gt;&lt;div style=&quot;margin-left: 4em;&quot;&gt;&lt;h1&gt;We're sorry...&lt;/h1&gt;&lt;p&gt;... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.&lt;/p&gt;&lt;/div&gt;&lt;div style=&quot;margin-left: 4em;&quot;&gt;See &lt;a href=&quot;https://support.google.com/websearch/answer/86640&quot;&gt;Google Help&lt;/a&gt; for more information.&lt;br/&gt;&lt;br/&gt;&lt;/div&gt;&lt;div style=&quot;text-align: center; border-top: 1px solid #dfdfdf;&quot;&gt;&lt;a href=&quot;https://
  www.google.com&quot;&gt;Google Home&lt;/a&gt;&lt;/div&gt;&lt;/body&gt;&lt;/html&gt;
  </code></pre>
  <p>This is the gist of the message</p>
  <pre><code>your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
  </code></pre>
  <p>Any one have any better ideas of how to get my images into cloudinary. This does need to function in one script that also uploads new records with the returned image to my database.</p>
  <p>Here is how that functions, but is not where the issue lies</p>
  <pre><code>import formidable from &quot;formidable&quot;;
  const fs = require(&quot;fs&quot;);
  const path = require(&quot;path&quot;);
  const csv = require(&quot;csvtojson&quot;);
  const cloudinary = require(&quot;cloudinary&quot;).v2;
  import { gql } from &quot;@apollo/client&quot;;
  import { getAuthCookie, removeAuthCookie } from &quot;../../utils/auth-cookies&quot;;
  import { authClient } from &quot;../../utils/fauna-auth&quot;;
  import { query as q } from &quot;faunadb&quot;;
  const { http, https } = require(&quot;follow-redirects&quot;);
  
  const CREATE_DECOR_ENTRY = gql`
    mutation CreateDecorEntry(
      $ownerID: ID!
      $description: String!
      $pieces: Int!
      $purchaser: String!
      $alterations: Boolean!
      $cost: Int!
      $purchaseDate: Date!
      $category: String!
      $image: String!
      $itemNum: Int!
      $visible: Boolean
    ) {
      createDecor(
        data: {
          description: $description
          pieces: $pieces
          purchaser: $purchaser
          alterations: $alterations
          cost: $cost
          purchaseDate: $purchaseDate
          category: $category
          image: $image
          itemNum: $itemNum
          visible: $visible
          owner: { connect: $ownerID }
        }
      ) {
        description
      }
    }
  `;
  
  import { client } from &quot;../_app&quot;;
  
  cloudinary.config({
    cloud_name: process.env.CLOUDINARY_NAME,
    api_key: process.env.CLOUDINARY_API_KEY,
    api_secret: process.env.CLOUDINARY_API_SECRET,
  });
  
  const apolloClient = client;
  
  // first we need to disable the default body parser
  export const config = {
    api: {
      bodyParser: false,
    },
  };
  
  export default async function image(req, res) {
    try {
      const form = new formidable.IncomingForm();
      form.uploadDir = &quot;./&quot;;
      form.keepExtensions = true;
      const test = form.parse(req, async (err, fields, files) =&gt; {
        var oldPath = files.file.path;
  
        const jsonArray = await csv().fromFile(oldPath);
  
        const databases = jsonArray;
  
        databases.shift();
  
        let test = [];
  
        const finalEvery = databases.every((v, i) =&gt; {
          if (i &gt; 5) {
            return false;
          }
          test.push(v);
          return true;
        });
  
        const final = databases.map((item, i) =&gt; {
          if (i &gt; 999) {
            return;
          }
          return item;
        });
  
        //console.log(test);
  
        const token = getAuthCookie(req);
  
        console.log(token);
  
        const { ref, data: dataId } = await authClient(token).query(
          q.Get(q.CurrentIdentity())
        );
  
        console.log(&quot;token &gt;&gt;&gt;&gt;&gt;&gt; &quot;, token);
        console.log(&quot;ref &gt;&gt;&gt;&gt;&gt;&gt; &quot;, ref);
        console.log(&quot;dataId &gt;&gt;&gt;&gt;&gt;&gt; &quot;, dataId);
  
        const itemNum = await authClient(token).query(
          //q.CurrentIdentity()
          q.Get(
            q.Match(q.Index(&quot;item_number_reversed_owner&quot;), q.CurrentIdentity())
          )
        );
  
        console.log(itemNum);
  
        let itemNumIndex = itemNum ? itemNum.data.itemNum : 0;
        itemNumIndex++;
  
        test.forEach(async (v, i) =&gt; {
          var params = v.field2 &amp;&amp; v.field2.split(&quot;?&quot;)[1].split(&quot;id=&quot;);
  
          console.log(params[1]);
  
          let imgSrc = `https://drive.google.com/uc?export=view&amp;id=${params[1]}`;
  
          https
            .get(imgSrc, (response) =&gt; {
              response.on(&quot;data&quot;, (chunk) =&gt; {
                // console.log(chunk);
                var buf = chunk.toString();
                console.log(buf);
              });
              // console.log(response);
            })
            .on(&quot;error&quot;, (err) =&gt; {
              console.error(err);
            });
  
          await cloudinary.uploader
            .upload(
              imgSrc,
              { folder: &quot;catalog&quot; },
              async function (error, result) {
                // console.log(result, error);
  
                let alterations = v.field8;
                if (alterations === &quot;No&quot;) {
                  alterations = false;
                } else {
                  alterations = true;
                }
  
                let test = result;
  
                if (result?.url === undefined) {
                  test = {
                    url:
                      &quot;https://cdn.sanity.io/images/9pb8ktqh/production/02ab6fbb6340e0514b95bb330b229c98d0b3d11c-4560x3648.jpg&quot;,
                  };
                }
  
                const { data } = await apolloClient.mutate({
                  mutation: CREATE_DECOR_ENTRY,
                  variables: {
                    ownerID: ref.id,
                    image: test.url,
                    pieces: parseInt(v.field6 ? v.field6 : 0),
                    cost: parseInt(v.field9 ? v.field9 : 0),
                    purchaseDate: new Date().toISOString().split(&quot;T&quot;)[0],
                    itemNum: itemNumIndex,
                    alterations,
                    visible: true,
                    description: v.field5,
                    purchaser: v.field7,
                    category: v.field13,
                  },
                  context: {
                    headers: {
                      authorization: `Bearer ${token}`,
                    },
                  },
                });
  
                console.log(data);
              }
            )
            .catch((error) =&gt; {
              console.log(error);
            });
        });
  
        res.status(200).json({ data: test });
      });
    } catch (error) {
      console.error(error);
      res.status(error.requestResult.statusCode).send(error.message);
    }
  }
  </code></pre>
  <p>Just a clarification, when I upload the imgSrc to cloudinary with the google drive link, at first it worked, but now it gives me a 400 unauthorized response, which is why I think google is blocking cloudinary to view the image, hence why I tried to follow the redirect link and get the image.</p>