title: Remove Duplicate Entries from JSON File - BeautifulSoup
body: <p>I am running a script to scape a website for textbook information and I have the script working.  However, when it writes to a JSON file it is giving me duplicate results.  I am trying to figure out how to remove the duplicates from the JSON file.  Here is my code:</p>
  
  <pre><code>from urllib.request import urlopen
  from bs4 import BeautifulSoup as soup
  import json
  
  urls = ['https://open.bccampus.ca/find-open-textbooks/', 
  'https://open.bccampus.ca/find-open-textbooks/?start=10']
  
  data = []
  #opening up connection and grabbing page
  for url in urls:
      uClient = urlopen(url)
      page_html = uClient.read()
      uClient.close()
  
      #html parsing
      page_soup = soup(page_html, "html.parser")
  
      #grabs info for each textbook
      containers = page_soup.findAll("h4")
  
      for container in containers:
         item = {}
         item['type'] = "Textbook"
         item['title'] = container.parent.a.text
         item['author'] = container.nextSibling.findNextSibling(text=True)
         item['link'] = "https://open.bccampus.ca/find-open-textbooks/" + container.parent.a["href"]
         item['source'] = "BC Campus"
         data.append(item) # add the item to the list
  
  with open("./json/bc.json", "w") as writeJSON:
      json.dump(data, writeJSON, ensure_ascii=False)
  </code></pre>
  
  <p>Here is a sample of the JSON output</p>
  
  <pre><code>{
  "type": "Textbook",
  "title": "Exploring Movie Construction and Production",
  "author": " John Reich, SUNY Genesee Community College",
  "link": "https://open.bccampus.ca/find-open-textbooks/?uuid=19892992-ae43-48c4-a832-59faa1d7108b&amp;contributor=&amp;keyword=&amp;subject=",
  "source": "BC Campus"
  }, {
  "type": "Textbook",
  "title": "Exploring Movie Construction and Production",
  "author": " John Reich, SUNY Genesee Community College",
  "link": "https://open.bccampus.ca/find-open-textbooks/?uuid=19892992-ae43-48c4-a832-59faa1d7108b&amp;contributor=&amp;keyword=&amp;subject=",
  "source": "BC Campus"
  }, {
  "type": "Textbook",
  "title": "Project Management",
  "author": " Adrienne Watt",
  "link": "https://open.bccampus.ca/find-open-textbooks/?uuid=8678fbae-6724-454c-a796-3c6667d826be&amp;contributor=&amp;keyword=&amp;subject=",
  "source": "BC Campus"
  }, {
  "type": "Textbook",
  "title": "Project Management",
  "author": " Adrienne Watt",
  "link": "https://open.bccampus.ca/find-open-textbooks/?uuid=8678fbae-6724-454c-a796-3c6667d826be&amp;contributor=&amp;keyword=&amp;subject=",
  "source": "BC Campus"
  }
  </code></pre>