title: "UnicodeEncodeError: 'charmap' codec can't encode character" When Writing to csv Using a Webscraper
body: <p>I've written a webscraper that scrapes NBA box score data off of <a href="https://www.basketball-reference.com" rel="nofollow noreferrer">basketball-reference</a>. The specific webpage that my error: </p>
  
  <p><code>UnicodeEncodeError: 'charmap' codec can't encode character '\u0107' in position 11: character maps to &lt;undefined&gt;</code> </p>
  
  <p>is occurring on is <a href="https://www.basketball-reference.com/boxscores/202003110MIA.html" rel="nofollow noreferrer">here</a>. Lastly, the specific player data that is tripping it up and throwing this specific UnicodeEncodeError is <a href="https://i.stack.imgur.com/hxqKl.png" rel="nofollow noreferrer">this one</a> (although I am sure the error is more generalized and will be produced with any character that contains an obscure accent mark).</p>
  
  <p>The minimal reproducible code:</p>
  
  <pre><code>def get_boxscore_basic_table(tag): #used to only get specific tables
      tag_id = tag.get("id")
      tag_class = tag.get("class")
      return (tag_id and tag_class) and ("basic" in tag_id and "section_wrapper" in tag_class and not "toggleable" in tag_class)
  
  import requests
  from bs4 import BeautifulSoup
  import lxml
  import csv
  import re
  
  website = 'https://www.basketball-reference.com/boxscores/202003110MIA.html'
  
  r = requests.get(website).text
  soup = BeautifulSoup(r, 'lxml')
  
  tables = soup.find_all(get_boxscore_basic_table)
  
  in_file = open('boxscore.csv', 'w', newline='')
  csv_writer = csv.writer(in_file)
  column_names = ['Player','Name','MP','FG','FGA','FG%','3P','3PA','3P%','FT','FTA','FT%','ORB','DRB','TRB','AST','STL','BLK','TOV','PF','PTS','+/-']
  csv_writer.writerow(column_names)
  
  for table in tables:    
      rows = table.select('tbody tr')
  
      for row in rows:
          building_player = [] #temporary container to hold player and stats
          player_name = row.th.text 
          if 'Reserves' not in player_name: 
              building_player.append(player_name)
  
          stats = row.select('td.right')
  
          for stat in stats:
              building_player.append(stat.text)
  
          csv_writer.writerow(building_player) #writing to csv
  
  in_file.close()
  </code></pre>
  
  <p>What is the best way around this? </p>
  
  <p>I've seen some stuff online about changing the encoding and specifically using the<code>.encode('utf-8')</code> method on the string before writing to the csv but it seems that this .encode() method, although it stops an error from being thrown, has several of its own problems. For instance; <code>player_name.encode('utf-8')</code> before writing to csv turns the name 'Willy Hernangómez' into 'b'Willy Hernang\xc3\xb3mez'' within by csv... not exactly a step in the right direction. </p>
  
  <p>Any help with this and an explanation as to what is happening would be much appreciated!</p>