title: Unicode files with mrjob
body: <p>I'm attempting to run a basic character count using mrjob. The file is a unicode UTF-8 text document that contains, among other symbols, Chinese characters. When I run the character count, I only get counts of symbols in the ASCII character set returned. </p>
  
  <p>As I understood it, mrjob worked with byte files and so should be able to process the unicode. Any idea how to make this work?</p>
  
  <p>Character count code:</p>
  
  <pre><code>import collections
  from mrjob.job import MRJob
  
  import pdb
  
  class MRCharCount(MRJob):
      def __init__(self, *args, **kwargs):
          super(MRCharCount, self).__init__(*args, **kwargs)
          self.charCount = collections.Counter()
  
      def mapper(self, _, value):
          #value = value.decode('utf-8')
          if 0:
              yield
          self.charCount += collections.Counter(value)
  
      def mapper_final(self):
          for (c, count) in self.charCount.iteritems():
              #c = c.decode('utf-8')
              #pdb.set_trace()
              yield c, count
  
      def reducer(self, key, values):
          #key = key.decode('utf-8')
          yield key, sum(values)
  
  if __name__ == '__main__':
      MRCharCount.run()
  </code></pre>
  
  <p>Sample file:</p>
  
  <pre><code>"WantedCropped/13.jpg"  " 
  
  蘑,-'此四郡咸瀉] 郡〝"同樹之吟[』赦〝連仙〝如五嶽〝Z蟲 總輝…埃 哪沛國樹也- 山嗚〕們義丘蜴〝紫陽鳳陽盎 枝也' 衡山泰山岱山也o委篤珊白俗因‵ 本踫順!〝 一抵以一 郡名而尊』冠〕，〝 夫[迥及砷洲咯l處嬤喵I奏 裔們彼此蕪汪一隃o4[姑聖叭}]五仰悔耐存蜘已吐琪〝 貴 天涯若北鄰 〝′ 聯盧吥以『千載重歡取硼」寫出子〝仃 『 之首) 苴〈垚思於此耳@ 仁 十] 本踫岫群屁〈屾}呱』力‵ 嚴〝'肋麻叭勘屾l支流喉巾清_I忖世鼻 { 次懸殊顯見舊本有歟. 征屾鮪以更) 例如'扣_『l屾 荳斗!閩之金釵溪踫喎、 慄沛國 郡文褶丕〞倣下十 屾
  
  
  
  
  
  
  
   】  【Iˋ 【】蔔lI_I(‵′l|忖|钀|ˋ| ˋ ˋ 〉_】IL 〝l
  
  
  
  〝 咽,世揀侍皿^加〞剴]長予曰夢金懷 眸哪′博慶) .詛屾'戌… 材丕派下十世孫允石刀』懦寫予、 予_.趴更 正o
  
  一】 蒼甫朱處派‵ 係友屾鞈忪派下廿四世孫公友 丕始遷) 其癆瞳鄢庸{父) 立以′公友一公 下五世蓀‵ 均求丕瀉肇基始淜) 紀以廿一 ll.lI屢」o 舊譜誤在公友雁者之上.啞闆荻〝'. 友一公是十』^世〝 稽穴』友蚕生於明嘉崝甲辰 即—扒』]兀一 五五如四年‵ 一世租 燃皿公居逕口』′ 是'公一兀「八七穴年‵ 其澗萋距有山〈百七十』^年
  
  陡聶繭卯缸臟曹[囊 止喋 .寬芡一兀一 光冗 一_一」'曄__
  
  '
  </code></pre>
  
  <p>Any ideas how Chinese character count could work? Thanks!</p>