title: Tokenization of Arabic words using NLTK
body: <p>I'm using <strong>NLTK</strong> <code>word_tokenizer</code> to split a sentence into words.</p>
  
  <p>I want to tokenize this sentence:</p>
  
  <pre><code>في_بيتنا كل شي لما تحتاجه يضيع ...ادور على شاحن فجأة يختفي ..لدرجة اني اسوي نفسي ادور شيء 
  </code></pre>
  
  <p>The code I'm writing is: </p>
  
  <pre><code>import re
  import nltk
  
  lex = u" في_بيتنا كل شي لما تحتاجه يضيع ...ادور على شاحن فجأة يختفي ..لدرجة اني اسوي نفسي ادور شيء"
  
  wordsArray = nltk.word_tokenize(lex)
  print " ".join(wordsArray)
  </code></pre>
  
  <p>The problem is that the <code>word_tokenize</code> function doesn't split by words. Instead, it splits by letters so that the output is:</p>
  
  <pre><code>"ف ي _ ب ي ت ن ا ك ل ش ي ل م ا ت ح ت ا ج ه ي ض ي ع ... ا د و ر ع ل ى ش ا ح ن ف ج أ ة ي خ ت ف ي .. ل د ر ج ة ا ن ي ا س و ي ن ف س ي ا د و ر ش ي ء"
  </code></pre>
  
  <p>Any ideas ? </p>
  
  <p>What I've reached so far:</p>
  
  <p>By trying the text in <a href="http://text-processing.com/demo/tokenize/">here</a>, it appeared to be tokenized by letters. Also, however, other tokenizers tokenized it correctly. Does that mean that <code>word_tokenize</code> is for English only? Does that go for most of NLTK functions?</p>