title: How to tokenize a Malayalam word?
body: <pre><code>ഇതുഒരുസ്ടലംമാണ്  
  </code></pre>
  
  <p><em>itu oru stalam anu</em></p>
  
  <p>This is a Unicode string meaning <em>this is a place</em> </p>
  
  <pre><code>import nltk
  nltk.wordpunct_tokenize('ഇതുഒരുസ്ഥാലമാണ് '.decode('utf8'))
  </code></pre>
  
  <p>is not working for me .</p>
  
  <pre><code>nltk.word_tokenize('ഇതുഒരുസ്ഥാലമാണ് '.decode('utf8'))
  </code></pre>
  
  <p>is also not working
  other examples</p>
  
  <pre><code>"കണ്ടില്ല "  = കണ്ടു +ഇല്ല,
  "വലിയൊരു"  = വലിയ + ഒരു
  </code></pre>
  
  <p>Right Split :</p>
  
  <pre><code>ഇത്  ഒരു സ്ഥാലം ആണ് 
  </code></pre>
  
  <p>output:</p>
  
  <pre><code>[u'\u0d07\u0d24\u0d4d\u0d12\u0d30\u0d41\u0d38\u0d4d\u0d25\u0d32\u0d02\u0d06\u0d23\u0d4d']
  </code></pre>
  
  <p>I just need to split the  words as shown in the other example. Other example section is for testing.The problem is not with Unicode. It is with morphology of language. for this purpose you need to use a morphological analyzer<br>
  Have a look at this paper.
  <a href="http://link.springer.com/chapter/10.1007%2F978-3-642-27872-3_38">http://link.springer.com/chapter/10.1007%2F978-3-642-27872-3_38</a></p>