title: Unicode support for non-English characters with Sqlite Full Text Search in Android
body: <p>Scroll to the end to skip the explanation.</p>
  
  <h2>Background</h2>
  
  <p>In my Android app, I want to use non-English Unicode text strings to search for matches in text documents/fields that are stored in a SQLite database. I've learned (so I thought) that what I need to do is implement a <a href="http://www.sqlite.org/fts3.html#section_1" rel="nofollow noreferrer">Full Text Search with fts3/fts4</a>, so that is what I have been working on learning for the past couple days. FTS is supported by Android, as is shown in the documentation <a href="http://developer.android.com/training/search/search.html" rel="nofollow noreferrer">Storing and Searching for Data</a> and in the blog post <a href="http://blog.andresteingress.com/2011/09/30/android-quick-tip-using-sqlite-fts-tables/" rel="nofollow noreferrer">Android Quick Tip: Using SQLite FTS Tables</a>.</p>
  
  <h2>Problem</h2>
  
  <p>Everything was looking good, but then I read the March 2012 blog post <a href="http://kmansoft.com/2012/03/17/the-sorry-state-of-sqlite-full-text-search-on-android/" rel="nofollow noreferrer">The sorry state of SQLite full text search on Android</a>, which said</p>
  
  <blockquote>
    <p>The first step when building a full text search index is to break down
    the textual content into words, aka tokens. Those tokens are then
    entered into a special index which lets SQLite perform very fast
    searches based on a token (or a set of tokens).</p>
    
    <p>SQLite has two built-in <a href="http://www.sqlite.org/fts3.html#tokenizer" rel="nofollow noreferrer">tokenizers</a>, and they both only consider tokens
    consisting of US ASCII characters. All other, non-US ASCII characters
    are considered whitespace.</p>
  </blockquote>
  
  <p>After that I also found <a href="https://stackoverflow.com/a/17399384/3681880">this StackOverflow answer</a> by <a href="https://stackoverflow.com/users/11654/cl">@CL.</a> (who, based on tags and reputation, appears to be an expert on SQLite) replying to a question about matching Vietnamese letters with different diacritics:</p>
  
  <blockquote>
    <p>You must create the FTS table with a tokenizer that can handle Unicode
    characters, i.e., ICU or UNICODE61.</p>
    
    <p>Please note that these tokenizers might not be available on all
    Android versions, and that the Android API does not expose any
    functions for adding user-defined tokenizers.</p>
  </blockquote>
  
  <p><a href="https://stackoverflow.com/a/8183890/3681880">This 2011 SO answer</a> seems to confirm that Android does not support tokenizers beyond the two basic <code>simple</code> and <code>porter</code> ones. </p>
  
  <p>This is 2015. Are there any updates to this situation? I need to have the full text search supported for everyone using my app, not just people with new phones (even if the newest Android version does support it now).</p>
  
  <h2>Potential partial solution?</h2>
  
  <p>I find it hard to believe that FTS does not work at all with Unicode. The <a href="http://www.sqlite.org/fts3.html#tokenizer" rel="nofollow noreferrer">documentation</a> for the <code>simple</code> tokenizer says </p>
  
  <blockquote>
    <p>A term is a contiguous sequence of eligible characters, where eligible
    characters are all alphanumeric characters <strong>and all characters with
    Unicode codepoint values greater than or equal to 128</strong>. All other
    characters are discarded when splitting a document into terms. Their
    only contribution is to separate adjacent terms. <em>(emphasis added)</em></p>
  </blockquote>
  
  <p>That gives me hope that some basic Unicode functionality could still be supported in Android, even if things like capitalization and diacritics (and various other equivalent letter forms that have different Unicode code points) are not supported.</p>
  
  <h2>My Main Question</h2>
  
  <p>Can I use SQLite FTS in Android with non-English Unicode text (codepoints > 128) if I am only using literal Unicode string tokens separated by spaces? (That is, I am searching for exact strings that occur in the text.)</p>
  
  <h2>Updates</h2>
  
  <ul>
  <li>The <a href="https://www.sqlite.org/fts3.html#unicode61" rel="nofollow noreferrer">unicode61 tokenizer</a> is available in SQLite version 3.7.13. This tokenizer supports "full unicode case folding" and "recognizes unicode space and punctuation characters." <a href="https://stackoverflow.com/questions/2421189/version-of-sqlite-used-in-android/4377116#4377116">Android Lollipop (API 20+) uses SQLite 3.8</a>. </li>
  </ul>