title: Solr custom tokenzier
body: <p>We have implemented a complex tokenizer for documents (using semantic analysis, etc). We want to insert our generated tokens for each document into solr. </p>
  
  <p>This is what we want. Define a solr custom tokenizer, such that it can accept our generated tokens, then pass them to filters. However, solr custom tokenizer is based on char and its offset, it's very complicated to modify it for our purpose.</p>
  
  <p>Is there any way to do this? I am using solr 5.0.</p>