title: Use boost::tokenizer with boost::iterator_range
body: <p>I'm using <code>boost::tokenizer</code> to read a CSV-like file. I'm storing the the tokens in a <code>std::vector</code>. It works well, but I want to store only a <code>boost::iterator</code> for each token. </p>
  
  <p>I tried: </p>
  
  <pre><code>#include &lt;string&gt;
  #include &lt;boost/tokenizer.hpp&gt;
  #include &lt;boost/range/iterator_range.hpp&gt;
  
  typedef std::string::const_iterator string_iter;
  typedef boost::iterator_range&lt;string_iter&gt; string_view;
  
  int main(){
      std::string line;
  
      std::vector&lt;string_view&gt; contents;
  
      boost::tokenizer&lt;boost::escaped_list_separator&lt;char&gt;, string_iter, string_view&gt; tok(line.begin(), line.end());
      contents.assing(tok.begin(), tok.end());
  }
  </code></pre>
  
  <p>But it fails to compile: </p>
  
  <blockquote>
    <p>/usr/include/boost/token_functions.hpp: In instantiation of ‘bool
    boost::escaped_list_separator::operator()(InputIterator&amp;, InputIterator, Token&amp;) [with
    InputIterator = __gnu_cxx::__normal_iterator >; Token =
    boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >; Char = char; Traits =
    std::char_traits]’: /usr/include/boost/token_iterator.hpp:70:11:
    required from ‘void boost::token_iterator::initialize() [with TokenizerFunc =
    boost::escaped_list_separator; Iterator =
    __gnu_cxx::__normal_iterator >; Type = boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >]’
    /usr/include/boost/token_iterator.hpp:77:63:   required from
    ‘boost::token_iterator::token_iterator(TokenizerFunc, Iterator, Iterator) [with
    TokenizerFunc = boost::escaped_list_separator; Iterator =
    __gnu_cxx::__normal_iterator >; Type = boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >]’ /usr/include/boost/tokenizer.hpp:86:53: 
    required from ‘boost::tokenizer::iter
    boost::tokenizer::begin() const [with
    TokenizerFunc = boost::escaped_list_separator; Iterator =
    __gnu_cxx::__normal_iterator >; Type = boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >; boost::tokenizer::iter =
    boost::token_iterator,
    __gnu_cxx::__normal_iterator >, boost::iterator_range&lt;__gnu_cxx::__normal_iterator > > >]’
    /home/wichtounet/dev/gooda-to-afdo-converter/src/gooda_reader.cpp:58:37:
    required from here /usr/include/boost/token_functions.hpp:187:16:
    error: no match for ‘operator+=’ in ‘tok += (&amp;
    next)->__gnu_cxx::__normal_iterator&lt;_Iterator,
    _Container>::operator* >()’ /usr/include/boost/token_functions.hpp:193:11: error: no match for
    ‘operator+=’ in ‘tok += (&amp;
    next)->__gnu_cxx::__normal_iterator&lt;_Iterator,
    _Container>::operator* >()’ /usr/include/boost/token_functions.hpp: In instantiation of ‘void
    boost::escaped_list_separator::do_escape(iterator&amp;,
    iterator, Token&amp;) [with iterator = __gnu_cxx::__normal_iterator >; Token =
    boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >; Char = char; Traits =
    std::char_traits]’:
    /usr/include/boost/token_functions.hpp:176:11:   required from ‘bool
    boost::escaped_list_separator::operator()(InputIterator&amp;, InputIterator, Token&amp;) [with
    InputIterator = __gnu_cxx::__normal_iterator >; Token =
    boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >; Char = char; Traits =
    std::char_traits]’ /usr/include/boost/token_iterator.hpp:70:11: 
    required from ‘void boost::token_iterator::initialize() [with TokenizerFunc =
    boost::escaped_list_separator; Iterator =
    __gnu_cxx::__normal_iterator >; Type = boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >]’
    /usr/include/boost/token_iterator.hpp:77:63:   required from
    ‘boost::token_iterator::token_iterator(TokenizerFunc, Iterator, Iterator) [with
    TokenizerFunc = boost::escaped_list_separator; Iterator =
    __gnu_cxx::__normal_iterator >; Type = boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >]’ /usr/include/boost/tokenizer.hpp:86:53: 
    required from ‘boost::tokenizer::iter
    boost::tokenizer::begin() const [with
    TokenizerFunc = boost::escaped_list_separator; Iterator =
    __gnu_cxx::__normal_iterator >; Type = boost::iterator_range&lt;__gnu_cxx::__normal_iterator > >; boost::tokenizer::iter =
    boost::token_iterator,
    __gnu_cxx::__normal_iterator >, boost::iterator_range&lt;__gnu_cxx::__normal_iterator > > >]’
    /home/wichtounet/dev/gooda-to-afdo-converter/src/gooda_reader.cpp:58:37:
    required from here /usr/include/boost/token_functions.hpp:130:9:
    error: no match for ‘operator+=’ in ‘tok += '\012'’
    /usr/include/boost/token_functions.hpp:134:9: error: no match for
    ‘operator+=’ in ‘tok += (&amp;
    next)->__gnu_cxx::__normal_iterator&lt;_Iterator,
    _Container>::operator* >()’ /usr/include/boost/token_functions.hpp:138:9: error: no match for
    ‘operator+=’ in ‘tok += (&amp;
    next)->__gnu_cxx::__normal_iterator&lt;_Iterator,
    _Container>::operator* >()’ /usr/include/boost/token_functions.hpp:142:9: error: no match for
    ‘operator+=’ in ‘tok += (&amp;
    next)->__gnu_cxx::__normal_iterator&lt;_Iterator,
    _Container>::operator* >()’</p>
  </blockquote>
  
  <p>I also simply tried to compute the two iterators by myself using the <code>boost::token_iterator</code>, but I haven't been successful so far. </p>
  
  <p>Is there a solution to get only the iterator range of each token instead of the string in order to save some performances ? </p>