title: Parse token from a file in c++
body: <p>//  How we can handle Acronyms(e.g  "U.S" "U.N") while parsing tokens from a file. I have used strtok to parse tokens and it separates U &amp; S from "U.S". Below code I am using to parse tokens from every line I read from file. I convert token into lower case and stores in a map. Thanks in advance.</p>
  
  <pre><code>char *pch = strtok (text," ,.-()/{}[]*&amp;#");
      while (pch != NULL)
      {
         for (char *iter = pch; *iter != '\0'; iter++)
         {
              *iter = tolower(*iter);
         }
           string str(pch);
           token_list.insert(pair&lt;T,K&gt;(str,fileName));//stores token with filename in which it is found
           word_count[str]++;//stores token with its frequency
           pch = strtok(NULL," ,.-()/{}[]*&amp;#");
      }
  </code></pre>