title: Can LDAvis analyse the results of vowpal_wabbit LDA?
body: <p><a href="https://github.com/cpsievert/LDAvis" rel="nofollow noreferrer">LDAvis</a> provides a excellent way of visualsing and exploring topic models. LDAvis requires 5 parameters:</p>
  <ol>
  <li><strong>phi</strong> (matrix with dimensions number of terms times number of topics)</li>
  <li><strong>theta</strong> (matrix with dimensions number of documents times number of topics)</li>
  <li>number of words per document (integer vector)</li>
  <li>the vocabulary (character vector)</li>
  <li>the word frequency in the whole corpus (integer vector)</li>
  </ol>
  <p>The short version of my question is: after fitting a LDA model with vowpal wabbit, how do one derive phi and theta?</p>
  <p>theta represents the mixture of topics per document, and must thus sum to 1 per document.
  phi represents the probability of a term given the topic, and must thus sum to 1 per topic.</p>
  <p>After running LDA with <a href="https://github.com/VowpalWabbit/vowpal_wabbit" rel="nofollow noreferrer">vowpal wabbit</a> (<code>vw</code>) some kind of weights are stored in a model. A human readable version of that model can be aquired by feeding a special file, with one document per term in the vocabulary while inactivating learning (by the <code>-t</code> parameter), e.g.</p>
  <pre><code>vw -t -i weights -d dictionary.vw --readable_model readable.model.txt
  </code></pre>
  <p>According to the <a href="https://github.com/VowpalWabbit/vowpal_wabbit/wiki/Latent-Dirichlet-Allocation" rel="nofollow noreferrer">documentation of vowpal wabbit</a>, all columns expect the first one of <code>readable.model.txt</code> now &quot;represent the per-word topic distributions.&quot;</p>
  <p>You can also generate predictions with <code>vw</code>, i.e. for a collection of documents</p>
  <pre><code>vw -t -i weights -d some-documents.txt -p predictions.txt
  </code></pre>
  <p>Both <code>predictions.txt</code> and <code>readable.model.txt</code> has a dimension that reflects the number of inputs (rows) and number of topics (columns), and none of them are <strong>probability distributions</strong>, because they do not sum to 1 (neither per row, nor per column).</p>
  <p>I understand that <code>vw</code> is not for the faint hearted and that some programming/scripting will be required on my part, but I'm sure there must be <strong>some</strong> way to derive theta and phi from some the output of <code>vw</code>. I've been stuck on this problem for days now, please give me some hints.</p>