title: R: problems applying LIME to quanteda text model
body: <p>it's a modified version of my <a href="https://stackoverflow.com/q/50273919/6327771">previous question</a>: I'm trying to run LIME on my <code>quanteda</code> text model that feeds off <a href="https://www.kaggle.com/benhamner/clinton-trump-tweets/data#" rel="nofollow noreferrer">Trump &amp; Clinton tweets data</a>. I run it following an example given by Thomas Pedersen in his <a href="https://cran.r-project.org/web/packages/lime/vignettes/Understanding_lime.html" rel="nofollow noreferrer">Understanding LIME</a> and useuful SO answer provided by <a href="https://stackoverflow.com/a/50275665/6327771">@Weihuang Wong</a>:</p>
  
  <pre><code>library(dplyr)
  library(stringr)
  library(quanteda)
  library(lime)
  
  #data prep
  tweet_csv &lt;- read_csv("tweets.csv")
  
  # creating corpus and dfm for train and test sets
  
  get_matrix &lt;- function(df){
    corpus &lt;- quanteda::corpus(df)
    dfm &lt;- quanteda::dfm(corpus, remove_url = TRUE, remove_punct = TRUE,     remove = stopwords("english"))
  }
  
  set.seed(32984)
  trainIndex &lt;- sample.int(n = nrow(tweet_csv), size =     floor(.8*nrow(tweet_csv)), replace = F)
  
  train_dfm &lt;- get_matrix(tweet_csv$text[trainIndex])
  train_raw &lt;- tweet_csv[, c("text", "tweet_num")][as.vector(trainIndex), ]
  train_labels &lt;- tweet_csv$author[as.vector(trainIndex)] == "realDonaldTrump"
  
  test_dfm &lt;- get_matrix(tweet_csv$text[-trainIndex])
  test_raw &lt;- tweet_csv[, c("text", "tweet_num")][-as.vector(trainIndex), ]
  test_labels &lt;- tweet_csv$author[-as.vector(trainIndex)] == "realDonaldTrump"
  
  #### make sure that train &amp; test sets have exactly same features
  test_dfm &lt;- dfm_select(test_dfm, train_dfm)
  
  ### Naive Bayes model using quanteda::textmodel_nb ####
  nb_model &lt;- quanteda::textmodel_nb(train_dfm, train_labels)
  nb_preds &lt;- predict(nb_model, test_dfm) #&gt; 0.5
  
  
  # select only correct predictions
  predictions_tbl &lt;- data.frame(predict_label = nb_preds$nb.predicted,
                            actual_label = test_labels,
                            tweet_name = rownames(nb_preds$posterior.prob)
  ) %&gt;%
    mutate(tweet_num = 
         as.integer(
           str_trim(
             str_replace_all(tweet_name, "text", ""))
       )) 
  
  
  correct_pred &lt;- predictions_tbl %&gt;%
    filter(actual_label == predict_label) 
  
  # pick a sample of tweets for explainer 
  tweets_to_explain &lt;- test_raw %&gt;%
    filter(tweet_num %in% correct_pred$tweet_num) %&gt;% 
    head(4)
  
  
  
  ### set up correct model class and predict functions 
  class(nb_model)
  
  model_type.textmodel_nb_fitted &lt;- function(x, ...) {
    return("classification")
  }
  
  
  # have to modify the textmodel_nb_fitted so that 
  
  predict_model.textmodel_nb_fitted &lt;- function(x, newdata, type, ...) {
    X &lt;- corpus(newdata)
    X &lt;- dfm_select(dfm(X), x$data$x)   
    res &lt;- predict(x, newdata = X, ...)
    switch(
      type,
      raw = data.frame(Response = res$nb.predicted, stringsAsFactors = FALSE),
      prob = as.data.frame(res$posterior.prob, check.names = FALSE)
    )  
  }
  
  
  ### run the explainer - no problems here 
  explainer &lt;- lime(tweets_to_explain$text, # lime returns error on different features in explainer and explanations, even if I use the same dataset in both. Raised an issue on Github and asked a question on SO
                model = nb_model,
                preprocess = get_matrix) 
  </code></pre>
  
  <p>But when I run the explainer...</p>
  
  <pre><code>corr_explanation &lt;- lime::explain(tweets_to_explain$text, 
                                explainer, 
                                n_labels = 1,
                                n_features = 6,
                                cols = 2,
                                verbose = 0)
  </code></pre>
  
  <p>... I get the following error:</p>
  
  <blockquote>
    <p>Error in UseMethod("corpus") : 
      no applicable method for 'corpus' applied to an object of class "c('dfm', 'dgCMatrix', 'CsparseMatrix', 'dsparseMatrix', 'generalMatrix', 'dCsparseMatrix', 'dMatrix', 'sparseMatrix', 'compMatrix', 'Matrix', 'xMatrix', 'mMatrix', 'Mnumeric', 'replValueSp')"</p>
  </blockquote>
  
  <p>It goes back to applying <code>corpus()</code> to <code>newdata</code>:</p>
  
  <pre><code>5.corpus(newdata) 
  4.predict_model.textmodel_nb_fitted(x = explainer$model, newdata = permutations_tokenized, 
  type = o_type) 
  3.predict_model(x = explainer$model, newdata = permutations_tokenized, 
  type = o_type) 
  2.explain.character(tweets_to_explain$text, explainer, n_labels = 1, 
  n_features = 6, cols = 2, verbose = 0) 
  1.lime::explain(tweets_to_explain$text, explainer, n_labels = 1, 
  n_features = 6, cols = 2, verbose = 0) 
  </code></pre>
  
  <p>But I don't understand why should this cause any issues as new data is a text vector?</p>
  
  <p>Thanks for any hints</p>