title: Machine Learning: Predict second dataset on behalf of first dataset trained classifier
body: <p>i am new to "Machine Learning" and tried to implement <a href="https://stackoverflow.com/questions/53740141/run-trained-machine-learning-model-on-a-different-dataset?rq=1">this question</a> but it is unclear to me. its been 2 months i am Seducing, so please help me to resolve my error.</p>
  
  <p>Actually, i am trying to:</p>
  
  <ol>
  <li><em>"Train svm classifer"</em> on <strong>TRAIN_features</strong> and <strong>TRAIN_labels</strong> extracted from <strong>TRAIN_dataset</strong> of shape <strong>(98962,)</strong> and size <strong>98962</strong></li>
  <li><em>"Test svm classifer"</em> on <strong>TEST_features</strong> extracted from another dataset i.e <strong>TEST_dataset</strong> of the same shape <strong>(98962,)</strong> and size <strong>98962</strong> as <strong>TRAIN_dataset</strong> is.</li>
  </ol>
  
  <p>After <em>"preprocessing"</em> of both <strong>"TRAIN_features"</strong> &amp; <strong>"TEST_features"</strong>, with the help of <strong>"TfidfVectorizer"</strong> i vectorized my both features. after that i again computed the shape and size of both features i.e</p>
  
  <pre><code>vectorizer = TfidfVectorizer(min_df=7, max_df=0.8, sublinear_tf = True, use_idf=True)
  processed_TRAIN_features = vectorizer.fit_transform(processed_TRAIN_features)
  </code></pre>
  
  <p><em>"processed_TRAIN_features"</em> size becomes <strong>1032665</strong> and <em>"shape"</em> becomes <strong>(98962, 9434)</strong></p>
  
  <pre><code>vectorizer1 = TfidfVectorizer(min_df=7, max_df=0.8, sublinear_tf = True, use_idf=True)
  processed_TEST_features = vectorizer1.fit_transform(processed_TEST_features)
  </code></pre>
  
  <p><em>"processed_TEST_features"</em> size becomes <strong>1457961</strong> and <em>"shape"</em> becomes <strong>(98962, 10782)</strong></p>
  
  <p>i know when i will <em>"TRAIN"</em> svm classifier on <strong>processed_TRAIN_features</strong> and when <em>"predict"</em> the <strong>"processed_TEST_features"</strong> using same classifier, it will produce error because <em>"shape"</em> and <em>"size"</em> of both features has become different.</p>
  
  <p>what i think, the only solution to this problem is to <em>"reshape"</em> sparse matrix (numpy.float64) either <strong>processed_TEST_features</strong> or <strong>processed_TRAIN_features</strong>... i think reshape to <em>"processed_TRAIN_features"</em> is possible only as its size is less than <em>"processed_TEST_features"</em> OR there is anyother way to implement my above points (1,2). I am unable to implement <a href="https://stackoverflow.com/questions/50088190/reshape-scipy-csr-matrix">this question</a> regarding to my problem and still in search that how it will become equal to <em>"processed_TEST_features"</em> w.r.t shape and size.</p>
  
  <p>please if anyone of you can do this for me... thanks in advance.</p>
  
  <p>Full code is below:</p>
  
  <pre><code>DataPath2     = ".../train.csv"
  TRAIN_dataset =   pd.read_csv(DataPath2)
  
  DataPath1     = "..../completeDATAset.csv"
  TEST_dataset  =   pd.read_csv(DataPath1)
  
  TRAIN_features = TRAIN_dataset.iloc[:, 1 ].values
  TRAIN_labels = TRAIN_dataset.iloc[:,0].values
  
  TEST_features = TEST_dataset.iloc[:, 1 ].values
  TEST_labeels = TEST_dataset.iloc[:,0].values
  lab_enc = preprocessing.LabelEncoder()
  TEST_labels = lab_enc.fit_transform(TEST_labeels)
  
  processed_TRAIN_features = []
  
  for sentence in range(0, len(TRAIN_features)):
      # Remove all the special characters
      processed_feature = re.sub(r'\W', ' ', str(TRAIN_features[sentence]))
  
      # remove all single characters
      processed_feature= re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature)
  
      #remove special symbols
      processed_feature = re.sub(r'\s+[xe2 x80 xa6]\s+', ' ', processed_feature)
  
      # remove special symbols
      processed_feature = re.sub(r'\s+[xe2 x80 x98]\s+', ' ', processed_feature)
  
      # remove special symbols
      processed_feature = re.sub(r'\s+[xe2 x80 x99]\s+', ' ', processed_feature)
  
      # Remove single characters from the start
      processed_feature = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature)
  
      # Substituting multiple spaces with single space
      processed_feature = re.sub(r'\s+', ' ', processed_feature, flags=re.I)
  
      #remove links
      processed_feature = re.sub(r"http\S+", "", processed_feature)
  
      # Removing prefixed 'b'
      processed_feature = re.sub(r'^b\s+', '', processed_feature)
  
      #removing rt
      processed_feature = re.sub(r'^rt\s+', '', processed_feature)
  
      # Converting to Lowercase
      processed_feature = processed_feature.lower()
  
      processed_TRAIN_features.append(processed_feature)
  
  vectorizer = TfidfVectorizer(min_df=7, max_df=0.8, sublinear_tf = True, use_idf=True)
  processed_TRAIN_features = vectorizer.fit_transform(processed_TRAIN_features)
  
  
  processed_TEST_features = []
  
  for sentence in range(0, len(TEST_features)):
      # Remove all the special characters
      processed_feature1 = re.sub(r'\W', ' ', str(TEST_features[sentence]))
  
      # remove all single characters
      processed_feature1 = re.sub(r'\s+[a-zA-Z]\s+', ' ', processed_feature1)
  
      #remove special symbols
      processed_feature1 = re.sub(r'\s+[xe2 x80 xa6]\s+', ' ', processed_feature1)
  
      # remove special symbols
      processed_feature1 = re.sub(r'\s+[xe2 x80 x98]\s+', ' ', processed_feature1)
  
      # remove special symbols
      processed_feature1 = re.sub(r'\s+[xe2 x80 x99]\s+', ' ', processed_feature1)
  
      # Remove single characters from the start
      processed_feature1 = re.sub(r'\^[a-zA-Z]\s+', ' ', processed_feature1)
  
      # Substituting multiple spaces with single space
      processed_feature1 = re.sub(r'\s+', ' ', processed_feature1, flags=re.I)
  
      #remove links
      processed_feature1 = re.sub(r"http\S+", "", processed_feature1)
  
      # Removing prefixed 'b'
      processed_feature1 = re.sub(r'^b\s+', '', processed_feature1)
  
      #removing rt
      processed_feature1 = re.sub(r'^rt\s+', '', processed_feature1)
  
      # Converting to Lowercase
      processed_feature1 = processed_feature1.lower()
  
      processed_TEST_features.append(processed_feature1)
  
  vectorizer1 = TfidfVectorizer(min_df=7, max_df=0.8, sublinear_tf = True, use_idf=True)
  processed_TEST_features = vectorizer1.fit_transform(processed_TEST_features)
  
  X_train_data, X_test_data, y_train_data, y_test_data = train_test_split(processed_TRAIN_features, TRAIN_labels, test_size=0.3, random_state=0)
  
  text_classifier = svm.SVC(kernel='linear', class_weight="balanced" ,probability=True ,C=1 , random_state=0)
  
  text_classifier.fit(X_train_data, y_train_data)
  
  text_classifier.predict(processed_TEST_features)
  </code></pre>
  
  <h2>Title EDIT: predict classification of dataset => predict dataset</h2>