I am looking for some general advice on where to start with this problem. There are 350 sparse (low positive integer) features. I have 2000 positives, 1000 negatives, and infinite unlabeled data, where the estimated true positive rate in the unlabeled data is ~1% (but this isn't known for certain). The goal is to predict the probability of an example being positive.

How would you approach the problem if you had roughly 40 hours to attain the best possible model?

What about if you had more time?

  • $\begingroup$ How are the unlabelled data related? Can it be removed? $\endgroup$
    – timekeeper
    Dec 28 '16 at 17:03
  • $\begingroup$ sklearn has some utility functions to try to guess a label for this kind of cases. I never used such things. You could try to see if that improves your model. $\endgroup$ Dec 29 '16 at 17:45

It's a binary semi-supervised classification problem. First, establish a base-line for the supervised case. Then try if the unlabeled data helps


  1. From your labeled data: create a training, validation and test set. Don't touch the test set until the very end.
  2. Try something simple, e.g. a multilayer Perceptron (MLP) with 350 input nodes and 1 output node (giving the probability of "true").
  3. Try more stuff (e.g. https://github.com/MartinThoma/algorithms/blob/master/ML/mnist/many-classifiers/python.py#L92).
  4. Try to combine classifiers in ensembles (see examples).
  5. A (naive) bayes classifier might be worth being investigated.


  • You could train an auto-encoder with a bottleneck on the unlabeled data. Then remove everything after the bottleneck. Use this network as a preprocessing step. The idea is that this network finds a more meaningful abstraction of the relevant data. However, similar as PCA can have a very low projection error and still destroy the possibility to distinguish the classes.
  • SVMs can also be used in a semi-supervised setting.
  • You could try to find clusters in the unlabeled data + labeled data. Then you can get an a priori probability of "true" for each cluster (ignoring the unlabeled data). You could train very small models (overfitting!) on the data in the clusters.

Usually, you can interpret features. This might help to develop new features.


Given your time budget and the potential challenges associated with class imbalance, I'd throw away the unlabelled data and use supervised learning on the labelled data. Try a simple classifier, e.g., logistic regression or random forests or xgBoost, and use cross-validation to see how well they perform. In advance put aside a held-out test data and don't touch it until the end. Use cross-validation on the rest of the data to try different classifiers and different approaches.

You have to deal with a severe class imbalance problem. My suspicion is that you might need to budget a large fraction of your time dealing with class imbalance. A true positive rate of 1% corresponds to a 100x class imbalance, which causes severe problems for many classifiers and might turn out to be a real headache.

Also the true underlying distribution doesn't match the distribution in your training set: the true positive rate in the wild is 1%, but 67% of your training set is positives. You'll need to adjust for this as well.

There are many ways to deal with class imbalance and to adjust for differences between the imbalance in your training set vs in the wild. I would suggest you start by setting class weights: you'll need to upweight the negatives by a factor of 198 (so that an error on a negative in your training set costs 198 times as much as an error on a positive in your positive set), since in the wild the distribution you expect to see is 2000 positives and 198000 negatives rather than 2000 positives and 1000 negatives.

The challenge is that some classifiers struggle to handle this level of class imbalance well. For instance, SVMs are known for sometimes failing badly in the presence of this kind of class imbalance; they can end up defaulting to always predict "negative", since this achieves an error rate of only 1%. In contrast, logistic regression tends to be better behaved, and provides an easy way to adjust for class imbalance.

Do some searching on class imbalance; you'll find lots written on multiple strategies you can use for handling it.

If class imbalance turns out not to be a problem and you end up with a decent classifier on your first try, and you have more time to try to improve the results, then the next thing I'd spend time on is seeing if you design better/more features, given domain knowledge about where these instances come from.

  • $\begingroup$ Thanks for this response, I ran some basic models on just the labeled data and got a great result, but unfortunately on the unlabeled data it is classifiying them mostly positive, so the model is wrong. I'm going to try the class weights and also add in unlabeled data assuming it is negative to try that $\endgroup$
    – Lucky711
    Jan 10 '17 at 18:45
  • $\begingroup$ @Lucky711, then it's likely that your labeled data isn't sampled from the same distribution as the unlabeled data. In that case, you have a data quality problem: before you spend any time applying ML techniques, you first need to understand how the data was sampled and what process was used to choose the samples and what biases have been introduced by that process (and then once you understand that, you can ask a new question about how to deal with it). $\endgroup$
    – D.W.
    Jan 10 '17 at 22:53
  • $\begingroup$ Unfortunately this is health based data, so it isn't easy / may be impossible to get more data. I have a feeling the negatives are not "random" and are known to be negative for a specific reasons. My thinking now is that since I know the unlabeled data is mostly negative, I'm going to build models assuming the unlabeled is negative (treat this almost like a PU learning problem) and go from there. $\endgroup$
    – Lucky711
    Jan 11 '17 at 3:57
  • $\begingroup$ You might not need to get more data, but you do need to know where the data you already have came from. If you don't understand the process by which the samples in your labelled set were chosen, it's going to be very difficult to draw any conclusions at all about any candidate scheme. $\endgroup$
    – D.W.
    Jan 11 '17 at 5:54

Your Answer

