{'fraction_non_alphanumeric': 0.02496626180836707, 'fraction_numerical': 0.001349527665317139, 'mean_word_length': 5.337606837606837, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Simultaneous Co-clustering and Learning with Dimensionality Reduction

In this project, we are interested in solving predictions problems involving dyadic data. The “Netflix problem”, where data may be organized into a matrix with the rows representing users, the columns representing movies and (some of) the matrix elements containing ratings, is an example of a setting where the data is naturally dyadic. The prediction problem here is essentially to fill in the missing entries of the matrix with predicted ratings. Recently, a technique called simultaneous co-clustering and learning (SCOAL) was proposed that was demonstrated to be effective in solving such dyadic prediction problems. SCOAL is essentially a divide-and-conquer approach that seeks to form clusters of feature vectors and build a separate local model for each one of these clusters. Such an approach inherently suffers from the risk of overfitting, especially when the number of clusters is large. Thus, to alleviate the effects of over-fitting, while at the same time, benefitting from the use of multiple models, we propose the use of dimensionality reduction in conjunction with SCOAL. We investigate two dimensionality reduction techniques: principle components regression and the least absolute shrinkage/selection operator (LASSO). We show that both these techniques provide significant reduction in test error (up to 10% in some settings) when linear predictive models are employed within each cluster.