{'fraction_non_alphanumeric': 0.01419698314108252, 'fraction_numerical': 0.0, 'mean_word_length': 4.814432989690721, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Accounting for Factor Variables in Big Data Regression

Continuous and factor explanatory variables are both important in linear regression. To fit a linear model with factor variables, the traditional implementation of the least squares approach is to define a number of dummy variables. This approach is difficult to apply to big data since the size of the design matrix can be significantly inflated by a factor variable, even if the number of factor levels is only moderately large. By treating the factor variable as an index, the article proposes a new approach, called the index least squares approach, to overcome the difficulty. Combining it with the technique of scanning data by rows, the index least squares approach can provide exact solutions to a group of linear models with factor variables simultaneously. Therefore, it avoids the memory barrier caused by the size of the design matrix. As the memory needed is unrelated to the number of observations, the index least squares approach can be used, even if the size of a massive data set is hundreds of times higher than the memory size of the computing system.