{'fraction_non_alphanumeric': 0.02450706474016125, 'fraction_numerical': 0.004550171629280754, 'mean_word_length': 4.78393351800554, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting Future Sales of Retail Products using Machine Learning

Techniques for making future predictions based upon the present and past data, has always been an area with direct application to various real life problems. We are discussing a similar problem in this paper. The problem statement is provided by Kaggle, which also serves as an ongoing competition on the Kaggle platform. In this project, we worked with a challenging time-series dataset consisting of daily sales data, kindly provided by one of the largest Russian software firms - 1C Company. The objective is to predict the total sales for every product and store in the next month given the past data. In order to perform forecasting for next month, we have deployed eXtreme Gradient Boosting (XGBoost) and Long Short Term Memory (LSTM) based network architecture to perform learning task. Root mean squared error (RMSE) between the actual and predicted target values is used to evaluate the performance, and make comparisons between the deployed algorithms. It has been found that XGBoost fared better than LSTM over this dataset which can be attributed to its relatively higher sparsity.


Introduction
Accurate sales forecasting is of utmost value to any organization, be it a small and medium enterprise or a Fortune 500 company. It provides an accurate estimation of company's top-line growth and can be utilized to prepare plans for near-term demand and supply actions. Also, it serves as a guidance to devise the bottom-line plans that keep up with the overall organizational goals for financial prudence. Finally, strategic decisions such as identification of key areas for short-term investment can also be made, taking cues from these forecasts.
Interesting and difficult sales forecasting problems are pretty common. We are very fortunate that a similar problem statement titled "Predict Future Sales" is provided by Kaggle. [1] As part of the problem statement, we need to work over a challenging time-series dataset consisting of daily sales data. A time series is a sequence of historical measurements of one or more observable variable(s) at equal time intervals. Time series can be studied for several purposes such as predicting future based on past knowledge, or understanding the latent variables behind the generation of measured values, or for simply providing a concise description of the salient features of the series. For this project, we need to forecast the total sales of every product and store combination for the next month, given the past data.
We have started working on the problem by performing exploratory data analysis (EDA) over the provided dataset, in order to understand the dataset better. For this task, we took help from the already available notebooks on Kaggle. Soon, we found out that our dataset is a combination of time series and static data. Columns such as item name, item category, shop ID, etc. do not change with time whereas item count and item price do vary with time. Having a better idea of the training dataset, we started to look for ideal candidates to perform this learning task. We split our team into two groups to simultaneously cover two different objectives. One group started to read more kaggle notebooks in order to understand what has worked for most people. Another group searched for what can be some novel techniques, that can be applied on this problem.
With time, we have realized that applying XGBoost with lagged features did work out pretty well for many people in the past. Also, we have found that Autoregressive Integrated Moving Average (ARIMA) and LSTM based networks have provided quality results over similar forecasting tasks. [2] [3] More about this is discussed in the subsequent section on related work. To summarize, we have decided to try XGBoost, ARIMA, and LSTM as learning algorithms, and to use RMSE for evaluating their performance.

Related work
The big inspiration for our model selection process was drawn from an outstanding Kaggle Blog Post "Time Series -ARIMA, DNN, XGBoost Comparison". [4] As the name says, the blog is about comparison of three models; ARIMA, DNN and XGBoost on Time Series Dataset. The format of the data set on which the comparisons were drawn largely matches the format of the dataset on which we had to make our predictions. And since all the above three models gave fairly appealing results in the blog, hence we were confident that we could not go wrong experimenting with these models if we did other parts of the job like data preparation and feature engineering right way.
The same blog also shows some good featuring engineering practices which suits best for each model. Hence, some of the ideas for feature engineering were also inspired from the same Kaggle Blog.
However, for some reason, we did not get impressive results using DNN with our first implementation. Meanwhile, we found another interesting paper which outlines how LSTM could be applied to a dataset of this format. More about the paper below, but the important thing to mention here is that instead of spending time playing around and fixing DNN, we thought to shift our focus on LSTM. Hence, our final set of models to experiment were; ARIMA, LSTM and XGBoost. Esteban, et al. (2016) have demonstrated in their paper, how static and dynamic features can be efficiently modeled with a combination of LSTM and fully connected neural network architecture. [5] We took inspiration from their architecture and decided to deploy an LSTM based neural network architecture, shown in figure 1, as one of our learning model.
Even after knowing the fact that XGBoost is a go-to model for data science competitions, we somewhere lacked the intuition on why XGBoost works magically well in such competitions and machine learning in general. And this gap was filled by another well articulated data science blog, 'XGBoost Algorithm: Long May She Reign!'. [6] 3 Data We followed the conventional data engineering process where we started with data cleaning followed with exploratory data analysis and feature engineering.

Data cleaning
First things first, we clipped the prediction feature Item Count to match the prediction output requirement [0,20]. Next step was to fix the invalid values and missing values. There weren't many corrupted values; the data set was quite legitimate in this sense. However, for the minor fixes that we did, our approach was to either remove it completely or to interpret the values by taking mean or median. We took the most appropriate approach on a case by case basis.

Exploratory Data Analysis (EDA)
To get a lead for the features to derive, EDA was a required step. The plots which gave us pretty good lead for data preparation and feature extraction are shown in figure 2. We noticed some key observations that we applied during feature engineering process. The most important finding was that the trend is normally distributed with just few peaks in between which are observed yearly (useful for getting intuition of extracting time lag). Some other observations we noticed were that sales is highest during weekends so features that might be helpful to capture the effect might be a count of each of the weekday in the date-block, and count of weekends. We further noticed that revenue peaks towards end of the year and is lower in mid-year. Moreover, the breakdown of revenue by category varies considerably month to month.

Feature engineering
Taking inspiration from the briefly mentioned EDA process, the following features were generated: 1. First step was to create merged data set by combining all the provided data sets i.e. shops, items and item categories data set into one master data set.

Transforming the combined master data set from Time Series Format into Grid Format
• Created a grid to store data with unique shop_id, item_id combinations for each date_block_number. • Since the sales data only has rows with transactions, the grid was filled with 0 for item shop combinations that are not sold for the month (date_block_number)

Revenue related features
• aggregated total revenue for the month by ['shop_id', 'date_block_number'] • aggregated total revenue for the month by shop_id + category_id

Item count related features
• aggregated total item_count_day for the month by shop_id.
• aggregated total item_count_day for the month by shop_id + category_id • aggregated total item sold after grouping it by ['date_block_number', 'item_id'] and cumulative item sold. • flag new items having 0 cumulative sale in previous month, but > 1 in current month.

Price related features.
• Computed weighted mean price feature for the [item_id, date_block_number].

Extracting lags and its interactions
• Lags of all the numeric features were calculated for the past 1-3 months and 12 months.
• To further get some more interaction between lags, we computed difference between the above extracted lags of numeric features and added that as an additional features. 7. One hot encoding 'month', 'year', 'item_category_id', 'shop_id' and their various combinations.
Finally, we concluded the data engineering part by splitting the data sets into training set and validation set. where we split the first 32 months for training and the 33rd month data was sliced out for validation.

XGBoost
XGBoost was one of the starting algorithms we chose. As per statistics, it is most widely used model for Kaggle competitions. It provides system optimization through parallelization and hardware optimization. XGBoost has advantages over general gradient boosting in terms of providing regularization through a combination of both the ridge and lasso regressions. It also handles different type of sparsity patterns in the data efficiently. Table 1 enumerates the hyperparameters we have tuned over the validation set using uniform random sampling over a grid search space. ARIMA modelling is usually a go-to approach for time series data. But ARIMA only works best for univariate time series data i.e. the one which has a single variable. For e.g., stock market prediction. To apply ARIMA for this dataset, our approach was to group the time series data with respect to the identifier columns and fit ARIMA for each group and make predictions. This required us to fit the model multiple times for as many unique values in identifier we have.

Long Short-Term Memory(LSTM) network
Given the static and dynamic nature of input features, a customized neural network architecture ( figure 3) is subsequently used for learning this regression task. The proposed architecture resulted in a decrease of space requirement by a factor of 12 over plain LSTM, where all static features are replicated across time steps. For the purpose of achieving optimal values for hyperparameters, we have created a grid search space. Because of the time and resource constraint, only batch size and L2 regularization coefficient (λ) is considered during hyperparameter tuning. Also, it is important to note here that the value of λ is assumed equal for all layers. The optimal value determined for batch size is 512 samples, and that for λ is 0.001. Adam optimizer and mean squared loss is used in optimization algorithm, which is executed over 5 epochs of training data.

Evaluation
The evaluation metric used to evaluate the performance of different models is the Root Mean Square Error (RMSE). The training and validation data were split based on description in Section 3.3. For the test set, we provided the prediction for November 2015 (month 34), and the same has been used to calculate our public leader board position on Kaggle.

Setup
We implemented our models in Google Colab with 25 GB RAM and GPU support. We have used python as the coding language for this project. The different libraries and packages that empowered our modelling process were: numpy, pandas, sklearn, keras, tensorflow, matplotlib, and xgboost.
Our best performing model is the XGBoost, whose performance along with the other models is listed in table 2. The similar values for training and validation RMSE suggests low variance across the models we tried.   Sales forecasting has become highly sophisticated and plays a vital role to the competitiveness of many companies. We have tried to deploy different learning algorithms for forecasting sales in this paper. XGBoost performed best in achieving our objective on this dataset. This can be attributed to the good feature engineering and the ability to try out a large range of hyperparameters during optimization.