{'fraction_non_alphanumeric': 0.025674186904248695, 'fraction_numerical': 0.004670211237246731, 'mean_word_length': 5.182262363044122, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Progressive Feature Upgrade in Semi-supervised Learning on Tabular Domain

Recent semi-supervised and self-supervised methods have shown great success in the image and text domain by utilizing augmentation techniques. Despite such success, it is not easy to transfer this success to tabular domains. It is not easy to adapt domain-specific transformations from image and language to tabular data due to mixing of different data types (continuous data and categorical data) in the tabular domain. There are a few semi-supervised works on the tabular domain that have focused on proposing new augmentation techniques for tabular data. These approaches may have shown some improvement on datasets with low-cardinality in categorical data. However, the fundamental challenges have not been tackled. The proposed methods either do not apply to datasets with high-cardinality or do not use an efficient encoding of categorical data. We propose using conditional probability representation and an efficient progressively feature upgrading framework to effectively learn representations for tabular data in semi-supervised applications. The extensive experiments show superior performance of the proposed framework and the potential application in semi-supervised settings.


I. INTRODUCTION
Since the major breakthrough in the ImageNet Large Scale Visual Recognition Challenge, deep learning has attracted much attention due to its superior performance in many applications, e.g. Speech Recognition, Computer Vision, and Natural Language Processing. Such great progress is largely driven by enormous datasets. Collecting and labeling such an enormous dataset is expensive, time-consuming, and often impossible.
Recently, semi-supervised learning [1]- [3] has gained a lot of attention due to superior performance in the image and language domain. Semi-supervised learning aims to leverage a small amount of labeled data as well as a huge amount of unlabeled data to perform learning tasks (classification and regression). Recently, researchers propose different augmentation techniques and regularizations (consistency regularization) in semi-supervised learning approaches [4]. Often, these augmentation techniques are domain specific. Take image domain augmentation as an example, these augmentation techniques help create images that can explicitly cover various perturbations/variances (viewpoint, lighting, occlusion, background in image domain) to challenge the learned model to better handle these difficult cases. Theoretically, the data augmentation technique in semi-supervised learning can help increase the generalization ability of the trained models by reducing the overfitting and expanding the decision boundary of the models.
However, the success of semi-supervised learning approach on image and language domain can hardly be transferred to tabular domain. Because, the success of augmentationbased semi-supervised learning algorithms heavily relies on the spatial or semantic structure of image or language data. For other data like tabular data which does not exhibit any explicit structure, the semi-supervised learning becomes much more challenging. The suspected reason is that augmentation technique like Mixup ( [5]) is usually a convex combination of the original samples. These augmentation techniques may only work well if the original data manifold is likely convex too. Meanwhile, tabular data is likely not convex. Directly applying augmentation techniques that have been used in image and language domain to tabular domain can easily create out-ofdistribution samples which may even hurt the learning process [6].
One straightforward idea towards solving semi-supervised learning in the tabular domain is to develop some customized augmentation and loss function for tabular data. This route is challenging because of the following reasons. Tabular data does not exhibit any explicit structure, coming up with any suitable augmentation is not an easy task. Also, tabular data usually contains both categorical data which is discrete and numerical data which is continuous. Mixing different data types altogether creates a severe challenge for the recent semi-supervised learning approaches. Some recent works on semi-supervised learning to tabular data have focused on proposing new augmentation operations suitable for tabular data or working on the latent space. However, even the new augmentation technique can not fundamentally solve the problem due to the high-cardinality of categorical data and inefficient representation.
We propose to change the representation of the tabular data especially categorical data from common approach (one-hot encoding, label encoding) to less known approach (conditional probability representation) to enjoy several unique benefits mentioned in the next paragraph. We argue that one should take a step back and carefully examine how tabular data and especially how categorical data can be represented in the semisupervised learning problem. Representation of the data is one important aspect that can easily be ignored. Our experiment in Table I also shows that the choice of representation for categorical data may greatly impact the performance. Also, such impact may even be agnostic to what follow-up model is being used.
We propose utilizing conditional probability representation (CPR) for semi-supervised learning in the tabular domain. CPR maps individual values of categorical feature to the probability estimate or the expected value of the target attribute. In another word, it computes the likelihood of a specific categorical value leading to a specific label. It has many unique benefits compared to other representations (one-hot encoding, label encoding). Firstly, it is an efficient representation in terms of how many bits are used to represent the feature, especially for high-cardinality categorical data. The reason being that the number of dimensions of CPR does not depend on the cardinality of the categorical feature. It only depends on the number of target labels. Secondly, label information has been baked into the representation. Label information is critical for semi-supervised learning algorithm. If one can inject label information into the feature, it may be easier for the model to learn meaningful representations. More importantly, it opens the door for utilizing pseudo-labels (predicted labels) in a novel way (constructing the feature). Thirdly, compared to other representations, CPR is closer to the numerical features, since it uses conditional probabilities as features. This property may open the door for better enabling leveraging various existing augmentation techniques for tabular data.
Beside employing CPR as feature representation for categorical data, more importantly, we propose to progressively upgrade the CPR during model training by leveraging the pseudolabels. Instead of only using true labels to construct CPR, we propose to use pseudo-labels to update the CPR during the model training process. Our initial study shows that even if we don't change how much data is used for training, but only increase the amount of data used for constructing the CPR, the prediction accuracy can be hugely boosted (Table II). Pseudolabels are defined as predicted class labels for unlabeled data. Self-training algorithm utilizes pseudo-labels of unlabeled data to continue improving the model. It is just one way of utilizing pseudo-labels which is treating pseudo-labels as if they are the ground-truth labels in training the model. However, pseudolabels can also be used in another way which is to update the CPR and then to influence the model training process. With model being trained progressively, more accurate pseudolabels will help generate more accurate representations for learning in the "feedback loop".
We propose a framework that can progressively upgrade the CPR representation. The proposed framework is flexible in the sense that it can act as an add-on component to the existing semi-supervised learning frameworks. For this framework to work, there needs to be a component that can produce pseudo-labels. This condition is not hard to satisfy.
Commonly, the semi-supervised learning algorithms always contain a component (predictor or label propagation) that can produce pseudo-labels. The pseudo-labels provided by these components can then be used to upgrade the CPR. One clear benefit is that even if certain category values do not exist in the labeled dataset, representation for those categories can still be calculated because of employing the generated pseudo-labels. Pseudo-labels being used for updating the CPR representation are not 100% correct, it may introduce additional noise to the model training process. We propose several refinement mechanisms to alleviate such issue by selecting only the pseudo-labels with high confidence that they are the correct label.
The main contribution of our paper can be summarized as follows.
• We propose using conditional probability representation for high-cardinality categorical data for efficient representation. To the best of our knowledge, our work is the first work that uses an encoding different from one-hot encoding for tabular domain in semi-supervised learning.
The proposed framework can also be extended to other encoding method such as target encoding ( [7]) which also bakes the target label into the representation. • We propose novel feature upgrading framework by leveraging pseudo-labels. To the best of our knowledge, we are the first paper to propose using pesudo-labels to update the CPR for categorical data in semi-supervised learning. • The proposed framework is flexible and complementary which can be easily embedded into the existing semisupervised learning algorithms to boost the learning performance. • We demonstrate the superior performance of the proposed framework in extensive experiments. The robustness of the proposed framework has been testified by superior performance on two different semi-supervised algorithms and three different tabular datasets. The paper is organized as follows. In Section 2, we provide a review of the related semi-supervised learning algorithms and works on the representation for categorical data, noting the importance of using and updating the conditional probability representation. In Section 3, we describe the details of the proposed framework. Then, we present the quantitative and qualitative experimental results in Section 4. Finally, Section 5 gives the conclusion.

A. Semi-supervised Learning
Semi-supervised learning in general is attempting to improve the performance of the learning algorithms by utilizing both the labeled and unlabeled data, such that the resulted classifier is better than the trained classifier on just labeled data [8]. Semi-supervised learning has shown considerable progress in the language and image domains in recent years. Most of these works resulted from the consistency regularization and pseudo-labeling on the unlabeled data.
Pseudo-labeling and self-training: The goal of pseudolabeling [15], [16] and self-training [17] refers to a classical semi-supervised approach where the model is being trained on the labeled and unlabeled samples using labels and pseudolabels associated with the unlabeled samples. The self-training [18], [19] has recently shown improved performance over supervised counterpart. Some works [20], [21] use calibration and uncertainty of predictions for the selection of samples to improve the pseudo-label selections. Also, disagreement-based models [22] use multiple learning algorithms and exploit the disagreement during the learning process to filter out wrong pseudo-labels.

B. Representation for tabular data
It is hard to transfer the semi-supervised learning algorithms proposed in image and language domains to tabular domain. Unlike image and language domains, tabular domain is a combination of different data types (numeric and non-numeric data). The non-numeric data can be unordered categories with a fixed set of possible values. General idea for processing tabular domain data is to encode it to the numerical representation to better consumed by machine learning algorithms. The classic approach to encode categorical variables is one-hot encoding, which is not suitable for high-cardinality categories due to generating high-dimensional vectors. This is a big problem in large datasets, which might have a very large number of categories, posing computational problems [23]. Despite the existence of data cleaning [24], [25] and similarity encoding techniques [26], it is hard to tackle the problem of high cardinality. For this purpose, Cerdar P., & Varoquaux G. [23] proposed a scalable encoding method for string categories using min-hash encoding and Gamma-Poisson factorization. They also proposed a similarity encoding technique [26] to encode dirty, non-curated categorical data. Also, Slakey A. et al. [27] proposed a CBM encoding approach to represent categorical features in low dimension.

C. Semi-supervised Learning in tabular domain
Recent advances in semi-supervised learning using deep networks have been applied to the tabular domain in some works. Darabi S. et al. [28] proposed a semi-supervised framework for the tabular domain, called "contrastiveMixup". They applied the one-hot encoding on raw categorical data, and the Mixup operation in the latent space among samples with the same labels and pseudo-labels on labeled and unlabeled data respectively. Supervised contrastive learning and mixup augmentation in the latent space are used to push the samples with the same label closer to each other in the latent space. Also Yoon J. et al. [6] proposed a semi-supervised method for the tabular domain using consistency loss among perturbed versions of the one-hot encoded input samples. The proposed framework introduced an augmentation technique in the input space, which is used to learn the latent space representation using an autoencoder. Then, the pre-trained encoder of autoencoder is used in the semi-supervised setting to learn from labeled and unlabeled samples using consistency losses and perturbation of samples. Also, Ucar T. Et al. [29] introduced a new framework that turns the tabular data into a multi-view representation learning. They claim that reconstructing the data from subsets of features captures a better latent representation rather than reconstructing the corrupted version of input in an autoencoder. Beside of using subset of features, they utilize the contrastive loss, the distance loss among the latent space of subsets, and the different augmentations in the input space to get the best performance.
Recent works only use one-hot encoding and test on small datasets containing low-cardinality categorical features. The existing works using one-hot encoding on these datasets usually work fine because the one-hot encoded features are not big. But in case of big datasets with high cardinality, the onehot encoding is not the best choice. There are many different encodings and several Python libraries cover them 1 and 2 . The different encodings have different performance on different datasets and it motivate us to think about using other encoding methods in semi-supervised learning. We consider using a conditional probability representation (CPR) that uses label of data for creating a numerical and continuous representation of the categorical data. We also propose a new framework that uses the CPR and progressively updates the representation for categorical features. Intuitively, this representation is more friendly to existing augmentation techniques than other representations (one-hot encoding or label encoding).
To the best of our knowledge, our work is the first work that uses an encoding different from one-hot encoding for tabular domain in semi-supervised learning. In this regard, we considered big datasets in experimental sections to show the effectiveness of our work.

III. METHODOLOGY
In this section, we describes our proposed framework. We describe the conditional probability representation (CPR), Update Policy and Refinement methods for the proposed framework. Then, we introduce progressive VIME and Progressive Contrastive Mixup.

A. Conditional Probability Representation
In case of big datasets contatining high-cardinality categorical data, the one-hot encoding is not the best choice because of the space consumption and curse of dimensionality problems [30]. In contrast, the CPR of the categorical data has a fixed representation w.r.t the number of targets in the classification problem, which creates a compact representation because usually the number of labels is much smaller than the cardinality of the features. Table I compares the performance of a Multi-layer perceptron (MLP) network on the CPR and two other encodings using Traffic Violations dataset. This experiment shows that the MLP using CPR can outperform the same model while using one-hot encoding or label encoding. Please note that different encoding methods may perform differently across different datasets/applications. To define the problem mathematically, let X be an N * M matrix with row vectors X n and column vectors X T m . Let Y be an N-dimensional target vector and Y n is the observed value correspond to X n . Then D = (X, Y ) is a dataset with N samples where Dn = (Xn, Y n) is the n th sample with label target Y n ∈ {0, 1, ..., C}. In context of the categorical data problem, column X T m with cardinality K m has a domain V (m) = {X n,m } n containing unique nominal values V m ∈ 1, ..., k m . Let C be the number of the values in the target variable. The CPR measures given each category value in a categorical feature, how likely for this category value lead to different target labels within the dataset. Therefore, for each categorical feature, a C-dimensional vector representation will be produced, where C is the number of target labels in the dataset. Following equation computes the CPR.

B. Preliminaries
To present the method, we formulate the semi-supervised problem. Consider a dataset with N samples. There are a small subset of labeled samples D L = {(X n , y n )} N L n=1 and a large set of the unlabeled samples The supervised training on labeled samples without learning from unlabeled samples mostly likely causes overfit. The unlabeled samples can be used to improve the generalization of the model to get better accuracy on unseen test samples. For this purpose, we use pseudo-labels and an Update Policy for better generalization in the training of the neural network, and also a better representation of data that helps boost the performance.

C. Update Policy
We design Update Policy that incorporates pseudo-labels of the unlabeled samples to update the CPR. More samples help generate more efficient representation. Our initial study in Table II shows that more labeled samples used for generating representation indeed drastically improve the model performance. This study shed a light on our approach that using more "labeled" samples may boost the model performance.
When we use more data with ground truth to generate the representation, the new representation of data influences the performance and improves the accuracy. The difference in this study is that we used the ground truth labels for statistics, however, in reality, pseudo-labels are being used.
In this approach, we use labeled samples D L at first to calculate the conditional probability on categorical features and generate the initial representation of all samples in the dataset (D). This representation will be updated using both labels for D L and pseudo-labels for D U , and keep on training the semi-supervised model on the updated representation. By updating the representation using more samples (D L + D U ), the model can obtain better generalization since the representation contains more information from the dataset.

D. Refinement
We propose refinement mechanism for handling noise in pseudo-labels. It works by filtering out likely incorrect pseudolabels. It helps generate more accurate representation in Update Policy and improve the performance of the trained model. Some methods are introduced to choose more accurate pseudo-labels. Note that, though how to find more accurate pseudo-labels have been discussed in these papers, our main idea of utilizing CPR and keep updating the representation progressively is different from the methods proposed in these papers. These works use weight of pseudo-labels in the graphbased label propagation [31], the confidence of pseudo-labels in a classifier [11], uncertainty weight for each sample [32], [33] or using all pseudo-labels without refinement [34].
If the label-propagation method is an component in the method, we use measured weights in the label-propagation method for filtering. If the classifier is available, we can use confidence of pseudo-labels in the classifier for filtering. When both label-propagation and classifier are used in the architecture, we propose a mechanism to leverage both components for filtering. Two steps of filtering are used in the proposed mechanism. First, we keep only those pseudo-labels agreed between classifier and label propagation methods. Then, the final pseudo-labels are selected based on a threshold on the measured weights by the label-propagation method. After the second step, the final pseudo-label and its corresponding When one of the components is available in the architecture, the agreement can not be used. Instead, a threshold on the confidence of prediction in the classifier or the weight of pseudo-label in label-propagation is used for filtering.

E. Progressive Training
Progressive training refers to updating the input feature representation to train the model. We believe that by updating representation using pseudo-labels, it is possible to train a more effective model. This section shows how Update Policy and Refinement are used in the progressive architecture. In this regard, the progressive architecture is plugged to two existing semi-supervised learning architectures [6], [28] that have extended the recent advances in semi-supervised learning to the tabular domain. In the following, we shortly describe the previous works, then expand them to the proposed progressive approach. We utilize Update Policy and Refinement for expansion. 1) VIME: Before introducing the proposed Progressive VIME, in this section, we first describe VIME [6]. VIME has two training steps. A self-supervised pretext task is proposed in the first step to learn the data representation. Then, consistency regularization is leveraged to fine-tune the prediction. These steps can be seen in Figure 1 inside right upper green box (step 1) and right lower blue box (step 2). The black box in Figure 1 is not part of the original architecture, it is the component proposed in this paper. It will be described in the next subsection.
In the first step, the representation of data is learned using a denoising autoencoder architecture. The input samples are corrupted by a augmentation method through mask generator and pretext generator components in both steps before input to the encoder. The autoencoder has one encoder and two decoders. One decoder (feature vector estimator) reconstructs the original input sample, while the other decoder (mask vector generator) learns to identify the inconsistency between feature values.
The second step is semi-supervised learning. The predictor uses the representation learned from pre-trained encoder. The predictor is trained in a semi-supervised manner using the consistency regularization loss. Several corrupted samples (augmented inputs) are used to compute the consistency loss for training the predictor.
2) Progressive VIME: We describe how the progressive architecture is added to VIME. The black box in Figure 1 shows the Update Policy Component. The Update Policy is used to generate the CPR before feeding samples to the model. We introduce the term run that stands for training the encoder and the predictor in both steps. So the first run is equivalent to the original VIME.
In the beginning, the Update Policy uses the labeled samples for generating representation because pseudo-labels for unlabeled samples are not available yet. The training follows two consecutive steps for the unsupervised training of autoencoder and semi-supervised training of the predictor, this is one run. After training the predictor (blue box) in the second step of a run, the pseudo-labels are generated from the predictor, which is then sent to Update Policy component (dotted red arrow in Figure 1). The refinement can be used inside the Update Policy. It uses the pseudo-labels' confidence produced from the classifier for filtering. Refinement will filter out unreliable pseudo-labels. Depending on whether using refinement or not, refined pseudo-labels or all pseudo-labels are added to the unlabeled samples set. The Representation Regeneration in Update Policy uses the labeled and unlabeled samples to generate the new representation for categorical features. When the representation is updated, the new training in the next run is started on the latest representation of the data. In other word, the representation is updated before the next run.
3) Contrastive Mixup: Before describing the progressive architecture, we review the Contrastive Mixup [28] first. Contrastive Mixup leverages mixup-based augmentation in latent space for contrastive learning. The Contrastive Mixup has two training steps like VIME. The first step is to train an autoencoder using labeled and unlabeled data by contrastive learning and reconstruction. The mixup operations are applied in the latent space on the samples with the same labels for contrastive learning. Note, unlike most mixup-based augmentation methods that they randomly select samples for mixup, Contrastive Mixup select samples with the same label for mixup operation. Supervised contrastive learning is applied between the original and mixed samples. The label-propagation [31] is used after warm-up to generate or update pseudo-labels on unlabeled samples. Pseudo-labels will eventually be used for contrastive learning. In the second step of the training, a predictor is trained using consistency loss as a regularization. The transparent components in Figure 2 show the original architecture of Contrastive Mixup. The black box component is proposed in progressive architecture.

4) Progressive Contrastive Mixup:
We introduce Progressive Contrastive Mixup where Update Policy and Refinement are plugged into the Contrastive MixUp [28]. All components in Figure 2 show the proposed progressive architecture. The highlighted components in Figure  2 show the changes compared to the original architecture. We propose adding a classifier connected to the encoder to offer an alternative way to produce pseudo-labels as well as confidence on the pesudo-labels.
Like Progressive VIME, we need to update the representation using Update Policy. In Update Policy, we can use all pseudo labels or a subset of them using Refinement component. We believe that the Refinement may improve the accuracy because of using more accurate pseudo-labels to generate the representation.
There are different strategies for performing Refinement in Progressive Contrastive Mixup because both label-propagation and the proposed classifier components are available in the architecture and can indicate the confidence on the pseudolabels. We propose a two-step filtering mechanism (illustrated in Pseudo-label Refinement component in Figure 2). Firstly, the pseudo-label agreement between the label-propagation and classifier is used to select more likely correct pseudo-labels. Then, the pseudo-labels are filtered based on a threshold on the weights measured by the label-propagation method. Based on our study, we find that label-propagation weights are more robust than the classifier's confidence scores.

IV. EXPERIMENTAL STUDY
This section shows the results of progressive architectures on different tabular datasets with the high-cardinality categorical data. Also, the progressive architecture is compared with VIME [6] and Contrastive Mixup [28] on three datasets. A detailed study of the performance of using different components in progressive architectures and the original architectures are presented.

A. Tabular Datasets
To demonstrate the efficacy of the CPR, Update Policy and Refinement on the progressive architectures, we conduct a series of experiments on three datasets: Traffic Violations [35], Drug Directory [36] and Display Advertising Challenge [37]. All datasets have multiple high-cardinality categorical data. We shortly introduce each dataset and describe the cardinality of some features in the following. More details of the datasets are shown in the Table V. The Traffic Violations dataset has 1,578,154 samples with multiple categorical features. The sum of the cardinality of all categorical features is higher than 200,000. This dataset contains DateTime, numerical, boolean, and string categorical features. We exclude DateTime features in our experiments.
The Display advertising challenge dataset is published by Criteo company. This dataset contains 11 numerical features (mostly count features) and 26 categorical features. The values of these features have been hashed onto 32 bits for anonymization purposes. One million samples are randomly sampled from the dataset for the experiment. This sampled dataset contains some categorical features that have cardinality higher than 200,000.
The Drug Directory dataset is the smallest in our experiments. It has 19,764 samples with several categorical features. In this dataset, the feature with maximum cardinality has 5,032 unique values. This dataset contains the numerical, date, and categorical features.
For preprocessing, the date features are converted to numerical values. All numerical features are normalized using standard scaler in scikit-learn library 3 . Also, categorical features are converted to numerical features using the CPR.
In the experiment, we use 80% of the data as the train set and 20% as the test set. Also, 10% of the train set is chosen as the labeled set and the rest as the unlabeled set. This division of dataset is used in all experiments. The prediction accuracy on the test set is used as the metric for evaluation. We use the existing baselines and their codebases in VIME and Contrastive Mixup to perform all experiments. Also, we use the same network architecture and training protocol including the optimizer, learning rate schedule, etc. The implementation of our proposed Progressive VIME and Progressive Contrastive Mixup can be found at GitHub page 4 .

B. Experiments
This section evaluates the proposed progressive architectures on all the aforementioned datasets. To explore the efficacy of our progressive semi-supervised framework on limited labeled data in practical setting, we compare the accuracy with the state-of-the-art methods by varying components in both proposed progressive architectures.
1) Progressive VIME: We report the performance of original VIME method along with Progressive VIME in Table III. We evaluate the performance gain of each component in VIME and Progressive VIME. This evaluation shows how VIME works on the CPR and how the Progressive VIME performs versus VIME.
The components and the training methods that are shown in the Table III for evaluation are described in the following: • Supervised Model: Train the predictor in the second step using the labeled data. • VIME:Self-Supervised: The encoder is trained in step 1 and the predictor is trained in step 2. Only labeled data is used in the second step to train the predictor. • VIME:Semi-Supervised: Similar to self-supervised method, there are 2 steps in training. Difference is that in the second step, the data augmentation and consistency loss are used for training the predictor. • Progressive VIME:Self-supervised with update: Adding Update Policy and several runs of training to the VIME:self-supervised method. All pseudo-labels are used for updating the representation (CPR). • Progressive VIME:Semi-supervised with update: Adding Update Policy and several runs of training to the VIME:semi-supervised method. All pseudo-labels are used for updating the representation. • Progressive VIME:Self-supervised with refinement: Adding Update Policy and several runs of training the VIME:self-supervised method. Updating the representation in Update Policy component using more confident pseudo-labels in the predictor. • Progressive VIME:Semi-supervised with refinement: Adding Update Policy and several runs of training to the VIME:semi-supervised method. Selection of pseudolabels based on the confidence of the predictor to update the representation in Update Policy component. Table III shows the proposed Progressive VIME with refinement outperforms the other methods, resulting in the best prediction performance. The VIME underperforms on Drug directory dataset in comparison with supervised method, while progressive method is more robust and provide consistent improvement on all datasets. In other words, the progressive methods with update and refinement always achieve better performance compared with the original VIME and supervised methods. The higher model's predictive power shows the advantage of the progressive approach in leveraging the unlabeled data and learning better representations. The progressive training with refinement performs slightly better than progressive training with update because it only keeps likely correct pseudo-labels, which results in better representation regeneration.
2) Progressive Contrastive Mixup: In this section, we compare our progressive architecture with the Contrastive Mixup using the same aforementioned datasets. We compare the performance of different models. All the models used for comparison are introduced as follows.  finement component filters the pseudo-labels only based on a threshold on the weights calculated by the label-propagation method.
-For with classifier method, refining the pseudolabels on the weights of the label-propagation method is used after the pseudo-labels agreement.
In other words, the pseudo-labels are refined twice. Table IV shows that the progressive training outperforms the original Contrastive Mixup [28] and supervised models. On Display Advertising Challenge and Drug Directory datasets, Refinement using classifier in the architecture performs the best, while on Traffic Violations dataset the best performance obtained by the proposed progressive training without a classifier and Refinement.
The results on the Display Advertising Challenge dataset show that Contrastive Mixup does not always perform better than the supervised method, but progressive training outperforms the supervised method.
Finally, both Tables III and IV show that VIME and Contrastive Mixup can not consistently improve the accuracy compared to the supervised approach on all datasets, while the progressive training outperforms the supervised, and nonprogressive methods on all datasets.
3) Ablation Study: We evaluate the Update Policy, Refinement and other components in Progressive Contrastive Mixup. The effect of each component (Classifier, Projection Network, Decode, and Label-propagation) is studied to examine how they affect final performance. Components changes are only made in the first training step, and the second step stays the same. Because the label-propagation is an important part of the Contrastive Mixup, it is used in all evaluations.
We use two terms: with update and with refinement. The with update is used when we add the update policy based on all pseudo-labels without refinement. The term with refinement means we use both update policy and refinement. In contrast, when we do not use these terms with update and with refinement, it means that they are not used and there is no regenerating data representation in the architecture. Twostep refinement mechanism is used in this experiment, where threshold on the weights of pseudo-labels generated by the label-propagation is set to 0.9 in our experiment.
We want to study the robustness of the propose framework under various combinations of the components. Mainly three different cases are compared: 1. without updating CPR; 2. updating CPR with all pseudo-labels; 3. refining pseudo-labels for updating. Three Tables VI, VII and VIII show that updating the conditional probability representation consistently outperforms the case when conditional probability representation is not updated across different models. It demonstrate the robustness of the proposed framework. In all three datasets, when refinement is added, Training with Refinement outperforms the Training without update and Training with update in majority cases. This shows that Training with Refinement uses more accurate pseudo-labels and improves the data representation regeneration. In Traffic Violations dataset, adding Training with update to the Contrastive Mixup performs best. In Display Advertising Challenge dataset, adding the proposed Classifier and Training with Refinement achieves the best result, which improves Contrastive Mixup by 5.28%. The proposed framework achieves the highest improvement in Display Advertising Challenge dataset because cardinality of categorical features are really high. In Drug Directory dataset, the best performing method adds the proposed Classifier and Training with Refinement but removes the contrastive learning component. In both Display Advertising Challenge and Drug Directory datasets, the proposed classifier brings positive effective to the model architecture. In Traffic Violation and Drug Directory dataset, we observe that Training with update underperforms Training without update probability because the noise introduced by wrong pseudo-labels.

V. CONCLUSION
In this paper, we advocate rethinking the semi-supervised learning problem on tabular domain from the feature representation perspective, especially for high-cardinality categorical data. dInstead of sticking to the most popular representation (one-hot encoding, label encoding), we propose using conditional probability representation and keep upgrading the representation during training. Upgrading representation is realized by leveraging labels for labeled data and pseudo-labels for unlabeled data. Refinement mechanism is also proposed to reduce the noise introduced to the feature representation through pseudo-labels. We demonstrate the effectiveness and robustness of the proposed framework by incorporating it with the different algorithms and evaluating it on different datasets. Note that, the progressive updating framework proposed in this work is not contradictory to the existing semi-supervised learning approaches, but complementary to help gain more understanding of the problem. We hope with the awareness of many encoding tools for tabular domain data, it becomes easier to learn meaningful representations in tabular domain.

A. Experimentation setup details
The Progressive Contrstive Mixup uses refinement threshold 0.9 on the weights of the label-propagation algorithm and the Progressive VIME uses different threshold in range [0.7, 0.9] for confidence of classifier in each dataset. In Progressive Contrastive Mixup, the weight of the loss for the classifier is 0.5.
All the setup follows what is used in the original papers (including all hyperparameters for loss functions, perturbation probability in mask generator of VIME, number of nearest neighbors in the label-propagation method of VIME and Contrastive Mixup). Except, for VIME related architectures, the size of the latent representation in the encoder is changed. We assigned number of dimensions as 46, 42, and 64 on Traffic Violations, Drug Directory, and Display Advertising Challenge datasets respectively. For reproducible results in Contrastive Mixup, we use random seed number 123,127,131,137.
All components except the label-propagation and Update Policy use fully connect layers like the settings in VIME and Contrastive Mixup.    Figure 3 shows that under different ratios for labeled samples the Progressive Contrastive Mixup outperforms Contrastive Mixup on Display Advertising Challenge dataset. Figure 4 demonstrates the effectiveness of Progressive VIME on Traffic Violations dataset on the majority of different ratios of the labeled samples. When less labeled sample are used for training (ratio smaller than 0.5), Progress VIME clearly outperforms VIME. When more labeled samples are involved in the training (ratio larger than 0.5), VIME and Progress VIME perform closely. Possible reason could be that 50% of data can already provide accurate estimation of real CPR, any additional data can not bring additional value for finding more accurate representations. It might be the reason for close performance on the Traffic Violations dataset. Overall, the proposed progressive method in most experiments on both datasets outperforms the non-progressive methods.
2) Performance on Target Encoding: The proposed framework is not restricted to only conditional probability representation. It is flexible as long as the representation method has baked target label information into the efficient representation. One other encoding method called target encoding (Target Encoding [7]) is very similar to CPR since it also considers target label in the representation. Additional experiments in Table IX have been performed to study the performance of Target Encoding [7]) in semi-supervised learning problems, similar effect has been observed.