{'fraction_non_alphanumeric': 0.03337481615567372, 'fraction_numerical': 0.01380246634234642, 'mean_word_length': 4.928459646769506, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Can lies be faked? Comparing low-stakes and high-stakes deception video datasets from a Machine Learning perspective

Despite the great impact of lies in human societies and a meager 54% human accuracy for Deception Detection (DD), Machine Learning systems that perform automated DD are still not viable for proper application in real-life settings due to data scarcity. Few publicly available DD datasets exist and the creation of new datasets is hindered by the conceptual distinction between low-stakes and high-stakes lies. Theoretically, the two kinds of lies are so distinct that a dataset of one kind could not be used for applications for the other kind. Even though it is easier to acquire data on low-stakes deception since it can be simulated (faked) in controlled settings, these lies do not hold the same significance or depth as genuine high-stakes lies, which are much harder to obtain and hold the practical interest of automated DD systems. To investigate whether this distinction holds true from a practical perspective, we design several experiments comparing a high-stakes DD dataset and a low-stakes DD dataset evaluating their results on a Deep Learning classifier working exclusively from video data. In our experiments, a network trained in low-stakes lies had better accuracy classifying high-stakes deception than low-stakes, although using low-stakes lies as an augmentation strategy for the high-stakes dataset decreased its accuracy.


Introduction
Lies are widespread throughout human societies and it is estimated that humans lie more than twice a day [13]. In contexts such as Justice, lies can have meaningful consequences, with 53% of the exonerations in the United States between 1989 and 2012 involving deception [24]. Therefore, researchers have been interested in Deception Detec-tion (DD) for centuries [39], and, since human accuracy for this task is merely 54% [8], attempts have been made to build automated systems for DD, the most successful being the polygraph. However, despite its widespread use in courts, the polygraph has proven to be an unreliable system [17,32] and its usage in official settings is currently not recommended.
The quest for automated DD systems has been reinvigorated by the success of Machine Learning (ML) applications and many researchers have directed efforts toward this field. Even though some papers achieved impressive accuracies above the 90% mark [9,11,14,20,36,40,58,61], the proper application of ML systems to DD in real-life is still not viable due to data scarcity [39,47]. Few video-based DD datasets have been created and even fewer made available for public use, hindering the development of proper techniques and experiments in the area. But most importantly, the existing datasets lack sufficient data for use in real-life settings, with no public DD dataset having more than 400 videos, which is far from enough to properly train a system capable of correctly identifying lies outside of experimental settings [39].
Aside from the inherent challenges of building datasets based on human subjects such as ethics and biases, DD has a conceptual barrier that complicates matters further: the distinction between low-stakes and high-stakes lies [59]. According to the literature [27], there is a significant distinction between lies told in different contexts since cues for deception will emerge in high-stakes situations in which the liars develop a stronger emotional response, a phenomenon known as non-verbal leakage. According to Frank and Ekman [19], "it is the presence of these emotions, such as guilt, fear of being caught, and disgust, that can betray the liar's deception when they are leaked through nonverbal behaviors such as facial expressions [...] or voice tone". In this sense, trials, interviews or negotiations could be considered as high-stakes scenarios, while contexts that do not elicit such strong reactions could be seen as low-stakes: truth games, roleplaying or even situations in which participants are instructed to lie by researchers.
Even though low-stakes lies are more frequent in human lives and can be easily simulated (faked) in controlled settings, the practical interest for automated DD falls on highstakes lies due to their potential impact. Based on that, solving the issue of data scarcity for DD would mean creating a large and diverse public dataset of high-stakes lies. However, the acquisition of this type of data poses ethical and privacy issues and, most importantly, given that human accuracy for DD is so low, any labeling of these samples should be based on unassailable proof that testimonies were either truthful or deceptive.
Ideally, ML systems would be trained on fake lies easily acquired in controlled settings (low-stakes lies) and applied to high-stakes DD scenarios, though the conceptual distinction between these two kinds of lies has blocked such possibility. We then arrive at an impasse since, on one hand, high-stakes datasets are needed for real-life applications but prove difficult to be built, and, on the other hand, low-stakes datasets can be created more easily from lies told in fake scenarios yet have limited practical relevance.
Despite being a possible solution to such an impasse, to the best of our knowledge, no study has evaluated if there is, indeed, a significant distinction between low-stakes and high-stakes deception from an ML perspective, from which two questions of practical significance arise: "Can an ML system trained in a low-stakes deception dataset be used to classify high-stakes lies?" and "Can low-stakes lies be used for data augmentation in high-stakes deception datasets?".
Hence, this paper investigates this distinction by comparing the performance of a Deep Learning system with two datasets: Real-life Trial (RLT) [53], a high-stakes dataset with footage from trials and currently considered the standard video-based DD dataset [48]; and Box of Lies (BoL) [55], a low-stakes dataset with videos taken from a truth game on the television show "The Tonight Show Starring Jimmy Fallon ®". Focusing exclusively on video data from each dataset and performing binary classification, a network based on the Slowfast architecture [16] is created and several experiments are performed to evaluate if the aforementioned distinction does indeed hold true based on experimental results. We hope that, by examining this distinction, researchers may redirect their efforts toward building largescale deception datasets while making the best use of the limited data available in this important application area.
Thus, this paper is organized as follows: Sec. 2 discusses the existing public video-based DD datasets and ML papers that use them, Sec. 3 describes our methodology, Sec. 4 presents and discusses the results from our experiments and Sec. 5 summarizes our findings.

Deception detection datasets
Back in 2012, Gokhman et al. [21] and Fitzpatrick and Bachenko [18] discussed the development of a standard Deception Detection (DD) dataset, since none was available at the time, which required every paper to build its own dataset, limiting comparisons between studies and hindering the advancement of the field. Since then, many papers have been published with experiments conducted in video-based datasets which were not publicly available [6,7,39,51,56] due to the sensitive nature of the task. However, four video-based public datasets (presented in Tab. 1) have been created and are discussed here.
We focus on datasets with videos for their simplicity of being deployed in real-life situations, in contrast to Electroencephalogram (EEG) or Functional near-infrared spectroscopy (fNIRS) which require special equipment to acquire data, and for the fact that they present the best results, in relation to audio and text, in papers that compare the performance of unimodal classification [40,48,61]. Yet, several papers have worked with DD datasets from other modalities (some of which are publicly available), such as EEG [1,3,15], fNIRS [30], audio [22,49,62] and text [10,12,29,31].

Low-stakes datasets
To the best of our knowledge, there are currently three publicly available video-based low-stakes DD datasets: Bag-of-Lies [26], Miami University Deception Detection Dataset (MU3D) [46] and Box of Lies (BoL) [55].
Bag-of-Lies [26] is described by its authors as a "casual deception" dataset, containing 325 videos of 35 volunteers that had to describe images from a selected set being free to be truthful or deceptive. Being a multimodal dataset, it has visual, audio and gaze data (acquired with an eye tracker) for all testimonies, and EEG data for 201 videos.
The MU3D [46] was created with the goal of being a standardized, unbiased and balanced dataset, being made of 320 videos from 80 subjects (20 Black female, 20 Black male, 20 White female, and 20 White male), each one having 2 lies and 2 truths. In each video, participants describe people they like and people they dislike, with all videos having audio and accompanying transcriptions.
The BoL dataset [55] features videos from participants playing the "Box of Lies" game in the "The Tonight Show Starring Jimmy Fallon ®" television show. Played between a celebrity guest and the show's host, the game consists of multiple rounds in which players describe (deceptively or truthfully) an object which is hidden from the other player, with the listener having to guess whether the description was a truth or a lie. Sample frames are presented in Fig. 1.
The BoL dataset provides transcriptions, annotations on gestures and facial displays, as well as labels for small segments (called utterances) within each round played, with the  videos being available on YouTube in an official playlist of the television show. When the Box of Lies paper [55] was published, 68 rounds were available, though currently this number has gone up to 93 rounds. Using these 68 rounds, the original Box of Lies paper achieves an accuracy of 65% with a Random Forest classifier on multimodal data, while Zhang et al. [64], also using Random Forest, achieves 73% accuracy for multimodal classification and 67% for classification solely on videos.

High-stakes datasets
To the best of our knowledge, the only publicly available video-based high-stakes DD dataset is the Real-life Trial dataset [53] (RLT), which makes it the standard dataset for papers that apply ML techniques to DD [48]. It originally consists of 121 videos from 56 unique individuals taken from trials in which truthful or deceptive testimonies were verified by the police as such, thus allowing the dataset creators to objectively label each video. It is available on one of its authors' page [50] and accompanies transcriptions for each video and annotations on non-verbal behavior such as facial displays and hand gestures. Sample frames from the dataset are presented in Fig. 2.
Several papers have used the RLT dataset for ML ex- periments, which have their results and techniques summarized in Tab. 2. Among these, data augmentation strategies, classifiers, feature selection and preprocessing techniques vary greatly. With a single exception [11], all papers that combine video, audio and transcripts achieve better results in multimodal classification than in unimodal classification from videos, showcasing the strength of combining multiple types of data for this task. But most importantly, all papers achieve results above human accuracy for DD (54% [8]) and the accuracy measured for this specific set of videos by the creators of RLT in a set of experiments with human volunteers (last row of Tab. 2).
However, according to Belavadi et al. [6], "there is insufficient evidence that AI systems for detecting deception are likely to achieve adequate accuracy in real-world use" and, given that the RLT dataset is small, such impressive experimental results do not provide such evidence. Moreover, Mambreyan et al. [47] have shown that the RLT dataset has significant gender bias which can be exploited by an ML classifier, further highlighting the need for bigger, more diverse and less biased publicly available video-based highstakes DD datasets.

Methodology
To evaluate a possible distinction between low-stakes and high-stakes deception from a Machine Learning (ML) perspective, the Box of Lies (BoL) [55] and Real-life Trial (RLT) [53] datasets were selected. While RLT is, to the best of our knowledge, the only publicly available video-based high-stakes Deception Detection (DD) dataset, BoL is the low-stakes dataset that has the closest number of videos to RLT (considering its current number of 93 rounds), making it a fairer comparison than other low-stakes datasets which have almost thrice the number of videos of RLT.

Dataset preparation
Following the steps taken by other papers that work with RLT [9,14,33,48,52,61], we remove videos from the dataset which are deemed unsuitable for classification based solely on visual data, such as those in which the speaker's face is hidden or out of focus during most of the video, or where there are multiple people in the foreground. After that, the dataset was reduced to 110 videos portraying 51 individuals. Among the remaining videos, some had noisy frames or frames showing other people instead of the speaker, so they were edited with Kdenlive [38] and these frames were removed. The full list of removed and edited videos is avail-able in the Supplementary Material.
While the BoL dataset was originally labeled by utterance, the RLT dataset is labeled by video, which required a change in labels in BoL. Therefore, each round was taken as a single video labeled according to the veracity of the object's description and, for each video, the frames not showing the speaker were removed with Kdenlive [38]. Despite assigning a single label to whole videos (which contain truthful and deceptive segments), such labeling scheme follows the scheme from RLT (allowing for better comparison between datasets) and removes subjectivity from the process. By contrast, the original labeling by utterance categorized segments with unclear veracity as deceptive, which resulted in 82.2% of the samples being labeled as deceptive.
After this dataset preparation step, the resulting BoL and RLT datasets were such as presented in Tab. 3, from which it can be inferred that an ML system trained in RLT should have better results than a similar one trained in BoL, since RLT has more data while also being more diverse (more individuals) and balanced. To illustrate such balance, the individual with the highest number of videos in RLT is depicted in 21 videos ( shown in Tab. 4. Despite not being a desirable property of the dataset, RLT has a significant gender bias, which should also make ML systems trained on it have better results due to bias exploitation. As a final preparation step, both datasets are enriched with copies of their videos flipped horizontally, doubling the number of videos in each dataset.

Network training
Focusing solely on DD from videos (as discussed in Sec. 2), we use the Slowfast [16] architecture to perform binary classification. This architecture was chosen for its balance between classification accuracy and low computational cost, achieving better results than other Deep Learning video recognition architectures that have similar computational cost, such as Hidden TSN [65], TSM [44], TEINet [45], MSNet [42], TEA [43], STM [34] and CSN [57].
Such criterion was used due to limited computational power and time allocation of resources for our experiments, which were performed in a shared computer with an Intel Core i3-10100F 3.6 GHz CPU, 16 GB 2666 MHz RAM and a GTX 1650 4 GB GPU. These limitations also prompted us to use the Slowfast implementation available in the Glu-onCV framework [25] for its optimizations and ease of implementation, as well as performing non-exhaustive hyperparameter search prior to k-fold testing.
For each network trained, hyperparameter search was performed independently, with the best 5 hyperparameter combinations later being used with 5-fold testing. Since non-exhaustive search was performed, the 5 best combinations were used for testing as a means of overcoming a possible deficit from the non-exhaustive search. An 80/20 split was performed for hyperparameter search, with an initial set of values being tested, and the following combinations being chosen according to the previous combinations' accuracy on the validation set. Therefore, hyperparameter values that did not perform well were assessed but soon discarded. Such strategy meant not all hyperparameter combinations were evaluated, but results on the validation set were considered satisfactory given that each combination had its results thoroughly analyzed.
For most trials, 100 epochs were used, but for combinations that had not converged by 100 epochs, greater numbers were also evaluated. The batch size was limited to 1 since there was not enough memory available for greater values, a limitation which also inhibited the use of different Slowfast backbones apart from ResNet50 [28]. The list of all hyperparameter combinations evaluated is available in the code repository for this paper.

Experimental setup
Our experimental setup is presented in Fig. 3. As a first step, hyperparameter search is performed considering each dataset independently. The 5 hyperparameter combinations with the highest validation accuracies are then used for 5fold testing with their respective datasets. Even though both use the Slowfast architecture, the combination that yields the best accuracy for RLT is named Net 1 while the one which yields the best accuracy for BoL is named Net 2. Both are considered optimized for their respective datasets. Figure 3. Experimental setup used to evaluate differences between a low-stakes deception dataset (Box of Lies -BoL [55]) and a highstakes one (Real-life Trial -RLT [53]). The network used was Slowfast [16], with three distinct training configurations based on the hyperparameter searches performed for the distinct training sets (RLT, BoL and RLT+BoL). Letter tags are used in each Experiment for later reference.
Trained on the RLT dataset, Net 1 is then used for inference on the BoL dataset while Net 2, having been trained on BoL, is used for inference on RLT (Step 2 in Fig. 3). Since both Net 1 and Net 2 were optimized for their datasets, this step allows us to evaluate whether each kind of deception dataset (high-stakes and low-stakes) can be used as training data for inference on the other kind.
Finally, both datasets are combined (Step 3 in Fig. 3) and hyperparameter search is performed for the mixed dataset, to evaluate the efficacy of combining the two kinds of deception datasets. The 5 best hyperparameter combinations are used for 5-fold testing and the best one is named Net 3.
Aiming to evaluate possible data augmentation strategies, both Nets 1 and 2 are retrained (using their previous set of hyperparameters) on the mixed dataset and tested with their original datasets.

Results and discussion
Given that 5-fold testing was performed for all experiments except cross-testing, a total of 27 runs were conducted for testing. Hyperparameter search for Nets 1, 2 and 3 required 44, 176 and 77 runs, respectively. Therefore, a total of 324 runs were performed. If an exhaustive hyperparameter search was performed along with 5-fold (exhaustive search being done for each fold), the number of runs would be above 10000, which would not be viable given our previously described computing power limitations. Even though our approach does not guarantee optimal results, it achieves adequate accuracies with a fraction of the resources needed. The results from hyperparameter search runs are available on the code repository of this paper.

Network optimization
The hyperparameter search for Nets 1, 2 and 3 was performed and the 5 best combinations of each were later used for 5-fold testing. The combination which had the best result was considered the one that better optimized its respective network. Tab. 5 presents the hyperparameter values for each of these combinations.
The results of the optimization step for each of the networks are presented in Tab. 6, from which the assumptions raised in Sec. 3.1 hold true: the RLT dataset (Exp. A) had better results than BoL (Exp. B). Both networks had results above human accuracy for DD (54%), however both were below the results achieved by most works presented in Sec. 2, which can be attributed to experimental limitations (such as a batch size of 1) and the fact that the Slowfast architecture was originally created for the Kinetics-400 [37] dataset, which has considerably more data with 306245 videos, and often showed overfitting in our experiments.
The combination of both datasets (Exp. E), however, had its results closer to those from BoL (Exp. B) than those from RLT (Exp. A), indicating that an increase in training data, by itself, is not enough to achieve better results. It is unclear to us whether these results occurred due to an improper combination of data with a significant semantic difference (low-stakes lies and high-stakes lies) or because of properties of these specific datasets (gender bias, category bias, imbalance of the number of videos by individual).

Real-life Trial (Net 1) results
The results from experiments done with Net 1 are presented in Tab. 7, which highlights the previous considerations on the difficulty of each dataset. While the best results are seen when RLT is trained and tested on its own (Exp. A), cross-testing on BoL (Exp. C) had results below human accuracy, and adding BoL videos as an augmentation strategy (Exp. F) ends up reducing the original accuracy (Exp. A).
Even though this cross-testing scenario does not hold much practical purpose since pragmatic interest lies in testing being performed with high-stakes lies and training with low-stakes lies, the evaluation of the augmentation strategy meets such interests. That being so, in our experiments, using low-stakes lies to augment a high-stakes deception dataset resulted in worse performance than working with the high-stakes dataset on its own. The results suggest that this behavior occurred due to semantic differences between these datasets (high-stakes and low-stakes deception). However, it is important to point out that the reduction in performance might have been caused by the aforementioned properties within the datasets which are unrelated to these semantic differences.

Box of Lies (Net 2) results
Experiments performed with Net 2 are shown in Tab. 8, which presents noteworthy results. Even though training and testing in BoL reached a low accuracy (Exp. B), crosstesting with RLT (Exp. D) increased this accuracy and using RLT to augment data on BoL (Exp. G) significantly improved results.
In view of the considerations previously presented on the difficulty of each dataset, the results from Exp. G can be understood from the perspective that videos from RLT make training easier, allowing the network to better identify patterns within the testing dataset (BoL). From another point of view, it might be argued that high-stakes deception are more easily identified than low-stakes, making them suitable for data augmentation.
However, Exp. D's results are not entirely conclusive. It might be argued that, since training was performed in a difficult dataset, testing in an easier one would improve results. Despite that, such an experimental scenario perhaps holds the greatest practical interest due to the ease of acquiring low-stakes deception data and the difficulty of doing so for high-stakes data (which gets the most practical interest for DD systems). From our experiments alone, it could be said that an ML system trained with low-stakes lies could be acceptably applied to a set of high-stakes lies. However, due to our experimental limitations and limited data, such findings cannot be generalized for all DD datasets.

Conclusion
We set out to tackle the issue of data scarcity in Deception Detection (DD), aiming to shed light on the conceptual distinction between high-stakes and low-stakes lies by presenting numerical data that might support researchers on the task of building new datasets for such an important application area. Currently, datasets for DD are either built with high-stakes or low-stakes lies and applications trained in one kind theoretically can only be used for inference on lies of that same kind. Through our investigation, we evaluated this distinction by experimentally comparing two datasets: Real-life Trial (RLT) [53], a high-stakes DD dataset, and Box of Lies (BoL) [55], a low-stakes DD dataset. Different experiments were performed with the Slowfast [16] architecture and their results were used to analyze whether such a distinction holds true. From these experiments we found that: 1. The network trained in high-stakes lies performed better than the network trained in low-stakes lies; 2. Combining both datasets into a single one had worse results than working with the high-stakes dataset on its own; 3. Using the network trained in high-stakes deception for inference on low-stakes deception had results below human accuracy; 4. Using low-stakes deception as a data augmentation strategy for the high-stakes dataset did not improve results; 5. The network trained in low-stakes lies had better accuracy classifying high-stakes deception than low-stakes; 6. Using high-stakes lies as a data augmentation strategy for the low-stakes dataset showed a significant improvement in results.
Given these findings, we conclude that there is a clear distinction between the RLT and BoL datasets. However, similar experiments should be performed with different datasets to assess whether these results stem from differences in these datasets' incidental properties (mainly bias and amount of data) or from a deep semantic difference in their data (low-stakes and high-stakes lies). Nonetheless, to the best of our knowledge, we have shown, for the first time, that a low-stakes DD dataset can be acceptably used to train a Machine Learning (ML) classifier created for inference on high-stakes deception data.
Finally, we highlight the need for bigger, less biased and more balanced publicly available video-based DD datasets. Despite impressive results in experimental settings, ML DD systems are not yet ready for deployment in real-life settings given the lack of sufficient data for their proper training.