{'fraction_non_alphanumeric': 0.023255813953488372, 'fraction_numerical': 0.0035778175313059034, 'mean_word_length': 5.373219373219373, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Techniques to Produce and Evaluate Realistic Multivariate Synthetic Data

Background Proper data modeling in biomedical research requires sufficient data for exploration and reproducibility purposes. A limited sample size can inhibit objective performance evaluation. Objective We are developing a synthetic population (SP) generation technique to address the limited sample size condition. We show how to estimate a multivariate empirical probability density function (pdf) by converting the task to multiple one-dimensional (1D) pdf estimations. Methods Kernel density estimation (KDE) in 1D was used to construct univariate maps that converted the input variables (X) to normally distributed variables (Y). Principal component analysis (PCA) was used to transform the variables in Y to the uncoupled representation (T), where the univariate pdfs were assumed normal with specified variances. A standard random number generator was used to create synthetic variables with specified variances in T. Applying the inverse PCA transform to the synthetic variables in T produced the SP in Y. Applying the inverse maps produced the respective SP in X. Multiple tests were developed to compare univariate and multivariate pdfs and covariance matrices between the input (sample) and synthetic samples. Three datasets were investigated (n = 667) each with 10 input variables. Results For all three datasets, both the univariate (in X, Y, and T) and multivariate (in X, Y, and T) tests showed that the univariate and multivariate pdfs from synthetic samples were statistically similar to their pdfs from the respective samples. Application of several tests for multivariate normality indicated that the SPs in Y were approximately normal. Covariance matrix comparisons (in X and Y) also indicated the same similarity. Conclusions The work demonstrates how to generate multivariate synthetic data that matches the real input data by converting the input into multiple 1D problems. The work also shows that it is possible to convert a multivariate input pdf to a form that approximates a multivariate normal, although the technique is not dependent upon this finding. Further studies are required to evaluate the generalizability of the approach.