{'fraction_non_alphanumeric': 0.026619343389529725, 'fraction_numerical': 0.001774622892635315, 'mean_word_length': 4.623441396508728, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Predicting the Treatment Status

Many studies in social sciences deal with treatment effect models. Usually there is a treatment variable which determines whether a particular individual (or more generally a unit of observation) was affected by some impact (the binary case) or the intensity of this impact. A researcher is interested in evaluating the effect of the treatment measured as some function of the joint distribution of y (i) 1 , the potential outcome for individual i conditional on being treated, and y (i) 0 , the potential outcome conditional on not being treated. There are many papers devoted to studying this setup due to a very fundamental problem that for each individual i either y (i) 1 or y (i) 0 is observed in the data, but never both. In my project I consider a somewhat different problem. What if we don’t know who has got the treatment and who has not? For example, suppose that we organize a lottery in which every participant draws a lottery ticket from a big jar. Some of the tickets are the winning ones (the individual is treated) while some aren’t (the individual is not treated). We cannot observe whether a particular person has won, but we can observe some outcome potentially affected by the realization of the lottery. Can we infer anything about the treatment? To be precise, I ask two questions: (i) Can we guess the treatment status? (ii) Is it possible to measure the treatment effect even though we don’t observe the treatment dummy? I approach this problem as an unsupervised learning problem. Although the treatment status is observed in the data that I use, I assume that in practice it won’t be and there will be no way to train the model. The main tool in my analysis is what I call the “Hidden Variable Model” (HVM). Essentially, this is a hidden Markov model with the main difference being that there is no Markov structure and instead of observing a number of consecutive realizations for one individual we observe a number of “independent” individuals. The hidden state is exactly the treatment status which we don’t observe. We may also know some additional details of the assignment process or how the treatment affects the outcome. For example, we may know that the treatment status was determined