{'fraction_non_alphanumeric': 0.03333333333333333, 'fraction_numerical': 0.025757575757575757, 'mean_word_length': 5.320574162679426, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Contrastive Divergence In Gaussian Diffusion Processes

Contrastive Divergence (CD) is a recent learning rule found to work very well in practice despite its still unclear theoretical underpinnings (Carreira-Perpiñan and Hinton, 2005; Hinton, 2002; Hinton and Salakhutdinov, 2006; Hyvärinen, 2006; MacKay, 2001; Roth and Black, 2005; Williams and Agakov, 2002; Yuille, 2004). This paper presents a theoretical analysis of CD in Gaussian Diffusion Processes (GDP), i.e, a linear, continuous, time, continuous state version of recurrent neural networks. These networks are of interest for two reasons: (1) Powerful analytical tools exist that allow comparing the behavior of CD to other algorithms, like Maximum Likelihood estimation. (2) Many non-linear systems of interest for which CD has proven useful have multiple attractors about which the systems locally behave like GDPs. Thus the analysis of the GDP case may provide some clues for a better understanding of CD in more general cases. The analysis presented here shows that convergence of CD is guaranteed if the first moment of the GDP is at equilibrium. If this is the case CD and maximum likelihood estimation converge to the same solution, otherwise CD may converge to arbitrarily different solutions from maximum likelihood estimation, or diverge altogether.