On Suspicious Coincidences and Pointwise Mutual Information
  Barlow (1985) hypothesized that the co-occurrence of two events $A$ and $B$
is "suspicious" if $P(A,B) \gg P(A) P(B)$. We first review classical measures
of association for $2 \times 2$ contingency tables, including Yule's $Y$ (Yule,
1912), which depends only on the odds ratio $\lambda$, and is independent of
the marginal probabilities of the table. We then discuss the mutual information
(MI) and pointwise mutual information (PMI), which depend on the ratio
$P(A,B)/P(A)P(B)$, as measures of association. We show that, once the effect of
the marginals is removed, MI and PMI behave similarly to $Y$ as functions of
$\lambda$. The pointwise mutual information is used extensively in some
research communities for flagging suspicious coincidences, but it is important
to bear in mind the sensitivity of the PMI to the marginals, with increased
scores for sparser events.
