Towards Learning Sparsely Used Dictionaries with Arbitrary Supports
  Dictionary learning is a popular approach for inferring a hidden basis or
dictionary in which data has a sparse representation. Data generated from the
dictionary A (an n by m matrix, with m > n in the over-complete setting) is
given by Y = AX where X is a matrix whose columns have supports chosen from a
distribution over k-sparse vectors, and the non-zero values chosen from a
symmetric distribution. Given Y, the goal is to recover A and X in polynomial
time. Existing algorithms give polytime guarantees for recovering incoherent
dictionaries, under strong distributional assumptions both on the supports of
the columns of X, and on the values of the non-zero entries. In this work, we
study the following question: Can we design efficient algorithms for recovering
dictionaries when the supports of the columns of X are arbitrary?
  To address this question while circumventing the issue of
non-identifiability, we study a natural semirandom model for dictionary
learning where there are a large number of samples $y=Ax$ with arbitrary
k-sparse supports for x, along with a few samples where the sparse supports are
chosen uniformly at random. While the few samples with random supports ensures
identifiability, the support distribution can look almost arbitrary in
aggregate. Hence existing algorithmic techniques seem to break down as they
make strong assumptions on the supports.
  Our main contribution is a new polynomial time algorithm for learning
incoherent over-complete dictionaries that works under the semirandom model.
Additionally the same algorithm provides polynomial time guarantees in new
parameter regimes when the supports are fully random. Finally using these
techniques, we also identify a minimal set of conditions on the supports under
which the dictionary can be (information theoretically) recovered from
polynomial samples for almost linear sparsity, i.e., $k=\tilde{O}(n)$.
