The phrase let alone is used to emphasize that if other more significant or pressing things are not possible  a lesser thing certainly is not or cannot either but in this context, the speaker is emphasizing that attracting anyone would not have been possible, except attracting a man which is contradictory.
Context: I wouldn't have believed myself capable of attracting anyone, except for such a man.
Hypothesis: I wouldn't have believed myself capable of attracting anyone, let alone such a man.

Given this context, is the hypothesis true (Yes/No/I don't know)?