Edit the following text to be less complicated:
I'm afraid that, even though I might try and deny it, all of our neighbours would have told you.
I fear that all our neighbours would tell you that, even if I were to attempt to deny it.