['I was afraid when my dad used to scare me and my brothers']
Was he just playing tricks on you, to get you to laugh? Or was he being malicious about it?
I was afraid when my dad used to scare me and  my brothers