["I'm anxious to meet my wife's ex-husband. Hopefully it will be civil and not awkward!", "I can definitely understand how you would be worried about that.  I think I'd have to decline - LOL!  You are a good person for being willing to meet with him.", 'Yeah, there is a kid in the picture so we have to meet at some point! I wish I could decline, haha.']
Well, yes if there is a child involved, then you've got to deal with him at some point.  I hope it goes well for you though.
I'm anxious to meet my wife's ex-husband.  Hopefully it will be civil and not awkward!