Both the speaker's referent and the semantic referent would be the kind lover; only if had misunderstood A's use as attributive could he have used 'he' to refer to the husband, but such a misunderstanding is excluded by the second part of B's utterance.