To not put too fine a point on it means to speak bluntly, which is what the speaker does in this context.
Consider the statement "Er... well, to speak frankly, yes, I said.". Based on this "Er... well, not to put too fine a point on it, yes,I said." a true statement? Yes, no, or I don't know?