Not to put too fine a point on it means to speak bluntly, which is what the speaker is doing when they say the man looked like a bloke who broke heads for a living.
Consider the statement "To speak frankly, sir, the man looked like a bloke who broke heads for a living.". Is the statement "not to put too fine a point on it, sir, the man looked like a bloke who broke heads for a living." false? Yes/no/unknown