You don't get to punch people in the face, even if their ideas are odious. You don't. We want a civil society, where ideas are met with other ideas. We don't want a society that encourages thuggish behavior, where if someone has politics different from yours, you get to beat them up. Aside from it just being morally wrong in itself to assault people, there's the practical consideration that in a society where ideas are met with fists, one is as likely to be the punched as the puncher, and it's no fun to be punched in the face.