Free speech means the government doesn't have the right to prosecute you for controversial opinions, not that you have the right to say whatever the fuck you want without consequence. Nor does anybody have the obligation to listen to your bullshit

Further, anyone who is openly a fascist or a Nazi is announcing, by definition, that they intend to bring physical harm to people. That's assault, and I have the right to defend myself against it