"free speech means we hear everything in the marketplace of ideas!" what if the market place of ideas decides that nazi ideas are bad?

A whole boatload of European countries outlawed the use of swastikas and other symbols appropriated by white power groups. Germany declared certain political parties unconstitutional. It seems like a small price to pay to protect the vulnerable.