Well, the opposition and their "No" friends, waged a very effective war of fear and misinformation. And, Australia has a long history of gross mistreatment and disregard of the indigenous population. Not to mention a solid history of racism in general.

White Australia policy - Wikipedia

Can't disagree with any of that. The few Australians I've known well all deny it's a racist country, but that's not how it looks from outside. Then again, many Brits deny the UK's racist and it very obviously fucking well is.