Obviously Vance's views are a dog whistle, but isn't there something to the idea that if American population actually started shrinking (like Japan) we might see some home values crater a bit?

Potentially, but keep in mind that local housing market conditions would probably play a larger role.