That's definitely valid, and I've done weeding work in national parks for exactly this reason. On the other hand, some invasives are thriving in places that we've changed too drastically (dams, etc) for native species to live anymore. We can't get them out without larger-scale restoration.