And you absolutely should. Mental energy is precious and limited. We should all refuse to spend it on people who do not add to our enjoyment of free time.