Rejection is hard. Past experience is difficult to overcome. And if I did ask if I could come, I would then feel like they only said "yes" to be polite

God yep. And that just feels awful having that hanging over your head.