To be fair, Raylan then stuffed his prisoner into the trunk of his car, and the show mostly consists of him running into assholes all day.

Right - It's been a minute since I've done a rewatch, but I thought it was Art's (if I'm remembering right) way of telling Raylan he was the asshole. I could have it wrong, but the sentiment still stands.

Raglan is an asshole though! He’s charming, but he frequently escalated confrontations unnecessarily and the show wasn’t blind to that.