I tried, but it does not remove them as a follower after I unblock them again.

They should probably sort that out since that's very dumb or just add that Remove Follower function