And when I think of non-binary people (I hope this isn’t a case of non-binary-phobia, if so please call me out on that!) I can think of having a relationship with an AFAB person but not an AMAB person.

Sadly this is a binary and cis-centric misunderstanding of what non-binary is. AFAB and AMAB refer to assignments at birth. They say nothing about what genitals a person has now, how their gender is presented, or how they would interact with you.