I call it "Mind your own business." It should never be a concern to anyone else. There are hundreds of reasons for not having a child, from being unable to have a child, to not finding it a necessary part of their life. No one else should be able to question it.

I don't think it is a concern and if people knew just how little everyone else actually thinks about them on a daily basis then they'd stop worrying about what other people think, because they really aren't thinking about anyone but themselves much at all.