From what I’ve heard, his wife is actually an only child, based on some sort of memoir. So this may be just a fantasy (though certainly a weird one). But take that with a grain of salt, I haven’t had a chance to confirm it.