So, since ML is just an auto-correlation tool, I then feed my model total garbage to see if there's any bad correlations it learned that kind of resembled my real set. I wanted to see if not "indecisive" for the garbage, at least no internal pattern.

What I instead found was that nature photos, particularly farm photos, were "metal buildings". (We called this the "metal cow" problem from the first high confidence photo of a cow in a field). Turns out those were generally poorly kept up in my training set, and had overgrowth visible.