Putting in manicured structures didn't tag as metal, and palm trees obscuring a skyscraper made that metal. My training set needed more work (and a "none of that above" bin as it turned out). Just saying. #crackergate

What I instead found was that nature photos, particularly farm photos, were "metal buildings". (We called this the "metal cow" problem from the first high confidence photo of a cow in a field). Turns out those were generally poorly kept up in my training set, and had overgrowth visible.

So, since ML is just an auto-correlation tool, I then feed my model total garbage to see if there's any bad correlations it learned that kind of resembled my real set. I wanted to see if not "indecisive" for the garbage, at least no internal pattern.

In recognition of Crackergate2024, here's a story. I once was training an ML model to visually identify building structure material to shore up holes in a dataset. It worked quite well on my out of sample validation set, scoring high 80s to low 90s on a set with a ~30% fill rate.

There are enough people doing an exodus that we'd have to be cross reading anyway! I create ML systems as part of my job and it's horrifying to me that my narrowly tailored systems apparently undergo more rigorous testing than FAANG production releases.