Thanks. Gonna fix that on my website now.
This isn’t really how crawlers work. Crawlers generally crawl the entire public web by default already, but you can add a text file onto your server that prevents it -when a crawler decides to care about that-. This page just provides a way to add that opt out (or not). It’s not an opt in.
Oh yikes! Thanks for the heads up~ I'll be changing that in my settings right now. 😬
Thank you for the heads up! I wish I was surprised but I'm really not.
Thank you!
Gross.
I don't think they opted you in. Rather, they gave you an option to opt out, which, when disabled, simply adds lines in robots.txt that tell AI bots they should not crawl your site.
Thanks for the heads up. That's also not a menu I can find on the mobile app so for others you have to log into desktop by the looks of it too.
"Crawlers"? I thought Spiders were bad enough.