Add Search Engine Group in feeds
Plus Target Version:
It would be good to get a search engine feed so you can either block them or use them as a whitelist. I have included links so people can check my research is valid.
Here are some to start the group:
- Googlebot https://developers.google.com/search/apis/ipranges/googlebot.json
- Bingbot https://www.bing.com/toolbox/bingbot.json
- DuckDuckBot https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/
If anyone has any other search engines or crawlers, please add them here