pfSense » pfSense Packages

It would be good to get a search engine feed so you can either block them or use them as a whitelist. I have included links so people can check my research is valid.

Here are some to start the group:

• Googlebot https://developers.google.com/search/apis/ipranges/googlebot.json
  • https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
  • https://searchengineland.com/google-publishes-the-list-of-googlebot-ip-addresses-375935
• Bingbot https://www.bing.com/toolbox/bingbot.json
  • https://www.bing.com/webmasters/help/Verify-Bingbot-2195837f
  • https://searchengineland.com/microsoft-list-of-bingbot-ip-addresses-released-376039
• DuckDuckBot https://help.duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/

If anyone has any other search engines or crawlers, please add them here