Website Crawlers
The website crawler index is an advanced feature that allows you to identify IP Addresses that are associated with various search engines and crawling services. Many crawlers can be malicious in nature so we keep an index of available services.
■ ■ ■
Category Applebot
Bot URL https://support.apple.com/en-us/HT204683
Vendor Apple Inc
Vendor URL https://www.apple.com
Category Yahoo! Slurp
Bot URL http://help.yahoo.com/ysearch/slurp
Vendor Yahoo! Inc.
Vendor URL http://www.yahoo.com
Category Flipboard
Bot URL http://flipboard.com/browserproxy
Vendor Flipboard
Vendor URL http://flipboard.com/
Category Twitterbot
Bot URL https://dev.twitter.com/docs/cards/getting-started
Vendor Twitter
Vendor URL http://www.twitter.com
Category PaperLiBot
Bot URL http://support.paper.li/entries/20023257-what-is-paper-li
Vendor Smallrivers SA
Vendor URL http://www.paper.li
Category BingBot
Bot URL http://search.msn.com/msnbot.htmn
Vendor Microsoft Corporation
Vendor URL http://www.microsoft.com
Category Facebook External Hit
Bot URL https://www.facebook.com/externalhit_uatext.php
Vendor Facebook
Vendor URL http://www.facebook.com
Category Googlebot
Bot URL http://www.google.com/bot.html
Vendor Google Inc.
Vendor URL http://www.google.com
Category MJ12 Bot
Bot URL http://majestic12.co.uk/bot.php
Vendor Majestic-12
Vendor URL http://majestic12.co.uk
Category BitlyBot
Bot URL https://bitly.com
Vendor Bitly, Inc.
Vendor URL https://bitly.com
Category uMBot
Bot URL https://www.ubermetrics-technologies.com/
Vendor Ubermetrics Technologies GmbH
Vendor URL https://www.ubermetrics-technologies.com/
Category Rogerbot
Bot URL http://moz.com/help/pro/what-is-rogerbot-
Vendor SEOmoz, Inc.
Vendor URL http://moz.com/
Category Hatena Bookmark
Bot URL https://www.hatena.ne.jp/faq/
Vendor Hatena Co., Ltd.
Vendor URL https://www.hatena.ne.jp
Category Baidu Spider
Bot URL http://www.baidu.com/search/spider.htm
Vendor Baidu
Vendor URL http://www.baidu.com
Category Mediatoolkit Bot
Bot URL https://mediatoolkit.com
Category Infegy
Bot URL https://infegy.com/
Vendor Infegy, Inc.
Vendor URL https://infegy.com/
Category aHrefs Bot
Bot URL https://ahrefs.com/robot
Vendor Ahrefs Pte Ltd
Vendor URL https://ahrefs.com/robot
Category Omgili bot
Bot URL http://www.omgili.com/Crawler.html
Vendor Omgili
Vendor URL http://www.omgili.com
Category Embedly
Bot URL https://support.embed.ly/hc/en-us
Vendor A Medium, Corp.
Vendor URL https://medium.com/
Category Tweetmeme Bot
Bot URL http://tweetmeme.com/
Vendor Mediasift
Category datagnionbot
Bot URL https://www.datagnion.com/bot.html
Vendor DATAGNION GMBH
Vendor URL https://www.datagnion.com/
Category Yandex Bot
Bot URL http://www.yandex.com/bots
Vendor Yandex LLC
Vendor URL http://company.yandex.com
Category Alexa Crawler
Bot URL https://support.alexa.com/hc/en-us/sections/200100794-Crawlers
Vendor Alexa Internet
Vendor URL https://www.alexa.com
Category WordPress
Bot URL https://wordpress.org/
Vendor WordPress.org
Vendor URL https://wordpress.org/
Category Gluten Free Crawler
Bot URL http://glutenfreepleasure.com/
Category BLEXBot Crawler
Bot URL http://webmeup-crawler.com
Vendor WebMeUp
Vendor URL http://webmeup.com
Category Seznam Bot
Bot URL http://www.mapy.cz/cz/seznambot.html
Vendor Seznam.cz, a.s.
Vendor URL http://www.seznam.cz/
Category Ask Jeeves
Vendor Ask Jeeves Inc.
Vendor URL http://www.ask.com
Category TelegramBot
Bot URL https://telegram.org/blog/bot-revolution
Category DotBot
Bot URL http://www.opensiteexplorer.org/dotbot
Vendor SEOmoz, Inc.
Vendor URL http://moz.com/
Category Pocket
Bot URL https://getpocket.com/pocketparser_ua
Vendor Read It Later, Inc.
Vendor URL https://getpocket.com/
Category Scraping Robot
Bot URL https://scrapingrobot.com/
Vendor Sprious LLC
Vendor URL https://sprious.com/
Category TurnitinBot
Bot URL http://www.turnitin.com/robot/crawlerinfo.html
Vendor iParadigms, LLC.
Vendor URL http://www.turnitin.com
Category Trendiction Bot
Bot URL http://www.trendiction.de/bot
Vendor Talkwalker Inc.
Vendor URL http://www.talkwalker.com
Category Mail.Ru Bot
Bot URL http://help.mail.ru/webmaster/indexing/robots/types_robots
Vendor Mail.Ru Group
Vendor URL http://corp.mail.ru
Category DuckDuckGo Bot
Bot URL https://duckduckgo.com/duckduckbot
Vendor DuckDuckGo
Vendor URL https://duckduckgo.com/
Category Qwantify
Bot URL https://www.qwant.com/
Vendor Qwant Corporation
Vendor URL https://www.qwant.com/
Category Grapeshot
Bot URL https://www.grapeshot.com/crawler
Vendor Grapeshot
Vendor URL https://www.grapeshot.com
Category ExaBot
Bot URL http://www.exabot.com/go/robot
Vendor Dassault Systèmes
Vendor URL http://www.3ds.com
Category Domain Re-Animator Bot
Vendor Domain Re-Animator, LLC
Vendor URL http://domainreanimator.com
Category Semrush Bot
Bot URL http://www.semrush.com/bot.html
Vendor SEMrush
Vendor URL http://www.semrush.com
Category MojeekBot
Bot URL http://www.mojeek.com/bot.html
Vendor Mojeek Ltd.
Vendor URL http://www.mojeek.com
Category Slackbot
Bot URL https://api.slack.com/robots
Vendor Slack Technologies
Vendor URL http://slack.com
Category HubSpot
Vendor HubSpot Inc.
Vendor URL https://www.hubspot.com
Category Yeti/Naverbot
Bot URL http://help.naver.com/robots/
Vendor Naver
Vendor URL http://www.naver.com
Category LinkedIn Bot
Bot URL http://www.linkedin.com
Vendor LinkedIn
Vendor URL http://www.linkedin.com
Category Heritrix
Bot URL https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
Vendor The Internet Archive
Vendor URL https://archive.org
Category SafeDNSBot
Bot URL https://www.safedns.com/searchbot
Vendor SafeDNS, Inc.
Vendor URL https://www.safedns.com/
Category Iframely
Bot URL https://iframely.com/
Vendor Itteco Software, Corp.
Vendor URL https://iframely.com/
Category Genieo Web filter
Bot URL http://www.genieo.com/webfilter.html
Vendor Genieo
Vendor URL http://www.genieo.com
Category YaK
Bot URL https://www.linkfluence.com/
Vendor Linkfluence SAS
Vendor URL https://www.linkfluence.com/
Category AddThis.com
Vendor Clearspring Technologies, Inc.
Vendor URL http://www.clearspring.com
Category Nutch-based Bot
Bot URL https://nutch.apache.org
Vendor The Apache Software Foundation
Vendor URL https://www.apache.org/foundation/
Category Linkdex Bot
Bot URL http://www.linkdex.com/bots
Vendor Mojeek Ltd.
Vendor URL http://www.mojeek.com
Category archive.org bot
Bot URL https://archive.org/details/archive.org_bot
Vendor The Internet Archive
Vendor URL https://archive.org
Category SISTRIX Crawler
Bot URL http://crawler.sistrix.net
Vendor SISTRIX GmbH
Vendor URL http://www.sistrix.de
Category MetaInspector
Bot URL https://github.com/jaimeiniesta/metainspector
Category Skype URI Preview
Vendor Skype Communications S.à.r.l.
Vendor URL https://www.skype.com
Category ICC-Crawler
Bot URL http://www.nict.go.jp/en/univ-com/plan/crawl.html
Category Snapchat Proxy
Bot URL https://www.snapchat.com
Vendor Snapchat Inc.
Vendor URL https://www.snapchat.com
Category Sprinklr
Vendor Sprinklr, Inc.
Vendor URL https://www.sprinklr.com/
Category BUbiNG
Bot URL http://law.di.unimi.it/BUbiNG.html
Vendor The Laboratory for Web Algorithmics (LAW)
Vendor URL http://law.di.unimi.it/software.php#buging
Category Daum
Bot URL http://tab.search.daum.net/aboutWebSearch_en.html
Vendor Daum Communications Corp.
Vendor URL http://www.kakaocorp.com/main
Category TraceMyFile
Bot URL https://www.tracemyfile.com/
Vendor Idee Inc.
Vendor URL http://ideeinc.com/
Category Serendeputy Bot
Bot URL https://serendeputy.com/about/serendeputy-bot
Category BrandVerity
Bot URL https://www.brandverity.com/why-is-brandverity-visiting-me
Vendor BrandVerity, Inc.
Vendor URL https://www.brandverity.com/
Category ccBot crawler
Bot URL http://commoncrawl.org/faq/
Vendor reddit inc.
Vendor URL http://www.reddit.com
Category Awario
Bot URL https://awario.com/bots.html
Vendor Awario
Vendor URL https://awario.com/
Category Lighthouse
Bot URL https://developers.google.com/web/tools/lighthouse
Vendor Lighthouse
Vendor URL https://developers.google.com/web/tools/lighthouse
Category Riddler
Bot URL https://riddler.io/about
Vendor F-Secure
Vendor URL https://www.f-secure.com
Category oBot
Bot URL https://www.xforce-security.com/crawler/
Vendor IBM Germany Research & Development GmbH
Vendor URL https://exchange.xforce.ibmcloud.com/
Category FreeWebMonitoring
Bot URL https://www.freewebmonitoring.com/bot.html
Vendor GreenWave Online, Inc.
Vendor URL http://www.greenwaveonline.com/
Category AspiegelBot
Bot URL https://aspiegel.com/
Vendor Huawei
Vendor URL https://www.huawei.com/
Category Femtosearch
Bot URL http://femtosearch.com/
Vendor Grier Forensics, LLC
Vendor URL https://www.grierforensics.com/
Category DataForSeoBot
Bot URL https://dataforseo.com/dataforseo-bot
Category Amazon Bot
Bot URL https://developer.amazon.com/support/amazonbot
Vendor Amazon.com, Inc.
Vendor URL https://www.amazon.com/
Category Bytespider
Bot URL https://bytedance.com/
Vendor ByteDance Ltd.
Vendor URL https://bytedance.com/
Category Buck
Bot URL https://hypefactors.com/
Vendor Hypefactors A/S
Vendor URL https://hypefactors.com/
Category Seobility
Bot URL https://www.seobility.net/en/faq/?category=crawling#!aboutourbot
Category GPTBot
Bot URL https://platform.openai.com/docs/gptbot
Vendor OpenAI OpCo, LLC
Vendor URL https://openai.com/
Category InternetMeasurement
Bot URL https://internet-measurement.com/
Category Sogou Spider
Bot URL http://www.sogou.com/docs/help/webmasters.htm
Vendor Sohu, Inc.
Vendor URL http://www.sogou.com
Category EFF Do Not Track Verifier
Bot URL https://www.eff.org/issues/do-not-track
Vendor Electronic Frontier Foundation
Vendor URL https://www.eff.org/
Category Expanse
Bot URL https://expanse.co/
Vendor Expanse Inc.
Vendor URL https://expanse.co/
Category BuiltWith
Bot URL https://builtwith.com/biup
Vendor BuiltWith Pty Ltd
Vendor URL https://builtwith.com/
Category Research Scan
Bot URL http://researchscan.comsys.rwth-aachen.de/
Vendor RWTH Aachen University
Vendor URL https://www.comsys.rwth-aachen.de/
Category Headline
Bot URL https://headline.com/legal/crawler
Vendor e.ventures Managementgesellschaft mbH
Vendor URL https://headline.com/
Category Pinterest
Bot URL https://help.pinterest.com/en/business/article/pinterest-crawler
Vendor Pinterest
Vendor URL https://www.pinterest.com/
Category start.me
Bot URL https://about.start.me/
Vendor start.me BV
Vendor URL https://about.start.me/
Category CensysInspect
Bot URL https://about.censys.io/
Vendor Censys, Inc.
Vendor URL https://censys.io/
Category Chrome Privacy Preserving Prefetch Proxy
Bot URL https://developer.chrome.com/blog/private-prefetch-proxy/
Vendor Google Inc.
Vendor URL https://www.google.com/
Category Dataprovider
Vendor Dataprovider B.V.
Vendor URL https://www.dataprovider.com/
Category Seekport
Bot URL http://www.seekport.com/
Vendor SISTRIX GmbH
Vendor URL https://www.sistrix.de/
Category 360Spider
Bot URL https://www.so.com/help/help_3_2.html
Vendor Online Media Group, Inc.
Category LeakIX
Bot URL https://leakix.net/
Vendor BaDaaS SRL
Vendor URL https://leakix.net/
Category Cốc Cốc Bot
Bot URL https://help.coccoc.com/en/search-engine/coccoc-robots
Vendor Cốc Cốc
Vendor URL https://coccoc.com/
Category ChatGPT
Bot URL https://platform.openai.com/docs/plugins/bot
Vendor OpenAI OpCo, LLC
Vendor URL https://openai.com/
Category Gmail Image Proxy
Vendor Google Inc.
Vendor URL http://www.google.com
Category Mixrank Bot
Bot URL http://mixrank.com
Vendor Online Media Group, Inc.
Category Velen Public Web Crawler
Bot URL https://hunter.io/robot
Category Netcraft Survey Bot
Vendor Netcraft
Vendor URL http://www.netcraft.com
Category AhrefsSiteAudit
Bot URL https://ahrefs.com/robot/site-audit
Vendor Ahrefs Pte Ltd
Vendor URL https://ahrefs.com/
Category Uptime Robot
Vendor Uptime Robot
Vendor URL http://uptimerobot.com
Category WhatCMS
Bot URL https://whatcms.org/
Vendor Nineteen Ten LLC
Vendor URL https://whatcms.org/
Category DomainStatsBot
Bot URL https://domainstats.com/pages/our-bot
Vendor Domainstats Ltd
Vendor URL https://domainstats.com/
Category NiceCrawler
Bot URL https://www.nicecrawler.com/
Vendor Intelium Corp.
Vendor URL https://www.intelium.com/
Category Repo Lookout
Bot URL https://www.repo-lookout.org/
Vendor Crissy Field GmbH
Vendor URL https://www.crissyfield.de/
Category W3C Markup Validation Service
Bot URL http://validator.w3.org/services
Vendor W3C
Vendor URL http://www.w3.org
Category Snap URL Preview Service
Bot URL https://developers.snap.com/robots
Vendor Snapchat Inc.
Vendor URL https://www.snapchat.com/
Category BrightEdge
Bot URL https://www.brightedge.com/
Vendor BrightEdge Technologies, Inc
Vendor URL https://www.brightedge.com/
Category Panscient
Bot URL https://www.panscient.com/faq.htm
Vendor Panscient, Inc.
Vendor URL https://www.panscient.com/
Category BitSight
Bot URL https://www.bitsight.com/
Vendor BitSight Technologies, Inc.
Vendor URL https://www.bitsight.com/
Category IPS Agent
Vendor VeriSign, Inc
Vendor URL http://www.verisign.com/
Category serpstatbot
Bot URL http://serpstatbot.com/
Vendor Netpeak Ltd
Vendor URL https://netpeak.net/
Category Screaming Frog SEO Spider
Bot URL http://www.screamingfrog.co.uk/seo-spider
Vendor Screaming Frog Ltd
Vendor URL http://www.screamingfrog.co.uk
Category Webwiki
Bot URL https://www.webwiki.com/
Vendor webwiki GmbH
Vendor URL https://www.webwiki.com/
Category Yahoo! Mail Proxy
Bot URL https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html
Vendor Yahoo! Inc.
Vendor URL http://www.yahoo.com
Category CATExplorador
Bot URL https://fundacio.cat/ca/domini/
Vendor Fundació puntCAT
Vendor URL https://fundacio.cat/ca/domini/
Category ZumBot
Bot URL http://help.zum.com/inquiry
Vendor ZUM internet
Vendor URL http://www.zuminternet.com/
Category sqlmap
Bot URL http://sqlmap.org/
Vendor sqlmap
Vendor URL http://sqlmap.org/
Category Amazon ELB
Bot URL https://aws.amazon.com/elasticloadbalancing/
Vendor Amazon.com, Inc.
Vendor URL https://www.amazon.com/
Category netEstate
Bot URL http://www.website-datenbank.de/Impressum
Vendor netEstate GmbH
Vendor URL https://www.netestate.de/en/
Category SMTBot
Bot URL https://www.similartech.com/smtbot
Vendor SimilarTech Ltd.
Vendor URL https://www.similartech.com/
Category Diffbot
Bot URL https://docs.diffbot.com/docs/getting-started-with-crawl
Vendor Diffbot Technologies Corp.
Vendor URL https://www.diffbot.com/
Category ContentKing
Bot URL https://whatis.contentkingapp.com/
Vendor ContentKing BV
Vendor URL https://www.contentkingapp.com/
Category Alexa Site Audit
Bot URL https://support.alexa.com/hc/en-us/articles/200450194
Vendor Alexa Internet
Vendor URL https://www.alexa.com
Category W3C CSS Validator
Bot URL http://jigsaw.w3.org/css-validator
Vendor W3C
Vendor URL http://www.w3.org
Category Cocolyzebot
Bot URL https://cocolyze.com/en/cocolyzebot
Vendor VSI INNOVATION SAS
Vendor URL https://vsi-innovation.com/
Category Gregarius
Bot URL https://web.archive.org/web/20100614011837/http://devlog.gregarius.net/docs/ua/
Category Wappalyzer
Bot URL https://github.com/AliasIO/Wappalyzer
Vendor AliasIO
Vendor URL https://github.com/AliasIO
Category Turnitin
Bot URL https://turnitin.com/robot/crawlerinfo.html
Category Nagios check_http
Bot URL https://nagios.org
Vendor Nagios Plugins Development Team
Vendor URL https://nagios.org
Category t3versions
Bot URL https://www.t3versions.com/bot
Vendor Torben Hansen
Vendor URL https://www.t3versions.com/
Category Datadog Agent
Bot URL https://github.com/DataDog/dd-agent
Vendor Datadog
Vendor URL https://www.datadoghq.com/
Category Crawlson
Bot URL https://www.crawlson.com/about
Vendor Crawlson
Vendor URL https://www.crawlson.com/
Category Asana
Bot URL https://asana.com/
Vendor Asana, Inc.
Vendor URL https://asana.com/
Category Nmap
Bot URL https://nmap.org/book/nse.html
Vendor Nmap
Vendor URL https://nmap.org/
Category CriteoBot
Bot URL https://www.criteo.com/criteo-crawler/
Category LCC
Bot URL https://corpora.uni-leipzig.de/crawler_faq.html
Vendor Universität Leipzig
Vendor URL https://www.uni-leipzig.de/
Category YaCy
Bot URL http://yacy.net/bot.html
Vendor YaCy
Vendor URL http://yacy.net
Category LinkWalker
Bot URL https://www.phishlabs.com/
Vendor PhishLabs, Inc.
Vendor URL https://www.phishlabs.com/
Category Adbeat
Bot URL https://www.adbeat.com/operation_policy
Vendor PPC Labs LLC
Vendor URL https://www.adbeat.com/
Category Everyfeed
Bot URL https://web.archive.org/web/20050930235914/http://www.everyfeed.com/
Category Datanyze
Vendor Datanyze
Vendor URL https://www.datanyze.com