Website Crawlers
The website crawler index is an advanced feature that allows you to identify IP Addresses that are associated with various search engines and crawling services. Many crawlers can be malicious in nature so we keep an index of available services.
■ ■ ■
Category Applebot
Bot URL https://support.apple.com/en-us/HT204683
Vendor Apple Inc
Vendor URL https://www.apple.com
Category Yahoo! Slurp
Bot URL http://help.yahoo.com/ysearch/slurp
Vendor Yahoo! Inc.
Vendor URL http://www.yahoo.com
Category Flipboard
Bot URL http://flipboard.com/browserproxy
Vendor Flipboard
Vendor URL http://flipboard.com/
Category Twitterbot
Bot URL https://dev.twitter.com/docs/cards/getting-started
Vendor Twitter
Vendor URL http://www.twitter.com
Category PaperLiBot
Bot URL http://support.paper.li/entries/20023257-what-is-paper-li
Vendor Smallrivers SA
Vendor URL http://www.paper.li
Category BingBot
Bot URL http://search.msn.com/msnbot.htmn
Vendor Microsoft Corporation
Vendor URL http://www.microsoft.com
Category Facebook External Hit
Bot URL https://www.facebook.com/externalhit_uatext.php
Vendor Facebook
Vendor URL http://www.facebook.com
Category Googlebot
Bot URL http://www.google.com/bot.html
Vendor Google Inc.
Vendor URL http://www.google.com
Category MJ12 Bot
Bot URL http://majestic12.co.uk/bot.php
Vendor Majestic-12
Vendor URL http://majestic12.co.uk
Category BitlyBot
Bot URL https://bitly.com
Vendor Bitly, Inc.
Vendor URL https://bitly.com
Category uMBot
Bot URL https://www.ubermetrics-technologies.com/
Vendor Ubermetrics Technologies GmbH
Vendor URL https://www.ubermetrics-technologies.com/
Category Rogerbot
Bot URL http://moz.com/help/pro/what-is-rogerbot-
Vendor SEOmoz, Inc.
Vendor URL http://moz.com/
Category Hatena Bookmark
Bot URL https://www.hatena.ne.jp/faq/
Vendor Hatena Co., Ltd.
Vendor URL https://www.hatena.ne.jp
Category Baidu Spider
Bot URL http://www.baidu.com/search/spider.htm
Vendor Baidu
Vendor URL http://www.baidu.com
Category Mediatoolkit Bot
Bot URL https://mediatoolkit.com
Category Infegy
Bot URL https://infegy.com/
Vendor Infegy, Inc.
Vendor URL https://infegy.com/
Category aHrefs Bot
Bot URL https://ahrefs.com/robot
Vendor Ahrefs Pte Ltd
Vendor URL https://ahrefs.com/robot
Category Omgili bot
Bot URL http://www.omgili.com/Crawler.html
Vendor Omgili
Vendor URL http://www.omgili.com
Category Embedly
Bot URL https://support.embed.ly/hc/en-us
Vendor A Medium, Corp.
Vendor URL https://medium.com/
Category Tweetmeme Bot
Bot URL http://tweetmeme.com/
Vendor Mediasift
Category datagnionbot
Bot URL https://www.datagnion.com/bot.html
Vendor DATAGNION GMBH
Vendor URL https://www.datagnion.com/
Category Yandex Bot
Bot URL http://www.yandex.com/bots
Vendor Yandex LLC
Vendor URL http://company.yandex.com
Category Alexa Crawler
Bot URL https://support.alexa.com/hc/en-us/sections/200100794-Crawlers
Vendor Alexa Internet
Vendor URL https://www.alexa.com
Category WordPress
Bot URL https://wordpress.org/
Vendor WordPress.org
Vendor URL https://wordpress.org/
Category Gluten Free Crawler
Bot URL http://glutenfreepleasure.com/
Category BLEXBot Crawler
Bot URL http://webmeup-crawler.com
Vendor WebMeUp
Vendor URL http://webmeup.com
Category Seznam Bot
Bot URL http://www.mapy.cz/cz/seznambot.html
Vendor Seznam.cz, a.s.
Vendor URL http://www.seznam.cz/
Category Ask Jeeves
Vendor Ask Jeeves Inc.
Vendor URL http://www.ask.com
Category TelegramBot
Bot URL https://telegram.org/blog/bot-revolution
Category DotBot
Bot URL http://www.opensiteexplorer.org/dotbot
Vendor SEOmoz, Inc.
Vendor URL http://moz.com/
Category Pocket
Bot URL https://getpocket.com/pocketparser_ua
Vendor Read It Later, Inc.
Vendor URL https://getpocket.com/
Category Scraping Robot
Bot URL https://scrapingrobot.com/
Vendor Sprious LLC
Vendor URL https://sprious.com/
Category TurnitinBot
Bot URL http://www.turnitin.com/robot/crawlerinfo.html
Vendor iParadigms, LLC.
Vendor URL http://www.turnitin.com
Category Trendiction Bot
Bot URL http://www.trendiction.de/bot
Vendor Talkwalker Inc.
Vendor URL http://www.talkwalker.com
Category Mail.Ru Bot
Bot URL http://help.mail.ru/webmaster/indexing/robots/types_robots
Vendor Mail.Ru Group
Vendor URL http://corp.mail.ru
Category DuckDuckGo Bot
Bot URL https://duckduckgo.com/duckduckbot
Vendor DuckDuckGo
Vendor URL https://duckduckgo.com/
Category Qwantify
Bot URL https://www.qwant.com/
Vendor Qwant Corporation
Vendor URL https://www.qwant.com/
Category Grapeshot
Bot URL https://www.grapeshot.com/crawler
Vendor Grapeshot
Vendor URL https://www.grapeshot.com
Category ExaBot
Bot URL http://www.exabot.com/go/robot
Vendor Dassault Systèmes
Vendor URL http://www.3ds.com
Category Domain Re-Animator Bot
Vendor Domain Re-Animator, LLC
Vendor URL http://domainreanimator.com
Category Semrush Bot
Bot URL http://www.semrush.com/bot.html
Vendor SEMrush
Vendor URL http://www.semrush.com
Category MojeekBot
Bot URL http://www.mojeek.com/bot.html
Vendor Mojeek Ltd.
Vendor URL http://www.mojeek.com
Category Slackbot
Bot URL https://api.slack.com/robots
Vendor Slack Technologies
Vendor URL http://slack.com
Category HubSpot
Vendor HubSpot Inc.
Vendor URL https://www.hubspot.com
Category Yeti/Naverbot
Bot URL http://help.naver.com/robots/
Vendor Naver
Vendor URL http://www.naver.com
Category LinkedIn Bot
Bot URL http://www.linkedin.com
Vendor LinkedIn
Vendor URL http://www.linkedin.com
Category Heritrix
Bot URL https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
Vendor The Internet Archive
Vendor URL https://archive.org
Category SafeDNSBot
Bot URL https://www.safedns.com/searchbot
Vendor SafeDNS, Inc.
Vendor URL https://www.safedns.com/
Category Iframely
Bot URL https://iframely.com/
Vendor Itteco Software, Corp.
Vendor URL https://iframely.com/
Category Genieo Web filter
Bot URL http://www.genieo.com/webfilter.html
Vendor Genieo
Vendor URL http://www.genieo.com
Category YaK
Bot URL https://www.linkfluence.com/
Vendor Linkfluence SAS
Vendor URL https://www.linkfluence.com/
Category AddThis.com
Vendor Clearspring Technologies, Inc.
Vendor URL http://www.clearspring.com
Category Nutch-based Bot
Bot URL https://nutch.apache.org
Vendor The Apache Software Foundation
Vendor URL https://www.apache.org/foundation/
Category Linkdex Bot
Bot URL http://www.linkdex.com/bots
Vendor Mojeek Ltd.
Vendor URL http://www.mojeek.com
Category archive.org bot
Bot URL https://archive.org/details/archive.org_bot
Vendor The Internet Archive
Vendor URL https://archive.org
Category SISTRIX Crawler
Bot URL http://crawler.sistrix.net
Vendor SISTRIX GmbH
Vendor URL http://www.sistrix.de
Category MetaInspector
Bot URL https://github.com/jaimeiniesta/metainspector
Category Skype URI Preview
Vendor Skype Communications S.à.r.l.
Vendor URL https://www.skype.com
Category ICC-Crawler
Bot URL http://www.nict.go.jp/en/univ-com/plan/crawl.html
Category Snapchat Proxy
Bot URL https://www.snapchat.com
Vendor Snapchat Inc.
Vendor URL https://www.snapchat.com
Category Sprinklr
Vendor Sprinklr, Inc.
Vendor URL https://www.sprinklr.com/
Category BUbiNG
Bot URL http://law.di.unimi.it/BUbiNG.html
Vendor The Laboratory for Web Algorithmics (LAW)
Vendor URL http://law.di.unimi.it/software.php#buging
Category Daum
Bot URL http://tab.search.daum.net/aboutWebSearch_en.html
Vendor Daum Communications Corp.
Vendor URL http://www.kakaocorp.com/main
Category TraceMyFile
Bot URL https://www.tracemyfile.com/
Vendor Idee Inc.
Vendor URL http://ideeinc.com/
Category Serendeputy Bot
Bot URL https://serendeputy.com/about/serendeputy-bot
Category BrandVerity
Bot URL https://www.brandverity.com/why-is-brandverity-visiting-me
Vendor BrandVerity, Inc.
Vendor URL https://www.brandverity.com/
Category ccBot crawler
Bot URL http://commoncrawl.org/faq/
Vendor reddit inc.
Vendor URL http://www.reddit.com
Category Awario
Bot URL https://awario.com/bots.html
Vendor Awario
Vendor URL https://awario.com/
Category Lighthouse
Bot URL https://developers.google.com/web/tools/lighthouse
Vendor Lighthouse
Vendor URL https://developers.google.com/web/tools/lighthouse
Category Riddler
Bot URL https://riddler.io/about
Vendor F-Secure
Vendor URL https://www.f-secure.com
Category oBot
Bot URL https://www.xforce-security.com/crawler/
Vendor IBM Germany Research & Development GmbH
Vendor URL https://exchange.xforce.ibmcloud.com/
Category FreeWebMonitoring
Bot URL https://www.freewebmonitoring.com/bot.html
Vendor GreenWave Online, Inc.
Vendor URL http://www.greenwaveonline.com/
Category AspiegelBot
Bot URL https://aspiegel.com/
Vendor Huawei
Vendor URL https://www.huawei.com/
Category Femtosearch
Bot URL http://femtosearch.com/
Vendor Grier Forensics, LLC
Vendor URL https://www.grierforensics.com/
Category DataForSeoBot
Bot URL https://dataforseo.com/dataforseo-bot
Category Amazon Bot
Bot URL https://developer.amazon.com/support/amazonbot
Vendor Amazon.com, Inc.
Vendor URL https://www.amazon.com/
Category Bytespider
Bot URL https://bytedance.com/
Vendor ByteDance Ltd.
Vendor URL https://bytedance.com/
Category Buck
Bot URL https://hypefactors.com/
Vendor Hypefactors A/S
Vendor URL https://hypefactors.com/
Category Seobility
Bot URL https://www.seobility.net/en/faq/?category=crawling#!aboutourbot
Category GPTBot
Bot URL https://platform.openai.com/docs/gptbot
Vendor OpenAI OpCo, LLC
Vendor URL https://openai.com/
Category InternetMeasurement
Bot URL https://internet-measurement.com/
Category Sogou Spider
Bot URL http://www.sogou.com/docs/help/webmasters.htm
Vendor Sohu, Inc.
Vendor URL http://www.sogou.com
Category EFF Do Not Track Verifier
Bot URL https://www.eff.org/issues/do-not-track
Vendor Electronic Frontier Foundation
Vendor URL https://www.eff.org/
Category Expanse
Bot URL https://expanse.co/
Vendor Expanse Inc.
Vendor URL https://expanse.co/
Category BuiltWith
Bot URL https://builtwith.com/biup
Vendor BuiltWith Pty Ltd
Vendor URL https://builtwith.com/
Category Research Scan
Bot URL http://researchscan.comsys.rwth-aachen.de/
Vendor RWTH Aachen University
Vendor URL https://www.comsys.rwth-aachen.de/
Category Headline
Bot URL https://headline.com/legal/crawler
Vendor e.ventures Managementgesellschaft mbH
Vendor URL https://headline.com/
Category Pinterest
Bot URL https://help.pinterest.com/en/business/article/pinterest-crawler
Vendor Pinterest
Vendor URL https://www.pinterest.com/
Category start.me
Bot URL https://about.start.me/
Vendor start.me BV
Vendor URL https://about.start.me/
Category CensysInspect
Bot URL https://about.censys.io/
Vendor Censys, Inc.
Vendor URL https://censys.io/
Category Chrome Privacy Preserving Prefetch Proxy
Bot URL https://developer.chrome.com/blog/private-prefetch-proxy/
Vendor Google Inc.
Vendor URL https://www.google.com/
Category Dataprovider
Vendor Dataprovider B.V.
Vendor URL https://www.dataprovider.com/
Category Seekport
Bot URL http://www.seekport.com/
Vendor SISTRIX GmbH
Vendor URL https://www.sistrix.de/
Category 360Spider
Bot URL https://www.so.com/help/help_3_2.html
Vendor Online Media Group, Inc.
Category LeakIX
Bot URL https://leakix.net/
Vendor BaDaaS SRL
Vendor URL https://leakix.net/
Category Cốc Cốc Bot
Bot URL https://help.coccoc.com/en/search-engine/coccoc-robots
Vendor Cốc Cốc
Vendor URL https://coccoc.com/
Category ChatGPT
Bot URL https://platform.openai.com/docs/plugins/bot
Vendor OpenAI OpCo, LLC
Vendor URL https://openai.com/
Category Gmail Image Proxy
Vendor Google Inc.
Vendor URL http://www.google.com
Category Mixrank Bot
Bot URL http://mixrank.com
Vendor Online Media Group, Inc.
Category Velen Public Web Crawler
Bot URL https://hunter.io/robot
Category Netcraft Survey Bot
Vendor Netcraft
Vendor URL http://www.netcraft.com
Category AhrefsSiteAudit
Bot URL https://ahrefs.com/robot/site-audit
Vendor Ahrefs Pte Ltd
Vendor URL https://ahrefs.com/
Category Uptime Robot
Vendor Uptime Robot
Vendor URL http://uptimerobot.com
Category WhatCMS
Bot URL https://whatcms.org/
Vendor Nineteen Ten LLC
Vendor URL https://whatcms.org/
Category DomainStatsBot
Bot URL https://domainstats.com/pages/our-bot
Vendor Domainstats Ltd
Vendor URL https://domainstats.com/
Category NiceCrawler
Bot URL https://www.nicecrawler.com/
Vendor Intelium Corp.
Vendor URL https://www.intelium.com/
Category Repo Lookout
Bot URL https://www.repo-lookout.org/
Vendor Crissy Field GmbH
Vendor URL https://www.crissyfield.de/
Category W3C Markup Validation Service
Bot URL http://validator.w3.org/services
Vendor W3C
Vendor URL http://www.w3.org
Category Snap URL Preview Service
Bot URL https://developers.snap.com/robots
Vendor Snapchat Inc.
Vendor URL https://www.snapchat.com/
Category BrightEdge
Bot URL https://www.brightedge.com/
Vendor BrightEdge Technologies, Inc
Vendor URL https://www.brightedge.com/
Category Panscient
Bot URL https://www.panscient.com/faq.htm
Vendor Panscient, Inc.
Vendor URL https://www.panscient.com/
Category BitSight
Bot URL https://www.bitsight.com/
Vendor BitSight Technologies, Inc.
Vendor URL https://www.bitsight.com/
Category IPS Agent
Vendor VeriSign, Inc
Vendor URL http://www.verisign.com/
Category serpstatbot
Bot URL http://serpstatbot.com/
Vendor Netpeak Ltd
Vendor URL https://netpeak.net/
Category Screaming Frog SEO Spider
Bot URL http://www.screamingfrog.co.uk/seo-spider
Vendor Screaming Frog Ltd
Vendor URL http://www.screamingfrog.co.uk