---
title: Crawlers reference 
description: "# Vrender crawlers reference  ## Introduction This document lists the crawlers whose requests your infrastructure can forward to Vrender, based on your company’s needs. Routing bots to Vrender ensures they receive fully pre-rendered pages, improving crawl efficiency and indexing quality.…"
url: "https://www.verbolia.com/knowledge/vrender-crawlers-reference/"
updated: "2026-04-24"
category: Vrender
---

# Crawlers reference 

# Vrender crawlers reference 

## Introduction

This document lists the crawlers whose requests your infrastructure can forward to Vrender, based on your company’s needs. Routing bots to Vrender ensures they receive fully pre-rendered pages, improving crawl efficiency and indexing quality.

Crawlers are grouped by type. Each table includes a relevance column to help you prioritize routing specific crawlers.

- **High** — Route these first. They have a direct impact on search visibility and AI discoverability.
- **Medium** — Secondary. Route these if the platform is relevant to your business.
- **Low** — Optional. Can be routed if needed for your business.

| **Note:**The bot names in the tables below refer to the user-agent string used to identify each crawler in your infrastructure configuration. You can confirm the exact user-agent string for any bot in its official documentation before configuring routing rules. |
| --- |

## 1. Search engine crawlers

These are the primary crawlers used by search engines to index your pages. 

| **Bot name** | **Company** | **Purpose** | **Relevance** |
| --- | --- | --- | --- |
| Googlebot | Google | Google's primary crawler for indexing web pages in Google Search. | **● High** |
| bingbot | Microsoft | Microsoft Bing's primary crawler for indexing web pages. | **● High** |
| Applebot | Apple | Used by Apple for Spotlight Search, Siri suggestions, and the Safari Reader. | **● Medium** |
| DuckDuckBot | DuckDuckGo | DuckDuckGo's web crawler for indexing pages in DuckDuckGo Search. | **● Medium** |
| yandexbot | Yandex | Primary crawler for Yandex, the leading search engine in Russia and several neighbouring markets. | **● Low** |
| baiduspider | Baidu | Primary crawler for Baidu, the leading search engine in China. | **● Low** |
| SeznamBot | Seznam.cz | Crawler for Seznam, the leading search engine in the Czech Republic. | **● Low** |
| Amzn-SearchBot | Amazon | Amazon's web crawler, used to improve search experiences in Amazon products and services.  | **● Low** |

## 2. AI indexing and training crawlers

These bots crawl your pages proactively to index content or train AI models. 

| **Bot name** | **Company** | **Purpose** | **Relevance** |
| --- | --- | --- | --- |
| GPTBot | OpenAI | Crawls web pages to improve OpenAI's language models and power ChatGPT's knowledge. | **● High** |
| ClaudeBot | Anthropic | Anthropic's crawler for indexing web content to improve Claude. | **● High** |
| PerplexityBot | Perplexity AI | Crawls web pages to answer user questions in Perplexity's AI search engine. | **● High** |
| Bytespider | ByteDance | ByteDance's web crawler, used for TikTok and other ByteDance AI training. | **● Low** |

## 3. AI retrieval agents (user-triggered)

Unlike indexing crawlers, these bots are triggered in real time when a user asks an AI assistant to browse or retrieve content. They visit your pages on demand and return the content directly to the user. 

| **Bot name** | **Company** | **Purpose** | **Relevance** |
| --- | --- | --- | --- |
| ChatGPT-User | OpenAI | Triggered when a ChatGPT user asks the assistant to browse a specific URL in real time. | **● High** |
| Claude-User | Anthropic | Triggered when a Claude user asks the assistant to retrieve content from a specific URL. | **● High** |
| OAI-SearchBot | OpenAI | Used to retrieve and surface web pages within ChatGPT's search features. | **● High** |
| Perplexity-User | Perplexity AI | Triggered when a Perplexity user submits a query that requires live web retrieval. | **● High** |

## 4. Social media and link preview crawlers

These bots fetch page metadata when a URL is shared on a social platform or messaging app. They generate the preview cards (title, description, image) that users see before clicking. 

| **Bot name** | **Company** | **Purpose** | **Relevance** |
| --- | --- | --- | --- |
| facebookexternalhit | Meta | Fetches metadata for link preview cards when a URL is shared on Facebook or Instagram. | **● Medium** |
| WhatsApp (Meta) | Meta | Fetches link previews when a URL is shared in a WhatsApp conversation. User-agent includes 'WhatsApp'. | **● Medium** |
| Twitterbot | X (Twitter) | Generates preview cards for URLs shared in tweets and direct messages. | **● Medium** |
| LinkedInBot | LinkedIn | Generates preview cards for URLs shared in LinkedIn posts and messages. | **● Medium** |
| Slackbot | Slack | Fetches link metadata for Slack's link-unfurling feature in channels and messages. | **● Medium** |
| Discordbot | Discord | Renders rich link previews in Discord chat channels. | **● Medium** |
| TelegramBot | Telegram | Fetches page metadata to generate link previews in Telegram chats. | **● Medium** |
| pinterest / PinterestBot | Pinterest | Fetches page metadata and images when users pin or share a URL. | **● Medium** |
| vkShare | VKontakte | Generates link previews on VKontakte, the leading social network in Russia. | **● Low** |
| SkypeUriPreview | Microsoft | Fetches link previews for URLs shared in Skype conversations. | **● Low** |
| embedly | Embed.ly | A general-purpose link preview service used by many third-party platforms. | **● Low** |
| flipboard | Flipboard | Fetches content for Flipboard's magazine-style aggregation feed. | **● Low** |
| bitlybot | Bitly | Fetches metadata for URL preview and analytics on Bitly's link shortening service. | **● Low** |
| Tumblr | Tumblr | Fetches content previews for URLs shared within Tumblr posts. | **● Low** |
| XING-contenttabreceiver | XING | Generates link previews on XING, a professional networking platform popular in German-speaking markets. | **● Low** |
| quora link preview | Quora | Fetches metadata when users paste URLs into Quora questions or answers. | **● Low** |
| nuzzel | Nuzzel | Used for link preview and content curation on the Nuzzel platform. | **● Low** |
| showyoubot | Showyou | Fetches link previews for the Showyou video sharing platform. | **● Low** |
| Bitrix link preview | Bitrix24 | Link preview bot for the Bitrix24 intranet and collaboration platform. | **● Low** |

## 5. Performance and validation tools

These tools audit your pages for performance, SEO, and standards compliance. 

| **Bot name** | **Company** | **Purpose** | **Relevance** |
| --- | --- | --- | --- |
| google-inspectiontool | Google | Used for on-demand page inspection and debugging via Google Search Console. | **● Medium** |
| Chrome-Lighthouse | Google | Google's open-source tool for auditing page performance, accessibility, and SEO. | **● Medium** |
| Google Page Speed | Google | Measures page speed and reports optimization opportunities via PageSpeed Insights. | **● Medium** |
| rogerbot | Moz | Moz's crawler used for SEO auditing and link analysis in Moz Pro tools. | **● Low** |
| W3C_Validator | W3C | Validates HTML and CSS against W3C web standards. | **● Low** |