Vrender crawlers reference
Introduction #
This document lists the crawlers whose requests your infrastructure can forward to Vrender, based on your company’s needs. Routing bots to Vrender ensures they receive fully pre-rendered pages, improving crawl efficiency and indexing quality.
Crawlers are grouped by type. Each table includes a relevance column to help you prioritize routing specific crawlers.
- High — Route these first. They have a direct impact on search visibility and AI discoverability.
- Medium — Secondary. Route these if the platform is relevant to your business.
- Low — Optional. Can be routed if needed for your business.
| Note: The bot names in the tables below refer to the user-agent string used to identify each crawler in your infrastructure configuration. You can confirm the exact user-agent string for any bot in its official documentation before configuring routing rules. |
1. Search engine crawlers #
These are the primary crawlers used by search engines to index your pages.
| Bot name | Company | Purpose | Relevance |
| Googlebot | Google’s primary crawler for indexing web pages in Google Search. | ● High | |
| bingbot | Microsoft | Microsoft Bing’s primary crawler for indexing web pages. | ● High |
| Applebot | Apple | Used by Apple for Spotlight Search, Siri suggestions, and the Safari Reader. | ● Medium |
| DuckDuckBot | DuckDuckGo | DuckDuckGo’s web crawler for indexing pages in DuckDuckGo Search. | ● Medium |
| yandexbot | Yandex | Primary crawler for Yandex, the leading search engine in Russia and several neighbouring markets. | ● Low |
| baiduspider | Baidu | Primary crawler for Baidu, the leading search engine in China. | ● Low |
| SeznamBot | Seznam.cz | Crawler for Seznam, the leading search engine in the Czech Republic. | ● Low |
| Amzn-SearchBot | Amazon | Amazon’s web crawler, used to improve search experiences in Amazon products and services. | ● Low |
2. AI indexing and training crawlers #
These bots crawl your pages proactively to index content or train AI models.
| Bot name | Company | Purpose | Relevance |
| GPTBot | OpenAI | Crawls web pages to improve OpenAI’s language models and power ChatGPT’s knowledge. | ● High |
| ClaudeBot | Anthropic | Anthropic’s crawler for indexing web content to improve Claude. | ● High |
| PerplexityBot | Perplexity AI | Crawls web pages to answer user questions in Perplexity’s AI search engine. | ● High |
| Bytespider | ByteDance | ByteDance’s web crawler, used for TikTok and other ByteDance AI training. | ● Low |
3. AI retrieval agents (user-triggered) #
Unlike indexing crawlers, these bots are triggered in real time when a user asks an AI assistant to browse or retrieve content. They visit your pages on demand and return the content directly to the user.
| Bot name | Company | Purpose | Relevance |
| ChatGPT-User | OpenAI | Triggered when a ChatGPT user asks the assistant to browse a specific URL in real time. | ● High |
| Claude-User | Anthropic | Triggered when a Claude user asks the assistant to retrieve content from a specific URL. | ● High |
| OAI-SearchBot | OpenAI | Used to retrieve and surface web pages within ChatGPT’s search features. | ● High |
| Perplexity-User | Perplexity AI | Triggered when a Perplexity user submits a query that requires live web retrieval. | ● High |
4. Social media and link preview crawlers #
These bots fetch page metadata when a URL is shared on a social platform or messaging app. They generate the preview cards (title, description, image) that users see before clicking.
| Bot name | Company | Purpose | Relevance |
| facebookexternalhit | Meta | Fetches metadata for link preview cards when a URL is shared on Facebook or Instagram. | ● Medium |
| WhatsApp (Meta) | Meta | Fetches link previews when a URL is shared in a WhatsApp conversation. User-agent includes ‘WhatsApp’. | ● Medium |
| Twitterbot | X (Twitter) | Generates preview cards for URLs shared in tweets and direct messages. | ● Medium |
| LinkedInBot | Generates preview cards for URLs shared in LinkedIn posts and messages. | ● Medium | |
| Slackbot | Slack | Fetches link metadata for Slack’s link-unfurling feature in channels and messages. | ● Medium |
| Discordbot | Discord | Renders rich link previews in Discord chat channels. | ● Medium |
| TelegramBot | Telegram | Fetches page metadata to generate link previews in Telegram chats. | ● Medium |
| pinterest / PinterestBot | Fetches page metadata and images when users pin or share a URL. | ● Medium | |
| vkShare | VKontakte | Generates link previews on VKontakte, the leading social network in Russia. | ● Low |
| SkypeUriPreview | Microsoft | Fetches link previews for URLs shared in Skype conversations. | ● Low |
| embedly | Embed.ly | A general-purpose link preview service used by many third-party platforms. | ● Low |
| Fetches content for Flipboard’s magazine-style aggregation feed. | ● Low | ||
| bitlybot | Bitly | Fetches metadata for URL preview and analytics on Bitly’s link shortening service. | ● Low |
| Tumblr | Tumblr | Fetches content previews for URLs shared within Tumblr posts. | ● Low |
| XING-contenttabreceiver | Generates link previews on XING, a professional networking platform popular in German-speaking markets. | ● Low | |
| quora link preview | Quora | Fetches metadata when users paste URLs into Quora questions or answers. | ● Low |
| nuzzel | Nuzzel | Used for link preview and content curation on the Nuzzel platform. | ● Low |
| showyoubot | Showyou | Fetches link previews for the Showyou video sharing platform. | ● Low |
| Bitrix link preview | Bitrix24 | Link preview bot for the Bitrix24 intranet and collaboration platform. | ● Low |
5. Performance and validation tools #
These tools audit your pages for performance, SEO, and standards compliance.
| Bot name | Company | Purpose | Relevance |
| google-inspectiontool | Used for on-demand page inspection and debugging via Google Search Console. | ● Medium | |
| Chrome-Lighthouse | Google’s open-source tool for auditing page performance, accessibility, and SEO. | ● Medium | |
| Google Page Speed | Measures page speed and reports optimization opportunities via PageSpeed Insights. | ● Medium | |
| rogerbot | Moz | Moz’s crawler used for SEO auditing and link analysis in Moz Pro tools. | ● Low |
| W3C_Validator | W3C | Validates HTML and CSS against W3C web standards. | ● Low |
