Google has quietly launched a new bot, Google-CloudVertexBot, which serves commercial clients of their Vertex AI platform. This bot is designed to crawl websites, specifically ingesting content for Vertex AI, distinguishing it from other Google bots associated with search or advertising.
What Does Google-CloudVertexBot Do?
- Crawling Scope: The bot appears to crawl websites at the request of site owners, focusing on data for Vertex AI Agents.
- Documentation Ambiguity: The official documentation is unclear, leaving questions about whether the bot crawls only verified domains or might crawl public sites as well.
Google’s documentation suggests two types of website indexing for this bot—Basic and Advanced—with Advanced requiring domain verification and imposing quotas. However, there’s some confusion as the bot is noted in the changelog to help site owners identify new crawler traffic, possibly implying it could crawl public sites.
What Should Site Owners Do?
Given the ambiguity in the documentation, it’s uncertain whether the bot will crawl unverified domains. Site owners might consider updating their robots.txt file to block the new crawler if they prefer not to have their content indexed by Google-CloudVertexBot.
User Agent Information:
- User Agent Tokens:
Google-CloudVertexBot,Googlebot - User Agent Substring:
Google-CloudVertexBot
Conclusion
While Google-CloudVertexBot seems to cater to Vertex AI clients, the unclear documentation suggests site owners should stay vigilant. If you’re unsure about allowing this bot to crawl your site, monitoring your server logs or adjusting your robots.txt may be prudent.
For more detailed information, you can refer to the official documentation provided by Google.