Cloudflare released a new Browser Rendering /crawl endpoint on March 10, 2026, enabling developers to crawl entire websites with a single API call. The service entered open beta immediately and is available to all Workers plan subscribers, including free tier users.
Single API Call Crawls Entire Websites With Headless Browsers
The /crawl endpoint automatically discovers pages through sitemaps or by following links, renders them in headless browsers, and delivers comprehensive results. Developers submit a URL, receive a job ID, and retrieve results asynchronously as processing completes—eliminating timeout issues when crawling large sites.
The service outputs data in three formats: HTML, Markdown, or structured JSON powered by Workers AI. This flexibility supports diverse use cases from AI model training to content monitoring.
Efficiency Features Include Modified-Since Parameters and Robots.txt Compliance
Cloudflare built several optimization features into the endpoint:
- modifiedSince parameter: Crawls only pages changed after a specified date
- maxAge parameter: Skips recently crawled pages to avoid redundant processing
- render: false option: Enables faster crawling of static sites without browser rendering overhead
- Robots.txt compliance: Honors directives including crawl-delay to respect site preferences
The service targets four primary use cases: training AI models, building RAG (Retrieval-Augmented Generation) pipelines, researching content across sites, and monitoring site changes.
Built on Cloudflare Workers AI Infrastructure
The endpoint leverages Cloudflare's existing Browser Rendering infrastructure combined with Workers AI for format conversion. Unlike traditional web scrapers that require managing headless browsers, proxies, and rate limiting, Cloudflare handles all infrastructure complexity behind a single API.
The announcement generated significant discussion on Hacker News, receiving 386 points and 148 comments focused on implications for web scraping, AI training data collection, and concerns about increased bot traffic. Cloudflare's positioning as an infrastructure provider—rather than a data consumer—differentiates this service from search engine crawlers operating at web scale.
Key Takeaways
- Cloudflare's /crawl endpoint enables entire website crawling through a single API call with automatic page discovery and headless browser rendering
- The service supports three output formats (HTML, Markdown, JSON) and includes efficiency features like modifiedSince and maxAge parameters
- Primary use cases include AI model training, RAG pipeline development, content research, and site change monitoring
- The endpoint is available in open beta for all Workers plan subscribers, including free tier users
- The service respects robots.txt directives and handles all infrastructure complexity behind a simple asynchronous API