SiteOne Crawler: Free Website Crawler for Technical SEO and Deployment Audits (2026)
SiteOne Crawler, developed by SiteOne, is a free website crawler for technical SEO teams that need local audits, offline exports and repeatable deployment checks. Its official website describes approximately 200 command-line options and a quality score across five areas rather than a paid, URL-metered SaaS dashboard.
The strongest reason to choose it is the combination of a desktop interface and a scriptable command-line tool. An SEO can inspect links and metadata while a developer can run the same underlying crawler in a release workflow. It is not a replacement for a backlink database, keyword research platform or managed rank tracker.
SiteOne Crawler: Key Specs at a Glance
| Feature | Verified position |
|---|
| Primary use case | Website crawling and technical quality audits |
| Developer | SiteOne; Jan Reges maintains the linked project |
| Price | Free; no paid subscription advertised on the official homepage |
| Access | Desktop GUI and standalone command-line binary |
| Platforms | Windows, macOS and Linux installation options |
| Configuration | Approximately 200 CLI options, according to the homepage |
| JavaScript rendering | Optional Chromium CDP browser mode for post-render DOM |
| Crawl limit | Configurable; published example reports use 1,000 URLs, not a documented product ceiling |
| Crawl speed | No universal URLs-per-second benchmark published on the reviewed homepage |
| Audit coverage | SEO, performance, security, accessibility and best practices |
| Outputs | Interactive HTML report, offline website copy and Markdown exports |
| Deployment integration | CI quality gate, JUnit XML and GitHub annotations |
| Scoring | 0–10 quality score across five audit areas |
| AI analysis | Optional user-supplied commercial or local LLM endpoint |
| Hosted API | No hosted crawl API plan documented on the reviewed homepage; CLI automation is explicit |
| White-label/client seats | No SaaS seat management or white-label entitlement documented on the reviewed homepage |
What SiteOne Crawler Does Well
A practical bridge between SEO audits and release checks
The crawler exposes a CI mode with configurable thresholds, JUnit XML and GitHub annotations. Its homepage documents exit code 10 when the quality gate fails. This gives development teams a concrete mechanism for blocking regressions instead of manually checking another dashboard after deployment.
For a technical SEO, that makes the tool particularly relevant when recurring metadata, link or accessibility regressions arise from application releases. Decide which findings should block a build and which should remain advisory before enabling a gate across production deployments.
Rendered crawling is available without pretending it is free of operational cost
Optional browser mode uses Chromium through CDP to crawl the rendered DOM of JavaScript applications. The site names React, Vue, Svelte, Angular and Next.js as relevant examples. This matters when a raw HTTP crawl misses links or metadata introduced by client-side code.
The mode can also collect page screenshots and browser diagnostics. Browser rendering requires more resources than fetching HTML, so compare a small representative set before running a large rendered crawl. No fixed throughput guarantee is published in the source reviewed for this guide.
Useful outputs beyond the issue list
SiteOne combines interactive reports with offline website copies and Markdown exports. These serve different workflows: the report supports diagnosis, a mirror supports archival inspection, and Markdown supports documentation or retrieval workflows. Exporting an application does not reproduce its server-side behaviour.
The official examples show reports limited to the first 1,000 URLs. Treat that as an example configuration rather than evidence of an unlimited machine or a restrictive subscription tier. Choose scope deliberately and check whether assets and third-party resources belong in the same run.
Optional AI with a choice of endpoint
The reviewed homepage describes support for commercial models and local endpoints such as Ollama, vLLM and LM Studio. AI analysis is optional and includes cost controls and a dry-run preview. Teams can therefore evaluate the core crawl before deciding whether model-assisted interpretation adds value.
Known Limitations
Compared with Screaming Frog SEO Spider: Screaming Frog explicitly documents connections to the Google Analytics, Search Console and PageSpeed Insights APIs to fetch user and performance data during a crawl. SiteOne's reviewed homepage does not document those account-data integrations. For an SEO who needs to combine crawl findings with GSC queries or Analytics engagement data in the crawler, that is a concrete documentation and workflow gap: verify SiteOne support before selecting it, or evaluate Screaming Frog's licensed integration features. This is not a claim that an undocumented capability can never exist. See Screaming Frog's official product description and official free-versus-paid feature comparison.
- No substitute for search-market data: this is not a keyword volume database, backlink index or competitive rank-tracking suite.
- JavaScript rendering needs resources: headless Chromium adds execution time and memory overhead. Test representative routes and ensure important states are reachable.
- Large-site sizing is your responsibility: hardware, network latency, crawl scope and target-server tolerance influence performance; no universal crawl-speed promise is documented.
- Local operation adds maintenance: downloads, architecture choices and updates must fit your organisation's software policy. Historical community installation concerns should not be mistaken for a verified current defect.
- Scores require interpretation: an audit quality score is not a Google ranking factor or proof of a Core Web Vitals field-data pass.
- AI is not automatically cost-free: commercial inference and locally hosted infrastructure can add costs even though the crawler itself is free.
Best For: Who Should Use SiteOne Crawler
- Technical SEOs who want a free local audit alongside their existing search-data subscriptions.
- Developers adding repeatable website checks to CI pipelines.
- Small agencies that can operate desktop software and interpret reports for clients.
- Documentation teams exporting public website material to Markdown or an offline copy.
Who Should Look Elsewhere
- Buyers needing a managed cloud workspace with documented client-seat and white-label contracts.
- Marketers primarily comparing competitor keywords, backlinks or daily rankings.
- Teams requiring a hosted crawl API and a published enterprise throughput SLA before procurement.
Pricing and Cost at Scale
The official homepage calls SiteOne Crawler free and provides desktop and CLI download paths. No paid tier or per-URL subscription is advertised there. A solo practitioner can therefore evaluate the software at a subscription cost of $0, excluding hardware, labour and any optional AI usage.
For an agency, the meaningful scaling cost is operational: machines, storage for reports and screenshots, scheduling and analyst review. Multiplying the software price by ten clients still produces $0 in advertised crawler subscription fees, but it does not establish zero total cost or unlimited practical throughput.
For an enterprise, evaluate deployment approval, resource isolation, data handling and support expectations separately. Publicly documented commercial support terms were not established in this research. Prices and licensing terms can change; check the official website and downloads before adopting.
Integrations and Workflow Fit
SiteOne is primarily a locally operated desktop/CLI tool, not a hosted analytics service. Its documented integrations are strongest around CI output, GitHub annotations and optional LLM endpoints. Google Search Console and GA4 connections are not documented on the reviewed homepage; do not assume the audit includes either account's data.
Getting Started
- Choose the desktop app or CLI for your operating system and CPU architecture using official downloads.
- Start with a website you control and a small URL scope to understand the output and load.
- Review the HTML report and prioritise actionable link, heading, metadata and response issues.
- If JavaScript controls your important content, compare a rendered crawl against the initial HTTP-oriented result.
- Only after establishing a baseline, add CI thresholds or recurring runs and retain reports for comparison.
Pro tip: keep your first CI gate narrow. A repeatable regression check for important templates is easier to maintain than a large collection of unexplained failing thresholds.
What Users Are Saying
Two verified Hacker News discussions provide external context rather than a large review sample. Participants highlighted the breadth of the free tool and its relevance to agency website audits; a 2024 discussion also raised macOS installation trust and packaging concerns, followed by replies from the maintainer. These are historical observations, not a claim that the same problem persists in 2026.
Have you tried SiteOne Crawler? Share your experience in the review section below to help other SEO professionals make the right choice.
FAQ
Is SiteOne Crawler free?
Yes. The reviewed official homepage explicitly describes it as free and links desktop and CLI downloads. Optional infrastructure and model usage can still cost money.
How does it compare with Screaming Frog SEO Spider?
SiteOne emphasises free local crawling, multi-area audit reports, website exports and CI gates. Evaluate both on your own templates rather than assuming feature parity or equivalent crawl performance.
Is it good for agencies?
It can fit technically capable agencies needing local audits without a crawler subscription. Managed client seats and white-label terms were not documented on the homepage reviewed here.
Does SiteOne have an API?
A hosted crawl API plan was not documented in the reviewed source. The CLI is explicitly designed for automation and CI integration.
Can it crawl JavaScript-heavy sites?
Yes, the official site documents optional Chromium CDP browser mode for crawling the rendered DOM. Resource usage and reachable application states still require testing.
What is the maximum number of URLs it can crawl?
A universal maximum was not established from the reviewed homepage. Published examples use a configurable 1,000-URL limit; that should not be presented as a product ceiling.
Sources
