Simple, intelligent sitemap generation for modern websites. Handles SSR and CSR with ease.
The generator respects robots.txt directives to ensure ethical crawling:
Parses robots.txt for existing sitemap declarations and prioritizes them for crawling.
Sitemap: https://example.com/sitemap.xmlRespects Disallow directives by checking URLs against disallowed paths before adding them to the crawl queue.
Disallow: /admin/
Disallow: /private/Automatically detects client-side rendered applications and uses Puppeteer when needed.
Follows robots.txt rules and existing sitemap references for ethical crawling.
Watch progress as pages are discovered with Server-Sent Events streaming.