Free tools
Robots.txt Generator
Generate a robots.txt file with user-agent rules, allow/disallow paths, optional crawl-delay, and Sitemap: lines.
How this works
- Add sitemap URLs you want advertised.
- Set allow and disallow paths (one per line).
- Copy the generated robots.txt to your site root.
robots.txt drafted. Track the launch in WorkFocus.
WorkFocus helps freelance developers plan today, run client projects, and show what they shipped — without juggling five apps.
FAQ
Does crawl-delay work everywhere?
Some crawlers honor it; Google generally does not. Use it only if you know your audience of bots.
Is this legal advice?
No. robots.txt is a hint to well-behaved crawlers, not access control.
Bring this work into a daily plan.
WorkFocus is the daily workspace for freelance developers — today's tasks, client projects, and bugs in one place.
More SEO & sitemaps tools
Sitemap Finder & Checker
Find XML sitemaps on any public website from robots.txt and common paths, then check whether they parse and how many URLs they list.
XML Sitemap Validator
Validate an XML sitemap for missing loc tags, bad dates, duplicate URLs, size limits, and sitemap vs index mix-ups — then get a simple score.
Sitemap URL Extractor
Pull every loc URL out of a sitemap.xml or sitemap index, with optional include and exclude path filters. Copy or download the list.
Sitemap URL Comparison
Compare two XML sitemaps and see which URLs were added, removed, or unchanged — useful for migrations and content audits.
Sitemap Split & Merge
Split a large urlset into chunks or merge several sitemaps into one XML file. Optionally emit a sitemap index for the chunks.
Sitemap Analytics
See path depth, file-type mix, changefreq, and priority distribution for a sitemap — plus simple coverage notes.