How to Generate a robots.txt File and a Starter XML Sitemap for Any Domain

Introduction

Search engine crawlers look for robots.txt at the root of a domain before fetching anything else, and a sitemap tells them which URLs matter. A malformed file can hide an entire site or expose admin paths that should stay private. The Robots.txt & Sitemap Generator builds both files from a base URL and a short list of disallowed paths, updating the output in the browser as fields change.

Step-by-Step Instructions

1. Enter the site root in the Domain / Website Base URL field, including https:// (a trailing slash is removed automatically).
2. List each folder or path to block in Disallowed Path Rules, one per line, such as /admin/ or /private/.
3. Review the robots.txt preview, which always starts with User-agent: * and Allow: / followed by your Disallow lines and a Sitemap directive.
4. Click Copy above robots.txt and save the text as robots.txt in the web root.
5. Select the sitemap.xml preview text, copy it, save it as sitemap.xml in the web root, then add further url entries for the pages you want indexed.

Key Use Cases

• New site launch: Produce a valid robots.txt in seconds so the crawl is not blocked by a missing or broken file.
• Staging cleanup: Add /staging/ or /test/ to the disallow list before the site goes public.
• Sitemap scaffold: Start from a correct urlset wrapper and append entries rather than typing the XML namespace by hand.

Frequently Asked Questions