Draft one all-agents group using comma-separated disallow paths and an optional sitemap URL. Review the rules before deployment; this does not secure content or guarantee deindexing.
The robots.txt generator formats a small crawl-rules draft from supplied path text and an optional sitemap address. It starts with User-agent: * and adds Disallow lines for the entries it recognizes. Begin by deciding which crawling behavior your site actually needs rather than pasting a generic list copied from another website. The tool does not crawl your routes, discover private areas or determine whether blocking a directory is appropriate for the content it contains.
This is a drafting aid, not a deployment operation. Selecting Generate Robots.txt places text in the output box but does not replace a server file. Keep the current live rules and the intended change available for review. A rule that looks simple can affect many paths, so assess it against your real route structure and required resources before publishing. The generator does not show the set of pages that a supplied prefix may match.
Although one interface label says one per line, the actual input is a single-line field and the function splits its value on commas. Enter “/admin/, /tmp/” to create two separate Disallow entries. Each comma-separated piece is trimmed and empty pieces are skipped. A newline-only list is not the parser's intended delimiter format. Follow the documented behavior rather than assuming the label means that a pasted multiline file will be interpreted as separate path rules.
For the sample above, the draft contains User-agent: *, then Disallow: /admin/ and Disallow: /tmp/ on separate lines. The function does not add missing leading slashes, resolve full URLs into paths or check for accidental spaces inside an entry. Supply the path form you intend and inspect every generated line. A correctly split list can still contain the wrong routes, because formatting entries and choosing an appropriate crawl policy are separate tasks.
With an empty disallow field, the generator adds a blank Disallow directive after the all-agents line. If the field contains only commas or spaces, its truthiness and filtering can produce a different draft with no actual path directive. Inspect the literal output rather than assuming every visually blank input follows the same branch. The tool does not explain all parser interpretations or consolidate an existing collection of groups and rules.
Only one all-agents group is produced. There are no controls for separate crawler-specific groups, Allow rules, comments or a complete merge with an existing file. If your site already has more complex rules, do not replace them blindly with this small generated block. Review the intended combination through your normal configuration workflow, because a simplified draft cannot infer which existing rules were essential or which consumers have more specific behavior.
Robots.txt communicates crawl preferences to compliant crawlers; it is not authentication or a permission barrier for visitors. Listing an administrative path here does not protect it from direct access. Keep sensitive resources behind the appropriate access controls in the application or server. The generator does not inspect those controls, and its output should not be described as making a route private or preventing every program from retrieving a resource.
Crawling and indexing are also different questions. A disallowed URL can still be known and appear in search without the page's crawled content. Do not use this draft as a guarantee of deindexing. If a page needs a separate indexing directive, review how a crawler can obtain that directive rather than combining blocks mechanically. Consult the linked guidance for the actual deployment question instead of expecting one Disallow line to solve access, crawling and search visibility at once.
The optional sitemap field is copied into a Sitemap line after a blank separator. It is not fetched, parsed or verified. Use the actual public sitemap address and inspect it before deployment. The function does not trim this field comprehensively or check whether it points to valid XML, an index file or a redirect. A plausible-looking address in the output is not evidence that a crawler can retrieve the intended resource.
Keep the sitemap reference aligned with the file your site maintains rather than a temporary example. If several sitemap references or a sitemap index are needed, handle that through your reviewed configuration; this small form exposes one optional address. It also does not generate the sitemap itself or submit it to a search service. Those are separate steps, so use the related sitemap tool only if its limited drafting behavior fits the actual URL inventory you need to publish.
Before publishing a path rule, list a few intended matches and a few routes that must remain crawlable. A broad slash or an overly general prefix can cover more of the site than you intended. The generator provides no matching simulator or warning based on those routes. Review the final syntax with appropriate robots tooling and compare it with your site map, especially when a directory mixes content pages and resources needed to render them.
Do not assume all crawler implementations support every extension or treat every unusual character identically. This form simply copies entries after comma splitting; it does not validate a complete rules specification. For wildcards, case differences, encoded paths or multiple groups, use the linked primary documentation and test the exact target behavior. Keep the simple sample separate from the production rules so a demonstration does not become an accidental policy change applied across the whole host.
A robots.txt file is served from the site's root location for the relevant origin. Copying a draft into an arbitrary tool directory does not establish the root policy. Use the server or publishing workflow responsible for that file and preserve necessary existing directives. Check the deployed response and its actual content after the change, rather than relying only on a local file or a successful paste into an editor.
Review related hosts and environments separately. A development rule intended to block everything should not be carried into a public site merely because the templates are shared. Conversely, a public draft is not access protection for a private environment. Keep the intended origin, previous file and reviewed change together in your deployment notes. The generator has no awareness of which environment will receive the output or whether another process regenerates and overwrites that file later.
Select Generate Robots.txt again after changing a field because the displayed draft is not a continuously refreshed preview. Copy the result manually and inspect representative allowed and disallowed cases using your normal validation workflow. Where available, the search service's robots reporting can help investigate processing after deployment. This page cannot predict when a crawler will next obtain an updated file, and generating a draft does not itself notify or force that external refresh.
Revisit rules when directories move, rendering resources change or a new content section is added. Keep each rule associated with a reason so obsolete blocks can be reviewed instead of accumulating indefinitely. The useful outcome is an intentional crawl policy implemented at the correct origin. It is not a guarantee that pages will rank, vanish from search immediately or become secure simply because a file containing their paths was produced by the generator.
It splits the input on commas, trims each piece and skips empty entries. The one-per-line label does not reflect the current parser.
It produces User-agent: * only. There are no separate controls for crawler-specific groups.
The empty-field branch adds a blank Disallow line. Review the output when entering only commas or whitespace because filtering follows a different branch.
No. Robots rules are not access controls or authentication.
No. Crawl restrictions and indexing are different mechanisms, and a blocked URL can still be known.
No. It does not crawl routes, simulate every match or confirm that the chosen paths are appropriate.
No. It is copied into a Sitemap line without retrieval or XML validation.
Not through the current controls. Maintain complex rules through an appropriate reviewed configuration workflow.
Deploy the checked policy through the root robots.txt location for the intended site origin, preserving necessary existing rules.
No. It only prepares text. Deployment and later crawler retrieval are separate steps.