What robots.txt actually controls
A robots.txt file sits at the root of a domain and gives crawling bots a set of rules about which paths they're allowed to request. It is a courtesy convention that well-behaved crawlers, including major search engines, choose to respect — it is not a security mechanism, and it does not remove a page from search results by itself. A disallowed page can still occasionally appear in search results (typically with no snippet) if enough other pages link to it, since blocking a crawl only stops the bot from reading the page, it doesn't stop the URL from being known to exist. For pages you genuinely want removed from search results, a "noindex" meta tag or HTTP header is the correct tool, used alongside or instead of a robots.txt rule.
Each rule set in a robots.txt file starts with a "User-agent" line naming which bot the following rules apply to, with an asterisk meaning "all bots that don't have a more specific rule elsewhere in the file." One or more "Disallow" lines then list path prefixes that bot should avoid requesting. An optional "Crawl-delay" line asks a bot to wait a given number of seconds between requests, which some crawlers respect and others ignore entirely; it's mainly useful for smaller or lesser-known bots that might otherwise hit a server harder than its hosting plan can comfortably handle. A "Sitemap" line, which can appear anywhere in the file, points crawlers to your XML sitemap so they can discover your full URL list efficiently.
Common mistakes this generator helps you avoid
A surprisingly common and costly mistake is a stray "Disallow: /" left over from a staging environment, which blocks a bot from crawling the entire site rather than one folder — this generator makes each disallowed path an explicit, visible row so it's harder to leave an overly broad rule in place by accident. Another frequent issue is applying a rule meant for one crawler to every bot by leaving the user-agent as an asterisk, when the intent was actually to slow down or block a specific aggressive bot while leaving major search engines untouched; adding a separate rule group with that bot's specific name, rather than editing the universal group, keeps the two concerns separate.
It's also worth double-checking the sitemap URL you enter uses the full absolute address, including the scheme and domain, rather than a relative path — crawlers expect a complete URL on the Sitemap line since the file can technically be referenced from robots.txt files on other domains in some multi-site setups, so a relative path would be ambiguous. Once generated, upload the resulting file to your domain's root directory as exactly "robots.txt" — placing it in a subfolder means most crawlers will never find it, since the convention is to look specifically at the root.
Testing your robots.txt before it goes live
Before uploading a new robots.txt to a production site, read through the generated rules one more time as if you were the crawler: for each user-agent group, would the listed disallow paths block anything you actually want indexed? A quick way to sanity-check this is to picture your site's main navigation and ask whether any of those primary pages fall under a disallowed path — if one does, that's usually a sign the path is either too broad or was meant for a different, more specific folder. Most search engines also provide a robots.txt testing tool in their webmaster consoles that lets you check a specific URL against your live file before or after publishing, which is worth using any time you make a significant change to the rules.