robots.txt
robots.txt is a small file at the root of your site that tells search engine crawlers which pages or sections they may or may not crawl. It is a set of polite instructions for automated visitors.
Why does robots.txt matter to your business?
Misconfigured, robots.txt can accidentally hide your whole site from search — a surprisingly common and costly mistake. Configured well, it keeps crawlers focused on the pages that matter and away from admin or duplicate areas. It is small but consequential.
At Nabee, robots.txt is set correctly as part of the build, so search engines can find what should be public and skip what should not. It is exactly the kind of easy-to-miss detail one partner takes off your plate.
A concrete example
A site with a public marketing area and a private admin area uses robots.txt to keep crawlers on the public pages. Getting it right protects both your visibility and your less-relevant areas from being indexed.
Related terms
Common questions
- Can robots.txt hurt my SEO?
- Yes, if misconfigured it can hide pages you want indexed. That is why Nabee sets it correctly during the build.
- Does robots.txt make pages private?
- No — it only asks crawlers not to index them. Truly private content needs proper access controls.
- Do I need to edit robots.txt myself?
- No. Nabee configures and maintains it as part of the build and maintenance.