What is robots.txt?
Photo: Avila Ops Tecnologia
robots.txt is a plain text file at your domain’s root (suaempresa.com.br/robots.txt), instructing search robots which website areas they may visit. It is the first thing Google’s robot reads on arrival. It keeps robots away from useless areas rather than hiding pages.
Its risks matter more than its benefits. One wrong instruction blocks the entire website from Google while it still opens normally for you.
What does robots.txt control and what does it not?
It controls crawling: “do not enter this folder” or “you may enter everything.” Common uses include blocking administrative areas, internal search results, shopping carts and filter pages generating thousands of useless combinations.
It does not control indexing. Google may index a blocked page receiving external links without reading its content, displaying only its address. To truly remove a page from Google, use noindex markup inside the page, which robots must be able to read. Blocking in robots.txt while adding noindex defeats noindex because robots never see it.
A Rio Preto language school changed websites in January. The developer left the new website’s test-setting Disallow: / (block everything) in place after publication. By March, the website had disappeared from Google.
How do I check mine?
Enter seudominio.com.br/robots.txt in the browser. If a short text opens, read it. The dangerous line is Disallow: / directly below User-agent: *. This blocks everything for everyone.
Include a Sitemap: line pointing to your sitemap so robots find both together.
What to do now
Open robots.txt now and look for Disallow: /. If present on a public website, remove it today. At Avila Ops, robots.txt is reviewed before every launch and after every migration as part of a website Google understands: avilaops.com/presenca-digital-para-pequenas-empresas or WhatsApp wa.me/5517997811471.
Frequently asked questions
Can robots.txt hide a page from Google?
Not reliably. It prevents reading rather than indexing. To keep a page out of results, use noindex and let robots read it.
Does robots.txt apply to AI tools too?
It applies to robots respecting the standard, and major AI crawlers say they do. A separate file, llms.txt, is designed to guide AI; see our llms.txt guide.
My website has no robots.txt. Do I need one?
It is not mandatory. Without it, Google crawls everything. Create one when you need to block areas such as login pages or store filters, or point to the sitemap.
Turn what you learned into your next step.
Tell us about your company and what you want to improve. We can explore the right tools, integrations or custom project together.