Preskoči na sadržaj
(+381) 60 3535 720
Moj nalog
BeoHosting
BeoHosting
Moj nalogKontakt
(+381) 60 3535 720
 

Free SEO tool

Robots.txt Generator - Create Robots.txt

Generate a properly formatted robots.txt file for your site. Choose user agents, set allow/disallow rules, specify a sitemap, and control how search engines index your content.

Create robots.txtSEO guide
TL;DR

How to create a robots.txt file for a site?

The Robots.txt Generator creates a properly formatted file in 4 steps: 1) Enter the Sitemap URL (e.g. https://yourdomain.com/sitemap.xml) — helps search engines discover all pages faster. 2) Configure the User-agent — choose from 12 options: Googlebot, Bingbot, DuckDuckBot, Yandexbot, Baiduspider, Facebook bot, Twitter bot, plus AI bots GPTBot (OpenAI), Google-Extended, CCBot (Common Crawl), anthropic-ai. 3) Add Disallow paths (admin, wp-admin, api, login, cart) and Allow exceptions. Optional Crawl-delay (Google ignores, Bing/Yandex respect). 4) Live preview shows the generated file, click Copy → paste into public_html/robots.txt. Useful for: optimizing crawl budget, blocking AI scraping (protecting copyrighted content), preventing duplicate indexing. The Robots.txt Generator is 100% free, with a drag-friendly visual editor. BeoHosting also offers cPanel File Manager for upload from $4.66/month.

  • Visual editor with 12 user agents + AI bot options
  • Allow/Disallow/Crawl-delay rules + Sitemap URL
  • Live preview with one-click copy
  • BeoHosting cPanel File Manager for upload from $4.66/month

BeoHosting Team

10+ godina iskustva — Stručnjaci za web hosting i infrastrukturu

Poslednje ažurirano:

Robots.txt Generator

Konfigurisite pravila za pretraživače i generisajte robots.txt za vaš sajt.

Pravilo #1

/admin//cgi-bin/
Brzo dodaj:

Google ne postuje crawl-delay. Bing i Yandex ga postuju.

robots.txt
# robots.txt generated by BeoHosting
# 2026-06-17

User-agent: *
Disallow: /admin/
Disallow: /cgi-bin/

Uputstvo:Kopirajte generisani sadržaj i sačuvajte ga kao robots.txt u root direktorijumu vašeg sajta (public_html/). Fajl mora biti dostupan na vassajt.rs/robots.txt.

Best practices

Robots.txt - What you need to know

Proper robots.txt configuration is key to SEO and controlling indexing of your site.

Always add a Sitemap

Specify the path to your sitemap.xml file in robots.txt. This helps search engines discover and index all pages of your site more efficiently.

Don't block CSS/JS

Google needs access to CSS and JavaScript files to properly render the page. Blocking these resources can negatively impact SEO ranking.

Use Allow for exceptions

If you block a directory, you can allow access to specific pages within it using the Allow directive. Allow takes priority over Disallow.

Block AI bots

If you don't want AI companies to use your content for training models, block GPTBot, Google-Extended, CCBot, and anthropic-ai user agents.

Test before publishing

Use the Google Search Console robots.txt testing tool before publishing. A wrong configuration can block indexing of the entire site.

Don't use for protection

robots.txt is only a recommendation — malicious bots ignore it. Don't use it to hide sensitive pages. Use a password, noindex meta tag, or HTTP authentication.

Robots.txt Guide - How to control site indexing

Robots.txt is a standard file used by web search engines (Google, Bing, Yandex) to learn which parts of your site they may visit and index. According to the Robots Exclusion Protocol, every well-behaved bot will first read your robots.txt before starting to crawl.

Proper robots.txt configuration is important for SEO because it helps search engines efficiently use their crawl budget — the number of pages Google visits during each crawl of your site. By blocking irrelevant pages (admin panels, search results, user pages), you direct bots to the most important content.

Robots.txt and BeoHosting

On all BeoHosting hosting plans, you can easily create a robots.txt file through cPanel File Manager. For WordPress sites, robots.txt can also be configured through a plugin like Yoast SEO or Rank Math. Our SEO guide for beginners covers this topic in detail.

It's important to note that robots.txt is not a security mechanism. Malicious bots ignore it. For real protection of private pages, use HTTP authentication, noindex meta tag, or password access. Robots.txt should be used exclusively to optimize indexing, not to hide content.

Related tools and services

Meta Tag AnalyzerSSL CheckerDNS CheckerGenerator LozinkiBase64 EnkoderWHOIS LookupSEO vodičWordPress HostingShared HostingKontakt

Spremni da pokrenete svoj sajt?

SSL zaštita
Brzina
24/7 podrška

Pridružite se 4.000+ zadovoljnih korisnika. Besplatna migracija i 15 dana garancije povrata novca.

Izaberite paketKontaktirajte nas
15 dana garancija povrata novca
Besplatna migracija15 dana garancija24/7 podrška

Frequently asked questions - Robots.txt Generator

Odgovori na najčešća pitanja o našim uslugama.

robots.txt is a text file located in the root directory of your site (e.g. site.com/robots.txt) and tells search engines (Googlebot, Bingbot) which pages they may and may not visit and index. It is part of the Robots Exclusion Protocol standard. Without robots.txt, search engines index all accessible pages. With it, you can prevent indexing of admin pages, duplicates, private sections, and unnecessary pages.

Yes, directly and indirectly. Robots.txt helps search engines efficiently use crawl budget (the number of pages a bot visits per session). By blocking irrelevant pages (admin, login, filters), you direct bots to important content. Also, specifying the sitemap URL speeds up discovery of new pages. However, blocked pages will NOT be removed from search results — for that use the noindex meta tag.

Robots.txt must be accessible at yourdomain/robots.txt (e.g. https://site.com/robots.txt). On BeoHosting: go to cPanel File Manager, open the public_html folder, create a new "robots.txt" file, and paste the generated content. The file must be in the site's root directory, not in a subfolder. Check that it is accessible by opening the URL in a browser.

Yes. Add a separate block for each AI bot with a Disallow: / directive. GPTBot (OpenAI/ChatGPT), Google-Extended (Google AI), CCBot (Common Crawl), and anthropic-ai (Anthropic/Claude) respect robots.txt. Our generator has an option for all these user agents. This prevents your content from being used to train AI models.

Crawl-delay tells the bot how many seconds to wait between requests. It's useful if you have a weak server that can't handle intense crawling. Google does NOT respect Crawl-delay (use Search Console to adjust), but Bing, Yandex, and other bots do. For BeoHosting customers, crawl-delay is usually not necessary because our servers have enough resources.

Naše garancije za vaš mir

Zaštićeni ste sa svake strane

15 dana garancije

Vraćamo novac bez pitanja u prvih 15 dana.

Besplatna migracija

Mi prebacimo vaš sajt bez prekida — vi ništa ne radite.

24/7 podrška

Naši stručnjaci su tu 24/7 kroz tikete i live chat.