HOW TO PROTECT YOUR WEBSITE FROM GOOGLE INDEXING ALL PAGES
These stop search engines from indexing private or duplicate content pages like login, user areas, or shopping carts.
Prevents thin content or sensitive data from being exposed in search results.
1. User-agent
[User-agent: *]
This applies to all crawlers (Googlebot, Bingbot, Ahrefs, etc.).
If you wanted to target only one crawler, you’d replace * with the bot name (e.g., User-agent: Googlebot).
2. Blocked Sensitive Routes
Disallow: /admin
Disallow: /dashboard
Disallow: /login
Disallow: /logout
Disallow: /register
Disallow: /password
Disallow: /cart
Disallow: /checkout
Disallow: /user
These stop search engines from indexing private or duplicate content pages like login, user areas, or shopping carts.
Prevents thin content or sensitive data from being exposed in search results.
3. Disallow Laravel Internals
Disallow: /api/
Disallow: /storage/
Disallow: /vendor/
Disallow: /resources/
Disallow: /node_modules/
Disallow: /cgi-bin/
Blocks crawlers from accessing Laravel’s system folders.
/storage, /vendor, /resources, /node_modules, /cgi-bin should never be crawled — they’re backend-only.
/api/ is blocked so that search engines don’t crawl JSON or API endpoints (which could waste crawl budget).
4. Query String Blocking (SEO cleanup)
Disallow: /*?*
Disallow: /*&*
Stops Google from crawling endless query string URLs (e.g., ?page=2, ?utm=ads, &sort=asc).
This prevents duplicate content issues and keeps your crawl budget focused on real pages.
Optional but highly recommended for SEO if your site generates lots of query variations.
5. Sitemap Location
Sitemap: https://yourdomain.com/sitemap.xml
Directs crawlers to your sitemap so they can efficiently discover your main pages.
Replace yourdomain.com with the actual domain name (or any real domain).
Why This is Good
Protects private areas (no login or dashboard in Google).
Optimizes crawl budget (Google won’t waste time on useless URLs).
Improves SEO by focusing bots on valuable content.
Security-conscious (keeps system folders hidden).
Final Version for Sailexhost Example
# robots.txt for Sailexhost (Laravel + Hosting)
User-agent: *
Disallow: /admin
Disallow: /dashboard
Disallow: /login
Disallow: /logout
Disallow: /register
Disallow: /password
Disallow: /cart
Disallow: /checkout
Disallow: /user
Disallow: /api/
Disallow: /storage/
Disallow: /vendor/
Disallow: /resources/
Disallow: /node_modules/
Disallow: /cgi-bin/
# Prevent crawling query strings (SEO cleanup)
Disallow: /*?*
Disallow: /*&*
# Sitemap
Sitemap: https://yourdomain.com/sitemap.xml
