Language: English
Request a service

Robots.txt for E-commerce: Best Practices and Examples for WooCommerce, Magento and Shopify

A good robots.txt helps search engines spend their time on your products and categories instead of filters, carts and search results. Here are best practices and examples per platform.

Hasan Gültekin4 min read

On an online store, search engines can find an almost endless number of URLs: filter combinations, sort orders, internal search results, cart and account pages. A well-written robots.txt file tells crawlers where not to go, so they spend their time on the pages that matter.

A badly written one can also remove your entire store from Google. This guide covers best practices and ready-to-use examples for WooCommerce, Magento 2 and Shopify.

What robots.txt does (and does not do)

robots.txt is a text file at the root of your domain (https://www.example.com/robots.txt) with rules for crawlers.

  • It controls crawling: which URLs bots may request
  • It does not reliably control indexing. A blocked URL can still appear in Google if other pages link to it, just without content
  • It is public. Never use it to hide sensitive areas; protect those with authentication

To keep a page out of search results, use a noindex meta tag or header and make sure the page is not blocked in robots.txt, otherwise Google cannot see the noindex.

Basic syntax

User-agent: *
Disallow: /checkout/
Allow: /checkout/assets/

Sitemap: https://www.example.com/sitemap.xml
  • User-agent – which crawler the rules apply to (* = all)
  • Disallow – path prefix that may not be crawled
  • Allow – exception inside a disallowed path
  • * matches any characters and $ marks the end of the URL (supported by Google and Bing)
  • Sitemap – absolute URL to your XML sitemap

What to block on an online store

Usually safe to block:

  • Cart, checkout and account pages
  • Internal search result pages
  • Sort, view and limit parameters
  • Filter parameter combinations that you do not want indexed
  • Compare, wishlist and “add to cart” URLs

What never to block

  • CSS and JavaScript files. Google needs them to render your pages
  • Product images (unless you deliberately do not want image search traffic)
  • Category and product pages
  • Pages with a noindex tag you want Google to see
  • Your whole site after launch. Check that the Disallow: / from staging is gone

WooCommerce robots.txt example

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?s=
Disallow: /*add-to-cart=
Disallow: /*?orderby=
Disallow: /*&orderby=
Disallow: /*?filter_
Disallow: /*&filter_

Sitemap: https://www.example.com/sitemap_index.xml

Adjust the cart, checkout and account paths if your store uses translated slugs (for example /winkelwagen/ or /warenkorb/). The sitemap URL depends on your SEO plugin.

Magento 2 robots.txt example

In Magento 2 you can edit robots.txt under Content → Design → Configuration → (store view) → Search Engine Robots.

User-agent: *
Disallow: /checkout/
Disallow: /customer/
Disallow: /wishlist/
Disallow: /catalog/product_compare/
Disallow: /catalogsearch/
Disallow: /sendfriend/
Disallow: /review/product/
Disallow: /*?dir=
Disallow: /*&dir=
Disallow: /*?product_list_order=
Disallow: /*&product_list_order=
Disallow: /*?product_list_limit=
Disallow: /*&product_list_limit=
Disallow: /*?product_list_mode=
Disallow: /*&product_list_mode=
Disallow: /*SID=

Sitemap: https://www.example.com/sitemap.xml

Layered navigation filters (?color=, ?size=) need a deliberate decision: block the combinations that have no search demand, and keep the valuable ones crawlable and indexable. See the faceted navigation section in my e-commerce technical SEO checklist.

Shopify robots.txt

Shopify generates a sensible default robots.txt that already blocks cart, checkout, account, internal search and several parameter URLs. You can customise it by adding a robots.txt.liquid template to your theme.

Only customise when there is a clear problem, for example a filter app creating thousands of crawlable URLs. Keep Shopify’s default rules and add your own on top. More in my Shopify SEO checklist.

AI crawlers

Besides search engine bots, AI companies run their own crawlers. You can address them by user agent in robots.txt. Decide deliberately:

  • Allow them if you want your products and content to appear in AI assistants (see GEO for online stores)
  • Block specific ones if you do not want your content used, keeping in mind this may reduce visibility in those tools

Check each provider’s documentation for the current user agent names, since they differ between training and live-search crawlers.

Common mistakes

  1. Disallow: / left over from staging. The single most damaging robots.txt mistake.
  2. Blocking pages that have noindex. Google never sees the noindex and the URLs may stay indexed.
  3. Blocking CSS/JS folders. Leads to poor rendering and evaluation.
  4. Forgetting language versions. Rules for /cart/ do not cover /de/warenkorb/.
  5. Using robots.txt to remove pages from Google. Use noindex or removal tools instead.
  6. Case and trailing slash errors. Paths are case-sensitive.

How to test your robots.txt

  • The robots.txt report in Google Search Console shows which robots.txt files Google found and any errors
  • Use URL Inspection to check whether a specific URL is blocked
  • Crawl your site with a crawler that respects robots.txt and compare the crawled URLs with what you expect

Frequently asked questions

Does robots.txt improve rankings?

Not directly. It helps search engines spend crawl time on important pages, which matters most on large stores with many filter URLs.

Should a small store worry about crawl budget?

Usually not much. For stores with thousands of products and filters, it matters a lot.

How quickly does Google pick up robots.txt changes?

Google generally caches robots.txt for up to about a day, so changes are usually picked up quickly.

Need help with crawling and indexing?

I fix crawl and indexing problems for WooCommerce, Magento and Shopify stores, including faceted navigation and robots.txt. See my technical SEO service or request an audit.