XML Sitemaps and Robots.txt: A Practical Guide for UK Business Websites
If a developer or SEO consultant has ever mentioned your robots.txt file, or asked whether your sitemap is up to date, and you weren't quite sure what either one actually does, you're not alone. XML sitemaps and robots.txt are two small text files that sit quietly in the background of almost every website, yet they have an outsized effect on how search engines crawl and index your pages. Get them right, and Google finds your important pages faster. Get them wrong, and you can accidentally hide pages you want ranking, or send search engines mixed signals about what matters on your site.

What Are XML Sitemaps and Robots.txt, and Why Do They Matter?
An XML sitemap is a file that lists the pages, and sometimes the images, videos or news articles, on your site, and shows how they relate to one another. Google's own documentation explains that a sitemap helps search engines crawl a site more efficiently, and can include details such as when a page was last updated. Robots.txt does a different job entirely: it's a plain text file, placed in your site's root directory, that tells search engine crawlers which parts of your site they're allowed to access. Google is explicit that robots.txt is mainly used to manage crawler traffic and to keep crawlers away from unimportant or duplicate pages, rather than to decide what does or doesn't appear in search results.
You're more likely to need a sitemap if your site is large, it's new with few external links pointing to it, or it has a lot of videos, images or news content you want to appear in Google's specialised search results.
You probably don't need one if your site has around 500 pages or fewer that should appear in search results, and those pages are already thoroughly linked to one another internally.
What Should Be in Your XML Sitemap
A good sitemap lists only the canonical URLs you actually want indexed, not duplicate versions, redirected URLs, or pages you've already blocked from indexing elsewhere. Google is clear that listing a URL in your sitemap doesn't guarantee it will be crawled or indexed, so it's best to think of a sitemap as a strong hint rather than an instruction. Most website platforms, including Wix, generate a basic sitemap automatically, usually at yourdomain.co.uk/sitemap.xml, which is one less thing for a busy business owner to build from scratch.
Once your sitemap exists, submit it through Google Search Console and check back on it periodically. We've covered how to read the Sitemaps report in Google Search Console, alongside the other reports that matter, which shows you how to spot URLs that are submitted but not indexed.
What Robots.txt Can (and Can't) Do for Your Site
This is where a lot of confusion creeps in. Robots.txt controls crawling, not indexing. Google's own guidance states plainly that robots.txt is not a way to keep a page out of Google: a URL you disallow can still appear in search results, often without a description, if other sites link to it. If you genuinely want a page kept out of search results, you need a noindex meta tag or password protection instead, and the page still needs to be crawlable for Google to see that noindex instruction in the first place.
User-agent names which crawler a group of rules applies to (an asterisk matches all crawlers, except a handful such as AdsBot that must be named directly).
Disallow blocks a specific path from being crawled.
Allow re-opens a path inside a directory you've otherwise disallowed.
Sitemap gives crawlers the full URL of your sitemap, and can appear more than once in the file.
The file must be named exactly robots.txt, saved as plain UTF-8 text, and placed at your domain's root (yourdomain.co.uk/robots.txt) — a copy in a subfolder has no effect, and each subdomain or protocol needs its own file.
If your site is built on Wix, you don't need to upload a robots.txt file by hand. Under SEO & GEO > Tools and settings in your site's dashboard, Wix provides a Robots.txt Editor where you can add your own allow and disallow directives on top of the sensible defaults Wix already applies. Wix's own support documentation describes this as an advanced feature that can affect how your site appears in search results, and notes that its support team can't troubleshoot changes you make yourself — so it's worth treating it with care, or asking an SEO specialist to check your directives before you save them.
Common XML Sitemaps and Robots.txt Mistakes to Avoid
Blocking pages you actually want to rank in robots.txt, then wondering why they never show up in search results.
Assuming a Disallow rule removes a page from Google altogether — it only stops crawling; use a noindex tag for genuine removal.
Editing robots.txt in Word or another word processor, which can insert curly quotes and formatting that breaks the rules.
Placing robots.txt anywhere other than your domain's root directory, where crawlers won't find it.
Listing URLs in your sitemap that are themselves blocked by robots.txt, redirected, or set to noindex — sending search engines mixed signals about what you actually want indexed.
Leaving an old sitemap in place after a redesign or site migration, full of URLs that no longer exist.
Blocking CSS or JavaScript files in robots.txt without realising it — if Google can't load your scripts and styles, it can struggle to understand what a page actually looks like and contains.
This overlaps with our broader technical SEO audit checklist, since sitemap and robots.txt errors are exactly the kind of thing that checklist is designed to catch early, before they quietly cost you rankings.
How to Check Your XML Sitemaps and Robots.txt
View your live robots.txt file by typing yourdomain.co.uk/robots.txt into a private or incognito browser window.
View your live sitemap the same way, at yourdomain.co.uk/sitemap.xml.
Use the robots.txt report inside Google Search Console to check Google's cached copy for errors, and the Sitemaps report to see which submitted URLs have actually been indexed.
If you're technically minded, Google's open-source robots.txt parsing library lets you test specific rules before you publish them.
If any of this throws up pages you didn't expect to see blocked, or a sitemap full of out-of-date URLs, it's worth getting your XML sitemaps and robots.txt checked by our team, rather than guessing at a fix.
Get Expert Help With Your Technical SEO
Sitemaps and robots.txt are only two pieces of a much bigger technical SEO picture — crawlability, site speed, structured data and internal linking all play a part too. If you're not sure where your site currently stands, our free SEO audit checks all of this for you and tells you exactly what to fix first.




Comments