Search
Hand crawlers a map of the pages you want found
Your robots file does not point at a sitemap, or the address it points at does not serve a reachable XML file, so crawlers have no list of your pages.
Why does it matter?
A sitemap is a plain list of the pages you want indexed, with the date each one last changed. Crawlers use it to find pages that few other pages link to, and to notice updates sooner. Without one the site can still be crawled, but new pages wait longer to appear, and pages reachable only through your own navigation may wait a very long time.
How would you notice it?
- New pages take weeks to appear in search results.
- Search Console reports that no sitemap was found, or that it could not be read.
- A page you know exists never gets indexed.
How do you fix it?
Our sitemap is written at build time from our own route data, and the first build taught us a lesson worth passing on.
Stage 1
- Make sure a real sitemap file exists at a stable address, usually /sitemap.xml. Most site builders and plugins generate one for you.
A hand-built site can start with the format above and add one url entry per page. List the pages you want found, not every file on the server.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.example.com/services/</loc>
</url>
</urlset>Stage 2
- Declare the sitemap in robots.txt with a Sitemap line naming the full address.
The address must be the one that actually answers, spelled exactly. A declared address that 404s is worse than no declaration, because the check reports it as broken rather than absent.
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlStage 3
- Open the declared address in a browser and confirm you see XML, starting with a urlset element, not a page-not-found screen and not your homepage.
Some hosts answer every unknown address with the homepage and a friendly success status. That is not a sitemap, and the check reads the first bytes of the file to tell the difference.
Check the bytes your generator writes
Our first sitemap build reported success while writing the literal text undefined/ as the host on every line, because the address we fed the generator arrived in a shape the code did not expect, and generators rarely shout about that. A sitemap full of addresses that do not resolve is worse than no sitemap at all, because crawlers record the failure. Open the file your own tool produced and read the first lines yourself before you publish it, and point one check at the bytes rather than at the tool exit message.
Source: Build-time sitemap generation audit on noblemarketglobal.com
How do you check it worked?
Confirm it worked
Open the sitemap address you declared and confirm the browser shows XML with your pages inside it. Add the same address to Search Console, which reports whether it was accepted. Re-run the Siege Test and confirm the Sitemap recommendation is gone.