How to find every page on a website
Published November 11, 2026 by Hill Country Bots
Whether you are planning a redesign, auditing SEO, or studying a competitor, you usually start with the same question: what pages does this site actually have? There is no single button for it, but there are six reliable methods. Here they are, from quickest to most thorough, with what each one misses.
1. Read the XML sitemap
Most sites publish a sitemap, a file that lists the pages the owner wants search engines to find. To locate it:
- Open
example.com/robots.txtand look for a line that starts withSitemap:. - If there is none, try
/sitemap.xmland/sitemap_index.xml. - Open the file. If it lists other sitemap files rather than pages, it is a sitemap index, and you need to open each child file too.
Do not skip step 1. When we audited docs.apify.com on November 11, 2026, its sitemap lived at /sitemap_base.xml, a path you would not guess. robots.txt pointed straight to it.
What it misses: anything the owner left out, and on some sites, pages a plugin forgot to include.
2. Export the sitemap to a spreadsheet
Reading XML in a browser is fine for 30 URLs. For 3,000 spread over a dozen gzip files, use a tool. Our sitemap to CSV extractor finds the sitemap through robots.txt, follows every nested index, unpacks gzip files and gives you one clean table. In a capped test run on sitemaps.org and crawlee.dev it pulled 40 URLs from three sitemap files, with the lastmod date and source file for each:
url,lastmod,changefreq,priority,sourceSitemap
https://www.sitemaps.org/protocol.html,2022-12-15,,,https://www.sitemaps.org/sitemap.xml
https://crawlee.dev/blog/archive,,weekly,0.5,https://crawlee.dev/sitemap.xml
https://crawlee.dev/python/blog,,weekly,0.5,https://crawlee.dev/python/sitemap.xmlThe lastmod column is worth sorting. It shows which sections the site has been updating lately.
3. Crawl the site by following links
Some sites have no sitemap at all. In our November 11 test, web.dev had no XML sitemap at the usual paths, so our SEO audit fell back to crawling: it started at the homepage and followed internal links, page by page. Crawling finds what a visitor can actually reach, and it also catches broken links along the way.
What it misses: orphan pages that no other page links to. If a page is in the sitemap but never shows up in a crawl, that is a signal in itself, because visitors and search engines will struggle to find it.
4. Check what Google has indexed
Search Google for site:example.com. The results are pages Google has indexed. The count at the top is a rough estimate, and Google will not page through thousands of results, so treat it as a sanity check. If you own the site, Google Search Console gives a much better list under Pages, including the ones Google found but chose not to index.
5. Try the default sitemap path for the platform
If robots.txt has no Sitemap line, the platform the site runs on usually gives it away. Each popular platform puts its sitemap in a predictable place:
| Platform | Where the sitemap usually is |
|---|---|
| WordPress (built in, version 5.5 and later) | /wp-sitemap.xml |
| WordPress with Yoast SEO or Rank Math | /sitemap_index.xml |
| Shopify | /sitemap.xml, an index of product, collection, page and blog sitemaps |
| Wix, Squarespace, Webflow | /sitemap.xml |
Not sure which platform a site uses? Our guide on how to find out what CMS a website uses shows the quick tells. A WordPress sitemap is split into files of up to 2,000 entries by default, so a big blog will show up as an index with many children.
6. Look up old pages in the Wayback Machine
The Internet Archive's Wayback Machine keeps copies of public pages it has crawled over the years. Search for example.com/* and it lists the addresses it has seen under that site, including pages that were removed or renamed. That is handy before a redesign, when you need to redirect old URLs that still get links. Two cautions: it only knows pages its crawler found, and the service is sometimes down. It was offline when we tried it on November 11, 2026, so have a second method ready.
Which method to use
| Method | Best for | Misses |
|---|---|---|
| XML sitemap | The fastest full list on most sites | Pages the owner or plugin left out |
| Platform sitemap path | Sites with no Sitemap line in robots.txt | Custom setups that moved the file |
| Sitemap export to CSV | Big sites, nested indexes, sorting by date | The same gaps as the sitemap |
| Crawling links | What visitors can actually reach, plus broken links | Orphan pages with no links pointing at them |
| Google site: search or Search Console | What Google has indexed | Pages Google skipped; site: counts are estimates |
| Wayback Machine | Old and removed URLs | New pages, and anything never archived |
Put the methods together
For a real audit, use at least two. Pull the sitemap, crawl the site, and compare. Pages in the sitemap but not in the crawl are orphans. Pages in the crawl but not in the sitemap are missing from the sitemap. Pages that return errors in either list need fixing first.
Once you have the list, the useful work starts. Check every page for speed with the bulk Core Web Vitals checker, and for titles, descriptions and headings with the website SEO audit. Want to see what the site is built on while you are at it? Check what technology it uses.
Questions
How do I find all pages on a website for free?
Open the site's robots.txt and look for a Sitemap line, then open that sitemap in your browser. For a quick rough count you can also search Google for site:example.com. Both are free. For big sites with nested sitemaps, a tool saves a lot of copying.
Does a site: search show every page?
No. It shows pages Google has indexed and chose to show, and the result count is an estimate. It is a useful sanity check, not a full list.
What if a website has no sitemap?
Then you crawl it: start at the homepage and follow every internal link. Pages that nothing links to, called orphan pages, will not show up this way, so compare against the sitemap whenever there is one.
Can I find hidden pages on someone else's site?
Only what is public. Sitemaps and links show what the owner chose to publish. Guessing private addresses or getting past logins is not something we do or recommend.
How do I find orphan pages?
Compare two lists. Pull every URL from the sitemap, then crawl the site from the homepage. URLs that are in the sitemap but never show up in the crawl have no internal links pointing at them. Those are your orphan pages.
How many pages can a sitemap hold?
One sitemap file can list up to 50,000 URLs and be up to 50 MB uncompressed. Bigger sites split their URLs across several files and list those files in a sitemap index.