How to Crawl XML Sitemaps with Python

While sitemaps make crawling easier, manually parsing them can be tricky:

• Many websites use index sitemaps that link to multiple smaller sitemaps, requiring additional steps to process.

• Large sitemaps can contain thousands of URLs, making extraction time-consuming and inefficient.

This is where ultimate-sitemap-parser (USP) comes in. It’s a powerful Python library built to handle complex sitemaps with ease. USP can:

• Fetch and parse XML sitemaps automatically.

• Handle nested index sitemaps without extra coding.

• Extract URLs efficiently with just a simple function call.

In this tutorial, we’ll walk you through how to use ultimate-sitemap-parser in Python to crawl the ASOS sitemap and extract all available URLs quickly. This approach saves time, reduces complexity, and provides a scalable way to handle sitemap crawling for SEO projects.

To run the sitemap crawler, you’ll need Python 3.x installed on your computer.

• Download the latest version of Python from the official Python website.

• After installation, confirm it’s working by running the command below in your terminal or command prompt:

    python3 --version

If Python is installed correctly, this will display the installed version number.

The ultimate-sitemap-parser library makes it simple to fetch and parse sitemaps in Python. Install it using pip:
    pip install ultimate-sitemap-parser
Once installed, you’re ready to start building your Python sitemap crawler.

The USP library makes it simple to fetch and parse XML sitemaps without manually handling XML files. With just a few lines of code, you can extract every URL from a sitemap.

from usp.tree import sitemap_tree_for_homepage

# Define the target website
url = "https://www.asos.com/"

# Fetch and parse the sitemap
tree = sitemap_tree_for_homepage(url)

# Extract and print all URLs
for page in tree.all_pages():
    print(page.url)

This script will automatically fetch the ASOS sitemap and print all available URLs.

A big advantage of USP is that it can automatically handle nested index sitemaps. Many websites (including ASOS) split their sitemaps into smaller files, such as:

• Product pages
• Category pages
• Blog or content pages

USP will:

• Detect index sitemaps.
• Fetch linked sub-sitemaps.
• Recursively extract URLs from all of them.

👉 The script above already handles this seamlessly — no extra code needed.

Sometimes you only want specific types of pages, such as product URLs. USP makes this easy with simple filtering.

For example, ASOS product URLs usually contain /product/. Here’s how you can filter only product pages:

# Extract and filter only product page URLs
product_urls = [page.url for page in tree.all_pages() if "/product/" in page.url]

# Print filtered URLs
for url in product_urls:
    print(url)

This way, you only crawl the URLs that matter to your project.

Instead of printing the URLs, you might want to save them for later use. Here’s how to export the extracted URLs into a CSV file:

import csv
from usp.tree import sitemap_tree_for_homepage

# Define the target website
url = "https://www.asos.com/"

# Fetch and parse the sitemap
tree = sitemap_tree_for_homepage(url)

# Extract all URLs
urls = [page.url for page in tree.all_pages()]

# Save URLs to a CSV file
csv_filename = "asos_sitemap_urls.csv"
with open(csv_filename, "w", newline="", encoding="utf-8") as file:
    writer = csv.writer(file)
    writer.writerow(["URL"])  # Write header
    for url in urls:
        writer.writerow([url])

print(f"Extracted {len(urls)} URLs and saved to {csv_filename}")

Running this script will extract all sitemap URLs and save them into a CSV file, making them easy to analyze or use in other projects.