How to Generate a Sitemap from a Protected API Using HTTP Headers

Why protected API sitemap generation matters

A lot of websites do not expose their content API openly. That does not always mean the content is private. Sometimes the API is protected simply because the backend requires authentication for every request, even when the returned data represents public website pages.

For example, an e-commerce website may have public product pages, but the product data may come from an API endpoint like:

And that endpoint may require a header like Authorization: Bearer YOUR_TOKEN or x-api-key: YOUR_API_KEY. Without that header, the API returns an error. With the header, it returns the product data needed to build the sitemap.

That is where a sitemap generator with HTTP headers becomes useful. Instead of making the endpoint public or writing custom sitemap code, you can authorize the API request and map the returned JSON fields into sitemap URLs.

Common use cases for protected API sitemaps

Protected API sitemap generation is useful when the URL source exists behind an authenticated or semi-private endpoint. Common use cases include:

The public page URL and the private data source are not always the same thing. Your sitemap should include public page URLs, but your API request may still need authorization to discover those URLs.

• Headless CMS APIs that require API keys
• Product catalog APIs protected by bearer tokens
• Private REST APIs used by public websites
• Staging APIs that need authorization
• Internal content APIs for SaaS websites
• Documentation systems behind custom headers
• Marketplace listings stored in protected backend routes
• Real estate listings from authorized data feeds
• Job board APIs that require tokens
• Backend services that require Accept, Authorization, or x-api-key headers

How SitemapFlow uses HTTP headers

SitemapFlow lets you define custom headers when connecting an API endpoint. After the API request is authorized, SitemapFlow reads the response and uses your URL pattern to generate sitemap entries.

For example, if the protected API returns a list of products with slugs, you can define the URL pattern as /products/[response.slug].

The header is used to access the API data. The final sitemap contains the public page URLs.

HTTP headers commonly used for sitemap APIs

Different APIs require different headers. The most common header types are:

SitemapFlow lets you define the header key and value, so the field name can match the exact requirement of the API you are connecting.

• Authorization bearer token: (Authorization: Bearer YOUR_TOKEN) Common for private APIs, CMS endpoints, and SaaS backends.
• API key header: (x-api-key: YOUR_API_KEY) Other systems may use api-key, x-access-token, or x-auth-token.
• Accept header: (Accept: application/json) Useful when an endpoint supports multiple response formats.
• Custom headers: (x-tenant-id: store-123 or x-locale: en) Used for internal routing, tenants, locales, or versions.

Examples of Protected API Sitemaps

Here are examples of how different platforms can generate sitemaps using protected data:

• E-commerce API: Requires 'x-api-key'. Maps /[response.category.slug]/[response.slug] into public product URLs without exposing the internal admin endpoint.
• Headless CMS: Requires 'Authorization: Bearer CMS_READ_TOKEN'. Maps /[response.type]/[response.slug] to generate URLs for blog posts, documentation, and case studies.
• Multi-locale CMS: Uses headers and maps /[response.locale]/[response.type]/[response.slug] to correctly build international sitemap URLs.

Relative paths and full URLs with protected APIs

The same URL pattern system works for protected APIs. SitemapFlow supports relative paths and full URLs.

Use a relative path (e.g., /products/[response.slug]) when the generated sitemap URLs should use the current scope domain. Use a full URL (e.g., https://store.example.com/products/[response.slug]) when the page belongs to a specific domain, subdomain, locale domain, or separate application.

Both formats can map fields from the protected API response.

What data should a protected API return?

A protected API used for sitemap generation should return only the records that can become public, indexable URLs.

Useful fields include id, slug, category, updatedAt, locale, canonicalUrl, image, title, description, and status.

The API should avoid returning records that should not be in search results, such as drafts, deleted records, private pages, internal dashboards, staging URLs, or duplicate URLs.

If possible, create a specific endpoint for sitemap generation (e.g., /sitemap/products). That endpoint can return a clean list of indexable records instead of exposing your entire internal data model.

Protected API sitemap security considerations

A protected API sitemap workflow should be handled carefully. Use API keys and tokens that are allowed to access only the data needed for sitemap generation. Avoid using admin tokens when a read-only token is enough.

SitemapFlow does not store your API endpoints, credentials, or headers on our servers. Processing happens during your active session. Good practices include:

• use read-only API keys when possible
• limit the token to public, indexable records
• avoid exposing private user data
• rotate tokens if they are leaked
• avoid entering sensitive production secrets on shared devices
• create a dedicated sitemap endpoint if possible

Protected APIs and private websites are not the same thing

A protected API does not always mean the website pages are private. A public e-commerce product page may be generated from a backend API that requires a token. The product page is public, but the data endpoint is protected. That is a good fit.

But if the final page itself requires login, it usually should not be in a public XML sitemap.

Examples of pages that usually should not be included: /dashboard, /account, /billing, /admin, /private-projects/[id]. Before generating a sitemap, check whether the final URLs are actually public and indexable.

Handling API pagination and errors

Many protected APIs return paginated results. If the endpoint only returns the first page of results, the sitemap may be incomplete. Make sure your workflow can access all records, which may require using pagination parameters or creating a dedicated sitemap endpoint.

If a protected API request fails, the sitemap cannot be generated correctly. Common errors include 401 Unauthorized (token missing/invalid) and 403 Forbidden (token lacks permission). SitemapFlow highlights these broken responses so you can fix the configuration before submitting.

Common mistakes with protected API sitemaps

The biggest mistake is putting the API endpoint itself into the sitemap. Your sitemap should contain public website URLs, not private API URLs.

Another mistake is using a token that returns too much data. If the response includes drafts or internal objects, those may accidentally become sitemap entries.

• using an expired API token
• forgetting the Bearer prefix
• using the wrong header key
• using an admin token instead of a read-only token
• generating URLs from unpublished records
• including pages blocked by login