SAG / ARCHITECTURE NOTE
Sitemap Discovery and Crawl Scope: How Does a URL List Become an Analysis Plan?
This is the process of turning a sitemap’s address list into crawl candidates within an approved domain. A URL list is not an analysis result. Unless language-specific duplicates, excluded pages, and other hosts are cleaned up, the workload grows while comparable evidence remains scarce.
What Is Sitemap Discovery?
It is the process of turning a sitemap’s address list into crawl candidates within an approved domain. This note treats sitemap discovery in terms of the responsibilities for inputs, transformations, and outputs, rather than as a feature name. To trust an analysis result, it must be possible to trace what data went in, what was checked, and how far the conclusions can go.
Why Is This Technology Needed?
A URL list is not an analysis result. Unless language-specific duplicates, excluded pages, and other hosts are cleaned up, the workload grows while comparable evidence remains scarce.
Design Principles and Data Flow
Collect XML URL lists and HTML links during the discovery stage, while limiting them by target domain, path, and maximum page count. Count discovery, successful requests, and completed diagnostics separately.
Sitemap URLs → Domain-scope validation → Crawl candidate list
Each stage should not relabel the success of the previous stage as its own achievement. Recording the identifiers, time period, and validation status of the data across stages makes it possible to locate omissions and errors and determine what needs to be checked again.
Connection to the SAG Architecture
SAG connects sitemap registration to domain-based data collection. Addresses of external competitors are not mixed into the company’s own scope; they are treated as a separate analysis target.
SAG’s operational value lies in connecting this relationship to pages and questions, comparison results, and improvement tasks. Rather than looking only at numbers, customers can review both what needs to be strengthened and the basis for the assessment. Patterns that require additional application should be interpreted according to the scope of the relevant paragraph.
Illustrative Example and Criteria for Assessment
Even if an illustrative sitemap contains 20 URLs, if only 18 are within the permitted scope and only 15 were collected, the counts must be distinguished: 20 discovered, 18 allowed, and 15 collected. Fixing the report’s denominator at 20 can conceal collection gaps.
The example above is provided to explain structure and calculation; it is not a measured result from a specific customer. In an actual report, the selected period, target, and observation conditions must be linked to the original records so the same assessment can be checked again.
Practical Validation Checklist
| Flow stage | Items to check |
|---|---|
| Sitemap URL | Confirm allowed domains and paths |
| Domain-scope validation | Separate discovery and success counts |
| Crawl candidate list | Validate differences between HTML and XML formats |
Check that the same meanings are preserved not only with valid inputs, but also with empty data, duplicate data, and data with different conditions. Connecting validation items to completion criteria can reduce the gap between the feature description and actual operations.
Limitations and Points to Consider When Applying It
A sitemap helps discover URLs; it does not guarantee search indexing or rankings. The source’s update time and actual HTTP responses must also be checked.
Research and Official Documentation
- Sitemaps protocol — A resource that defines the format and scope of sitemap URLs and metadata.
The external resource provides background for the design topic above; it does not certify every SAG implementation or customer outcome. The application interpretation and illustrative example in this note are organized according to SAG’s operational structure. Sources checked: 2026-10-06.
Further Reading and Feature Review
- Related architecture note
- Try the service connected to sitemap discovery
- Feature-specific FAQ
- Discuss implementation scope
How to Continue Reading About This Technology
Follow the flow: HTML ZIP and sitemap → normalization → page version → evidence record.
SAG / KNOWLEDGE LINKS