Configure Canonical URLs, Crawling Directives, and Sitemaps

Organize duplicate and language-specific addresses

A new operating notice without links to its URL has few discovery paths. Even with the address, crawlers cannot get the body if robots or the server blocks requests. Multiple addresses for one article require canonical selection. Internal links and sitemaps, crawling directives, and canonical declarations address these different problems. Read the settings in the order documents are discovered and processed.

If one article opens at multiple addresses, first choose its canonical URL. List relationships among HTTP and HTTPS, www and non-www, trailing slashes, and old slugs. Permanently redirect equivalent addresses no longer in use through the server or hosting. URLs differing only in tracking queries can share a canonical if their bodies match. If filters produce different content, determine first whether it is actually duplicate.

Pagination page 2 differs from page 1, so do not automatically canonicalize every page to the first. Provide real a href links to subsequent pages. For multilingual pages, give each language a self-referencing canonical and reciprocal hreflang relationships. Do not emit nonexistent translation URLs.

Align sitemaps, internal links, canonicals, and redirects so they do not nominate different canonical URLs. Use Google’s canonical URL guidance to configure them, then check the search system’s actual selection in URL Inspection after deployment.

Map old URLs to their post-redesign destinations individually

Before redesign, save an inventory of existing addresses and mark each as retained, moved, preserved as a historical record, or ended. Move identical content with 301/308 redirects. Distinguish current guidance from historical records by validity period and purpose. Historical records can keep separate addresses with their reference period and a link to current guidance.

For example, moving /notice/old-guide to /guides/apply requires updating redirects, menu and body links, canonical, and sitemap together. If preserving it as /archive/2024/apply, do not overwrite the historical body with current guidance. Search exclusion of archives is a publication-policy decision; age alone does not justify blanket noindex.

Record old URL, new URL, expected status, canonical, sitemap inclusion, and inbound links in the route map. Check for chains through other old addresses and redirect loops. When the production domain changes, update absolute-URL settings, sharing-image addresses, and search-tool properties and ownership verification too. See Google’s guide to site moves with URL changes.

Old addresses may remain in search results, external articles, and saved links. Permanent redirects for moved articles guide visitors and crawlers to the new document. Historical operating records answer questions about their period and should retain their own address and period. Redirecting every old URL to the home page loses the connection to what visitors sought.

Edit robots.txt and response headers

In Next.js, check whether app/robots.ts or public/robots.txt actually serves the file. For other servers, inspect root robots.txt and CDN or proxy blocking rules. Narrow Disallow rules covering public paths according to publication policy. Inspect remaining noindex in HTML meta robots and HTTP X-Robots-Tag too. Google does not support noindex in robots.txt as an indexing-exclusion directive.

Decide separately for search bots and training bots. The following example permits search crawling but restricts OpenAI training crawling. Sites allowing training should change GPTBot’s Disallow rule accordingly.

User-agent: * Allow: / Disallow: /admin$ Disallow: /admin? Disallow: /admin/ Disallow: /preview$ Disallow: /preview? Disallow: /preview/ User-agent: OAI-SearchBot Allow: / Disallow: /admin$ Disallow: /admin? Disallow: /admin/ Disallow: /preview$ Disallow: /preview? Disallow: /preview/ User-agent: GPTBot Disallow: / Sitemap: https://example.com/sitemap.xml

Bot-specific groups must also state that bot’s excluded paths. The search-bot example repeats exclusions so they apply even where general-group rules are not merged. Protect preview paths with real authentication and check that broad blocks do not catch CSS, JavaScript, images, or public APIs needed for the public body. robots permission alone does not solve a WAF delivering login, challenges, or 403 responses.

Google’s noindex directive goes in the page response, which the crawler must receive to read it. OAI-SearchBot collects search material; GPTBot collects training material. Set allowed paths for each purpose. Google noindex · OpenAI bots

/admin/ alone does not match /admin or /admin?mode=edit. The rules above use $ for the exact root’s end, ? for its query, and / for descendants. They do not include /administrator. This example follows Google’s supported * and $ matching rules; check other providers’ supported behavior separately. robots path rules

Generate sitemaps from public data

Generating sitemaps from public queries reduces omissions compared with maintaining a manual list. app/sitemap.ts in Next.js returns URLs and modification dates; other stacks can generate XML from the same data. listPublicGuides below queries the entire public list. If database results are paginated, traverse through the last page.

The next block shows the transformation function. The complete executable app/sitemap.ts and imports follow in the next section.

export default async function sitemap() { const guides = await listPublicGuides() return guides.map(g => ({ url: g.info.canonical, ...(g.info.dateModified ? { lastModified: g.info.dateModified } : {}) })) }

guides is the full public-article array; map creates one sitemap item per article. url uses validated info.canonical, with lastModified added only when dateModified exists. ... merges conditionally created properties into the object. Keep date strings unchanged instead of adding invented midnight to a date-only modification value.

lastModified records when the body or important links actually changed, not deployment time or the current time on every request. Omit it without a reliable record. Sitemap URLs must be working public canonicals; exclude withdrawn, error, and redirect addresses. Follow sitemap creation guidance when serving the generated file, then check submission and processing status in search tools.

RSS and Atom are document formats for subscriptions and external integration. A feed is an address collecting titles, links, and dates in such a format. Apply the same publication conditions as the sitemap and output title, canonical link, date, and stable item ID. Define the feed address for subscription or integration tools and update it on publication, editing, and withdrawal.

Public-query-generated sitemaps add discovery addresses as CMS articles grow. If there are 120 detail articles but the sitemap contains only the first list’s 20, discovery of the other 100 depends on other links. Connecting complete queries with valid-address checks reveals this omission. lastmod signals changed documents, so align it with significant actual changes.

Complete robots and sitemap for the example

app/robots.ts guides crawling of public paths and excludes administration and preview. Authentication is applied separately by the relevant server functions. app/sitemap.ts includes public-query results and the actual archive page.

import type { MetadataRoute } from 'next' import { SITE_ORIGIN } from '../lib/site' export default function robots(): MetadataRoute.Robots { return { rules: [{ userAgent: '*', allow: '/', disallow: ['/admin$', '/admin?', '/admin/', '/preview$', '/preview?', '/preview/'] }], sitemap: new URL('/sitemap.xml', SITE_ORIGIN).href } }

import type { MetadataRoute } from 'next' import { listPublicGuides } from '../lib/guides' import { SITE_ORIGIN } from '../lib/site' export const dynamic = 'force-dynamic' export default async function sitemap(): Promise<MetadataRoute.Sitemap> { const guides = await listPublicGuides() return [ { url: SITE_ORIGIN }, { url: new URL('/policies/opening-hours', SITE_ORIGIN).href }, { url: new URL('/archive/opening-2025', SITE_ORIGIN).href }, ...guides.map(g => ({ url: g.info.canonical, ...(g.info.dateModified ? { lastModified: g.info.dateModified } : {}) })), ] }

Check that drafts, missing URLs, and redirects are absent and every URL returns a valid public document. Split large URL sets into files connected by a sitemap index. Google’s ordinary sitemap limit is 50,000 URLs and 50MB uncompressed per file. Use sitemap creation guidance to check file sizes and omissions from the full list.

The first code is app/robots.ts, the second app/sitemap.ts. The robots function gives Next.js rules to serialize as robots.txt; sitemap gives a public-address array to serialize as XML. Home, archive, evidence page, and current public articles are included; drafts are excluded. Read actual output with curl http://localhost:3027/robots.txt and curl http://localhost:3027/sitemap.xml.

This example permits general crawling. To adopt the earlier GPTBot restriction, reflect it in this robots function’s rules too. Do not create both public/robots.txt and app/robots.ts and leave the responding file ambiguous.

Check crawlability separately from indexability

Crawlability concerns whether a crawler can fetch material; indexability concerns whether fetched material can enter the index. Even HTTP 200 and robots permission do not ensure inclusion if noindex, canonical selection, or content assessment says otherwise. Blocking crawling may prevent reading noindex inside the page.

To exclude an accessible public file only from search, provide noindex through meta robots or X-Robots-Tag. Protect private material with authentication and authorization and verify that unauthorized requests are actually blocked. robots.txt is public guidance, not a means to hide secret paths.

Crawl budget concerns search-engine resources and demand constraints. On very large or rapidly changing sites, duplicates, infinite URLs, and server-response problems can affect crawling efficiency. On small information sites, first check discovery links, errors, bodies, and reasons important documents are excluded from indexing. Crawling guidance for large sites

Local businesses provide branch addresses, opening hours, and contacts. Products provide price, stock, and purchasing conditions; videos provide playback URLs, thumbnails, and duration. Translation pages need actual content in that language and reciprocal hreflang URLs. Additional information varies by content type.

Set generative-search participation and content-use boundaries

Pages participating in Google generative search must be indexed and eligible for snippets. In Search Console Settings → Search generative AI, check values for the property and its parent. Include is the default and child properties can inherit. If ordinary search visibility exists but generative participation does not, compare indexing, snippet directives, and this setting in order. Participation requirements · Search generative AI settings

<!-- Option excluding snippets for the entire page; not the example site default --> <meta name="robots" content="nosnippet"> <!-- Alternative: limit maximum snippet length --> <meta name="robots" content="max-snippet:160"> <!-- Option excluding part of a page from snippets --> <p>Sundays 10–17; closed on the first Sunday.</p> <div data-nosnippet>Additional wording for screen use</div>

These are separate choices. nosnippet prevents text snippets and restricts direct use of content in Google AI Overviews and AI Mode. max-snippet:0 means the same exclusion; positive values affect snippet and direct-input length. data-nosnippet on span, div, or section excludes that part. Putting the core closure condition inside it can remove evidence needed for an answer. Separate permissions, agreements, or structured data may have exceptions; check each directive’s documented scope. Google snippet directives

Googlebot crawls for Google Search. Google-Extended is a robots product token controlling some Gemini training and designated grounding uses outside Google Search. That token alone does not control Search ranking or generative-search participation. The Search generative AI exclusion applies to specified generative search features, not other rankings or model training. Google crawler scope · Setting scope

OpenAI’s OAI-SearchBot and GPTBot likewise serve separate search and training purposes. After allowing public paths, read actual robots headers and bodies and preserve server-log outcomes. Combining search participation and training permission into one ‘Allow AI’ switch makes the restricted purpose hard to explain.

Provide recent articles as RSS and update it after edits and withdrawals

RSS lets subscription tools periodically read one address to discover new articles. It delivers news to readers who do not visit the home page each time and to other services. A sitemap informs search tools of public addresses; this exercise’s RSS sends titles, canonical URLs, and summaries for the 20 latest posts. Google accepts RSS and Atom as sitemaps, but a recent-post feed cannot replace the complete URL list. Sitemap-format guidance

Save feed.mjs below beside identity-time.mjs. renderFeed selects public documents from all source rows, then takes 20 by publication date. guid holds the document ID and link the current canonical, preserving identity through title or slug changes. RSS 2.0 specification

import { describeDocument, instant } from './identity-time.mjs'; export function escapeXml(value) { // Reject characters disallowed by XML 1.0 and lone surrogates. for (const char of String(value)) { const n = char.codePointAt(0); if (!(n === 9 || n === 10 || n === 13 || (n >= 32 && n <= 0xD7FF) || (n >= 0xE000 && n <= 0xFFFD) || n >= 0x10000)) throw new Error('Invalid XML character'); } return String(value).replace(/[&<>"']/g, c => ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&apos;' })[c]); } export function renderFeed(rows, origin, asOf) { const x = escapeXml; const publicRows = rows.map(doc => ({ doc, identity: describeDocument(doc, asOf, origin) })) .filter(row => row.identity) .sort((a, b) => order(b.doc.publishedAt) - order(a.doc.publishedAt)) .slice(0, 20); const items = publicRows.map(({ doc, identity }) => { // RSS pubDate requires a time. Omit this optional field for documents with only a date. const date = identity.datePublished; const pubDate = date?.includes('T') ? `<pubDate>${new Date(instant(date)).toUTCString()}</pubDate>` : ''; return `<item><title>${x(doc.title)}</title><link>${x(identity.canonical)}</link> <guid isPermaLink="false">${x(identity.id)}</guid>${pubDate} <description>${x(x(doc.summary))}</description></item>`; }).join(''); return `<?xml version="1.0" encoding="UTF-8"?> <rss version="2.0"><channel><title>Fictional Library News</title> <link>${x(origin)}</link><description>20 latest public updates</description>${items}</channel></rss>`; } // Use the start of the Korean calendar day to sort date-only entries, not for pubDate. function order(value) { return value === null ? -Infinity : instant(value.length === 10 ? value + 'T00:00:00+09:00' : value); }

XML uses characters such as & and < as syntax, so escapeXml escapes text. RSS description may be read as HTML, so summaries are escaped twice: once to preserve HTML text, then to preserve that HTML inside XML. Forbidden control characters produce errors. pubDate uses the original publication time; omit this optional field when only the date is known.

Editing the body updates the summary and link under the same guid. This RSS does not include a per-item modification time. Integrations needing it should use a receiver-supported field such as Atom updated. Withdrawn articles disappear from the next feed, but subscribers may retain saved copies. Block original access and decide whether to publish a separate correction notice.

The image-exercise server returns this at /rss.xml as application/rss+xml. The home-page head has a rel="alternate" link. Separately test reading the feed XML directly and how an actual subscription tool handles new articles, edits under the same ID, and withdrawal.