Diagnose an existing site and prioritize fixes
Compare the public URL inventory with actual responses to find access, content, URL, and factual errors. Record the baseline, locate the cause, and define the expected result before implementation.
Define public scope and the URL inventory
Applying the same fix to every page missing from search can leave the cause untouched. An unlinked article needs a discovery path; a noindex article needs indexing instructions corrected; an article without an answer needs better content. First compare actual responses and search-processing records for the same URL to locate where material was lost.
Agree with the operator on public access and search inclusion first. If policy allows all public content in search, do not add noindex solely because content is an article, notice, introduction, or photo description. Distinguish login-only material, drafts, not-yet-published scheduled material, and withdrawn articles. Search exclusion and access restriction are different settings; protect private material with authentication and authorization.
Extract public URLs from the repository or CMS and compare them with sitemap URLs and URLs found in actual menus and body links. Investigating only the sitemap cannot reveal articles missing from it. Record route types such as listings, details, pagination, images, and attachments. Public articles absent from both body links and sitemaps are candidates for added connections.
For each URL, record content ID, type, publication status, expected response code, canonical URL, publication and substantive modification dates, and related questions. Begin sampling with a representative URL for each template, but apply the final comparison to the entire target inventory. Include draft, withdrawn, and missing URLs as separate samples to check that they do not mix with public articles.
Create a pre-deployment baseline and observation table
Fix the target URLs and questions before observation begins. Compare the same set with pre-deployment records if available; otherwise use the first observation as baseline. Version the set and mark additions when URLs or questions change. A single overall total makes new-content growth hard to distinguish from improvements to existing pages.
Technical records capture what the site actually returned. Search records capture the search engine's latest processed state. AI records preserve the answer and citations for a question. These three records may have different timestamps even for one URL, so keep each original with its observation time. The following is a fictional record of the initial library error.
Immediately after deployment, check that responses contain the new body and canonical URL. In later search and AI observations, read the latest crawl date and the dates of cited documents. Whether old material remains helps determine whether to await recrawling or improve the document's explanation of its applicable period.
Separate initial setup, redesign, and new-feature scope
Initial setup creates public URLs and basic content, page-specific titles, descriptions and canonicals, robots, sitemap, and search-property registration. A redesign also examines how old URLs move or remain, how current and historical information differ, and the publication conditions in new data queries.
When adding a menu or page area, first document what content it reads, where from, and how it links to details. Check its initial response for content, title, detail links, images, and original sources. For material appearing only after tabs, carousels, or scrolling, examine browser behavior and crawlable paths together.
Collect server responses and browser views together
The following is a read-only investigation in a macOS or Linux shell. example.com is an illustrative domain. Replace it with the target and select one public detail article. Start without cookies or administrator authentication.
AUDIT_ORIGIN='https://example.com'
AUDIT_PAGE="$AUDIT_ORIGIN/guides/opening-hours"
mkdir -p seo-audit
curl -sS -D seo-audit/page.headers -o seo-audit/page.html "$AUDIT_PAGE"
curl -sS "$AUDIT_ORIGIN/robots.txt" -o seo-audit/robots.txt
curl -sS "$AUDIT_ORIGIN/sitemap.xml" -o seo-audit/sitemap.xml
The first request does not automatically follow redirects. Read status code, Location, X-Robots-Tag, and Content-Type from the headers file. For 3xx, use the next command to record the final URL and redirect count too. A working canonical URL should return the intended public body rather than a login screen.
curl -sS -L -o /dev/null -w 'status=%{http_code} redirects=%{num_redirects} final=%{url_effective}\n' "$AUDIT_PAGE"
rg -n -i 'noindex|canonical|<title|application/ld\+json' seo-audit/page.html
rg searches files for strings. Locate the settings found in its results. It can match single-line HTML and JavaScript strings, so distinguish actual tags and body content. Open the same URL in a new unauthenticated browser window and check text, photos, and links. If HTML contains only loading UI rather than the answer, trace where data arrives through the rendering configuration's query flow. See also Google's JavaScript-processing guide.
If the saved HTML omits closure dates but the browser shows them, investigate JavaScript execution and data requests between those results. If both omit the dates, fix the manuscript or server query. Establishing this difference helps avoid changing frameworks merely because search exposure is low.
Choose one static photo and one CMS photo and open them at the same viewport width. If the static photo requests a width-specific transformation URL but the CMS photo requests the original, trace the CMS field through to img.src. Comparing CSS widths alone does not reveal how many bytes the server sends. In the CMS image exercise, original and derivative views use the same material and layout, allowing delivery methods to be compared.
Connect symptoms to fix locations
In the repository, use the searches below to locate where settings are created. Adjust paths to the project structure. Follow not just defaults in search results but per-page overrides, response headers, and deployment-environment settings. The same noindex can exist in both HTML and HTTP headers.
rg -n 'noindex|X-Robots-Tag|generateMetadata|canonical' src app public
rg -n 'sitemap|robots|dateModified|datePublished|application/ld\+json' src app public
rg -n 'published|visibility|deletedAt|publishedAt' src app
If there is an external peer review, first copy its target URLs, times, conditions, and observed values. Reproduce findings in actual responses and code, then link them to fix locations. Distinguish items already fixed, remaining items, and suggestions not adopted because they conflict with operating policy. Compare performance findings before and after on the same page, device, and network conditions.
Why a photo is slow also depends on request timing. If its URL appears late in HTML, inspect rendering and data queries. If the URL appears early but download takes long, inspect file size and connection. If display lags after download, inspect main-thread work and decoding. This sequence can locate a fix before replacing an entire framework.
Set the fix order and its basis
First handle issues affecting the whole target set: private-data exposure, inaccessible public articles, and widespread wrong noindex or canonical values. Then fix missing content, broken links, wrong dates, and factual errors. Improve titles, summaries, tables, and image descriptions starting with pages where readers struggle to find answers. Record one pre-change baseline for external answers, then continue under the same conditions after deployment.
Record each task as “URL / symptom / original response or screen / cause location / fix / pass condition / repeat-observation question.” If public details inherit noindex from a shared layout, for example, fix the default and public-page override, then check normal articles, drafts, and missing URLs together. Keep unsupported hypotheses as investigation items, separate from confirmed fixes.
Google's generative search uses basic search-quality and technical requirements. Fix public content, access, and sources first. If operating a separate tool that reads llms.txt, define what to deliver to it and when to update it. Generative-search optimization guide
Material for the next stage
After diagnosis, retain the question–URL map, publication-state table, response and screen records, and prioritized fix list. Implementers should be able to find code locations and pass conditions directly. If external reports are inaccessible, record accessible items and those not yet observed, explaining blank values for the next observer.
Connect each diagnosis item to the files, settings, or content to change and the expected response. Before changing rendering, define data update frequency and publication conditions. For URL issues, first define canonical URLs and the old-address map. Recheck with the same URLs and questions after fixing.
Turn the fictional library's three symptoms into fix tasks
Request the opening-hours detail URL without authentication and save the body. If it shows only loading UI, investigate query timing. If canonical points to the homepage, locate detail-metadata generation. Compare dates and applicable periods to judge whether an older timetable can answer the current question.
The table links three fictional library errors to fix locations. Reading response HTML, canonical value, and the old notice together distinguishes the required changes.
Separate evidence for technical and content defects
The same symptom, “little search traffic,” can have different causes. Missing server-response content calls for delivery-path investigation; missing application conditions call for manuscript work; errors after the Apply button call for conversion-path investigation. Include questions from keyword research and actual inquiries in the diagnosis table and check whether each page answers them.
Record target question, URL, published version, applicable period, and observation time in the baseline. Rather than directly comparing today's screen with a month-old search report, establish which manuscript may actually have appeared. Use document roles in the search-intent map to find gaps.
Narrow down pages indexed but absent from generative answers
Even when URL Inspection shows the fictional opening-hours page as indexed, an answer may omit its closure conditions. Save the answer's actual cited URL first. If it cites current guidance, check whether exceptions exist and whether nosnippet or data-nosnippet hides them. If it cites an archive, compare the period and current-guidance link. With no citation, check Search generative AI participation settings and content matching the question.
Record the question and cited URL together, adding the latest crawl date and snippet instructions from URL Inspection. If AI cites a 2025 notice, first inspect its applicable period and current-guidance link. If it cites current guidance but omits closures, compare the cited sentence and exception explanation.
If the last crawl was October 20 and closure conditions were added on the 24th, an answer based on the 20th may lack them. If it cites material from the 24th onward yet omits exceptions, review answer/condition placement, the cited passage, and question interpretation. Comparing crawl dates separates undelivered changes from misuse of delivered content.