Basics of Reading Web Documents, Requests, and Responses

Connect the web, servers, and infrastructure through one request

A browser logged in as an administrator may display the guide, while an external crawler requesting the same address receives a login page. Keywords or structured data entered on the page then never reach the crawler. To decide whether to fix the body or access settings, determine which servers the request passed through and which response it ended with.

The frontend handles the documents visitors read and the buttons they press. The backend selects data to publish and processes requests on the server. Infrastructure means the DNS, hosting, networks, and storage that run this code and carry internet requests. One Next.js application can handle both frontend and backend.

Browsers and crawlers are both clients that request material from servers. A request to https://example.com/guides/opening-hours first uses DNS to find the connection address, establishes HTTPS, then requests the document at that path. A CDN in front may return a stored response or forward the request to the origin server. The origin reads public data, builds HTML, and responds.

A CDN (Content Delivery Network) delivers files or responses from servers in multiple regions. The origin server produces the original response behind the CDN. A cache stores previous results for reuse. A server can also respond directly without a CDN, and a static site can deliver prebuilt files instead of using an application server.

Browser or search crawler → Look up the connection address in DNS → Establish HTTPS → CDN, firewall, or proxy handles the request, if present → Origin application queries public articles → Return HTML, status code, and headers → Browser requests CSS, JavaScript, and images → Display the page and attach button behavior

If the body is empty, start with the page and server query. If the connection itself fails, inspect DNS, certificates, and blocking at the edge. If an edited article still shows old content, compare the query result and each cache. Compare status codes, bodies, and cache records to locate where the request stopped or the content changed.

Read URLs, HTTP, headers, and bodies separately

A URL is the address of an internet resource. https is the communication scheme, example.com is the host to connect to, and /guides/opening-hours is the path identifying the document. page=2 after the question mark is a query, an additional condition sent to the server. A fragment such as #hours points to a location within the document and is not sent in an ordinary HTTP request.

A domain is a human-readable name such as example.com. A port identifies the service to connect to on the same server; HTTPS normally uses 443. In browser security, an origin is the combination of scheme, host, and port. Distinguish this from the origin server described above.

HTTP is the protocol for exchanging requests and responses between clients and servers. GET requests a resource, while HEAD requests headers without a body. A response includes a status code describing the outcome, headers describing format and caching, and a body containing the actual material. SEO checks should fetch the actual body with GET to detect replacements by login screens or error messages.

Content-Type is the header identifying the body’s format: text/html for HTML documents, application/json for JSON, or image/webp for WebP images. These format names are also called MIME types. Location contains a destination address, X-Robots-Tag contains search directives for the whole response, and Cache-Control contains caching conditions.

Read the status code together with the body. A 200 means a successful response was sent, but only the body shows whether it contains opening hours or a bot-verification screen. Codes 301 and 308 provide an address to request next; 404 says the requested document does not exist. Search systems process these responses for crawling and indexing, so returning 200 and the home page for every address obscures document existence and redirects. Google’s handling of HTTP responses

Separate the roles of HTML, CSS, and JavaScript

HTML (HyperText Markup Language) represents document structure such as headings, paragraphs, and links. CSS (Cascading Style Sheets) determines fonts, spacing, colors, and layouts for different screens. JavaScript runs behavior in the browser, including buttons, search, and data updates. Provide public article titles and bodies as HTML elements, and connect presentation and interaction through CSS and JavaScript.

A tag identifies the type of element, as in <p>. An element includes the tags and their content. An attribute is a setting attached to an element, such as href or lang. The DOM (Document Object Model) is the structure a browser builds in memory from a document. JavaScript can later add to or remove from the DOM built from server HTML, so check both the initial response and the screen after interaction.

The head contains the document title and metadata; the body contains the readable content. main identifies the principal area, article an independent article, h1, h2, and h3 heading levels, and p a paragraph. Use headings according to the content hierarchy and control their size with CSS. Turning paragraphs into pictures makes direct text reading harder for crawlers and screen readers.

A link puts its destination URL in an a element’s href. If the address exists only in a button’s click handler, crawlers may fail to discover the next document. Use link text describing the destination, such as ‘Documents required to apply for borrowing.’ Google’s link guidance explains crawlable a and href structures and link text.

A responsive layout changes with screen width. Keep the body, sources, and navigation links readable on mobile, and preserve which column heading belongs to each table value. Accessibility makes content usable despite differences in disability or input method. Image alt text, heading hierarchy, and keyboard-accessible links also aid document understanding.

Distinguish metadata and search-setting terminology

Metadata describes a document through its title, description, and canonical URL. title supplies a candidate title for search results and browser tabs, while meta description supplies a candidate summary. Search systems may display different titles and summaries depending on the question and context, so inspect actual results too.

A canonical URL is the address selected to represent duplicate or very similar documents. An HTML link rel="canonical" is one way to suggest it. Material without an HTML head, such as a PDF, can suggest it through an HTTP Link response header. Google combines several signals to select the final canonical URL. Redirects move visitors to a new address; canonical declarations do not. Canonical URL guidelines

robots.txt is a file at the site root that guides crawlers about retrieval paths. Allow permits access and Disallow restricts it. noindex in meta robots or X-Robots-Tag requests exclusion from the search index. nofollow is a directive about following links. Protect private material with server authentication and authorization checks; configure search directives according to the crawling and indexing policy for public material.

A sitemap lists URLs to crawl and information such as modification dates. XML records data using a tag structure and is used for sitemaps. Structured data expresses relationships such as title, author, date, and subject as machine-readable properties. JSON records data as names and values, while JSON-LD expresses linked entities and relationships in JSON. Schema.org is a vocabulary defining types and properties such as Article and Person.

Open Graph (OG) is metadata for titles, descriptions, and images used in sharing cards. hreflang indicates relationships between pages providing the same content for different languages or regions. Both should contain addresses and languages matching the actual pages. author, publisher, about, and sameAs describe different relationships: the writer, publishing entity, subject, and another reference to the same entity.

BreadcrumbList is a Schema.org type expressing the document’s place in the site hierarchy, such as ‘Home → Visitor information → Opening hours.’ Use the same actual navigation path as the visible breadcrumb. It identifies the current document’s location and lets readers return to parent documents; it differs from citation, which identifies a source for another article. Breadcrumb structured data

HTTP/1.1 200 OK Content-Type: application/pdf Link: <https://example.com/guides/opening-hours>; rel="canonical"

This response suggests an HTML page as the canonical URL for a printable PDF containing the same guidance. If the PDF contains a timetable from a different year, use the canonical address for that record instead. Set the Link header on the server or storage service delivering the PDF.

Read the same document in a terminal and browser

A terminal is a program for entering commands and reading results; a shell interprets and runs commands. curl sends a request to a URL and saves the response. rg searches files for strings. dig queries DNS address information. Check that the commands are installed, then select the public URL to work on.

In curl, -sS reduces progress output while displaying errors, -D saves headers, and -o saves the body. -L follows redirects. First inspect redirect instructions without -L, then follow to the final address and compare the redirect count and result. --max-time sets the maximum wait in seconds.

In browser developer tools, Network shows requests, responses, and transfer times; Console shows code errors; Elements shows the current DOM. If the initial HTML differs from the DOM after JavaScript runs, find the request or code that added the content. Distinguish the displayed CSS size from the pixel dimensions of the downloaded image.

Read meaning and metadata together in ordinary HTML

Save the following as opening.html and open it in a browser to see document structure without a server or framework. In developer tools’ Elements panel, compare title, description, and canonical in head with the heading, dates, and evidence link in body. Replace example.com with the production domain when deploying.

<!doctype html> <html lang="en"> <head> <meta charset="utf-8"> <title>Library Sunday Opening Hours and Closures</title> <meta name="description" content="Open Sundays 10–17; closed on the first Sunday and public holidays."> <link rel="canonical" href="https://example.com/guides/opening-hours"> </head> <body> <main><article> <h1>Library Sunday Opening Hours and Closures</h1> <p>Published <time datetime="2026-10-01T00:00:00+09:00">October 1, 2026</time></p> <h2>Opening hours and exceptions</h2> <p>Open Sundays from 10 to 17. Closed on the first Sunday and public holidays.</p> <p>Effective from October 1, 2026.</p> <a href="https://example.com/policies/opening-hours">Original opening-hours notice</a> </article></main> </body> </html>

h1 identifies the document’s subject and h2 a subsection. An a element’s href is the actual destination, while time’s datetime is the machine-readable value. Plain HTML uses datetime; React JSX uses dateTime. title is a title signal for search and tabs and differs in role from the visible heading. Search engines may determine their own title link and snippet.

Check that the heading, hours, and closure exceptions remain when CSS or JavaScript is disabled. The later Next.js exercise checks missing-URL 404 handling through HTTP responses; Documents and time checks scheduled-publication boundaries with fixed-time tests.

In this HTML, h1 identifies the opening-hours subject, p gives the hours and closure conditions, and a provides the path to the evidence document. Tools reading the body obtain the sentences needed for an answer from these elements. time’s datetime supplies a consistently formatted date, but the adjacent label and sentence explain whether it is a publication or effective date. This is why tags and human-readable explanations appear together.

Image values also have different meanings. A CSS display width of 358px is the space occupied on screen. The 640w in srcset is the pixel width of the file to download. naturalWidth is a density-corrected CSS pixel value, so inspect the response file with an image tool to determine its raw width. currentSrc is the address the browser selected. Reading all four values on the same screen reveals cases where a large original is sent into a small slot. Definition of naturalWidth