AI crawler
A crawler operated by an AI company, such as GPTBot (OpenAI) or ClaudeBot (Anthropic), that fetches pages for model training or AI search.
Read more about AI crawlerPlain-English definitions of the metrics, headers and tags you meet in a website audit, each with a link to the guide or free tool that goes deeper.
66 terms, A to Z
A crawler operated by an AI company, such as GPTBot (OpenAI) or ClaudeBot (Anthropic), that fetches pages for model training or AI search.
Read more about AI crawlerA short text description of an image, written in the alt attribute.
Read more about Alt textAccessible Rich Internet Applications: a set of HTML attributes that add information for assistive technology, such as roles and labels.
Read more about ARIAA link whose target returns an error, most often 404 Not Found, or cannot be reached at all.
Read more about Broken linkA compression format for text files such as HTML, CSS and JavaScript.
Read more about BrotliAn HTTP response header that tells browsers and CDNs whether and for how long they may reuse a stored copy of a file.
Read more about Cache-ControlThe address you want search engines to treat as the main version of a page when several URLs show the same or very similar content.
Read more about Canonical URLA network of servers in many locations that keeps copies of your files and serves each visitor from a nearby one.
Read more about CDNProtection against another site embedding yours in a hidden frame to trick visitors into clicking.
Read more about Clickjacking protectionA Core Web Vitals metric that scores how much visible content jumps around while a page loads.
Read more about CLSAn HTTP response header that lists which sources a page may load scripts, styles, images and other resources from.
Read more about Content-Security-PolicyFlags that protect cookies.
Read more about Cookie attributesGoogle's set of three user-experience metrics: Largest Contentful Paint (loading), Interaction to Next Paint (responsiveness) and Cumulative Layout Shift (visual stability).
Read more about Core Web VitalsThe number of URLs a search engine is willing and able to crawl on a site in a given period.
Read more about Crawl budgetA program that fetches web pages automatically.
Read more about CrawlerA public dataset of real-user performance measurements from Chrome users who have opted in.
Read more about CrUXThe tree-like structure the browser builds from a page's HTML. Scripts read and change the DOM, and a very large DOM makes rendering and interactions slower.
Read more about DOMThe same or very similar content available at more than one URL. It is not a penalty in itself, but search engines have to pick one version.
Read more about Duplicate contentThe small icon shown in browser tabs, bookmarks and some search results.
Read more about FaviconThe time until the browser first draws any text or image from the page.
Read more about FCPField data is measured from real visitors' browsers over time, for example in the Chrome User Experience Report.
Read more about Field data and lab dataA widely supported compression format for text responses.
Read more about GzipThe outline of a page made by its heading elements.
Read more about Heading structureAn annotation that tells search engines which language or regional version of a page to show to which audience.
Read more about hreflangAn HTTP response header that tells browsers to use only HTTPS for your domain for a set number of seconds, even if a visitor types an http:// address.
Read more about HSTSA name and value pair sent along with a request or a response.
Read more about HTTP headerThe three-digit number a server returns with every response. 200 means success, 301 and 302 are redirects, 404 means not found and 503 means the service is temporarily unavailable.
Read more about HTTP status codeNewer versions of the protocol browsers use to talk to servers.
Read more about HTTP/2 and HTTP/3A search engine storing a page in its index so that it can appear in results.
Read more about IndexingA Core Web Vitals metric for responsiveness.
Read more about INPA way to add structured data to a page as a script block of JSON that describes the page's content to search engines, using the schema.org vocabulary.
Read more about JSON-LDDelaying the loading of images or frames until they are near the visible area, for example with loading="lazy".
Read more about Lazy loadingA Core Web Vitals metric for loading.
Read more about LCPGoogle's open-source tool that loads a page in a controlled browser and scores performance, accessibility, best practices and SEO. Our speed test runs Lighthouse on our own servers.
Read more about LighthouseA proposed plain-text Markdown file at /llms.txt that gives AI assistants a short, curated guide to a site's most useful pages.
Read more about llms.txtA short summary of a page in a meta tag.
Read more about Meta descriptionRemoving whitespace, comments and other unneeded characters from HTML, CSS and JavaScript so the files are smaller.
Read more about MinificationAn HTTPS page that loads some resources, such as images, scripts or styles, over plain HTTP. Browsers may block these or show a warning.
Read more about Mixed contentGoogle predominantly uses the mobile version of a page's content for indexing and ranking.
Read more about Mobile-first indexingImage formats that usually give smaller files than JPEG or PNG at similar visual quality.
Read more about Modern image formatsA directive, set in a robots meta tag or an X-Robots-Tag HTTP header, that asks search engines not to include a page in their results.
Read more about noindexA set of meta tags (og:title, og:description, og:image and others) that control how a page looks when its link is shared on social networks and in chat apps.
Read more about Open GraphThe total size of all files a browser downloads to show a page: HTML, CSS, JavaScript, images, fonts and more.
Read more about Page weightAn HTTP response header that switches browser features such as the camera, microphone or geolocation on or off for a page and the frames embedded in it.
Read more about Permissions-PolicyAn instruction that sends visitors and crawlers from one address to another.
Read more about RedirectSeveral redirects in a row before the final page, for example http to https to www.
Read more about Redirect chainAn HTTP response header that controls how much of the current page's address is sent to another site when a visitor follows a link or loads a resource from it.
Read more about Referrer-PolicyA CSS or JavaScript file the browser must download and process before it can show the page.
Read more about Render-blocking resourceBuilding a page so the same HTML adapts to any screen size using flexible layouts and CSS media queries.
Read more about Responsive designThe srcset and sizes attributes list several sizes of an image so the browser downloads one that fits the screen.
Read more about Responsive imagesA plain-text file at the root of a site that tells crawlers which paths they may crawl.
Read more about robots.txtA shared vocabulary of types and properties, such as Article, Product and FAQPage, used to describe page content in structured data.
Read more about Schema.orgA file that lists several sitemap files, for sites with more URLs than one sitemap can hold.
Read more about Sitemap indexA Lighthouse metric that shows how quickly the visible parts of a page are filled in while it loads.
Read more about Speed IndexA Lighthouse lab metric that adds up the time the main thread was blocked by long tasks (over 50 ms) while the page was loading.
Read more about TBTA page with very little useful text of its own.
Read more about Thin contentCode loaded from another domain, such as analytics, ads, chat widgets or fonts.
Read more about Third-party scriptThe title element in a page's head.
Read more about Title tagTransport Layer Security, the protocol that encrypts the connection between browser and server.
Read more about TLSThe time between a browser requesting a page and receiving the first byte of the response.
Read more about TTFBMeta tags such as twitter:card that control how a link looks when shared on X. When they are missing, X can fall back to the Open Graph tags.
Read more about Twitter/X CardA meta tag that tells mobile browsers to match the page width to the device width.
Read more about Viewport meta tagThe Web Content Accessibility Guidelines, published by the W3C. They define testable success criteria at three levels (A, AA and AAA).
Read more about WCAGAn HTTP response header whose only valid value, nosniff, stops browsers from guessing a file's type and forces them to respect the declared Content-Type.
Read more about X-Content-Type-OptionsAn HTTP response header that carries the same directives as the robots meta tag, such as noindex.
Read more about X-Robots-TagA file that lists the URLs you want search engines to know about, optionally with last-modified dates.
Read more about XML sitemapNo terms match. Try a shorter word, or search the whole site.