Key pages, schedules and limits

Key pages are the pages that must never drop out of Google: your pricing page, your best products, sign-up, checkout. Your homepage always counts, on top of the ones you name. A site can have up to 10 on Free and 100 on Pro.

1. Open the site, then Settings. Setup is the first tab.
2. Under "Key pages", put one full address on each line, such as https://example.com/pricing.
3. Click "Save". Changes apply from the next crawl.

Each address has to be on the site's own addresses. Only the team's owner and admins can change a site's settings.

A site that moves to a smaller plan keeps its whole list, and the first as many as the plan allows are checked.

Once the site is verified, we check your key pages once a day, and 30 seconds after each deploy if your developer has added the deploy step. Each time we also read robots.txt and your sitemaps.

• A noindex on your homepage or a key page is a critical issue, and alerts.
• A new error on a key page alerts.
• Issues on key pages rank higher in "Fix first" on the overview.

These checks of your key pages never use the monthly page allowance.

A full crawl goes through the site the way Google does, following its links and its sitemap, up to the plan's size: 500 pages on Free and 10,000 pages on Pro. Choose how often under "How often to crawl the whole site", on the same Setup tab. Free offers "Only when I ask", "Monthly" and "Weekly". Pro offers "Only when I ask", "Monthly", "Weekly" and "Daily".

A site is crawled on a schedule only once it is verified. If you leave it on "Only when I ask" until then, verifying it the first time turns on "Weekly". The line under the choice gives the date of the next crawl.

Both buttons are at the top of the site's Overview.

• Crawl now starts a full crawl at once. One crawl of a site runs at a time, and 2 at a time for the whole team. Until the site is verified it can be crawled once a day, of up to 50 pages.
• Quick check reads robots.txt, the homepage and the sitemap in a few seconds. Use it to see whether a fix to one of those is live.

A team's monthly page allowance is 5,000 on Free and 500,000 on Pro pages, across all its sites. A month is a calendar month in UTC.

What counts

• Every page of every full crawl, on the schedule or started by you.
• The images, scripts and stylesheets a full crawl checks, up to 500 a crawl, and a few test requests, such as a made-up address to see how the site answers for a missing page.
• Rechecks of fixes.
• The pages a check looks at besides your key pages, such as pages new in your sitemap.

What doesn't

The daily and deploy checks of your homepage and key pages, with robots.txt and the sitemaps, and the quick check.

When it runs out

From 80% of the allowance, the site's pages show a notice and the team's owner gets an e-mail. Once it is all used, full crawls pause until the next month starts, whether scheduled or started by hand. The checks of your key pages go on, and so do rechecks of your fixes.

To see how much is used, open Team settings, then Plan, from the menu under your name. The same number is under "How often to crawl the whole site" in each site's settings. Your team and plan says how to move to a bigger plan.

Settings, then Advanced, holds "What to crawl". Most sites never need it, and changes apply from the next crawl.

• Site addresses to include: the names your site answers on, such as example.com and www.example.com. Tick "And addresses under them" to add ones like shop.example.com. Another domain is a site of its own: add it and verify it there.
• Only pages starting with and Skip pages starting with: one per line, each starting with a slash. * means anything, so /blog/* is every page under /blog/.
• Test-site addresses: such as staging.example.com. We warn if your live site points at one of them.
• Most pages per crawl: fewer than the plan's size, if you want smaller crawls. Left empty, a crawl goes up to the plan's size.
• Clicks from the homepage to follow: how many links deep a crawl goes, 10 unless you set another.