SEOUptimeBot

You are probably here because this turned up in your logs. It is the crawler for SEO Uptime, and this page says what it is doing, how to control it, and how to reach a person.

How to recognise it

User-Agent

SEOUptimeBot/1.0 (+https://seouptime.com/bot)

robots.txt token

SEOUptimeBot

Comes from

167.233.176.200

Anyone can put that string in a User-Agent header. If something claiming to be us is behaving badly, the addresses above are the only ones that are us. Where this page does not list them yet, we would rather you told us than assumed.

Why it fetched your site

One of two reasons, and no others:

• Someone with an account asked us to watch this site. We re-crawl it on a schedule, and after a deploy if they wired that up, so we can tell them when a page that search engines could index stops being indexable.
• Someone ran a free check at whynotindexed.com, which crawls up to 50 pages of a site once a day and then stops. After that, a page someone asks about that the crawl did not reach is fetched once, on its own.

We do not crawl to build an index, to train a model, or to resell what we find. We read pages the way a search engine reads them and report what would stop one indexing them.

How to slow it down or stop it

It reads robots.txt before anything else and obeys it. To keep it out entirely:

User-agent: SEOUptimeBot
Disallow: /

To keep it out of one part of the site, or to make it wait between requests:

User-agent: SEOUptimeBot
Disallow: /admin/
Crawl-delay: 10

Crawl-delay is honoured up to 10 seconds; a larger number is read as 10. A 429 or 503 with Retry-After is honoured up to 60 seconds.

How hard it pulls

• At most 8 requests in flight against your host at any moment, counted across everyone who has asked us to watch it, not per customer.
• A single crawl opens 2 connections by default, waits for what it asked for, and stops at the page cap its plan sets.
• A failed request is retried 2 times, then counted as a failure. If more than half of the last 50 requests fail, the crawl stops rather than keeps knocking.
• It waits 5 seconds to connect and 20 seconds in total for a response, so a slow page is abandoned rather than held open.

What it keeps

What was fetched, so a report can quote it: the response headers, the HTML, and what the rules made of them. It is how a finding can say what we saw on this URL rather than asking you to take our word for it.

A free check at whynotindexed.com is deleted after a day, automatically, and belongs to no account while it exists. For a site someone is watching, the crawl is theirs and is removed when they remove the site.

It does not sign in, submit forms, or run your JavaScript. It fetches pages, and the files a page needs, over HTTP.

If it is causing you trouble

Write to abuse@seouptime.com with the host and roughly when, and we will stop crawling it. You do not need an account with us, and you do not have to edit robots.txt first, though the rule above takes effect faster than we can read e-mail.