How we check websites
Up’n’Top checks sites on a schedule, slowly and politely. Here is exactly what we check, how often, and how to recognise our bot.
Which pages we check and how often is decided by our own algorithm: pages that matter more are checked more often.
How often
| What | How often |
|---|---|
| Is the site up (home page) | every 5 minutes, or every 15 if you choose; every minute on the Pro and Agency plans once you confirm the site is yours |
| Home page and key pages you choose | every hour by default; every 15 or 30 minutes if you choose |
| SSL certificate | twice a day |
| Domain registration | every week; every day in the last 30 days |
| DNS records | every 6 hours |
| robots.txt | every hour |
| Pages of each type | a sample every day |
| Every other page | about every 8 days; important pages and pages with errors more often |
| Pages closed to search engines (noindex) | every 30 days |
| New pages | a full crawl every 30 days |
| Site-wide checks | every day: analytics tags, scripts, feeds, legal pages; every week: exposed files, plugin vulnerabilities; every month: cookies before consent |
Before we call something a problem
- Site down: after the first failed check we try again in 20 seconds, then every minute. We report the site as down only after two failures in a row and at least 3 minutes, and only if checks from other locations don’t see it working.
- Broken page: the error must last at least 4 hours and repeat on a check made 4 hours later. If many pages fail at once, we wait 8 hours — it is usually a server-wide problem.
Alerts
- Urgent problems — the site is down, the SSL certificate expires in 3 days, the domain in 7 days — are sent at once.
- Everything else is grouped into one summary every 15 minutes by default, so ten broken pages are one message, not ten.
- Channels: email, Telegram, Slack and webhooks.
Gentle on your server
- At most 2 requests per second to one site, one request at a time.
- If the server answers 429, or 503 with a Retry-After header, we pause for up to 60 seconds and slow down step by step, to one request every 5 seconds at most.
- New pages we discover are visited only if robots.txt allows it. Pages you add to monitoring are checked because you asked us to.
Our bot
- Monitoring:
Mozilla/5.0 (compatible; UpntopBot/1.0; +https://upntop.com/bot) - Free website check:
UpntopCheck/1.0 (+https://upntop.com/bot) - Checks in a browser:
Chrome/… UpntopBot/1.0 (+https://upntop.com/bot) - Once a week a check of whether your CDN or firewall lets search bots in adds
ai-access-check/<name>to our User-Agent, where the name is OAI-SearchBot, PerplexityBot or Claude-SearchBot. We never send other companies’ bot names as our own User-Agent. - To block it, add
User-agent: UpntopBot / Disallow: /to robots.txt, or write to support@upntop.com and we will stop checking your site.
What we check
115 checks in eight areas: availability, security, SEO, speed, content, accessibility, email setup and search answers.