a parent asked us to watch one page on your site for a new school calendar. when the pdf behind it changes, we read it once and update their subscribed calendar. that is the whole job.
how often
once a week per watched page, and we send if-none-match and if-modified-since, so an unchanged page costs you a 304 and nothing more. we download the pdf itself only when its bytes have actually changed. a page that keeps failing is backed off exponentially rather than retried on schedule.
robots.txt
we read it before anything else, on every origin we touch, and we obey it — including crawl-delay. if your file returns a server error we treat the site as closed until it recovers.
how to stop us
add this to your robots.txt and we will stop on the next check:
User-agent: onthefridge-watcher
Disallow: /no appeal, no exception, and nothing to email us about first — though you are welcome to.
the full user-agent
onthefridge-watcher/1.0 (+https://onthefridge.app/bot)what we do not do
- · we do not crawl. we fetch the one page we were given.
- · we do not follow links except to the calendar pdf on it.
- · we do not submit forms or touch anything behind a login.
- · we do not republish your pdf — we read the dates and store those.