Our crawler

TuppenceResearchBot

If you found this page in your server logs, our crawler visited your website. Here is what it reads, what it never does and how to stop it.

1. Who it is

TuppenceResearchBot is run by Tuppence AI Ltd. It reads the public pages of business websites to understand how a business takes payments online, so that we only write to businesses that Tuppence might save money. Every request it makes carries this user agent:

TuppenceResearchBot/1.0 (+https://tuppence.ai/crawler)

It reads robots.txt rules written for TuppenceResearchBot, and the rules for every crawler (User-agent: *) when there are none for it.

2. What it reads

Only pages anyone can see without signing in, and only the ones that say something about payments or about how to reach the business: the home page, shop and product pages, the basket or cart, booking and pricing pages, the terms and legal pages, and the contact, about and team pages. From them it notes:

  • which payment provider's checkout, scripts or booking tool the site uses;
  • prices and the currency they are in, and where the business delivers;
  • the card and wallet logos shown;
  • whether the site sells to consumers or to other businesses;
  • the legal name and registration details in the footer or terms;
  • the contact details the business publishes: email addresses at its own domain, telephone numbers, its contact form and its address, and the names and job titles shown beside them.

A few of those pages, such as a checkout, booking or product page, are also opened in a sandboxed browser, because many payment forms only appear once the page's own scripts have run. In that browser it only looks. It may click a link or button that leads to the checkout, and it stops when it reaches the payment form. Any form submission is blocked before it leaves the browser, and it sends nothing but ordinary page requests to your site.

Every fact is kept with the address of the page it came from and when it was read. The words on a page are treated as information about the business, never as instructions to our software.

3. What it never does

  • It never logs in, and never reads anything behind a sign-in.
  • It never submits a form, never types into a field and never enters card details.
  • It never adds anything to a basket or cart, and never makes a purchase or a booking.
  • It never clicks a cookie banner. Not accepting it means nothing is agreed to.
  • It never visits addresses inside private networks, and it does not follow links to other websites.
  • It does not crawl social media or review sites.

4. How often and how fast

  • It reads robots.txt first and follows it. A path it disallows is never requested. If robots.txt cannot be read because your server is having trouble, the crawler reads nothing and tries another day.
  • It sends at most one request to your server every second, from all of our machines together, over one connection it keeps open. If your robots.txt sets a longer Crawl-delay, it waits that long instead, and reads fewer pages.
  • It reads at most 25 pages of a site per visit, and usually far fewer: it stops once it has what it came for. It reads at most the first 1.5 million characters of a page. On a later visit it asks whether a page has changed before reading it again. It opens at most 6 of them in the sandboxed browser, for no more than 20 seconds each. Like any browser, it then loads the images and scripts those pages ask for.
  • It visits a site again no sooner than 14 days later, unless a member of our team asks for a fresh look. For a business we keep following, it looks again about once a month, to see whether the way it takes payment has changed.

5. How to keep it out

Either of these works, and you can do both.

With robots.txt. Add these lines to robots.txt at the root of your site. The crawler reads the file at the start of every visit, so it takes effect on the next one:

User-agent: TuppenceResearchBot
Disallow: /

By email. Email contact@tuppence.ai with “Crawler opt-out” in the subject line and your website’s domain. We add the domain to our suppression list: the crawler stops visiting it, and we never email anyone at that domain about Tuppence. We keep only a one-way code made from the domain, so we can keep that promise without keeping a list of names.

6. What we keep

We keep the facts listed above about the business, with where and when each was seen, and a record of each visit: the pages read and what robots.txt said. If a page names a person, such as a director on an about page, our Prospect Privacy Notice explains what we do with it and how to ask us to delete it.