Our crawler
What Boys Don't Know is an independent UK jewellery magazine. Once a night we read the public product catalogues of the jewellery brands we cover, so we can compare prices and materials fairly. This page says what that crawler does and how to reach us about it.
How to recognise it
Its user agent starts with WBDK-catalogue and links to this page. Every request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). The Signature-Agent is https://whatboysdontknow.com, and our public key is at /.well-known/http-message-signatures-directory.
What it reads
Public catalogue data only: a store's /products.json, and for some stores a product's own page to read its UK price. It never logs in, adds to a basket, submits forms or reads anything behind a login. It doesn't download images, and it doesn't keep any personal data.
How often and how fast
Once a night, starting at 02:30 UK time, from one machine:
- One request at a time to any store, at least 5 seconds apart.
- No more than one request every 3 seconds in total, across every store.
- It follows robots.txt.
- When a site answers 429, it slows down and pauses for longer each time, and it respects
Retry-After. - A product page is only fetched again when the product has changed, or once a week.
Asking it to stop
Disallow WBDK-catalogue in your robots.txt and it stops reading your site from the next night. Or email tellme@whatboysdontknow.com and we'll remove you by hand.
Questions about how we use what we read go to the same address.