ShopproofBot
The crawler of Shopproof, a project that checks publicly available facts about online shops in the EU.
What it does
ShopproofBot visits websites of newly registered domains to find out whether they are online shops and to record publicly visible facts about them, for example the shop platform, the payment methods offered and the details published in the legal notice (Impressum). These facts are used for an automated, explainable assessment of online shops.
What it requests
/robots.txt, the home page, the legal notice page and the favicon: at most four requests per domain per visit;- repeat visits: about once a day while a shop is younger than 60 days, about once a week after that, and less often for sites that are not shops;
- plain HTTP GET requests only. It does not run JavaScript, submit forms, log in, add items to a cart or place orders.
From the legal notice page it stores the business details published there, such as the company name, VAT ID, commercial register number, e-mail address and phone number. Requests about this data: [email protected].
How to identify it
| User-Agent | ShopproofBot/0.1 (+https://shopproof.net/bot) |
| IPv4 | 2.31.8.54 |
| IPv6 | 2a01:4f8:1c18:36de::1 |
| Reverse DNS | crawler.shopproof.net, which resolves back to the same addresses |
Requests that carry this User-Agent but come from other addresses are not from ShopproofBot.
How to block it
ShopproofBot follows the Robots Exclusion Protocol (RFC 9309). To exclude your whole site, add to /robots.txt:
User-agent: ShopproofBot Disallow: /
Rules for User-agent: * apply if there is no group for ShopproofBot. Changes take effect within 24 hours. If robots.txt answers with a server error (5xx) or 429, the site is not crawled.
Contact
Questions and data requests: [email protected]
Abuse reports: [email protected]
Deutsch
ShopproofBot ist der Crawler von Shopproof, einem Projekt, das öffentlich verfügbare Fakten über Online-Shops in der EU prüft. Er besucht Websites neu registrierter Domains und erfasst öffentlich sichtbare Angaben, etwa Shop-System, angebotene Zahlungsarten und die im Impressum veröffentlichten Daten (Firmenname, USt-IdNr., Handelsregisternummer, E-Mail, Telefon).
Pro Besuch ruft er höchstens /robots.txt, die Startseite, das Impressum und das Favicon ab – bei Shops, die jünger als 60 Tage sind, etwa einmal täglich, danach etwa einmal pro Woche –, nur per HTTP GET, ohne JavaScript, Formulare, Logins oder Bestellungen. Er hält sich an das Robots Exclusion Protocol (RFC 9309). Ihre gesamte Website schließen Sie so aus:
User-agent: ShopproofBot Disallow: /
Änderungen werden innerhalb von 24 Stunden wirksam. Echte Anfragen kommen nur von 2.31.8.54 und 2a01:4f8:1c18:36de::1 (Reverse DNS crawler.shopproof.net).
Fragen und Auskunftsersuchen: [email protected] · Missbrauch: [email protected]