tidyfetch-bot

Requests with the user agent tidyfetch-bot/1.0 (+https://data.tidyfetch.com/bot) come from our collectors. They read only logged-out, public files.

What it fetches

• Shopify stores: /robots.txt, /meta.json and /products.json (first page), once when a store is first seen and occasionally to refresh.
• Job boards: the public job-board APIs of Greenhouse, Lever and Ashby, and the board page title.

Rules it follows

• It obeys robots.txt (cached up to 7 days) and skips stores that disallow it.
• At most 1 request per second per platform from one IP address, slower after any HTTP 429. No proxies or IP rotation.
• Conditional requests (ETag, If-Modified-Since) on job boards.

Opt out

Disallow tidyfetch-bot in robots.txt, or email [email protected] with your domain or board and we add it to our suppression list within 2 business days.