DisclosedBot

How to block it

Put this in your robots.txt. It is honoured on every request, including requests to redirect endpoints, and it is re-checked at every redirect hop rather than once at the start.

User-agent: DisclosedBot
Disallow: /

A path-level rule works the same way. If your robots.txt disallows a path, that path is not requested — and the refusal is recorded as a finding rather than as a gap, because “the operator said no” and “we did not look” are different facts and we publish them apart. Wirecutter’s Disallow: /wirecutter/out/ is the largest such refusal in our data and it is published as one.

If your robots.txt is unreadable — a 403, a bot interstitial, a timeout — we treat that as a refusal too. That is our own rule, not your instruction, and we say so wherever we report it.

What it does

It fetches public product-review articles and records, for each one, the outbound links on the page after the page’s own JavaScript has run, where any disclosure text sits relative to the first outbound link, and the ranked order of the products the article recommends. It exists to measure one thing: whether the commission a publisher earns on a product predicts how highly that publisher ranks it.

What it does not do

• It never follows an affiliate link. Not to check where it goes, not to verify a merchant, not once. Following one registers a click and credits a publisher for traffic that never happened.
• It never transacts. No sign-ups, no carts, no purchases, no accounts.
• It never logs in and never goes behind a paywall. Public pages only.
• It does not walk your site graph. Candidate URLs come from your own sitemap. We read the list you published; we do not guess URLs and we do not follow links from one page of yours to another.
• It sends no cookies and keeps no cookie jar between requests.
• It does not modify anything. Every request is a GET or a HEAD.

Rate

It honours Crawl-delay where you set one, and never goes faster than a floor of one request every 1.5 seconds even when your robots.txt is silent about it. It is a single process. There is no distributed fleet.

Redirect resolution

Where we need to know where a shortened or cloaked link goes, that is resolved from our own servers, cookielessly, with HEAD where the host allows it, reading the forwarding address out of the response header and stopping — the destination itself is not requested. robots.txt is re-checked before every hop. No reader’s browser is ever involved, which is the point: a reader’s browser firing an affiliate cookie would hijack a commission from the creator who earned it, and that is the practice this project exists to expose.

If you are a publisher we have crawled

Every number we publish about a publisher is recomputable by a third party from inputs we publish, and every rate we cite carries the page it came from and the date we fetched it. If something we published is wrong, we want to know precisely which number and why, and we will correct it and say that we did.

Every publisher named in anything we publish gets a right of reply, in writing, before publication, and the reply is published on that publisher’s own page.

Contact

bot@disclosed.info — crawler conduct, block requests, and corrections.

A block request is honoured on receipt and does not need a reason. We would rather have the refusal than the argument.