Crawler
DisclosedBot
You are probably here because you found this in a server log:
DisclosedBot/1.0 (+https://disclosed.info/bot)That is the only user agent this project has ever sent. It is never disguised as a browser, and it is never changed to get a different answer out of a server: where a site has refused us, we record the refusal and stop, rather than asking again under another name.
How to block it
Put this in your robots.txt. It is honoured on every request, and it is
re-checked at every redirect hop rather than once at the start.
User-agent: DisclosedBot
Disallow: /
A path-level rule works the same way. If your robots.txt disallows a path, that
path is not requested — and the refusal is recorded as a finding rather than as a gap,
because “the operator said no” and “we did not look” are different
facts and we publish them apart.
If your robots.txt is unreadable to us — a 403, a bot interstitial, a
timeout — we treat that as a refusal too. That is our own rule rather than your
instruction, and we say so, in those terms, wherever we report it.
What it does
It fetches public product-review articles and records, for each one, the outbound links on the page after the page’s own JavaScript has run, where any disclosure text sits relative to the first outbound link, and the ranked order of the products the article recommends. It exists to measure one thing: whether the commission a publisher earns on a product predicts how highly that publisher ranks it.
What it does not do
- It never follows an affiliate link. Not to check where it goes, not to verify a merchant, not once. Following one registers a click and credits a publisher for traffic that never happened.
- It never transacts. No sign-ups, no carts, no purchases, no accounts.
- It never logs in and never goes behind a paywall. Public pages only.
- It does not walk your site graph. Candidate URLs come from your own sitemap. We read the list you published; we do not guess URLs and we do not follow links from one page of yours to another.
- It sends no cookies and keeps no cookie jar between requests.
-
It changes nothing. Every request is a
GETor aHEAD.
Rate
It honours Crawl-delay where you set one, and never goes faster than a floor of
one request every 1.5 seconds even where your robots.txt is silent about it. It
is a single process. There is no distributed fleet.
Redirect resolution
Where we need to know where a shortened or cloaked link goes, that is resolved from our own
servers, cookielessly, with HEAD where the host allows it, reading the
forwarding address out of the response header and stopping — the destination itself is
not requested. robots.txt is re-checked before every hop, against the host of
that hop. No reader’s browser is ever involved, which is the point: a reader’s
browser firing an affiliate cookie would hijack a commission from the creator who earned it,
and that is the practice this project exists to expose.
If we have crawled you
Every number published about a site is recomputable by a third party from inputs published beside it, and every commission rate carries the page it came from and the date we fetched it. If something is wrong, we want to know precisely which number and why; we correct it and say that we did.
Anyone named in anything we publish gets a right of reply, in writing, before publication, and the reply is published beside the finding.
Contact
bot@disclosed.info — crawler conduct, block requests, and corrections.
A block request is honoured on receipt and does not need a reason. We would rather have the refusal than the argument.
Nothing else is served at this address.