Privacy Policy

Last updated: August 30, 2026

What happens when you check a URL

The crawler-access check necessarily runs on our server: when you submit a URL, our server fetches that site's robots.txt and its homepage a handful of times (with different user-agent strings) to produce your report. The URL you check:

• is processed transiently to run the check and briefly cached (minutes) so repeat checks don't re-fetch the target;
• is not stored in our analytics, not linked to you, and not sold or shared;
• is not used to build any crawl index or dataset.

The robots.txt analyzer tool is different: text you paste there is parsed entirely in your browser and never sent to our server.

Anonymous usage statistics

To understand which tools help people, we record anonymous, first-party usage events (a tool page was viewed; a check or processing run succeeded or failed; duration). These events never include the URLs you check, or any personal information. We use a first-party random identifier stored in your browser to count returning visitors; no third-party advertising cookies are used for this, and the Do Not Track setting is honored for pageview analytics.

For aggregate statistics, your IP address is used at request time to derive a country code (using a local copy of the DB-IP database, CC BY 4.0) and is then discarded — we do not store IP addresses with analytics data.

Advertising

This site may display advertising in the future to keep the tools free. If that happens, this policy will be updated first, and advertisers will never receive the URLs you check.

Who operates this site

Am I Crawlable is operated by Webmaster and More LLC (Los Angeles, California, USA). It is an independent utility and is not affiliated with the companies or organizations whose crawlers it describes.