DatafalkBot

DatafalkBot is the web crawler operated by Datafalk. It collects publicly available web pages about companies and their supply chains, in several languages, to build the Datafalk platform.

This page describes how the crawler behaves. It applies the Datafalk Public Data Collection Principles, section 3.2.

User agent

DatafalkBot sends the following User-Agent header:

DatafalkBot/1.0 (+https://www.datafalk.com/bot)

When a page is rendered with the Lightpanda engine, the header reads:

Lightpanda/<version> DatafalkBot/1.0 (+https://www.datafalk.com/bot)

The DatafalkBot token is always present, so you can match it in your server rules and in robots.txt.

robots.txt

DatafalkBot reads /robots.txt before crawling a site and follows it as defined in RFC 9309:

  • It applies the rules of the User-agent: DatafalkBot group. If there is no such group, it applies the User-agent: * group.
  • It honours Crawl-delay, up to a maximum of 30 seconds.
  • If robots.txt cannot be reached or returns a server error (5xx), DatafalkBot does not crawl the site.
  • If robots.txt returns a client error (4xx, for example 404), the site is treated as having no restriction.

Text and data mining reservations

Under Article 4 of Directive (EU) 2019/790, a rights holder can reserve their content against text and data mining in a machine-readable way. DatafalkBot recognises the following signals, including those defined by the W3C TDMRep specification:

  • the HTTP header tdm-reservation: 1;
  • the HTML tag <meta name="tdm-reservation" content="1">;
  • a reservation in /.well-known/tdmrep.json that covers the requested path;
  • the noai directive in the X-Robots-Tag HTTP header or in a robots <meta> tag.

When one of these signals applies, the page is not extracted and not stored.

Request rate

DatafalkBot waits at least 1 second between two requests to the same host. A longer Crawl-delay in robots.txt takes precedence (up to 30 seconds).

What DatafalkBot does not do

  • It never logs in to a site.
  • It does not bypass paywalls.
  • It does not solve captchas or anti-bot challenges. A page that shows one is skipped.

Refusal log

Every page that DatafalkBot does not fetch or does not keep because of robots.txt or a TDM reservation is recorded in a log. This log lets us show that these rules were applied.

How to block DatafalkBot

To block DatafalkBot from your whole site, add the following to your robots.txt:

User-agent: DatafalkBot
Disallow: /

To block only part of the site, list the paths concerned with Disallow, for example Disallow: /private/.

Contact

For any question, restriction or removal request, write to data@datafalk.com. Please include the domain or URLs concerned.