YardimoBot

YardimoBot only crawls domains whose ownership has been verified. A customer cannot start a crawl before proving ownership with a meta tag on the home page or a verification file on the server.

The text of crawled pages is stored only in that customer knowledge base and is used only to answer their own visitors.

It follows robots.txt rules and respects Crawl-delay. By default it sends at most one request per second to a site and opens no concurrent connections.

It reads HTML pages only; images, video and responses larger than 5 MB are not downloaded. Pages marked noindex are skipped.

Add the following lines to your robots.txt file. The block takes effect on the next crawl.