We simulate this by parsing robots.txt and blocking access to the Disallow lines in robots.txt
Attach a file by drag & drop or click to upload
