Searched refs:Robots (Results 1 – 6 of 6) sorted by relevance
5 name NFTR - Not for the Robots!6 …e search engines (set robots to "noindex,follow" and send HTTP header "X-Robots-Tag: noindex") but…
7 class Robots class
17 use ComboStrap\Robots; alias337 $follow = Robots::canBeIndexedAndGetFollowValue($page, $executionContext);
568 …ml><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8"><meta name="Robots" ');
104 Robots will follow every link, which means that they will generate all pdfs for all pages and names…
1 …Robots.txt is a file which is usually placed in the root of any website. It decides whether crawle…