Results are in….
Does Google crawl links that have the “nofollow” tag appended to them?
Does Google crawl links that reside in a folder disallowed in the robots.txt?
How we tested…
…….lets find out:
The following links are filtered based on the user agent of the visit, if the visit is from the googlebot then the link will show a specific file that records when it has been accessed. Otherwise if the user agent is not the googlebot then it shows the links you see here.
The first link is in the root directory but has the “nofollow” tag appended to it.
the second link is in a directory that has been set to disallow in the robots.txt.
Now to wait for google to crawl it…
Results to follow: