Re: robots.txt question
Thanks for at least trying. Yes, I'm aware of robotstxt.org. My question was
not regarding user-agents, but about whether a string such as /foo blocks
only http://website.org/foo, or if it would block
http://website.org/blah/foo. I think it would not, but I'm looking for
documentation, references, or examples that specifically address that.
From reading Google's reference on robots.txt as extended by Google,
Microsoft, and Yahoo, the most obvious interpretation would be that I need to
block /*foo, but again, no specific confirmation.
As for the trailing slash, again, no specific reference to that format so far.
KCLUG mailing list