What about a robot.txt file?

[odin_dax]

Admins, what's your opinion? I clicked on "Currently Active Members" and just about a 1/3 of the guests were spiders.

[DoctaD]

What's the problem with search engines crawling the site?

[odin_dax]

What's the problem with search engines crawling the site?

Nothing inherently. Some site owners take issue, so I was just asking if that was at all a concern. Nothing big.

[Stone]

They use such little bandwidth at the moment there's no need. Also, there's nothing we want to hide from them so we leave it. It's something to consider in the future though. I wonder does the phx.gbl bot obey a robots.txt file? I doubt it but if it did then that would be a way to stop it... Hmm...