Comments on: robots.txt: People Don’t Always Want Search Engines to Crawl Their Content http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/ Social Media Marketing Mon, 15 Mar 2010 10:32:45 -0700 http://wordpress.org/?v=2.9.2 hourly 1 By: lee http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-455750 lee Fri, 11 Sep 2009 21:35:58 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-455750 hi i have been waiting for google and bing to index my blog <a href="http://www.diyanswerdirect.com/blog" target="_blank">www.diyanswerdirect.com/blog</a> for two weeks now and am getting very frustrated with them. aswell as iam still waiting for them to update my site <a href="http://www.diyanswerdirect.com." target="_blank">www.diyanswerdirect.com.</a> My question is when do they update and how can i get them to crawl or spider my site. hi i have been waiting for google and bing to index my blog http://www.diyanswerdirect.com/blog for two weeks now and am getting very frustrated with them. aswell as iam still waiting for them to update my site http://www.diyanswerdirect.com. My question is when do they update and how can i get them to crawl or spider my site.

]]> By: Mexabet http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-371595 Mexabet Fri, 13 Mar 2009 22:22:26 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-371595 Normally I don't block search engines from crawling my sites, except the Admin directories. However, I started lately to worry about few duplicate title tags and Meta tags in one of my CMS-based sites, and I had to block spiders form crawling the affected pages. Normally I don't block search engines from crawling my sites, except the Admin directories. However, I started lately to worry about few duplicate title tags and Meta tags in one of my CMS-based sites, and I had to block spiders form crawling the affected pages.

]]> By: internet marketing http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-370933 internet marketing Thu, 12 Mar 2009 19:18:29 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-370933 I really liked your post. I will have to bookmark your site and come back later. I really liked your post. I will have to bookmark your site and come back later.

]]> By: Elmer Cagape http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-34659 Elmer Cagape Tue, 22 May 2007 08:51:58 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-34659 Is this just a link bait for people to wonder what could be the reason they have this message? Is this just a link bait for people to wonder what could be the reason they have this message?

]]> By: Search Engine Land: News About Search Engines & Search Marketing http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-34487 Search Engine Land: News About Search Engines & Search Marketing Mon, 21 May 2007 18:10:39 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-34487 <strong>SearchCap: The Day In Search, May 21, 2007...</strong> Below is what happened in search today, as reported on Search Engine Land and from other places across the web:...... SearchCap: The Day In Search, May 21, 2007…

Below is what happened in search today, as reported on Search Engine Land and from other places across the web:……

]]> By: share.websitemagazine.com http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-34440 share.websitemagazine.com Mon, 21 May 2007 14:34:50 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-34440 <strong>robots.txt: People Don’t Want SE's to Crawl Content...</strong> What goes through your mind when you read about the silly lawsuits against Google accessing portions of your website? What do you think when you visit the Internet Wayback Machine and find hundreds of pages of your site in its full form (almost)?... robots.txt: People Don’t Want SE’s to Crawl Content…

What goes through your mind when you read about the silly lawsuits against Google accessing portions of your website? What do you think when you visit the Internet Wayback Machine and find hundreds of pages of your site in its full form (almost)?…

]]> By: Tamar Weinberg http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-34033 Tamar Weinberg Sun, 20 May 2007 04:48:44 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-34033 anty: I've seen this example a number of times. I don't think so. :) It's late Saturday night (Sunday morning?) and I see the same file! anty: I’ve seen this example a number of times. I don’t think so. :) It’s late Saturday night (Sunday morning?) and I see the same file!

]]> By: anty http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-33779 anty Sat, 19 May 2007 07:32:02 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-33779 Another idea why the webmaster may could have added the comment-lines about the daytimes: Maybe this robots.txt-file is dynamically served and during daytimes (whatever daytimes are on the internet) this file is displayed, else robots are allowed to crawle the site. Just a thought, but this would explain this comment. Another idea why the webmaster may could have added the comment-lines about the daytimes:

Maybe this robots.txt-file is dynamically served and during daytimes (whatever daytimes are on the internet) this file is displayed, else robots are allowed to crawle the site.

Just a thought, but this would explain this comment.

]]> By: anty http://www.10e20.com/blog/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/comment-page-1/#comment-33778 anty Sat, 19 May 2007 07:26:33 +0000 http://www.10e20.com/2007/05/18/robotstxt-people-dont-always-want-search-engines-to-crawl-their-content/#comment-33778 This robots.txt file-line must be a joke. I don't think that a webmaster can be so stupid to build a website but not knowing that spiders aren't human. None the less a funny an helpful post! This robots.txt file-line must be a joke. I don’t think that a webmaster can be so stupid to build a website but not knowing that spiders aren’t human.

None the less a funny an helpful post!

]]>