TopStyle, exclusions, robots.txt, robots meta tags and sitemaps.

Newsgator Forums
Home       Members    Calendar    Who's On
Welcome Guest ( Login | Register )
        



TopStyle, exclusions, robots.txt, robots meta tags and sitemaps. Expand / Collapse
Author
Message
Posted 10/21/2007 3:48:57 PM
Forum Member

Forum Member

Group: Forum Members
Last Login: 12/28/2008 6:14:43 PM
Posts: 13, Visits: 33
After my recent tests of TopStyle 3.5 I found that I had inadvertently triggered some side effects when TopStyle created a site.

In other words TopStyle fired a page invisible to browsers and Search Engines that I considered "safe".

Fortunately I have a "checking robot" that detected the problem and sent me an email alert.

(There may be a feature that already does what follows.  I didn't find it.  If it's there my apologies in advance.)

It seems to me that TopStyle acts like a robot, but with privileged access:  it can see the filing system.  The web site in question has a robots.txt which prohibits access to the files I don't want touched.

To guard against such problems l suggest:

  1. TopStyle reads the robots.txt file and, by default, adhere to it instructions.  (When using server mappings to access pages.)
  2. TopStyle should be individually addressable as a user agent in robots.txt.

(There are possible applications of the robots meta tag, for a similar purpose.  My current concern cannot be addressed that way.  Sitemaps are not a mechanism for doing this as they only list pages you can access.)

I'm usually careful about this sort of thing but let my guard down when testing!!

Post #32346
Posted 10/22/2007 1:51:16 PM
NewsGator

NewsGator

Group: Administrators
Last Login: 12/18/2008 8:43:14 AM
Posts: 2,264, Visits: 54,129
Thanks for the feedback - I've passed it through to the developer to take a look at it. (113735)


Jonathon McDougall
NewsGator Support
Post #32368
Posted 11/14/2007 4:37:17 PM
NewsGator

NewsGator

Group: Administrators
Last Login: 12/18/2008 8:43:14 AM
Posts: 2,264, Visits: 54,129
At this point we are not likely to implement this feature.


Jonathon McDougall
NewsGator Support
Post #33376
Posted 11/14/2007 5:53:23 PM
Forum Member

Forum Member

Group: Forum Members
Last Login: 12/28/2008 6:14:43 PM
Posts: 13, Visits: 33
Thanks Jonathon.

I'll keep Topstyle away from web sites that have this issue until it's sorted.

Post #33378
« Prev Topic | Next Topic »


All times are GMT -7:00, Time now is 6:29am

Powered By InstantForum.NET v4.1.4 © 2009
Execution: 0.172. 9 queries. Compression Disabled.

NewsGator.com | Support | Developers | Partners | Press | Company
© Copyright 2003-2009  NewsGator Technologies, Inc. All Rights Reserved.
Copyright Information | Privacy Policy | Terms of Service
Microsoft Gold Certified Partner