Archive
Monthly
Go
|
|
DNN Blog
Jun
3
Posted by:
cathal connolly
6/3/2008 9:39 PM
Since 4.5.2 DotNetNuke has had the capability to automatically generate an xml sitemap , a useful way to help search engines index your website. Typically you have to go to a website such as www.google.com/webmasters/tools to submit your sitemap url, however theres an easier way to do this. Search engines will automatically check for the existance of a robots.txt file in your site root to see what sections of a site are allowed to be indexed. A number of the major search engines such as Live, Google, Yahoo and Ask also support picking up the locations of sitemap's via an autodiscovery instruction. To save yourself the effort of manually registering your sitemap page, just create a robots.txt file and add a line such as:
Sitemap: http://www.yourwebsite.com/sitemap.aspx
The next time a search engine that supports autodiscovery browses your site they will see where the sitemap can be found and use that to help index site content.
Note: whilst the core sitemap does a good job of indexing pages, there are a number of other community donated sitemap's that work well with dynamic modules, the http://www.dotnetnuke.com/Community/Forums/tabid/795/forumid/108/threadid/207139/scope/posts/threadpage/1/Default.aspx thread has some good information on this.
7 comment(s) so far...
Re: use robots.txt to get your sitemap automatically discovered by search engines
great tip, I had applied it in our website!
By sunwangji on
6/5/2008 9:28 AM
|
Re: use robots.txt to get your sitemap automatically discovered by search engines
How does an administrator do this for a multiple site install? Since the URL has to have the full path to the site it would make all sites point to only one site's site map. Any ideas?
Dax
By daxdavis on
6/8/2008 6:52 PM
|
Re: use robots.txt to get your sitemap automatically discovered by search engines
@Dax, for multiple sites you have to manually register via the search engines themselves.
By cathal on
6/9/2008 10:34 PM
|
Re: use robots.txt to get your sitemap automatically discovered by search engines
What is new in recent version and how does it handle the Robot.txt. My site is hosted on powerdnn.com. How do you go to the root directory to see your robot.txt file. Thank you
By Ganti on
1/27/2009 4:07 PM
|
Re: use robots.txt to get your sitemap automatically discovered by search engines
@Ganti, there was a sitemap change for the protocol(support.dotnetnuke.com/issue/ViewIssue.aspx?id=9210&PROJID=2), but i don't believe there were any other changes. To get to your robots.txt file just type www.yourdomain.com/robots.txt
By Ganti on
1/27/2009 4:56 PM
|
Re: use robots.txt to get your sitemap automatically discovered by search engines
We host multiple portals, and some of our customers want to use SEO services requiring a custom "ROBOTS.TXT" file for each of their sites. So far, the only "robots.txt" file that is visible is the one sitting in the main DNN installation root folder, so when someone types //www.mysite1.com/robots.txt or //www.mysite2.com/robots.txt they always get the same file from the root DNN folder. This looks like a major development oversight -- each site gets its own SiteMap but not the "robots'txt" file?
By SelfNet on
8/30/2010 3:10 PM
|
Re: use robots.txt to get your sitemap automatically discovered by search engines
@Selfnet, no, this is not a development oversight, it's a limitation of the robots.txt capabilites. Search engines only look for 1 per site and always in the site root folder (i.e. you can create site1_robots,txt etc.), meaning it's not suited to multiportal setups -if your customers need that level of filtering you will have to create seperate sites for them.
By cathal connolly on
8/30/2010 3:13 PM
|
|