Unlike most articles on search engines this has nothing to do with meta tags. Instead we are going to discuss the rights and wrongs of redirects. The use of redirects in Adult Websites is very popular, but just how many people are using them the right way?
Being a tech with a well known web host I recently got to watch the error log on a google index spider. The spider entered our network and started listing pages, however much to my dismay I found it doing something else. The spider also started following the redirects. The problem here is it wasn't following the links everybody had hoped for.
First you may ask how I knew it was the google spider as google doesn't announce when it's going to spider. That was easy. The monitoring alarms for our servers started going off. I was finding disk space problems. While checking these disk space problems I found the error.log for apache growing on all our servers. Co-incidence? I think not.
While watching the logs grow, I was getting a feel for the most common errors that I was seeing. It was redirect errors in .htaccess files. Upon further investigation I was finding the SE spider was skipping these sites when it received the redirect error. I then started looking at the .htaccess files to find where the spider was 'erroring.' The things I found there prompted me to write this article and try to educate webmasters on the correct usage of .htaccess files. Listed below in order are the problems I found and the correct way of fixing the problem.
Redirects: This was the number one error the spiders found:
ErrorDocument 404 http://www.mydomain.com
Any problems with the above statement? Most webmasters will say no. However the above statement is completely wrong and will force a spider off your page and into an error. This will not only make the spider not list your site, but it will also force the spider to stop following your links, thus hurting you in Search Engine Listings. The proper way to make this statement is as follows: If you're sending people from your current domain on the error to another domain, then the correct syntax is:
Redirect 404 http://www.mydomain.com
If you're keeping them on your domain but sending them to a custom error page, then the proper syntax is as follows:
ErrorDocument 404 /errors/404.html
The above path means there is a folder called 'errors' and a html document called '404.html' in that folder. That folder would be the first branch off where the .htaccess file is located.
Problem 2 with redirects:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|Gif|jpg|JPG|jpeg)$
http://www.mydomain.com [R,L]
Some people try to use the above for capturing hot-linkers to their site. This syntax does not work at all. The correct way to do this is as follows:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|Gif|jpg|JPG|jpeg)$ - [F]
If anybody is hot-linking to you then it will create a 403 error (Forbidden). So under this statement you use either Redirect or Errordocument from the above lesson. IE:
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|Gif|jpg|JPG|jpeg)$ - [F]
ErrorDocument 403 /error/403.html
You won't only reap the rewards in better directed traffic, but it will help reduce your bandwidth and server crashes also.
I hope this helps you in your quest for making sure your traffic gets to your most profitable pages. If you find your .htaccess file is in error, please fix it. You won't only reap the rewards in better directed traffic, but it will help reduce your bandwidth and server crashes also. Your web server software won't be working as hard to figure out where you really wanted your traffic to go.
