Home > Features > Getting Listed With Search Engines • Bookmark   • Newsletters   • Register Search Options

FEATURE

Getting Listed With Search Engines

Getting Listed With Search Engines

April 23, 2002
Text size: 
Get XBIZ News
XBIZ Research
Will virtual reality boost the paysite market?
Yes, it will soon
  39.66%
Yes, but in a few years
  36.87%
No
  23.46%
Out of 179 votes. Results based on votes submitted by members of XBIZ.net social network.

" While checking these disk space problems I found the error.log for apache growing on all our servers. Co-incidence? I think not. "

Unlike most articles on search engines this has nothing to do with meta tags. Instead we are going to discuss the rights and wrongs of redirects. The use of redirects in Adult Websites is very popular, but just how many people are using them the right way?

Being a tech with a well known web host I recently got to watch the error log on a google index spider. The spider entered our network and started listing pages, however much to my dismay I found it doing something else. The spider also started following the redirects. The problem here is it wasn't following the links everybody had hoped for.

First you may ask how I knew it was the google spider as google doesn't announce when it's going to spider. That was easy. The monitoring alarms for our servers started going off. I was finding disk space problems. While checking these disk space problems I found the error.log for apache growing on all our servers. Co-incidence? I think not.

While watching the logs grow, I was getting a feel for the most common errors that I was seeing. It was redirect errors in .htaccess files. Upon further investigation I was finding the SE spider was skipping these sites when it received the redirect error. I then started looking at the .htaccess files to find where the spider was 'erroring.' The things I found there prompted me to write this article and try to educate webmasters on the correct usage of .htaccess files. Listed below in order are the problems I found and the correct way of fixing the problem.

Redirects: This was the number one error the spiders found:

ErrorDocument 404 http://www.mydomain.com

Any problems with the above statement? Most webmasters will say no. However the above statement is completely wrong and will force a spider off your page and into an error. This will not only make the spider not list your site, but it will also force the spider to stop following your links, thus hurting you in Search Engine Listings. The proper way to make this statement is as follows: If you're sending people from your current domain on the error to another domain, then the correct syntax is:

Redirect 404 http://www.mydomain.com

If you're keeping them on your domain but sending them to a custom error page, then the proper syntax is as follows:

ErrorDocument 404 /errors/404.html

The above path means there is a folder called 'errors' and a html document called '404.html' in that folder. That folder would be the first branch off where the .htaccess file is located.

Problem 2 with redirects:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|Gif|jpg|JPG|jpeg)$
http://www.mydomain.com [R,L]

Some people try to use the above for capturing hot-linkers to their site. This syntax does not work at all. The correct way to do this is as follows:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|Gif|jpg|JPG|jpeg)$ - [F]

If anybody is hot-linking to you then it will create a 403 error (Forbidden). So under this statement you use either Redirect or Errordocument from the above lesson. IE:

RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com$ [NC]
RewriteCond %{HTTP_REFERER} !^http://yourdomain.com/.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.yourdomain.com/.*$ [NC]
RewriteRule .*\.(gif|GIF|Gif|jpg|JPG|jpeg)$ - [F]
ErrorDocument 403 /error/403.html
You won't only reap the rewards in better directed traffic, but it will help reduce your bandwidth and server crashes also.

I hope this helps you in your quest for making sure your traffic gets to your most profitable pages. If you find your .htaccess file is in error, please fix it. You won't only reap the rewards in better directed traffic, but it will help reduce your bandwidth and server crashes also. Your web server software won't be working as hard to figure out where you really wanted your traffic to go.


LEGAL PERSPECTIVES

Porn ‘Public Health Crisis’ Is False, Dangerous

We are witnessing the early stages of a moral panic, the likes of which we have not seen since the Reagan Administration. As I write this, there are 27 states in the process of passing mandatory porn... More »

The Skinny on the New DMCA Requirement

The requirements under the Digital Millennium Copyright Act (DMCA) have changed, but the sky is not falling. We have, even as a law firm, been hit with an unbelievable amount of spam sent out by law firms... More »

2017 Outlook: Legal Matters Around the Corner

The adult entertainment business’ legal community recently weighed in the top issues that companies and the industry, as a whole, should be concerned about. XBIZ asked industry attorneys to discuss... More »
XBIZ NEWSLETTERS
Stay informed of the latest industry developments. Get XBIZ newsletters delivered to your inbox. Subscribe today!
Enter email address:

* To manage existing subscriptions click here.






POPULAR PRODUCTS & SERVICES
Submit your press release to
multiple news outlets with 1 click.
Subscribe to RSS news feeds or
add free content to your website.
Access XBIZ news and articles
with your mobile device.
Subscribe to XBIZ World magazine, the industry's leading e-commerce trade publication, delivering in-depth coverage of the online, mobile and ancillary digital markets.

UPCOMING EVENTS

XBIZ.net Paysite Meetup

May 04 - May 04
Prague, Czech Republic

Eurowebtainment 2017

May 17 - May 20
Majorca, Spain

XBIZ Retreat

May 30 - Jun 03
Miami, Florida

XBIZ Miami 2017

May 30 - Jun 02
Miami, Florida
Everyday thousands of business professionals browse XBIZ's industry directory for quality products and services. Not listed yet? Your company could be losing potential new business. Submit your company today!
Use XBIZ RSS feeds to stay informed of the latest industry developments or as a content syndication tool for your website!