xbiz world
xbiz premiere
educational

Robot and Spider Control

Editor’s note: Search engine spiders are typically the only kind of spiders that Webmasters want to see hanging around. These robots quietly crawl their way around the World Wide Web seeking out every page they can find, and reporting their contents back to their search engine masters. This is usually a welcome operation as it often leads to more ‘free’ traffic – but occasionally robots find their way into places we wish they wouldn’t, exposing sensitive information for the world to see… Here’s how to help prevent this from happening: ~ Stephen

Before submitting your site to the search engines, you will want to consider what pages and links you want the search engine "robot" (the program that indexes your site) to "spider" (follow), and what pages you don’t want it to follow – since you may have pages with sensitive information, a ‘scrap directory’ full of "work in progress," or a protected "members area" that you would not like listed.

This goal can easily be achieved in two ways. The first way is with a robots.txt file placed in the root directory of your Website, but you must have full domain privileges in order for this to work. While this article is not meant to deal with the intricacies of the robots.txt file, a quick word of warning is in order: never leave this file empty, as it will indicate to some robots that you do not want any part of your site indexed.

The other way to stop most ‘bots’ from searching or indexing your page is to use META exclusion tags. This is often the only way that Webmasters on virtual or free hosts without full server access can hope to control a spider’s wanderings and reports on a page-by-page basis. The syntax is simple:

<META name="ROBOTS" content="ALL,NONE,INDEX,FOLLOW,NOINDEX,NOFOLLOW">

The default value for the robots tag is "ALL" which allows the robot to index the page, then spider all links, indexing the linked pages too. "NONE" performs the opposite, disallowing the robot from either indexing the page, or spidering the links on it, in essence ignoring the page altogether.

"INDEX" indicates that robots should include this page in their search engines, while "FOLLOW" means that robots should follow (spider) the links on this page. Conversely, a value of "NOINDEX" allows links from the page to be spidered, even though the page itself is not indexed, while a value of "NOFOLLOW" allows the page to be indexed, but no links from the page are to be spidered.

Some Sample Snippets
Here’s some example robot controlling META tags, which would be put in between your document’s <HEAD> and </HEAD> tags:

<META name="ROBOTS" content="NOINDEX">
- This will prevent the bot from indexing that page.

<META name="ROBOTS" content="NOFOLLOW">
- This allows the page to be indexed, but any hyperlinks in that page will not be spidered.

<META name="ROBOTS" content="NOINDEX,NOFOLLOW">
- Is a combination of the two, where the page will not be indexed, and other links will not be followed. This tag may also prevent some mirroring software from downloading the page.

While there are many other META tags that can be used to improve your rankings, controlling what’s ranked is the first step, after which it’s wiser to invest your time in optimizing your description and keywords tags in order to boost your search engine rankings, which is the subject of my next article…

Copyright © 2026 Adnet Media. All Rights Reserved. XBIZ is a trademark of Adnet Media.
Reproduction in whole or in part in any form or medium without express written permission is prohibited.

More Articles

opinion

How Adult Businesses Can Navigate Global Compliance Demands

The internet has made the world feel small. Case in point: Adult websites based in the U.S. are now getting letters from regulators demanding compliance with foreign laws, even if they don’t operate in those countries. Meanwhile, some U.S. website operators dealing with the patchwork of state-level age verification laws have considered incorporating offshore in the hopes of avoiding these new obligations — but even operators with no physical presence in the U.S. have been sued or threatened with claims for not following state AV laws.

Larry Walters ·
opinion

Top Tips for Bulletproof Creator Management Contracts

The creator management business is booming. Every week, it seems, a new agency emerges, promising to turn creators into stars, automate their fan interactions or triple their revenue through “secret” social strategies. The reality? Many of these agencies are operating with contracts that wouldn’t survive a single serious dispute — if they even have contracts at all.

Corey D. Silverstein ·
opinion

Building Sustainable Revenue Without Opt-Out Cross-Sales

Over the past year, we’ve seen growing pushback from acquirers on merchants using opt-out cross-sales — also known as negative option offers. This has been especially noticeable in the U.S. In fact, one of our acquirers now declines new merchants during onboarding if an opt-out flow is detected. Existing merchants submitting new URLs with opt-out cross-sales are being asked to remove them.

Cathy Beardsley ·
trends

How to Handle Payment Disputes Without Sacrificing Trust

You can run the best-managed and most compliant website out there, but that still doesn’t completely shield you from the risks tied to payment disputes. Buyer’s remorse, an unclear billing description or even a simple misunderstanding can lead a customer to dispute a transaction. Accumulate enough disputes, and both your reputation and revenue could be at risk.

Jonathan Corona ·
trends

WIA Profile: Taylor Moore

With a 70-person team and a growing slate of tools for content creators, the Teasy Agency has developed a reputation for putting talent first. That commitment owes a lot to co-founder Taylor Moore’s own experiences as a cam model.

Jackie Backman ·
profile

WIA Profile: Cathy Turns Creator Platform Experience Into a Model-First Playbook

As both a model and industry executive, Cathy lives in two worlds at once. “Since I do both things, I can act as the liaison between the model community and the rest of the SextPanther team,” she tells XBIZ.

Jackie Backman ·
opinion

From Compliance to Confidence: The Future of Safety in Adult Platforms

In numerous countries and U.S. states, laws now require platforms to prevent minors from accessing age-inappropriate material. But the need for safeguarding doesn’t end with age verification. Today’s online landscape also places adult companies at uniquely high risk for inadvertently facilitating exploitation, abuse or reputational harm, or of being accused of doing so.

Andy Lulham ·
opinion

What Adult Businesses Need to Know About Florida's Age Verification Law

The rise and proliferation of age verification laws has changed the landscape for the online adult industry. A recent and compelling example is the state of Florida, where Attorney General James Uthmeier has filed multiple complaints against major platforms as well as affiliates accused of violating the state’s AV law.

Corey D. Silverstein ·
opinion

Maintaining Brand Trust in the Face of Negative Press

Over the last year, several of our merchants have found themselves caught up in litigation over compliance with state age verification laws. Recently, Segpay itself was pulled into the spotlight, facing scrutiny over Florida’s AV statute, HB 3. These stories inevitably get picked up by both industry and mainstream news outlets.

Cathy Beardsley ·
opinion

How to Switch Payment Processors Without Disrupting Business

For many merchants, the idea of switching payment processors can feel pretty overwhelming. That’s understandable. After all, downtime can stall sales, recurring subscriptions can suddenly fail, or compliance gaps can put accounts at risk. Operating in a high-risk sector like the adult industry can further amplify the stress of transition.

Jonathan Corona ·
Show More