Web Crawlers Get Smarter, Harder to Stop

WASHINGTON — Web crawlers are getting more human, which, according to engineers presenting at this year’s ShmooCon hacker conference, makes them progressively more difficult to trace, track and, if need be, fend off.

Crawlers traditionally have been fairly easy to spot because they don’t behave like human surfers. Google, for example, sends its spidering bots out into the world with a blast that encapsulates millions of sites an hour, culling information and spitting it back into Google’s mainframe in a highly automated fashion. Software used by spammers and other hackers is similarly designed.

But engineer Billy Hoffman at SPI Dynamics is just one of several researchers who has designed advanced crawlers that mimic human behavior: crawlers that click slowly through sites and pause on certain pages longer than on others as if they were human, and are thus practically undetectable.

“Basically this nullifies any traditional form of forensics,” Hoffman said during a presentation at the conference this week.

Hoffman said that every crawler, all of which are automatically launched from several different Internet addresses, has its own browsing style.

“You can assign the different threads a personality,” he said. “This crawler, you're the slow reader, you read the entire page. Each individual crawler has its own browser habits.”

Despite their human-like qualities, Hoffman said the crawlers download everything they can find on a page, from images to JavaScript and even Flash components, providing far more detailed website captures than traditional spidering software.

This week marks the second annual ShmooCon conference, which hosts three separate security-focused tracks for professionals interested in building, bettering and breaking the latest security advancements. More than 500 hacking experts, as well as several dozen federal law enforcement officials, attended this year’s event.

Copyright © 2025 Adnet Media. All Rights Reserved. XBIZ is a trademark of Adnet Media.
Reproduction in whole or in part in any form or medium without express written permission is prohibited.

More News

Sex Work CEO Launches 'Teams Plan' for AI Assistant

Sex Work CEO has introduced the new Teams Plan for its AI-powered, NSFW text generator, GPTease.

2025 XBIZ Amsterdam Website Launches With Call for Speakers

XBIZ is pleased to announce that the website for its annual European conference, XBIZ Amsterdam, is now live.

NC Governor Vetoes Bill Targeting Adult Industry, Override Possible

North Carolina Governor Josh Stein today vetoed a bill imposing new regulations that adult industry observers have warned could push adult websites and platforms to ban most adult creators and content.

25,000 Sign Petition to Legalize Pornography in Ukraine

An OnlyFans model’s petition to decriminalize pornography in Ukraine has amassed the 25,000 signatures required for official consideration by President Volodymyr Zelensky.

WannaCollab Joins Pineapple Support as Supporter-Level Sponsor

WannaCollab has joined the ranks of over 70 adult businesses and organizations committing funds and resources to Pineapple Support.

FSC Unpacks SCOTUS Age Verification Ruling in Webinar

The Free Speech Coalition conducted a public webinar Tuesday to help adult industry stakeholders understand the Supreme Court’s recent decision in FSC v. Paxton, and its potential implications.

UK Lawmaker Calls for Appointment of 'Porn Minister'

Baroness Gabrielle Bertin, the Conservative member of Parliament who recently convened a new anti-pornography task force, is calling for the appointment of a “minister for porn,” according to British news outlet The Guardian.

FSC Toasts Jeffrey Douglas for 30 Years of Service

n the very same evening when the adult industry was hit hard by the Supreme Court ruling supporting Texas’ controversial age verification law, HB 1181, members of the Free Speech Coalition board, staff and supporters gathered to celebrate Jeffrey Douglas’ 30 years as board chair — a fitting reflection of his reputation as an eternal optimist.

TTS Opens UK Testing Location

Talent Testing Service (TTS) has opened a new U.K. location in Ware, Hertfordshire.

FSC: Age-Verification Laws Go Into Effect in South Dakota, Georgia, Wyoming on July 1

The Free Speech Coalition (FSC) has published a statement regarding new age verification laws set to go into effect tomorrow in South Dakota, Georgia, and Wyoming.

Show More