Crawlers traditionally have been fairly easy to spot because they don’t behave like human surfers. Google, for example, sends its spidering bots out into the world with a blast that encapsulates millions of sites an hour, culling information and spitting it back into Google’s mainframe in a highly automated fashion. Software used by spammers and other hackers is similarly designed.
But engineer Billy Hoffman at SPI Dynamics is just one of several researchers who has designed advanced crawlers that mimic human behavior: crawlers that click slowly through sites and pause on certain pages longer than on others as if they were human, and are thus practically undetectable.
“Basically this nullifies any traditional form of forensics,” Hoffman said during a presentation at the conference this week.
Hoffman said that every crawler, all of which are automatically launched from several different Internet addresses, has its own browsing style.
“You can assign the different threads a personality,” he said. “This crawler, you're the slow reader, you read the entire page. Each individual crawler has its own browser habits.”
Despite their human-like qualities, Hoffman said the crawlers download everything they can find on a page, from images to JavaScript and even Flash components, providing far more detailed website captures than traditional spidering software.
This week marks the second annual ShmooCon conference, which hosts three separate security-focused tracks for professionals interested in building, bettering and breaking the latest security advancements. More than 500 hacking experts, as well as several dozen federal law enforcement officials, attended this year’s event.
