what are spiders?

Home Archive Serious Business what are spiders?
ricola's avatar

ricola

Senior Member

855 posts
Jan 24, 2011 9:03 PM
As a techno-dolt, I noticed on the user list at the bottom of the page, that at the end of the list are bunch of "spiders" (google,yahoo,msn etc). What exactly are they and what do they do?
Jan 24, 2011 9:03pm
wes_mantooth's avatar

wes_mantooth

Tomfoolery & shenanigans

17,977 posts
Jan 24, 2011 9:05 PM
things that scare the shit out of Heretic
Jan 24, 2011 9:05pm
sleeper's avatar

sleeper

Legend

27,879 posts
Jan 25, 2011 12:54 AM
They collect information about a website so when you google something, if the website is deemed relevant, it will pop up in the search results.

That's probably a terrible explanation, but I have a lot of money, so I don't worry about being technical.
Jan 25, 2011 12:54am
I

I Wear Pants

Senior Member

16,223 posts
Jan 25, 2011 12:58 AM
They are what collect a lot of the information that you search on Google, Bing, Yahoo, etc. They collect information from sites that is used to make search results more relevant.

From wikipedia:
This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.
http://en.wikipedia.org/wiki/Web_crawler
Jan 25, 2011 12:58am
ytownfootball's avatar

ytownfootball

Bold faced liar...

6,978 posts
Jan 25, 2011 12:59 AM
you can probably go ahead and take that to the bank too, in case you were curious.
Jan 25, 2011 12:59am