Many applications largely search-engines, crawl sites everyday so that you can find up-to-date information.

A lot of the net spiders save yourself a of the visited page so they can simply index it later […]