A Conversation for A Guide to Using Search Engines
The Grub project
Peet (the Pedantic Punctuation Policeman, Muse of Lateral Programming Ideas, Eggcups-Spurtle-and-Spoonswinner, BBC Cheese Namer & Zaphodista) Started conversation May 20, 2003
An exciting development is "Grub's Distributed Web Crawling Project". Funded by LookSmart, it is a distributed application in similar vein to the "SETI@Home" client, but instead of using your spare processor time it utilises your spare bandwidth to crawl URLs in batches of 500, notifying the central server of which URLs have changed since the last time they were crawled, and forwarding a snapshot of the HTML content of the page. The idea is that, when fully deployed, it will be able to crawl and refresh ***every page on the web, every day***! The upshot of this will be an accurate search engine with no "expired" links. 
Check out http://www.grub.org for details; if you have a 24/7 flat fee broadband connection, you might consider running it overnight. It makes a good match with SETI@Home for that sort of use, as while it uses a lot of bandwidth but very little CPU, SETI@Home (http://setiathome.ssl.berkeley.edu/) uses a lot of CPU but virtually no bandwidth. 
The Grub project
Martin Belam Posted May 23, 2003
we had a bit of a chat about this over on collective [F99145?thread=268091]
the trouble with Grub is that the spider that actually collects the pages is pretty badly behaved - doesn't obey robots.txt properly (which is a way webmasters can give instructions to search engines on what they can and can't index) - and doesn't seem to have a proper way of avoiding hammering sites by repeatedly requesting pages from them over a short period of time.
The Grub project
Peet (the Pedantic Punctuation Policeman, Muse of Lateral Programming Ideas, Eggcups-Spurtle-and-Spoonswinner, BBC Cheese Namer & Zaphodista) Posted May 24, 2003
True. It's in beta at the moment... There seems to be a lot of coding activity going on, though, so the most anti-social behaviour is being weeded out first. If nobody signs up, they'll never have any incentive to improve it... 
Key: Complain about this post
The Grub project
- 1: Peet (the Pedantic Punctuation Policeman, Muse of Lateral Programming Ideas, Eggcups-Spurtle-and-Spoonswinner, BBC Cheese Namer & Zaphodista) (May 20, 2003)
- 2: Martin Belam (May 23, 2003)
- 3: Peet (the Pedantic Punctuation Policeman, Muse of Lateral Programming Ideas, Eggcups-Spurtle-and-Spoonswinner, BBC Cheese Namer & Zaphodista) (May 24, 2003)
More Conversations for A Guide to Using Search Engines
Write an Entry
"The Hitchhiker's Guide to the Galaxy is a wholly remarkable book. It has been compiled and recompiled many times and under many different editorships. It contains contributions from countless numbers of travellers and researchers."