|
Abstract : |
We report our experience in implementing UbiCrawler, a scalable distributed web crawler, using the Java programming language. The main features of Ubi-Crawler are platform independence, fault tolerance, a very effective assignment function for partitioning the domain to crawl, and more in general the complete decentralization of every task. The necessity of handling very large sets of data has highlighted some limitation of the Java APIs, which prompted the authors to partially reimplement them. 1, |
