Research on Agent collaboration and Web partition in WAN-based distributed Web crawlers

This paper focuses on agent collaboration and Web partition, the two core issues in WAN-based distributed crawling. First, a new consultant-service-based agent collaboration method and the corresponding system model are proposed. The new method has a lower communication overhead than the central-coordinator-based crawling systems and exploits location proximity better than the ones based on Distributed Hash Table (DHT). Second, the detailed definitions of Web partition are presented. The selection of Web partition unit and the Web partition strategy are discussed. The experiment under the real Internet environment shows that WAN-based distributed Web crawling systems have better performance than the LAN-based ones. The experiment also shows that the impact of Internet service providers interconnectivity on the system performance is greater than that of the geographical locality.