Distributed high-performance web crawler based on peer-to-peer network

Liu Fei; Ma Fan-Yuan; Ye Yun-Ming; Li Ming-Lu; Yu Jia-Di

doi:10.1007/978-3-540-30501-9_13

Distributed high-performance web crawler based on peer-to-peer network

Liu Fei^*
, Ma Fan-Yuan
, Ye Yun-Ming
, Li Ming-Lu
, Yu Jia-Di

^*Corresponding author for this work

Shanghai Jiao Tong University

Research output: Contribution to journal › Conference article › peer-review

1 Scopus citations

Abstract

Distributing the crawling activity among multiple machines can distribute processing to reduce the analysis of web page. This paper presents the design of a distributed web crawler based on Peer-to-Peer network. The distributed crawler harnesses the excess bandwidth and computing resources of nodes in system to crawl the web. Each crawler is deployed in a computing node of P2P to analyze web page and generate indices. Control node is another node to being in charge of distributing URLs to balance the load of the crawler. Control nodes are organized as P2P network. The crawler nodes managed by the same control node is a group. According to the ID of crawler and average load of the group, crawler can decide whether transmits the URL to control node or hold itself. We present an implementation of the distributed crawler based on Igloo and simulate the environment to evaluate the balancing load on the crawlers and crawl speed.

Original language	English
Pages (from-to)	50-53
Number of pages	4
Journal	Lecture Notes in Computer Science
Volume	3320
DOIs	https://doi.org/10.1007/978-3-540-30501-9_13
State	Published - 2004
Externally published	Yes
Event	5th International Conference, PDCAT 2004 - , Singapore Duration: 8 Dec 2004 → 10 Dec 2004

Access to Document

10.1007/978-3-540-30501-9_13

Cite this

@article{0355832567dd431d818454808e2ef231,

title = "Distributed high-performance web crawler based on peer-to-peer network",

abstract = "Distributing the crawling activity among multiple machines can distribute processing to reduce the analysis of web page. This paper presents the design of a distributed web crawler based on Peer-to-Peer network. The distributed crawler harnesses the excess bandwidth and computing resources of nodes in system to crawl the web. Each crawler is deployed in a computing node of P2P to analyze web page and generate indices. Control node is another node to being in charge of distributing URLs to balance the load of the crawler. Control nodes are organized as P2P network. The crawler nodes managed by the same control node is a group. According to the ID of crawler and average load of the group, crawler can decide whether transmits the URL to control node or hold itself. We present an implementation of the distributed crawler based on Igloo and simulate the environment to evaluate the balancing load on the crawlers and crawl speed.",

author = "Liu Fei and Ma Fan-Yuan and Ye Yun-Ming and Li Ming-Lu and Yu Jia-Di",

year = "2004",

doi = "10.1007/978-3-540-30501-9\_13",

language = "英语",

volume = "3320",

pages = "50--53",

journal = "Lecture Notes in Computer Science",

issn = "0302-9743",

publisher = "Springer Science and Business Media Deutschland GmbH",

note = "5th International Conference, PDCAT 2004 ; Conference date: 08-12-2004 Through 10-12-2004",

}

TY - JOUR

T1 - Distributed high-performance web crawler based on peer-to-peer network

AU - Fei, Liu

AU - Fan-Yuan, Ma

AU - Yun-Ming, Ye

AU - Ming-Lu, Li

AU - Jia-Di, Yu

PY - 2004

Y1 - 2004

N2 - Distributing the crawling activity among multiple machines can distribute processing to reduce the analysis of web page. This paper presents the design of a distributed web crawler based on Peer-to-Peer network. The distributed crawler harnesses the excess bandwidth and computing resources of nodes in system to crawl the web. Each crawler is deployed in a computing node of P2P to analyze web page and generate indices. Control node is another node to being in charge of distributing URLs to balance the load of the crawler. Control nodes are organized as P2P network. The crawler nodes managed by the same control node is a group. According to the ID of crawler and average load of the group, crawler can decide whether transmits the URL to control node or hold itself. We present an implementation of the distributed crawler based on Igloo and simulate the environment to evaluate the balancing load on the crawlers and crawl speed.

AB - Distributing the crawling activity among multiple machines can distribute processing to reduce the analysis of web page. This paper presents the design of a distributed web crawler based on Peer-to-Peer network. The distributed crawler harnesses the excess bandwidth and computing resources of nodes in system to crawl the web. Each crawler is deployed in a computing node of P2P to analyze web page and generate indices. Control node is another node to being in charge of distributing URLs to balance the load of the crawler. Control nodes are organized as P2P network. The crawler nodes managed by the same control node is a group. According to the ID of crawler and average load of the group, crawler can decide whether transmits the URL to control node or hold itself. We present an implementation of the distributed crawler based on Igloo and simulate the environment to evaluate the balancing load on the crawlers and crawl speed.

UR - https://www.scopus.com/pages/publications/13944270304

U2 - 10.1007/978-3-540-30501-9_13

DO - 10.1007/978-3-540-30501-9_13

M3 - 会议文章

AN - SCOPUS:13944270304

SN - 0302-9743

VL - 3320

SP - 50

EP - 53

JO - Lecture Notes in Computer Science

JF - Lecture Notes in Computer Science

T2 - 5th International Conference, PDCAT 2004

Y2 - 8 December 2004 through 10 December 2004

ER -

Distributed high-performance web crawler based on peer-to-peer network

Abstract

Access to Document

Other files and links

Fingerprint

Cite this