Design and implementation of a distributed high-performance web crawler

Ling Zhang; Yun Ming Ye; Hui Song; Shui Yu; Fan Yuan Ma

Design and implementation of a distributed high-performance web crawler

Ling Zhang
, Yun Ming Ye
, Hui Song
, Shui Yu
, Fan Yuan Ma^*

^*Corresponding author for this work

Shanghai Jiao Tong University

Research output: Contribution to journal › Article › peer-review

1 Scopus citations

Abstract

Web crawler is the core component of WWW search engine and information retrieval systems. This paper discussed the architecture of a distributed Web crawler and the design ideas about the Web crawler data structure, system modules and related algorithms. The key problems encountered in the design and implementations were also commented, and the solutions to those problems were presented.

Original language	English
Pages (from-to)	59-61
Number of pages	3
Journal	Shanghai Jiaotong Daxue Xuebao/Journal of Shanghai Jiaotong University
Volume	38
Issue number	1
State	Published - Jan 2004
Externally published	Yes

Keywords

Distributed system
Java
Search engine
Web crawler

Cite this

@article{5085d664d5d847ea89d74f96c305913e,

title = "Design and implementation of a distributed high-performance web crawler",

abstract = "Web crawler is the core component of WWW search engine and information retrieval systems. This paper discussed the architecture of a distributed Web crawler and the design ideas about the Web crawler data structure, system modules and related algorithms. The key problems encountered in the design and implementations were also commented, and the solutions to those problems were presented.",

keywords = "Distributed system, Java, Search engine, Web crawler",

author = "Ling Zhang and Ye, \{Yun Ming\} and Hui Song and Shui Yu and Ma, \{Fan Yuan\}",

year = "2004",

month = jan,

language = "英语",

volume = "38",

pages = "59--61",

journal = "Shanghai Jiaotong Daxue Xuebao/Journal of Shanghai Jiaotong University",

issn = "1006-2467",

publisher = "Shanghai Jiaotong University",

number = "1",

}

TY - JOUR

T1 - Design and implementation of a distributed high-performance web crawler

AU - Zhang, Ling

AU - Ye, Yun Ming

AU - Song, Hui

AU - Yu, Shui

AU - Ma, Fan Yuan

PY - 2004/1

Y1 - 2004/1

N2 - Web crawler is the core component of WWW search engine and information retrieval systems. This paper discussed the architecture of a distributed Web crawler and the design ideas about the Web crawler data structure, system modules and related algorithms. The key problems encountered in the design and implementations were also commented, and the solutions to those problems were presented.

AB - Web crawler is the core component of WWW search engine and information retrieval systems. This paper discussed the architecture of a distributed Web crawler and the design ideas about the Web crawler data structure, system modules and related algorithms. The key problems encountered in the design and implementations were also commented, and the solutions to those problems were presented.

KW - Distributed system

KW - Java

KW - Search engine

KW - Web crawler

UR - https://www.scopus.com/pages/publications/2942699999

M3 - 文章

AN - SCOPUS:2942699999

SN - 1006-2467

VL - 38

SP - 59

EP - 61

JO - Shanghai Jiaotong Daxue Xuebao/Journal of Shanghai Jiaotong University

JF - Shanghai Jiaotong Daxue Xuebao/Journal of Shanghai Jiaotong University

IS - 1

ER -

Design and implementation of a distributed high-performance web crawler

Abstract

Keywords

Other files and links

Fingerprint

Cite this