Abstract
This paper outlines the design and implementation of Insun Crawler: a C/S Structured distributed multi-thread Crawler Using ACE and PVFS, which is an improvement of the previous Inar Crawler. The Insun Crawler is structured carefully to make it flexible and extensible. Compared to no-distributed crawlers there are more challenges as how to cooperate the distributed crawling threads, how to deal with exchanging of URLs between threads and so on. This paper describes the architecture of Insun Crawler and present our solutions for a distribute Web Crawler without exchanging of URLs. The privilege of ACE and PVFS alleviate the complex of network programming and the implementation of a distributed file system.
| Original language | English |
|---|---|
| Pages (from-to) | 883-890 |
| Number of pages | 8 |
| Journal | Journal of Computational Information Systems |
| Volume | 4 |
| Issue number | 3 |
| State | Published - Jun 2008 |
| Externally published | Yes |
Keywords
- ACE
- Crawler
- Distributed
- Multi-thread
- PVFS
Fingerprint
Dive into the research topics of 'Design and implementation of a C/S structured distributed multi-thread crawler using ACE and PVFS'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver