Exploring large-scale small file storage for search engines

Weizhe Zhang; Gangzhao Lu; Hui He; Qizhen Zhang; Chuanliang Yu

doi:10.1007/s11227-015-1394-z

Exploring large-scale small file storage for search engines

Weizhe Zhang^*
, Gangzhao Lu
, Hui He
, Qizhen Zhang
, Chuanliang Yu

^*Corresponding author for this work

School of Computer Science and Technology, Harbin Institute of Technology

Research output: Contribution to journal › Article › peer-review

8 Scopus citations

Abstract

Large-scale small file storage for original pages degrades performance of search engines. In this paper, we first analyze the disadvantages of the existing EXT3 file system in accessing small files. Then, the rate and speed of compression algorithms are verified to choose a proper storage compression algorithm. Meanwhile, we design an original page oriented file organization structure and a read–write query tree to store the large-scale small files which need no modification. The accessing response time and disk space waste are remarkably decreased when search engines use these techniques to store original-page small files.

Original language	English
Pages (from-to)	2911-2923
Number of pages	13
Journal	Journal of Supercomputing
Volume	72
Issue number	8
DOIs	https://doi.org/10.1007/s11227-015-1394-z
State	Published - 1 Aug 2016
Externally published	Yes

Keywords

Search engine
Small file storage
Storage space
Storage time

Access to Document

10.1007/s11227-015-1394-z

Cite this

@article{6021f74677ca4da8b4703c042b8b5ff4,

title = "Exploring large-scale small file storage for search engines",

abstract = "Large-scale small file storage for original pages degrades performance of search engines. In this paper, we first analyze the disadvantages of the existing EXT3 file system in accessing small files. Then, the rate and speed of compression algorithms are verified to choose a proper storage compression algorithm. Meanwhile, we design an original page oriented file organization structure and a read–write query tree to store the large-scale small files which need no modification. The accessing response time and disk space waste are remarkably decreased when search engines use these techniques to store original-page small files.",

keywords = "Search engine, Small file storage, Storage space, Storage time",

author = "Weizhe Zhang and Gangzhao Lu and Hui He and Qizhen Zhang and Chuanliang Yu",

note = "Publisher Copyright: {\textcopyright} 2015, Springer Science+Business Media New York.",

year = "2016",

month = aug,

day = "1",

doi = "10.1007/s11227-015-1394-z",

language = "英语",

volume = "72",

pages = "2911--2923",

journal = "Journal of Supercomputing",

issn = "0920-8542",

publisher = "Springer Netherlands",

number = "8",

}

TY - JOUR

T1 - Exploring large-scale small file storage for search engines

AU - Zhang, Weizhe

AU - Lu, Gangzhao

AU - He, Hui

AU - Zhang, Qizhen

AU - Yu, Chuanliang

PY - 2016/8/1

Y1 - 2016/8/1

N2 - Large-scale small file storage for original pages degrades performance of search engines. In this paper, we first analyze the disadvantages of the existing EXT3 file system in accessing small files. Then, the rate and speed of compression algorithms are verified to choose a proper storage compression algorithm. Meanwhile, we design an original page oriented file organization structure and a read–write query tree to store the large-scale small files which need no modification. The accessing response time and disk space waste are remarkably decreased when search engines use these techniques to store original-page small files.

AB - Large-scale small file storage for original pages degrades performance of search engines. In this paper, we first analyze the disadvantages of the existing EXT3 file system in accessing small files. Then, the rate and speed of compression algorithms are verified to choose a proper storage compression algorithm. Meanwhile, we design an original page oriented file organization structure and a read–write query tree to store the large-scale small files which need no modification. The accessing response time and disk space waste are remarkably decreased when search engines use these techniques to store original-page small files.

KW - Search engine

KW - Small file storage

KW - Storage space

KW - Storage time

UR - https://www.scopus.com/pages/publications/84923365974

U2 - 10.1007/s11227-015-1394-z

DO - 10.1007/s11227-015-1394-z

M3 - 文章

AN - SCOPUS:84923365974

SN - 0920-8542

VL - 72

SP - 2911

EP - 2923

JO - Journal of Supercomputing

JF - Journal of Supercomputing

IS - 8

ER -

Exploring large-scale small file storage for search engines

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this