Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0

Haoran Zhang; Jianzhong Li; Hongzhi Wang

doi:10.1007/978-981-13-2203-7_31

Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0

Haoran Zhang
, Jianzhong Li
, Hongzhi Wang^*

^*Corresponding author for this work

Faculty of Computing

Harbin Institute of Technology

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

This paper is mainly to predict the running time of data-intensive MapReduce program under Hadoop2.0 environment. Although MapReduce programs are diverse, they can be divided into data-intensive and computationally intensive, depending on the time complexity and the nature of the program. The prediction of computationally intensive programs has always been difficult, and Hadoop has exhibited certain database attributes that are basically data-intensive. Moreover, the relationship between data-intensive programs and the amount of data is more closely related and shows certain statistical characteristics. So the method of statistical learning is applied to predict the execution time. This paper first generates training data and test data according to requirements, and then selects the appropriate features through the analysis of the logs. The prediction was first performed using the KCCA algorithm. However, the deficiencies were found. Then based on the characteristics of the kernel function, a prediction method based on deep learning was proposed, and the result was significant.

Original language	English
Title of host publication	Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings
Editors	Yong Gan, Xianhua Song, Qinglei Zhou, Weipeng Jing, Yan Wang, Zeguang Lu
Publisher	Springer Verlag
Pages	403-414
Number of pages	12
ISBN (Print)	9789811322020
DOIs	https://doi.org/10.1007/978-981-13-2203-7_31
State	Published - 2018
Event	4th International Conference of Pioneer Computer Scientists, Engineers and Educators, ICPCSEE 2018 - Zhengzhou, China Duration: 21 Sep 2018 → 23 Sep 2018

Publication series

Name	Communications in Computer and Information Science
Volume	901
ISSN (Print)	1865-0929

Conference

Conference	4th International Conference of Pioneer Computer Scientists, Engineers and Educators, ICPCSEE 2018
Country/Territory	China
City	Zhengzhou
Period	21/09/18 → 23/09/18

Keywords

Deep learning
Feature extraction
KCCA
Training data

Access to Document

10.1007/978-981-13-2203-7_31

Cite this

Zhang, H., Li, J., & Wang, H. (2018). Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0. In Y. Gan, X. Song, Q. Zhou, W. Jing, Y. Wang, & Z. Lu (Eds.), Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings (pp. 403-414). (Communications in Computer and Information Science; Vol. 901). Springer Verlag. https://doi.org/10.1007/978-981-13-2203-7_31

Zhang, Haoran ; Li, Jianzhong ; Wang, Hongzhi. / Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0. Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings. editor / Yong Gan ; Xianhua Song ; Qinglei Zhou ; Weipeng Jing ; Yan Wang ; Zeguang Lu. Springer Verlag, 2018. pp. 403-414 (Communications in Computer and Information Science).

@inproceedings{f006e9a751f44ff5bd07c96551416fc4,

title = "Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0",

abstract = "This paper is mainly to predict the running time of data-intensive MapReduce program under Hadoop2.0 environment. Although MapReduce programs are diverse, they can be divided into data-intensive and computationally intensive, depending on the time complexity and the nature of the program. The prediction of computationally intensive programs has always been difficult, and Hadoop has exhibited certain database attributes that are basically data-intensive. Moreover, the relationship between data-intensive programs and the amount of data is more closely related and shows certain statistical characteristics. So the method of statistical learning is applied to predict the execution time. This paper first generates training data and test data according to requirements, and then selects the appropriate features through the analysis of the logs. The prediction was first performed using the KCCA algorithm. However, the deficiencies were found. Then based on the characteristics of the kernel function, a prediction method based on deep learning was proposed, and the result was significant.",

keywords = "Deep learning, Feature extraction, KCCA, Training data",

author = "Haoran Zhang and Jianzhong Li and Hongzhi Wang",

note = "Publisher Copyright: {\textcopyright} Springer Nature Singapore Pte Ltd. 2018.; 4th International Conference of Pioneer Computer Scientists, Engineers and Educators, ICPCSEE 2018 ; Conference date: 21-09-2018 Through 23-09-2018",

year = "2018",

doi = "10.1007/978-981-13-2203-7\_31",

language = "英语",

isbn = "9789811322020",

series = "Communications in Computer and Information Science",

publisher = "Springer Verlag",

pages = "403--414",

editor = "Yong Gan and Xianhua Song and Qinglei Zhou and Weipeng Jing and Yan Wang and Zeguang Lu",

booktitle = "Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings",

address = "德国",

}

Zhang, H, Li, J & Wang, H 2018, Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0. in Y Gan, X Song, Q Zhou, W Jing, Y Wang & Z Lu (eds), Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings. Communications in Computer and Information Science, vol. 901, Springer Verlag, pp. 403-414, 4th International Conference of Pioneer Computer Scientists, Engineers and Educators, ICPCSEE 2018, Zhengzhou, China, 21/09/18. https://doi.org/10.1007/978-981-13-2203-7_31

Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0. / Zhang, Haoran; Li, Jianzhong; Wang, Hongzhi.
Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings. ed. / Yong Gan; Xianhua Song; Qinglei Zhou; Weipeng Jing; Yan Wang; Zeguang Lu. Springer Verlag, 2018. p. 403-414 (Communications in Computer and Information Science; Vol. 901).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0

AU - Zhang, Haoran

AU - Li, Jianzhong

AU - Wang, Hongzhi

N1 - Publisher Copyright: © Springer Nature Singapore Pte Ltd. 2018.

PY - 2018

Y1 - 2018

N2 - This paper is mainly to predict the running time of data-intensive MapReduce program under Hadoop2.0 environment. Although MapReduce programs are diverse, they can be divided into data-intensive and computationally intensive, depending on the time complexity and the nature of the program. The prediction of computationally intensive programs has always been difficult, and Hadoop has exhibited certain database attributes that are basically data-intensive. Moreover, the relationship between data-intensive programs and the amount of data is more closely related and shows certain statistical characteristics. So the method of statistical learning is applied to predict the execution time. This paper first generates training data and test data according to requirements, and then selects the appropriate features through the analysis of the logs. The prediction was first performed using the KCCA algorithm. However, the deficiencies were found. Then based on the characteristics of the kernel function, a prediction method based on deep learning was proposed, and the result was significant.

AB - This paper is mainly to predict the running time of data-intensive MapReduce program under Hadoop2.0 environment. Although MapReduce programs are diverse, they can be divided into data-intensive and computationally intensive, depending on the time complexity and the nature of the program. The prediction of computationally intensive programs has always been difficult, and Hadoop has exhibited certain database attributes that are basically data-intensive. Moreover, the relationship between data-intensive programs and the amount of data is more closely related and shows certain statistical characteristics. So the method of statistical learning is applied to predict the execution time. This paper first generates training data and test data according to requirements, and then selects the appropriate features through the analysis of the logs. The prediction was first performed using the KCCA algorithm. However, the deficiencies were found. Then based on the characteristics of the kernel function, a prediction method based on deep learning was proposed, and the result was significant.

KW - Deep learning

KW - Feature extraction

KW - KCCA

KW - Training data

UR - https://www.scopus.com/pages/publications/85053891751

U2 - 10.1007/978-981-13-2203-7_31

DO - 10.1007/978-981-13-2203-7_31

M3 - 会议稿件

AN - SCOPUS:85053891751

SN - 9789811322020

T3 - Communications in Computer and Information Science

SP - 403

EP - 414

BT - Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings

A2 - Gan, Yong

A2 - Song, Xianhua

A2 - Zhou, Qinglei

A2 - Jing, Weipeng

A2 - Wang, Yan

A2 - Lu, Zeguang

PB - Springer Verlag

T2 - 4th International Conference of Pioneer Computer Scientists, Engineers and Educators, ICPCSEE 2018

Y2 - 21 September 2018 through 23 September 2018

ER -

Zhang H, Li J, Wang H. Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0. In Gan Y, Song X, Zhou Q, Jing W, Wang Y, Lu Z, editors, Data Science - 4th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2018, Proceedings. Springer Verlag. 2018. p. 403-414. (Communications in Computer and Information Science). doi: 10.1007/978-981-13-2203-7_31

Statistical learning-based prediction of execution time of data-intensive program under hadoop2.0

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this