Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis

Yan Li; Peiyi Han; Chuanyi Liu; Binxing Fang

doi:10.1109/DSC.2018.00042

Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis

Yan Li
, Peiyi Han
, Chuanyi Liu^*
, Binxing Fang

^*Corresponding author for this work

Beijing University of Posts and Telecommunications
Harbin Institute of Technology Shenzhen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

According to the statistics of BrightPlanet in 2012, the information contained in Deep Web is 400-500 times more than those in the Surface Web. Automatic data collection under Deep Web has become one of the research hotspots in the domain of web crawlers. During the exploration of automatic dynamic web crawlers, we discover two inevitable situations which have not been processed properly by existing tools. One is that some websites adopt CSS pseudo-class to locate clickable elements, which prevents the existing technologies from simulating user actions. Another is that pop-up windows can be triggered during automatically elements clicking, where the existing web crawler procedures will be terminated. Addressing the CSS pseudo-class problem, we adopt a proxy server in our system to analyze JavaScript code of target websites both statically and dynamically. In the meantime, we propose a checking mechanism to handle the pop-up windows. Finally, we implement an automatic web crawler AutoCrawler with the aid of the proxy server. Evaluation results indicate that AutoCrawler can not only cover more dynamic interactions but also obtain more valuable data from the real-world web applications.

Original language	English
Title of host publication	Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	242-249
Number of pages	8
ISBN (Electronic)	9781538642108
DOIs	https://doi.org/10.1109/DSC.2018.00042
State	Published - 16 Jul 2018
Externally published	Yes
Event	3rd IEEE International Conference on Data Science in Cyberspace, DSC 2018 - Guangzhou, Guangdong, China Duration: 18 Jun 2018 → 21 Jun 2018

Publication series

Name	Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018

Conference

Conference	3rd IEEE International Conference on Data Science in Cyberspace, DSC 2018
Country/Territory	China
City	Guangzhou, Guangdong
Period	18/06/18 → 21/06/18

Keywords

Dynamic websites
JavaScript analysis
Proxy server
Web crawler

Access to Document

10.1109/DSC.2018.00042

Cite this

Li, Y., Han, P., Liu, C., & Fang, B. (2018). Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis. In Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018 (pp. 242-249). Article 8411863 (Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DSC.2018.00042

Li, Yan ; Han, Peiyi ; Liu, Chuanyi et al. / Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis. Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 242-249 (Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018).

@inproceedings{80127dc768c24b9ab336ee47da81ae0d,

title = "Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis",

abstract = "According to the statistics of BrightPlanet in 2012, the information contained in Deep Web is 400-500 times more than those in the Surface Web. Automatic data collection under Deep Web has become one of the research hotspots in the domain of web crawlers. During the exploration of automatic dynamic web crawlers, we discover two inevitable situations which have not been processed properly by existing tools. One is that some websites adopt CSS pseudo-class to locate clickable elements, which prevents the existing technologies from simulating user actions. Another is that pop-up windows can be triggered during automatically elements clicking, where the existing web crawler procedures will be terminated. Addressing the CSS pseudo-class problem, we adopt a proxy server in our system to analyze JavaScript code of target websites both statically and dynamically. In the meantime, we propose a checking mechanism to handle the pop-up windows. Finally, we implement an automatic web crawler AutoCrawler with the aid of the proxy server. Evaluation results indicate that AutoCrawler can not only cover more dynamic interactions but also obtain more valuable data from the real-world web applications.",

keywords = "Dynamic websites, JavaScript analysis, Proxy server, Web crawler",

author = "Yan Li and Peiyi Han and Chuanyi Liu and Binxing Fang",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 3rd IEEE International Conference on Data Science in Cyberspace, DSC 2018 ; Conference date: 18-06-2018 Through 21-06-2018",

year = "2018",

month = jul,

day = "16",

doi = "10.1109/DSC.2018.00042",

language = "英语",

series = "Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "242--249",

booktitle = "Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018",

address = "美国",

}

Li, Y, Han, P, Liu, C & Fang, B 2018, Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis. in Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018., 8411863, Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018, Institute of Electrical and Electronics Engineers Inc., pp. 242-249, 3rd IEEE International Conference on Data Science in Cyberspace, DSC 2018, Guangzhou, Guangdong, China, 18/06/18. https://doi.org/10.1109/DSC.2018.00042

Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis. / Li, Yan; Han, Peiyi; Liu, Chuanyi et al.
Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 242-249 8411863 (Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis

AU - Li, Yan

AU - Han, Peiyi

AU - Liu, Chuanyi

AU - Fang, Binxing

PY - 2018/7/16

Y1 - 2018/7/16

N2 - According to the statistics of BrightPlanet in 2012, the information contained in Deep Web is 400-500 times more than those in the Surface Web. Automatic data collection under Deep Web has become one of the research hotspots in the domain of web crawlers. During the exploration of automatic dynamic web crawlers, we discover two inevitable situations which have not been processed properly by existing tools. One is that some websites adopt CSS pseudo-class to locate clickable elements, which prevents the existing technologies from simulating user actions. Another is that pop-up windows can be triggered during automatically elements clicking, where the existing web crawler procedures will be terminated. Addressing the CSS pseudo-class problem, we adopt a proxy server in our system to analyze JavaScript code of target websites both statically and dynamically. In the meantime, we propose a checking mechanism to handle the pop-up windows. Finally, we implement an automatic web crawler AutoCrawler with the aid of the proxy server. Evaluation results indicate that AutoCrawler can not only cover more dynamic interactions but also obtain more valuable data from the real-world web applications.

AB - According to the statistics of BrightPlanet in 2012, the information contained in Deep Web is 400-500 times more than those in the Surface Web. Automatic data collection under Deep Web has become one of the research hotspots in the domain of web crawlers. During the exploration of automatic dynamic web crawlers, we discover two inevitable situations which have not been processed properly by existing tools. One is that some websites adopt CSS pseudo-class to locate clickable elements, which prevents the existing technologies from simulating user actions. Another is that pop-up windows can be triggered during automatically elements clicking, where the existing web crawler procedures will be terminated. Addressing the CSS pseudo-class problem, we adopt a proxy server in our system to analyze JavaScript code of target websites both statically and dynamically. In the meantime, we propose a checking mechanism to handle the pop-up windows. Finally, we implement an automatic web crawler AutoCrawler with the aid of the proxy server. Evaluation results indicate that AutoCrawler can not only cover more dynamic interactions but also obtain more valuable data from the real-world web applications.

KW - Dynamic websites

KW - JavaScript analysis

KW - Proxy server

KW - Web crawler

UR - https://www.scopus.com/pages/publications/85051026152

U2 - 10.1109/DSC.2018.00042

DO - 10.1109/DSC.2018.00042

M3 - 会议稿件

AN - SCOPUS:85051026152

T3 - Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018

SP - 242

EP - 249

BT - Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 3rd IEEE International Conference on Data Science in Cyberspace, DSC 2018

Y2 - 18 June 2018 through 21 June 2018

ER -

Li Y, Han P, Liu C, Fang B. Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis. In Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 242-249. 8411863. (Proceedings - 2018 IEEE 3rd International Conference on Data Science in Cyberspace, DSC 2018). doi: 10.1109/DSC.2018.00042

Automatically crawling dynamic web applications via proxy-based javascript injection and runtime analysis

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this