分布式流处理系统的时间感知分组算法

The increasing demand for real-time data stream processing is driving the development of distributed stream processing systems. The large amount of skew data streams and the heterogeneity of complex distributed systems pose challenges to the current grouping strategies of distributed stream processing systems. The existing distributed stream processing grouping strategies usually focus on balancing the number of tuples between parallel instances, while ignoring the impact of system heterogeneity on the grouping strategy. This paper presents a time-aware grouping algorithm that analyzes the network heterogeneity and the processing capability in a distributed stream processing system that considers the processing time of each downstream operator instance in the stream processing system. The algorithm also takes into account the communication time between the upstream and downstream operators with various routing strategies formulated according to the frequency of the key, so that the system achieves load balancing with little overhead. Tests on an Apache Flink distributed stream processing system show that the time-aware grouping algorithm increases the throughput by 10% while the average processing latency is reduced by 33% compared to the existing grouping algorithm.