SepJoin: A Distributed Stream Join System with Low Latency and High Throughput

In the field of real-time analytics, stream joins are the basis for complex queries and greatly affect system performance. In order to satisfy the real-time requirements of streaming applications, the system imposes high requirements on the latency and throughput of the stream join operator. In this paper, we model the latency and throughput of distributed stream join systems based on queuing theory. Based on the analysis of this model, we demonstrate the impact of indexing-related overhead on the latency and throughput of stream join systems and propose a new distributed stream join system, SepJoin, which is oriented to the hash join problem. SepJoin reduces the number of tuples stored in each processing unit belonging to each input stream by designing a novel partitioning scheme that uses as many processing units as possible to store tuples belonging to each input stream, thereby reducing the index-related overhead of each processing unit when performing join operations and ultimately achieving performance benefits in terms of latency and throughput. We provide both theoretical analysis and extensive experimental evaluations to evaluate the processing latency and max throughput of SepJoin.