Hierarchical Rate-Splitting for Non-Orthogonal Broadcast Multicast and Unicast Transmission in Multibeam Satellite System

Multibeam satellite systems, benefiting from wide coverage, enable ubiquitous connectivity. Meanwhile, the promising hierarchical rate-splitting multiple access (HRSMA) technique, with robust interference management capabilities, can effectively handle both inter-beam and intra-beam interference caused by multibeam satellite systems. To simultaneously support diverse information services in multibeam satellite systems, this paper proposes a non-orthogonal broadcast, multicast, and unicast (NOBMU) joint transmission scheme based on HRSMA. Specifically, we formulate a maximum sum rate (MSR) problem and develop a block coordinate descent (BCD) iterative optimization (IO) algorithm based on weighted minimum mean square error (WMMSE). The closed-form expressions for beamforming vectors are derived using the Lagrangian expression and Karush-Kuhn-Tucker (KKT) conditions. To enhance the sum rate while ensuring fairness, a two-stage optimization framework is proposed: the first stage solves a max-min fairness (MMF) problem to obtain a guaranteed minimum user rate, which is then applied as a constraint in the second-stage MSR problem. Finally, simulation results demonstrate that the proposed HRSMA-assisted NOBMU transmission scheme significantly outperforms other multiple access schemes.