ISAC Systems With Realistic Compound-Gaussian Clutter: Detection Performance Analysis and Input Distribution Parameter Design

Integrated sensing and communication (ISAC) enables unified waveforms to simultaneously support high-rate communication and high-precision sensing. However, the communication symbols are drawn from a specific input distribution, whose inherent randomness inevitably degrades sensing accuracy, resulting in a deterministic–random trade-off (DRT). Moreover, most existing analyses assume clutter-free or idealized Gaussian environments, which diverge from realistic clutter conditions and significantly compromise the reliability of performance characterization. To address these issues, in this paper, we model clutter as a compound-Gaussian process that accounts for realistic amplitude fluctuations. Under this model, we analyze both detection performance and achievable communication rate. Specifically, we propose simplified likelihood ratio test (LRT)-based detectors for strong clutter regimes and derive analytical expressions for the probability of detection (PD) and probability of false alarm (PFA). In more general clutter scenarios, the analysis is extended to the generalized LRT (GLRT) framework using both maximum likelihood (ML) and maximum a posteriori (MAP) estimators. Since the MAP-GLRT distribution lacks a closed-form expression, a moment-matched Gaussian approximation is adopted to enable analytical tractability. Then, to theoretically substantiate the DRT, we investigate the impact of input distribution parameters on the average sensing signal-to-clutter-plus-noise ratio (SCNR), which serves as a key metric for evaluating the derived detection performance. Building on this relationship, we further formulate a joint optimization over pilot power and data symbol statistics to enhance sensing performance under communication rate constraints. Simulation results validate the theoretical analysis and the performance advantage over conventional Gaussian-based detectors.