Waveform Design and Successive Parameter Estimation for OTFS-ISAC in Clutter Environment

Integrated sensing and communication (ISAC) opens new opportunities for next-generation systems by supporting simultaneous communication and sensing within a unified framework. In this paper, we investigate the waveform design and parameter estimation problem for orthogonal time-frequency-space (OTFS)-based ISAC in monostatic multiuser scenarios under dynamic multipath and cluttered environments. A key challenge lies in developing an OTFS-ISAC waveform to establish a structured input-output relationship, which is critical for enabling high-accuracy parameter estimation. To overcome this, we construct a periodic convolution sum spectrum (CSS) by designing practical transceiver filters, yielding a tractable received signal model. The corresponding optimization problem is solved efficiently within the alternating direction method of multipliers (ADMM) framework, where closed-form solutions are derived to avoid reliance on high-complexity numerical solvers. Furthermore, strong clutter in the environment makes it difficult for the base station (BS) to detect non-line-of-sight (NLoS) echoes, hindering full communication channel reconstruction. To address this issue, we adopt a hierarchical estimation scheme and embed pilot sequences (PSs) within a tailored delay-Doppler (DD) pattern. In addition, a weighted integrated sidelobe level (WISL)-based PS design is introduced, which leverages coordinate descent (CD) and discrete Fourier transform (DFT) techniques to support short sequence lengths. Building on the proposed OTFS-ISAC waveform, which integrates transceiver filters, DD pattern, and optimized PS, we propose a successive parameter estimation (SPE) algorithm with a dual-direction estimation strategy. By exploiting inter-Doppler interference (IDI) dispersion, the algorithm derives closed-form channel parameter estimates, applicable at both the BS and user equipments (UEs). Simulation results demonstrate that the proposed PS design method exhibits near-optimal correlation properties, while the estimation algorithm achieves competitive performance with significantly reduced complexity.