Penalty-based second-order multiagent approaches to distributed nonconvex optimization

Distributed nonconvex optimization has found broad application in machine learning and related areas. In distributed nonconvex constrained optimization, the combination of both nonconvex and convex inequality constraints with coupling constraints yields a nonconvex feasible set, while the objective function is also nonconvex, thereby complicating coordination. By employing penalty-function methods and distributed tracking techniques, this paper develops two second-order optimization frameworks to solve distributed nonconvex problems, addressing mixtures of nonconvex and convex inequality constraints, with one framework designed for consensus constraints and the other for coupling constraints. By leveraging equivalent reformulations derived from penalty function methods, two second-order multi-agent systems interconnected by communication networks are designed to solve the nonconvex models in a distributed manner. The convergence analysis is provided, demonstrating that the proposed multi-agent systems converge to the critical points set of the nonconvex models, effectively overcoming the potential divergence caused by nonconvexity and hybrid constraints. Compared to existing approaches, the proposed multi-agent system exhibits enhanced stability and features lower-dimensional dynamics. The effectiveness of the proposed multi-agent systems is validated through four numerical examples.