A Novel Boundary-Aware Hybrid CNN–Transformer Framework for Breast Cancer Detection in Ultrasound Images

Recent advancements in breast cancer detection using deep CNNs and vision transformers (ViTs) on breast ultrasound images (BUSI) have demonstrated strong performance. However, model complexity and variations in contrast, texture, and morphology continue to limit effectiveness. This study introduces the CB-Res-RBCMT, a hybrid framework that combines customized residual CNNs with ViT components for detailed BUSI cancer analysis. The RBCMT integrates stem convolution blocks and CNN-Meet-Transformer (CMT) modules, followed by regional-boundary (RB) feature operations. These operations apply the Laplacian of Gaussian (LoG) filter to enhance homogeneity, reduce speckle noise, and highlight structural transitions, while boundary operations capture malignant morphological changes. The CMT module utilizes multi-head attention for global context interactions, improving computational efficiency. New inverse residual blocks and stem CNNs further extract malignant texture information and address vanishing gradients. A multiscale channel fusion and attention (MSCFA) block enhances feature representation by combining global context with boundary-aware cues. The Channel-Boosted (CB) strategy fuses RBCMT and residual CNN feature maps to increase feature diversity for the limited BUSI dataset. Finally, a spatial attention block refines channels for optimal pixel selection and reduced redundancy. The CB-Res-RBCMT achieves 95.63% accuracy, 95.57% F1-score, 96.42% precision, and 94.79% sensitivity, outperforming existing CNN and ViT-based approaches. Furthermore, cross-dataset generalization on mammographic images validates the model's robust clinical applicability across diverse screening scenarios.