Sparse Patch Routing for Efficient Remote Sensing Scene Segmentation

Abstract

High-resolution remote sensing images contain large homogeneous regions such as water, farmland, roofs, roads, and vegetation. Processing all image patches with the same computational depth increases the cost of transformer segmentation models and may provide limited benefit for visually simple regions. This study investigates sparse patch routing for efficient remote sensing scene segmentation. We propose the Sparse Patch Routing Transformer (SPRT), which separates image patches into three computational routes: a shallow route for low-texture background regions, a standard route for common land-cover areas, and a detailed route for boundary-sensitive regions such as road edges, building contours, and small objects. Patch routing is determined by a lightweight saliency estimator using local texture variance, class-token response, and boundary uncertainty. Unlike direct token deletion, SPRT preserves the spatial grid and assigns different processing depths to different patch groups. The model was evaluated on LoveDA, ISPRS Vaihingen, and WHU Building datasets, covering 7,426 high-resolution images, 38 urban and rural scenes, and more than 12.8 billion labeled pixels. On LoveDA, SPRT achieved 53.7% mIoU, compared with 54.2% for the full SegFormer-B2 baseline, while reducing FLOPs by 34.6%. On WHU Building, boundary F1-score decreased by only 0.4 percentage points after patch routing, showing that small structural details were largely preserved. Runtime testing on 2048 × 2048 image tiles showed that inference time decreased from 186 ms to 119 ms on an RTX 4090 and from 742 ms to 486 ms on Jetson Orin. Stratified analysis further showed that farmland and water regions used the shallow route in over 71% of patches, while building-edge regions were routed to detailed computation in 64% of boundary patches. These results suggest that patch-level routing can reduce transformer segmentation cost while maintaining reliable spatial accuracy in remote sensing applications