{'fraction_non_alphanumeric': 0.04765225999505029, 'fraction_numerical': 0.01817895471010417, 'mean_word_length': 4.9653737753321705, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SVAM: Saliency-guided Visual Attention Modeling by Autonomous Underwater Robots

This paper presents a holistic approach to saliency-guided visual attention modeling (SVAM) for use by autonomous underwater robots. Our proposed model, named SVAM-Net, integrates deep visual features at various scales and semantics for effective salient object detection (SOD) in natural underwater images. The SVAM-Net architecture is configured in a unique way to jointly accommodate bottom-up and top-down learning within two separate branches of the network while sharing the same encoding layers. We design dedicated spatial attention modules (SAMs) along these learning pathways to exploit the coarse-level and fine-level semantic features for SOD at four stages of abstractions. The bottom-up branch performs a rough yet reasonably accurate saliency estimation at a fast rate, whereas the deeper top-down branch incorporates a residual refinement module (RRM) that provides fine-grained localization of the salient objects. Extensive performance evaluation of SVAM-Net on benchmark datasets clearly demonstrates its effectiveness for underwater SOD. We also validate its generalization performance by several ocean trials' data that include test images of diverse underwater scenes and waterbodies, and also images with unseen natural objects. Moreover, we analyze its computational feasibility for robotic deployments and demonstrate its utility in several important use cases of visual attention modeling.


I. INTRODUCTION
Salient object detection (SOD) aims at modeling human visual attention behavior to highlight the most important and distinct objects in a scene. It is a well-studied problem in the domains of robotics and computer vision [1], [2], [3], [4] for its usefulness in identifying regions of interest (RoI) in an image for fast and effective visual perception. The SOD capability is essential for visually-guided robots because they need to make critical navigational and operational decisions based on the relative importance of various objects in their field-of-view (FOV). The autonomous underwater vehicles (AUVs), in particular, rely heavily on visual saliency estimation for tasks such as exploration and surveying [5], [6], [7], [8], [9], ship-hull inspection [2], event detection [10], place recognition [11], target localization [12], [13], and more.
In the pioneering work on SOD, Itti et al. [14] used local feature contrast in image regions to infer visual saliency. Numerous methods have been subsequently proposed [10], [15], [16] that utilize local point-based features and also global contextual information as reference for saliency estimation. In recent years, the state-of-the-art (SOTA) approaches have used powerful deep visual models [17], [18] to imitate human visual information processing through top-down or bottomup computational pipelines. The bottom-up models learn to gradually infer high-level semantically rich features [18]; hence the shallow layers' structural knowledge drives their multi-scale saliency learning. Conversely, the the top-down approaches [3], [19] progressively integrate high-level semantic knowledge with low-level features for learning coarse-tofine saliency estimation. Moreover, the contemporary models have introduced various techniques to learn boundary refinement [20], [19], [21], pyramid feature attention [17], and contextual awareness [3], which significantly boost the SOD performance on benchmark datasets.
However, the applicability of such powerful learning-based SOD models in real-time underwater robotic vision has been rather limited. The underlying challenges and practicalities are twofold. First, the visual content of underwater imagery is uniquely diverse due to domain-specific object categories, background waterbody patterns, and a host of optical distortion artifacts [22], [23]; hence, the SOTA models trained on terrestrial data are not transferable off-the-shelf. A lack of large-scale annotated underwater datasets aggravates the problem; the existing datasets and relevant methodologies are tied to specific applications such as coral reef classification and coverage estimation [24], [25], [26], object detection [27], [28], [29], and foreground segmentation [30], [31], [32]. Consequently, these do not provide a comprehensive data representation for effective learning of underwater SOD. Secondly, learning a generalizable SOD function demands the extrapolation of multi-scale hierarchical features by high-capacity deep network models. This results in a heavy computational load and makes real-time inference impossible, particularly on single-board robotic platforms.
To this end, traditional approaches based on various feature contrast evaluation techniques [5], [12], [33] are often practical choices for saliency estimation by visually-guided underwater robots. These techniques encode low-level image-based features (e.g., color, texture, object shapes or contours) into superpixel descriptors [34], [35], [33], [36], [9] to subsequently infer saliency by quantifying their relative distinctness on a global scale. Such bottom-up approaches are computationally light and are useful as pre-processing steps for faster visual search [34], [2] and exploration tasks [5], [11]. However, they do not provide a standalone generalizable solution for SOD in underwater imagery. A few recently proposed approaches attempt to address this issue by learning more generalizable SOD solutions from large collection of annotated underwater data [13], [37], [38], [39], [40]. These approaches and other SOTA deep visual models have reported inspiring results for underwater SOD and relevant problems [35], [38], [13]. Nevertheless, their utility and performance margins for realtime underwater robotic applications have not been explored in-depth in the literature.
In this paper, we formulate a robust and efficient solution for saliency-guided visual attention modeling (SVAM) by harnessing the power of both bottom-up and top-down learning in a novel encoder-decoder model named SVAM-Net (see §III-A). We design two spatial attention modules (SAMs) named SAM bu and SAM td to effectively exploit the coarselevel and fine-level semantic features along the bottom-up and top-down learning pathways, respectively. SAM bu utilizes the semantically rich low-dimensional features extracted by the encoder to perform an abstract yet reasonably accurate saliency estimation. Concurrently, SAM td combines the multiscale hierarchical features of the encoder to progressively decode the information for robust SOD. A residual refinement module (RRM) further sharpens the initial SAM td predictions to provide fine-grained localization of the salient objects. To balance the high degree of refined gradient flows from the later SVAM-Net layers, we deploy an auxiliary SAM named SAM aux that guides the spatial activations of early encoding layers and ensures smooth end-to-end learning.
In addition to sketching the conceptual design, we present a holistic training pipeline of SVAM-Net and its variants. The end-to-end learning is supervised by six loss functions which are selectively applied at the final stages of SAM aux , SAM bu , SAM td , and RRM. These functions evaluate information loss and boundary localization errors in the respective SVAM-Net predictions and collectively ensure effective SOD learning (see §III-B). In our evaluation, we analyze SVAM-Net's performance in standard quantitative and qualitative terms on three benchmark datasets named UFO-120 [38], MUED [37], and SUIM [13]. We also conduct performance evaluation on USOD, which we prepare as a new challenging test set for underwater SOD. Without data-specific tuning or task-specific model adaptation, SVAM-Net outperforms other existing solutions on these benchmark datasets (see §IV); more importantly, it exhibits considerably better generalization performance on random unseen test cases of natural underwater scenes.
Lastly, we present several design choices of SVAM-Net, analyze their computational aspects, and discuss the corresponding use cases. The end-to-end SVAM-Net model offers over 20 frames per second (FPS) inference rate on a single GPU 1 . Moreover, the decoupled SAM bu branch offers significantly faster rates, e.g., over 86 FPS on a GPU and over 21 FPS on single-board computers 2 . As illustrated in Fig. 1, robust saliency estimates of SVAM-Net at such speeds are ideal for fast visual attention modeling in robotic deployments. We further demonstrate its usability benefits for important applications such as object detection, image enhancement, and image super-resolution by visually-guided underwater robots (see §VI). The SVAM-Net model, USOD dataset, and relevant resources will be released at http://irvlab.cs.umn. edu/visual-attention-modeling/svam.

A. Salient Object Detection (SOD)
SOD is a successor to the human fixation prediction (FP) problem [14] that aims to identify fixation points that human viewers would focus on at first glance. While FP originates from research in cognition and psychology [41], [42], [43], SOD is more of a visual perception problem explored by the computer vision and robotics community [1], [2], [3], [4]. The history of SOD dates back to the work of Liu et al. [44] and Achanta et al. [45], which make use of multi-scale contrast, center-surround histogram, and frequency-domain cues to (learn to) infer saliency in image space. Other traditional SOD models rely on various low-level saliency cues such as point-based features [10], local and global contrast [16], [15], background prior [46], etc. Please refer to [47] for a more comprehensive overview of non-deep learning-based SOD models.
In particular, Wang et al. [43] use multi-stage convolutional LSTMs for saliency estimation guided by fixation maps. Later in [18], they explore the benefits of integrating bottom-up and top-down recurrent modules for co-operative SOD learning. Since the feed-forward computational pipelines lack a feedback strategy [54], [42], recurrent modules offer more learning capacity via self-correction. However, they are prone to the vanishing gradient problem and also require meticulous design choices in their feedback loops [55], [18]. To this end, topdown models with UNet-like architectures [3], [19], [56], [57], [58] provide more consistent learning behavior. These models typically use a powerful backbone network (e.g., VGG [59], ResNet [60]) to extract a hierarchical pyramid of features, then perform a coarse-to-fine feature distillation via mirrored skip-connections. Subsequent research introduces the notions of short connections [61] and guided super-pixel filtering [62] to learn to infer compact and uniform saliency maps.
Moreover, various attention mechanisms are incorporated by contemporary models to intelligently guide the SOD learning, particularly for tasks such as image captioning and visual question answering [63], [64], [65], [66]. Additionally, techniques like pyramid feature attention learning [17], [67], boundary refinement modules [20], [19], [21], contextual awareness [3], and cascaded partial decoding [68] have significantly boosted the SOTA SOD performance margins. However, this domain knowledge has not been applied or explored in-depth for saliency-guided visual attention modeling (SVAM) by underwater robots, which we attempt to address in this paper.

B. SOD and SVAM by Underwater Robots
The most essential capability of visually-guided AUVs is to identify interesting and relevant image regions to make effective operational decisions. As shown in Fig. 2, the existing systems and solutions for visual saliency estimation can be categorically discussed from the perspectives of model adaptation [6], [13], high-level robot tasks [5], [69], and feature evaluation pipeline [35], [11]. Since we already discussed the particulars of bottom-up and top-down computational pipelines, our following discussion is schematized based on the model and task perspectives.
Visual saliency estimation approaches can be termed as either model-based or model-free, depending on whether the robot models any prior knowledge of the target salient objects and features. The model-based techniques are particularly beneficial for fast visual search [6], [9], enhanced object detection [12], [40], and monitoring applications [70], [71]. For instance, Maldonado-Ramírez et al. [11]  visual descriptors learned by a convolutional autoencoder to identify salient landmarks for fast place recognition. Moreover, Koreitem et al. [6] use a bank of pre-specified image patches (containing interesting objects or relevant scenes) to learn a similarity operator that guides the robot's visual search in an unconstrained setting. Such similarity operators are essentially spatial saliency predictors which assign a degree of relevance to the visual scene based on the prior model-driven knowledge of what may constitute as salient, e.g., coral reefs [25], [72], companion divers [28], [12], wrecks [13], fish [27], etc.
On the other hand, model-free approaches are more feasible for autonomous exploratory applications [73], [74]. The early approaches date back to the work of Edgington et al. [10] that uses binary morphology filters to extract salient features for automated event detection. Subsequent approaches adopt various feature contrast evaluation techniques that encode lowlevel image-based features (e.g., color, luminance, texture, object shapes) into super-pixel descriptors [33], [34], [36]. These low-dimensional representations are then exploited by heuristics or learning-based models to infer global saliency. For instance, Girdhar et al. [5] formulate an online topicmodeling scheme that encodes visible features into a lowdimensional semantic descriptor, then adopt a probabilistic approach to compute a surprise score for the current observation based on the presence of high-level patterns in the scene. Moreover, Kim et al. [2] introduce an online bag-of-words scheme to measure intra-and inter-image saliency estimation for robust key-frame selection in SLAM-based navigation. Wang et al. [36] encode multi-scale image features into a topographical descriptor, then apply Bhattacharyya [75] measure to extract salient RoIs by segmenting out the background. These bottom-up approaches are effective in pre-processing raw visual data to identify point-based or region-based salient features; however, they do not provide a generalizable objectlevel solution for underwater SOD.
Nevertheless, several contemporary research [76], [35], [32], [77] report inspiring results for object-level saliency estimation and foreground segmentation in underwater imagery. Chen et al. [76] use a level set-based formulation that exploits various low-level features for underwater SOD. Moreover, Jian et al. [35] perform principal components analysis (PCA) in quaternionic space to compute pattern distinctness and local contrast to infer directional saliency. These methods are also model-free and adopt a bottom-up feature evaluation pipeline. In contrast, our earlier work [38] incorporates multi-scale hierarchical features extracted by a top-down deep residual model to identify salient foreground pixels for global contrast enhancement. In this paper, we formulate a generalized solution for underwater SOD and demonstrate its utility for SVAM by visually-guided underwater robots. It combines the benefits of bottom-up and top-town feature evaluation in a compact end-to-end pipeline, provides SOTA performance, and ensures computational efficiency for robotic deployments in both search-based and exploration-based applications.

III. MODEL & TRAINING PIPELINE
A. SVAM-Net Architecture As illustrated in Fig. 3, the major components of our SVAM-Net model are: the backbone encoder network, the top-down SAM (SAM td ), the residual refinement module (RRM), the bottom-up SAM (SAM bu ), and the auxiliary SAM (SAM aux ). These components are tied to an end-to-end architecture for a supervised SOD learning.

1) Backbone Encoder Network:
We use the first five sequential blocks of a standard VGG-16 network [59] as the backbone encoder in our model. Each of these blocks consist of two or three convolutional (Conv) layers for feature extraction, which are then followed by a pooling (Pool) layer for spatial down-sampling. For an input dimension of 256 × 256 × 3, the composite encoder blocks e 1 → e 5 learn 128×128×64, 64×64×128, 32×32×256, 16×16×512, and 8 × 8 × 512 feature-maps, respectively. These multi-scale deep visual features are jointly exploited by the attention modules of SVAM-Net for effective learning.
Such asymmetric pairing facilitates the use of a standalone deconvolutional (DeConv) layer [78] following d 2 rather than using another composite decoder block, which we have found to be redundant (during ablation experiments These feature-maps are passed along two branches (see Fig. 3); on the shallow branch, a Sigmoid layer is applied to generate an intermediate SOD prediction Y td , while the other deeper branch incorporates residual layers for subsequent refinement.

3) Residual Refinement Module (RRM):
We further design a residual module to effectively refine the top-down coarse saliency predictions by learning the desired residuals as Such refinement modules [79], [20], [21] are designed to address the loss of regional probabilities and boundary localization in intermediate SOD predictions. While the existing methodologies use iterative recurrent modules [79] or additional residual encoder-decoder networks [20], we deploy only two sequential residual blocks and a Conv layer for the refinement. Each residual block consists of a Conv layer followed by batch normalization (BN) [80] and a rectified linear unit (ReLU) activation [81]. The entire RRM operates on a feature dimension of 256 × 256 × 128; following refinement, a Sigmoid layer squashes the feature-maps to generate a single-channel output Y tdr , which is the final SOD prediction of SVAM-Net.

4) Bottom-Up SAM (SAM bu ):
A high degree of supervision at the final layers of RRM and SAM td forces the backbone encoding layers to learn effective multi-scale features. In SAM bu , we exploit these low-resolution yet semantically rich features for efficient bottom-up SOD learning. Specifically, we combine the feature-maps of dimension 16 × 16 × 512 from e 4 (Pool4) and e 5 (Conv53), and subsequently learn the bottom-up spatial attention as On the combined input feature-maps, SAM bu incorporates 4× bilinear interpolation (BI) followed by two Conv layers with ReLU activation to learn 64 × 64 × 256 feature-maps. Subsequently, another BI layer performs 4× spatial up-sampling to generate S bu ; lastly, a Sigmoid layer is applied to generate the single-channel output Y bu .

5) Auxiliary SAM (SAM aux ):
We excluded the features of early encoding layers for bottom-up SOD learning in SAM bu for two reasons: i) they lack important semantic details despite their higher resolutions [67], [68], and ii) it is counterintuitive to our goal of achieving fast bottom-up inference. Nevertheless, we adopt a separate attention module that refines the features of e 2 (Conv22) and e 3 (Conv33) as Here, a Conv layer with ReLU activation is applied separately on these inputs, followed by a 2× or 4× BI layer (see Fig. 3). Their combined output features are passed to a Conv layer to subsequently generate S aux of dimension 256 × 256 × 128. The sole purpose of this module is to backpropagate additional refined gradients via supervised loss applied to the Sigmoid output Y aux . This auxiliary refinement facilitates smooth feature learning while adding no computational overhead in the bottom-up inference through SAM bu (as we discard SAM aux after training).

B. Learning Objectives and Training
SOD is a pixel-wise binary classification problem that refers to the task of identifying all salient pixels in a given image. We formulate the problem as learning a function f : X → Y , where X is the input image domain and Y is the target saliency map, i.e., saliency probability for each pixel. As illustrated in Fig. 3, SVAM-Net generates saliency maps from four output layers, namely We adopt six loss components to collectively evaluate the information loss and boundary localization error for the supervised training of SVAM-Net. To quantify the information loss, we use the standard binary cross-entropy (BCE) function [82] that measures the disparity between predicted saliency mapŶ and ground truth Y as We also use the analogous weighted cross-entropy loss function L W CE (Ŷ , Y ), which is widely adopted in SOD literature to handle the imbalance problem of the number of salient pixels [1], [43], [67]. While L W CE provides general guidance for accurate saliency estimation, we use the 2D Laplace operator [83] to further ensure robust boundary localization of salient objects. Specifically, we utilize the 2D Laplacian kernel  Table I). Information loss and/or boundary localization error terms applied at various output layers are annotated by purple letters.
K Laplace to evaluate the divergence of image gradients [67] in the predicted saliency map and respective ground truth as Then, we measure the boundary localization error as As demonstrated in Fig. 4, we deploy a two-step training process for SVAM-Net to ensure robust and effective SOD learning. First, the backbone encoder and SAM td are pretrained holistically with combined terrestrial (DUTS [84]) and underwater data (SUIM [13], UFO-120 [38]). The DUTS training set (DUTS-TR) has 10553 terrestrial images, whereas the SUIM and UFO-120 datasets contain a total of 3025 underwater images for training and validation. This large collection of diverse training instances facilitates a comprehensive learning of a generic SOD function (more details in §V). We supervise the training by applying L P T ≡ L BCE (Y td , Y ) loss at the sole output layer of SAM td . The SGD optimizer [85] is used for the iterative learning with an initial rate of 1e −2 and 0.9 momentum, which is decayed exponentially by a drop rate of 0.5 after every 8 epochs; other hyper-parameters are listed in Table I.
Subsequently, the pre-trained weights are exported into the SVAM-Net model for its end-to-end training on underwater imagery. The loss components applied at the output layers of We formulate the combined objective function as a linear combination of these loss terms as follows Here, λ symbols are scaling factors that represent the contributions of respective loss components; their values are empirically tuned as hyper-parameters. In our evaluation, the selected values of λ w , λ b , λ aux , λ bu , λ td , and λ tdr are 0.7, 0.3, 0.5, 1.0, 2.0, and 4.0, respectively. As shown in Table I, we use the Adam optimizer [85] for the global optimization of L E2E with a learning rate of 3e −4 and a momentum of 0.5.

C. SVAM-Net Inference
Once the end-to-end training is completed, we decouple a bottom-up and a top-down branch of SVAM-Net for fast inference. As illustrated in Fig. 5, the {e 1:5 → SAM td → RRM} branch is the default SVAM-Net top-down pipeline that generates fine-grained saliency maps; here, we discard the SAM aux and SAM bu modules to avoid unnecessary computation. On the other hand, we exploit the shallow bottom-up branch, i.e., the {e 1:5 → SAM bu } pipeline to generate rough yet reasonably accurate saliency maps at a significantly faster rate. Here, we discard SAM aux and both the top-down modules (SAM td and RRM); we denote this computationally light pipeline as SVAM-Net Light . Next, we analyze the SOD performance of SVAM-Net and SVAM-Net Light , demonstrate potential use cases, and discuss various operational considerations.

A. Implementation Details and Ablation Studies
As mentioned in §III-B, SVAM-Net training is supervised by paired data ({X}, {Y }) to learn a pixel-wise predictive function f : X → Y aux , Y bu , Y td , Y tdr . TensorFlow and Keras libraries [86] are used to implement its network architecture and optimization pipelines (Eq. 1-9). A Linux machine with two NVIDIA™ GTX 1080 graphics cards is used for its backbone pre-training and end-to-end training with the learning parameters provided in Table I.

Y aux Y bu Y td Y tdr
(b) Snapshots of SVAM-Net output after 40 epochs of subsequent end-to-end training; notice the spatial attention of early encoding layers (in Y aux ) and the gradual progression and refinement by the deeper layers (through Y bu → Y td → Y tdr ).

(iii) (ii) (iv) (i) (v) (c) Results of ablation experiments (for the same input images)
showing contributions of various attention modules and loss functions in the SOD learning: (i) without LBLE (λ b = 0, λw = 1), (ii) without SAM aux and SAM bu (λaux = λ bu = 0), (iii) without SAM td and RRM (λ td = λ tdr = 0), (iv) without RRM (λ tdr = 0), and (v) without backbone pre-training. We demonstrate the progression of SOD learning by SVAM-Net and visualize the contributions of its learning components in Fig. 6. The first stage of learning is guided by supervised pre-training with over 13.5K instances including both terrestrial and underwater images. This large-scale training facilitates effective feature learning in the backbone encoding layers and by SAM td . As Fig. 6a shows, the {e 1:5 → SAM td } pipeline learns spatial attention with a reasonable precision within 90 epochs. We found that it is crucial to not over-train the backbone for ensuring a smooth and effective end-to-end learning with the integration of SAM aux , SAM bu , and RRM. As illustrated in Fig. 6b, the subsequent end-to-end training on underwater imagery enables more accurate and fine-grained saliency estimation by SVAM-Net.
Moreover, we conduct a series of ablation experiments to visually inspect the effects of various loss functions and attention modules in the learning. As Fig. 6c demonstrates, the boundary awareness (enforced by L BLE ) and bottomup attention modules (SAM aux and SAM bu ) are essential to achieve precise localization and sharp contours of the salient objects. It also shows that important details are missed when we incorporate only bottom-up learning, i.e., without SAM td SUIM [13] UFO-120 [38] MUED [36] 0  Fig. 8: Comparisons of PR curves on three benchmark datasets are shown; to maintain clarity, we consider the top ten SOD models based on the results shown in Table II. and subsequent delicate refinements by RRM. Besides, the backbone pre-training step is important to ensure generalizability in the SOD learning and as an effective way to combat the lack of large-scale annotated underwater datasets.

B. Evaluation Data Preparation
We conduct benchmark evaluation on three publicly available datasets: SUIM [13], UFO-120 [38], and MUED [37]. As mentioned, SVAM-Net is jointly supervised on 3025 training instances of SUIM and UFO-120; their test sets contain an additional 110 and 120 instances, respectively. These datasets contain a diverse collection of natural underwater images with important object categories such as fish, coral reefs, humans, robots, wrecks/ruins, etc. Besides, MUED dataset contains 8600 images in 430 groups of conspicuous objects; although it includes a wide variety of complex backgrounds, the images lack diversity in terms of object categories and water-body types. Moreover, MUED provides bounding-box annotations only. Hence, to maintain consistency in our quantitative evaluation, we select 300 diverse groups and perform pixel-level annotations on those images.
In addition to the existing datasets, we prepare a challenging test set named USOD to evaluate underwater SOD methods. It contains 300 natural underwater images which we exhaustively compiled to ensure diversity in the objects categories, waterbody, optical distortions, and aspect ratio of the salient objects. We collect these images from two major sources: (i) Existing unlabeled datasets: we utilize benchmark datasets that are generally used for underwater image enhancement and superresolution tasks; specifically, we select subsets of images from datasets named USR-248 [87], UIEB [88], and EUVP [23].
(ii) Field trials: we have collected data from several oceanic trials and explorations in the Caribbean sea at Barbados. The selected images include diverse underwater scenes and setups for human-robot cooperative experiments (see §VI). Once the images are compiled, we annotate the salient pixels to generate ground truth labels; a few samples are provided in Fig. 7. Fig. 9: Comparison of PR curves on USOD dataset is shown for the top ten SOD models based on the results shown in Table II. C. SOD Performance Evaluation 1) Metrics: We evaluate the performance of SVAM-Net and other existing SOD methods based on four widely-used evaluation criteria [1], [19], [20], [21]: is an overall performance measurement that is computed by the weighted harmonic mean of the precision and recall as: Here, β 2 is set to 0.3 as per the SOD literature to weight precision more than recall. Also, the maximum scores (F max β ) are reported for quantitative comparison.
• S-measure (S m ) is a recently proposed metric [89] that simultaneously evaluates region-aware and object-aware structural similarities between the predicted and ground truth saliency maps.
• Mean absolute error (MAE) is a generic metric that measures the average pixel-wise differences between the predicted and ground truth saliency maps.
• Precision-recall (PR) curve is a standard performance metric and is complementary to MAE. It is evaluated by binarizing the predicted saliency maps with a threshold sliding from 0 to 255 and then performing bin-wise comparison with the ground truth values. 2) Quantitative and Qualitative Analysis: For performance comparison, we consider the following six methods that are widely used for underwater SOD and/or saliency estimation: (i) SOD by Quaternionic Distance-based Weber Descriptor (QDWD) [35], (ii) saliency estimation by the Segmentation of Underwater IMagery Network (SUIM-Net) [13], (iii) saliency prediction by the Deep Simultaneous Enhancement and Super-Resolution (Deep SESR) model [38], (iv) SOD by a Level Set-guided Method (LSM) [76], (v) Saliency Segmentation by evaluating Region Contrast (SSRC) [77], and (vi) SOD by Saliency-based Adaptive Object Extraction (SAOE) [36]. We also include the performance margins of four SOTA SOD models: (i) Boundary-Aware Saliency Network (BASNet) [20], (ii) Pyramid Attentive and salient edGE-aware Network (PAGE-Net) [17], (iii) Attentive Saliency Network (ASNet) [43], and (iv) Cascaded Partial Decoder (CPD) [68]. We use their publicly released weights (pre-trained on terrestrial imagery) and further train them on combined SUIM and UFO-120 data by following the same setup as SVAM-Net (see Table I). We present detailed results for this comprehensive performance analysis in Table II. As the results in the first part of Table II suggest, SVAM-Net outperforms all the underwater SOD models in comparison with significant margins. Although QDWD and SUIM-Net perform reasonably well on particular datasets (e.g., SUIM, and MUED, respectively), their F max β , S m , and MAE scores are much lower; in fact, their scores are comparable to and often lower than those of SVAM-Net Light . The LSM, Deep SESR, SSRC, and SAOE models offer even lower scores than SVAM-Net Light . The respective comparisons of PR curves shown in Fig. 8 and Fig. 9 further validate the superior performance of SVAM-Net and SVAM-Net Light by an area-underthe-curve (AUC)-based analysis. Moreover, Fig. 10 demonstrates that SVAM-Net-generated saliency maps are accurate with precisely segmented boundary pixels in general. Although not as fine-grained, SVAM-Net Light also generates reasonably well-localized saliency maps that are still more accurate and consistent compared to the existing models. These results corroborate our earlier discussion on the challenges and lack of advancements of underwater SOD literature (see §II).
For a comprehensive validation of SVAM-Net, we compare the performance margins of SOTA SOD models trained through the same learning pipeline. As shown in Fig. 10, the saliency maps of BASNet, PAGE-Net, ASNet, and CPD are mostly accurate and often comparable to SVAM-Net-generated maps. The quantitative results of Table II and Fig. 8-9 also confirm their competitive performance over all datasets. Given the substantial learning capacities of these models, one may exhaustively find a better choice of hyper-parameters that further improves their baseline performances. Nevertheless, unlike these standard models, SVAM-Net incorporates a considerably shallow computational pipeline and offers an even lighter bottom-up sub-network (SVAM-Net Light ) that ensures fast inference on single-board devices. Next, we demonstrate SVAM-Net's generalization performance and discuss its utility for underwater robotic deployments.

V. GENERALIZATION PERFORMANCE
Underwater imagery suffers from a wide range of non-linear distortions caused by the waterbody-specific properties of light propagation [23], [38]. The image quality and statistics also vary depending on visibility conditions, background patterns, and the presence of artificial light sources and unknown objects in a scene. Consequently, learning-based SOD solutions oftentimes fail to generalize beyond supervised data. To address this issue, SVAM-Net adopts a two-step training pipeline (see §III-B) that includes supervision by (i) a large collection of samples with diverse scenes and object categories to learn a generalizable SOD function, and (ii) a wide variety of natural underwater images to learn to capture the inherent optical distortions. In Fig. 11, we demonstrate the robustness of SVAM-Net with a series of challenging test cases.  Table II). From the top: first four images belong to the test sets of SUIM [13] and UFO-120 [38], the next one to MUED [37], whereas the last three images belong to the proposed USOD dataset.
As shown in Fig. 11a, underwater images tend to have a dominating green or blue hue because red wavelengths get absorbed in deep water (as light travels further) [22]. Such wavelength dependent attenuation, scattering, and other optical properties of the waterbodies cause irregular and non-linear distortions which result in low-contrast, often blurred, and color-degraded images [23], [90]. We notice that both SVAM-Net and SVAM-Net Light can overcome the noise and image distortions and successfully localize the salient objects. They are also robust to other pervasive issues such as occlusion and cluttered backgrounds with confusing textures. As Fig. 11b demonstrates, the salient objects are mostly well-segmented from the confusing background pixels having similar colors and textures. Here, we observe that although SVAM-Net Light introduces a few false-positive pixels, SVAM-Net's predictions are rather accurate and fine-grained.
Another important feature of general-purpose SOD models is the ability to identify novel salient objects, particularly with complicated shapes. As shown in Fig. 11c, objects such as wrecked/submerged cars, planes, statues, and cages are accurately segmented by both SVAM-Net and SVAM-Net Light . Their SOD performance is also invariant to the scale and orientation of salient objects. We postulate that the large-scale supervised pre-training step contributes to this robustness as the terrestrial datasets include a variety of object categories. In fact, we find that they also perform reasonably well on arbitrary terrestrial images (see Fig. 11d), which suggest that with domain-specific end-to-end training, SVAM-Net could be effectively used in terrestrial applications as well.

VI. OPERATIONAL FEASIBILITY & USE CASES A. Single-board Deployments
As Table III shows, SVAM-Net offers an end-to-end runtime of 49.82 milliseconds (ms) per-frame, i.e., 20.07 framesper-second (FPS) on a single NVIDIA™ GTX 1080 GPU. Moreover, SVAM-Net Light operates at a much faster rate of 11.60 ms per-frame (86.15 FPS). These inference rates surpass the reported speeds of SOTA SOD models [48], [1] and are adequate for GPU-based use in real-time applications. More importantly, SVAM-Net Light runs at 21.77 FPS rate on a singleboard computer named NVIDIA™ Jetson AGX Xavier with an on-board memory requirement of only 65 MB. These computational aspects make SVAM-Net Light ideally suited for singleboard robotic deployments, and justify our design intuition of decoupling the bottom-up pipeline {e 1:5 → SAM bu } from the SVAM-Net architecture (see §III-C).

B. Practical Use Cases
In the last two sections, we discussed the practicalities involved in designing a generalized underwater SOD model and identified several drawbacks of existing solutions such as QDWD, SUIM-Net, LSM, and Deep SESR. Specifically, we showed that their predicted saliency maps lack important details, exhibit improperly segmented object boundaries, and

SVAM-Net
(a) Lack of contrast and/or color distortions.

SVAM-Net
(c) Unseen objects/shapes and variations in scale.

SVAM-Net
(d) Unseen terrestrial images with arbitrary objects. incur plenty of false-positive pixels (see §IV-C and Fig. 10).
Although such sparse detection of salient pixels can be useful in specific higher-level tasks (e.g., contrast enhancement [38], rough foreground extraction [77]), these models are not as effective for general-purpose SOD. It is evident from our experimental results that the proposed SVAM-Net model overcomes these limitations and offers a robust SOD solution for underwater imagery. For underwater robot vision, in particular, SVAM-Net Light can facilitate faster processing in a host of visual perception tasks. As seen in Fig. 12, we demonstrate its effectiveness for two such important use cases. 1) Salient RoI Enhancement: AUVs and ROVs operating in noisy visual conditions frequently use various image en-hancement models to restore the perceptual image qualities for improved visual perception [90], [91], [92]. However, these models typically have a low-resolution input reception, e.g., 224 × 224, 256 × 256, or 320 × 240. Hence, despite the robustness of SOTA underwater image enhancement models [23], [88], [91], their applicability to high-resolution robotic visual data is limited. For instance, the fastest available model, FUnIE-GAN [23], has an input resolution of 256 × 256, and it takes 20 ms processing time to generate 256 × 256 outputs (on AGX Xavier). As a result, it eventually requires 250 ms to enhance and combine all patches of a 1080×768 input image, which is too slow to be useful in near real-time applications.
An effective alternative is to adopt a salient RoI enhancement mechanism to intelligently enhance useful image regions only. As shown in Fig. 12a, SVAM-Net Light -generated saliency maps are used to pool salient image RoIs, which are then reshaped to convenient image patches for subsequent enhancement. Although this process requires an additional 46 ms of processing time (by SVAM-Net Light ), it is still considerably faster than enhancing the entire image. As demonstrated in Fig. 12a, we can save over 45% processing time even when the salient RoI occupies more than half the input image.
2) Effective Image Super-Resolution: Single image superresolution (SISR) [87], [93] and simultaneous enhancement and super-resolution (SESR) [38] techniques enable visuallyguided robots to zoom into interesting image regions for detailed visual perception. Since performing SISR/SESR on the entire input image is not computationally feasible, the challenge here is to determine which image regions are salient. As shown in Fig. 12b, SVAM-Net Light can be used to find the salient image RoIs for effective SISR/SESR. Moreover, the super-resolution scale (e.g., 2×, 3×, or 4×) can be readily determined based on the shape/pixel-area of a salient RoI. Hence, a class-agnostic SOD module is of paramount importance to gain the operational benefits of image super-resolution, especially in vision-based tasks such as tracking/following fast-moving targets [12], [94], [95] and surveying distant coral reefs [70], [71], [72]. For its computational efficiency and robustness, SVAM-Net Light is an ideal choice to be used alongside a SISR/SESR module in practical applications.

3) Fast Visual Search and Attention Modeling:
In §II-B, we discussed various saliency-guided approaches for fast visual search [6], [9] and spatial attention modeling [73]. Robust identification of salient pixels is the most essential first step in these approaches irrespective of the high-level application-specific tasks, e.g., enhanced object detection [12], [40], [27], place recognition [11], coral reef monitoring [70], [72], autonomous exploration [73], [74], etc. SVAM-Net Light offers a general-purpose solution to this, while ensuring fast inference rates on single-board devices. As shown in Fig. 13, SVAM-Net Light reliably detects humans, robots, wrecks/ruins, instruments, and other salient objects in a scene. Additionally, it accurately discards all background (waterbody) pixels and focuses on salient foreground pixels only. Such precise segmentation of salient image regions enables fast and effective spatial attention modeling, which is key to the operational success of visually-guided underwater robots. (a) Benefits of salient RoI enhancement are shown for two high-resolution input images. On the left: (i) SVAM-Net Light -generated saliency maps are used for RoI pooling, (ii) the salient RoIs are reshaped based on their area, and then (iii) FUnIE-GAN [23] is applied on all 256 × 256 patches; the total processing time is 88 ms for a 512 × 256 RoI (top image) and 131 ms for a 512 × 512 RoI (bottom image). In comparison, as shown on the right, it takes 250 ms to enhance the entire image at 1024 × 768 resolution.  [38] on the salient image RoI is potentially more useful for detailed perception rather than SESR on the entire image. Moreover, as seen on the right, SVAM-Net Light -generated saliency maps can also be used to determine the scale for super-resolution; here, we use 2× and 4× SRDRM [87] on two salient RoIs based on their respective resolutions.

VII. CONCLUDING REMARKS
"Where to look" is a challenging and open problem in underwater robot vision. An essential capability of visuallyguided AUVs is to identify interesting and salient objects in the scene to accurately make important operational decisions. In this paper, we present a novel deep visual model named SVAM-Net, which combines the power of bottom-up and topdown SOD learning in a holistic encoder-decoder architecture. We design dedicated spatial attention modules to effectively exploit the coarse-level and fine-level semantic features along the two learning pathways. In particular, we configure the bottom-up pipeline to extract semantically rich hierarchical features from early encoding layers, which facilitates an abstract yet accurate saliency prediction at a fast rate; we denote this decoupled bottom-up pipeline as SVAM-Net Light . On the other hand, we design a residual refinement module that ensures fine-grained saliency estimation through the deeper top-down pipeline.
In the implementation, we incorporate comprehensive endto-end supervision of SVAM-Net by large-scale diverse training data consisting of both terrestrial and underwater imagery. Subsequently, we validate the effectiveness of its learning components and various loss functions by extensive ablation experiments. In addition to using existing datasets, we release a new challenging test set named USOD for the benchmark evaluation of SVAM-Net and other underwater SOD models.
By a series of qualitative and quantitative analyses, we show that SVAM-Net provides SOTA performance for SOD on underwater imagery, exhibits significantly better generalization performance on challenging test cases than existing solutions, and achieves fast end-to-end inference on single-board devices. Moreover, we demonstrate that a delicate balance between robust performance and computational efficiency makes SVAM-Net Light suitable for real-time use by visually-guided underwater robots. In the near future, we plan to optimize the endto-end SVAM-Net architecture further to achieve a faster runtime. The subsequent pursuit will be to analyze its feasibility in online learning pipelines for task-specific model adaptation.