{'fraction_non_alphanumeric': 0.03093244451670596, 'fraction_numerical': 0.03690756849036664, 'mean_word_length': 5.126245019920319, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modeling Bottom-Up and Top-Down Attention with a Neurodynamic Model of V1

Previous studies in that line suggested that lateral interactions of V1 cells are responsible, among other visual effects, of bottom-up visual attention (alternatively named visual salience or saliency). Our objective is to mimic these connections in the visual system with a neurodynamic network of firing-rate neurons. Early subcortical processes (i.e. retinal and thalamic) are functionally simulated. An implementation of the cortical magnification function is included to define the retinotopical projections towards V1, processing neuronal activity for each distinct view during scene observation. Novel computational definitions of top-down inhibition (in terms of inhibition of return and selection mechanisms), are also proposed to predict attention in Free-Viewing and Visual Search conditions. Results show that our model outpeforms other biologically-inpired models of saliency prediction as well as to predict visual saccade sequences during free viewing. We also show how temporal and spatial characteristics of inhibition of return can improve prediction of saccades, as well as how distinct search strategies (in terms of feature-selective or category-specific inhibition) predict attention at distinct image contexts. Author summary Saliency maps are the representations of how certain visual regions attract attention in a visual scene, and these can be measured with eye movements. A myriad of computational models with artificial and biological inspiration have been able to acquire outstanding predictions of human fixations. However, most of these models have been built specifically for visual saliency, a characteristic that denies their biological plausibility for modeling distinct visual processing mechanisms or other visual processes simultaneously. In addition to saliency, our approach is also able to efficiently work for other tasks (without applying any type of training or optimization and keeping the same parametrization) such as Visual Search, Visual Discomfort [1], Brightness [2] and Color Induction [3]. By performing simulations of human physiology and its mechanisms, we propose to build a unified model that could be extended to predict and understand distinct perceptual processes in which V1 is responsible.

The human visual system (HVS) structure has evolved in a way to efficiently Objectives 56 Initial hypotheses by Li [27,28] suggested that visual saliency is processed by the lateral 57 interactions of V1 cells. In their work, pyramidal cells and interneurons in the primary 58 visual cortex (V1, Brodmann Area 17 or striate cortex) and their horizontal 59 intracortical connections are seen to modulate activity in V1. Li's neurodynamic 60 model [29] of excitatory and inhibitory firing-rate neurons was able to determine how 61 contextual influences of visual scenes contribute to the formation of saliency. In this 62 model, interactions between neurons tuned to specific orientation sensitivities served as 63 predictors of pop-out effects and search asymmetries [30]. Li's neurodynamic model was 64 later extended by Penacchio et al. [2] proposing the aforementioned lateral interactions 65 to also be responsible for brightness induction mechanisms. By considering neuron 66 orientation selectivity at distinct spatial scales, this model can act as a contrast 67 enhancement mechanism of a particular visual area depending of induced activity from 68 surrounding regions. Latest work from Berga & Otazu [31] has shown that the same 69 model (without changing its parametrization) is able to predict saliency using real and 70 synthetic color images. We propose to extend the model providing saliency 71 computations with foveation, concerning distinct viewpoints during scene observation 72 (mapping retinal projections towards V1 retinotopy) as a main hypothesis for predicting 73 visual scanpaths. Furthermore, we also test how the model is able to provide predictions 74 considering recurrent feedback mechanisms of already visited regions, as well as from 75 visual feature and exemplar search tasks with top-down inhibition mechanisms. 76 A unified model of V1 predicts several perceptual processes 77 Here we present a novel neurodynamic model of visual attention and we remark its 78 biological plausability as being able to simultaneously reproduce other effects such as 79 Brightness Induction [2], Chromatic Induction [3] and Visual Discomfort [1] effects. 80 Brightness and Chromatic induction stand for the variation of perceived luminance and 81 color of a visual target depending on its luminance and/or chromatic properties as well 82 as for its surrounding area respectively. Thus, a visual target can be perceived as being 83 different (contrast) or similar (assimilation) to its physical properties by varying its 84 surrounding context. With the simulations of our model, the output of V1's neuronal 85 activity (coded as firing-rates), after several cycles of excitatory-inhibitory V1 86 interneuron interactions, is used as predictors of induction and saliency respectively. 87 These responses will act as a contrast enhancement mechanism, which for the case of 88 saliency, are integrated towards projections in the superior colliculus (SC) for eye 89 movement control. Therewith, our model has also been able to reproduce visual 90 discomfort, as relative contrast energy of particular region on a scene is found to produce 91 hyperexcitability in V1 [32,33], one of possible causes of producing certain conditions 92 such as malaise, nausea or even migraine. Previous neurodynamic [34][35][36][37][38] and saliency 93 models [11,12] are able to reproduce attention processes and predict eye movements [39] 94 but are uniquely presented to work for that specific task. On behalf of model biological 95 plasusibility on V1 function and its computations, we present a unified model of lateral 96 connections in V1, able to predict attention from real and synthetic color images while 97 mimicking physiological properties of the neural circuitry stated previously. 98 Model 99 Retinal and LGN responses 100 The HVS perceives to light at distinct wavelengths of the visual spectrum and separates 101 them to distinct channels for further processing in the cortex. First, retinal photoreceptors (or RP, corresponding to rod and cone cells) are photosensitive to 103 luminance (rhodopsin-pigmented) and color (photopsin-pigmented) [40,41]. Mammal 104 cone cells are photosensitive to distinct wavelengths between a range of ∼ 400-700nm, 105 corresponding to three cell types, measured to be maximally responsive to Long (L, 106 λ max 560nm), Medium (M, λ max 530nm) and Short (S, λ max 430nm) 107 wavelengths respectively [42]. RP signals are received by retinal ganglion cells (or RGC) 108 forming an opponent process [43]. This opponent process allows to model midget, 109 bistratified and parasol cells as "Red vs Green", "Blue vs Yellow", and "Light vs Dark" 110 channels. In order to simulate these chromatic and light intensity opponencies using 111 digital images, we transformed the RGB color space to the CIELAB (Lab or L * a * b * ) 112 space (including a gamma correction of γ RGB =1/2.2), as exemplified in Figure 1.

Fig 2.
Representation of wavelet coefficients (ω isoθ ), in conjunction with the output of "a-trous" wavelet transform applied to components (o = L * , a * , b * ) shown in Figure 1.
We modeled V1's simple cell responses with a 2D "a-trous" wavelet transform [45]. 133 Discrete wavelet transforms allow to process signals by extracting information of 134 orientation and scale-dependent features in the visual space, filtering each of the 135 aforementioned opponencies. By building feature maps of orientation sensitivities at 136 distinct spatial frequencies, it is possible to represent V1 RF input activity (which we 137 applied separately to each pathway of the LGN), shown in Figure 2. The "a-trous" 138 transform is undecimated and allows to perform a transform where its basis functions 139 remain similar to Gabor filters. The "a trous" wavelet transform can be defined as: By transposing the wavelet filter (h s , expressed in Fig. 2 visual field send inputs of the LGN towards the right hemifield of V1, similarly for the 168 case of the right visual field to the left hemifield of V1 (Figure 3-Right).   Figure 3-Left). The visual space is transformed to a cortically-magnified 172 space (with its correspondence of millimeter for each degree of visual angle) with a 173 logarithmic mapping function. The pixel-wise cartesian visual space is transformed to 174 polar coordinates in terms of eccentricity and azimuth for a specific foveation instance, 175 then transformed to coordinates in mm of cortical space. Acknowledging that the visual 176 space for digital images is represented with either a squared or rectangular shape, we 177 computed the continuation of cortical coordinates by symmetrically mirroring existing 178 coordinates of the image with their correspondence of visual space outside boundaries in 179 the cortical space. In that manner, we exclude possible effects of zero-padding over 180 recurrent processing while preserving 2D shapes for our feature representations. For this 181 case, these effects were minimized by the inverse and repeating the same process at 182 specific interaction cycles. Schwartz's mapping has been applied over the wavelet 183 coefficients represented in Figure 2, as basis functions are convolved in the visual space, 184 later magnified to the cortical space for representing V1 signals. These signals will serve 185 as input to excitatory pyramidal cells, projected to their respective iso-orientation 186 domains at distinct RF sizes.

188
Li's hypotheses suggest that V1 computations are responsible of generating a bottom-up 189 saliency map [27,28]. These hypotheses state that intracortical interactions between  Our excitatory-inhibitory model is described in Table 1. Horizontal connections 208 (lateral and reciprocal) are schematized in Figure 4 and Table 1C, where excitatory cells 209 have self-directed (J 0 ) and monosynaptic connections (J) between each other, whereas 210 dysynaptically connected through (W ) inhibitory interneurons. Axonal field projections 211 follow a concentric toroid of radius ∆ s = 15 × 2 s−1 and radial distance ∆ θ (accounting 212 for RF size d s and radial distance β). Membrane potentials of excitatory (ẋ isθ ) and 213 inhibitory (ẏ isθ ) cells are obtained with partial derivative equations defined in Table 1D, 214 composed by a chain of functions that consider firing-rates (obtained by piece-wise 215 linear functions g x and g y ) and membrane potentials from previous membrane cycles 216 (modulated by α x , α y constants), current lateral connection potentials (J and W ) and 217 spread of inhibitory activity within hypercolumns (ψ). Background inputs (I noise and 218 I norm ) correspond to simulating random noise and divisive normalization signals (i.e.

219
accounting for local nonorientation-specific cortical normalization and nonlinearities).

Combining the output of all components by
we can describe the changes of the model (resulting from the simulated lateral 231 interactions of V1) with respect the original wavelet coefficients ω t isoθ . Our result (S t i;o ) 232 will define the saliency map as an average conspicuity map or feature-wise 233 distinctiveness (RF firing rates across scales and orientations for each pathway). These 234 changes in firing-rate alternatively define the contrast enhancement seen on the 235 brightness and chromatic induction cases [1][2][3] Latest hypotheses about neural correlates of saliency [56,57] state that the superior 242 colliculus is responsible for encoding visual saliency and to guide eye movements [23,58]. 243 Acknowledging that the superficial layers of the SC (sSC) receive inputs from the early 244 stages of visual processing (V1, retina), the SC selects these as the root of bottom-up 245 activity to be selected in the intermediate and deep layers (iSC, dSC). In accordance to 246 the previous stated hypotheses [27], saccadic eye movements modulated by saliency The behavioral quantity of the unique 2D saliency map has been defined by . External current in lower (I) or higher (I c ) cortical areas and random noise (I 0 ) Measurements Firing-rate (g x and g y ) B Populations Name Elements Description Sensory    Top-down selection Goal-directed or memory-guided saccades imply executive 292 control mechanisms that account for task requirements during stimulus perception. The 293 dorsolateral prefrontal cortex (DLPFC) is known to be responsible for short-term 294 spatial memory, to retrieve long-term memory signals of object representations (through 295 projections towards the para-and hippocampal formations) as well as to perform 296 reflective saccade inhibition, among other functions. These inhibitory signals, later 297 projected to the frontal eye field (FEF), are able to direct gaze during search and 298 smooth pursuit tasks [63,67,68] (also suggested to be crucial for planning intentional or 299 endogenously-guided saccades), where its signals are sent to the SC. By feeding our 300 model with inhibitory signals (I c shown in Figure 4 and Table 1E) we can simulate 301 top-down feedback control mechanisms in V1 (initially proposed by Li [29,Sec. 3.7]). In 302 this case, a new term I {vs} is added to the top-down inhibition of our V1 cortical 303 signals that will be projected to the SC during each gaze.
In this implementation, we can perform distinct search tasks such as feature search 305 (by manually selecting the features, or selecting features with maximal responses, 306 similarly to a boolean selection [26]), exemplar and categorical object search (by 307 processing the mean of responsesω from wavelet coefficients of a single or several image 308 samples "N"). These low-level computations would serve as cortical activations to be 309 stored as weights in our low-level memory representations, that will be used as 310 inhibitory modulation for the task execution.

311
Inhibition of Return During scene viewing, saccadic eye movements show distinct 312 patterns of fixations [69], directed by exploratory purposes or either towards putting the 313 attentional focus on specific objects in the scene. For the former case, the HVS needs to 314 ignore already visited regions (triggering anti-saccades away from these memorized 315 regions, as a consequence of inhibition) during a period of time before gazing again 316 towards them. This phenomena is named inhibition of return [70], and similarly involves 317 extracting sensory information and short-term memory during scene perception. As 318 mentioned before, DLPFC is responsible of memory-guided saccades, and this function 319 might be done in conjunction with the parietal cortex and the FEF. The parietal areas 320 (LIP and PEF) [63,67,71] are known to be responsible of visuospatial integration and 321 preparation of saccade sequences. These areas conjunctively interact with the FEF and 322 DLPFC for planning these reflexive visually-guided saccades. Acknowledging that LIP 323 receives inputs from FEF and DLPFC, the role of each cannot be disentangled as a 324 unique functional correlate for the IoR. Following the above, we have modeled return 325 mechanisms as top-down cortical inhibition feedback control accounting for 326 previously-viewed saccade locations. Thus, we added an inhibition input I {IoR} at the 327 start of each saccade, which will determine our IoR mechanism: This term is modulated with a constant power factor α {IoR} and a decay factor 329 β {IoR} , which in every cycle will progressively reduce inhibition. The spatial region of 330 the IoR has been defined as a gaussian function centered to the previous gaze (g), with 331 a spatial standard deviation σ {IoR} dependent on a specific spatial scale and a peak 332 with an amplitude of the maximal RF firing rate of our model's output (Ŝ). Inhibitory 333 activity is accumulated to the same map and can be shown how is progressively reduced 334 during viewing time (Fig. 14).   These are calculated using euclidean distance between fixation coordinates (distance 382 between saccade length for SA and distance between locations of saccades for SL).

383
Initial investigations on visual attention [7,8] during visual search tasks formulated 384 that reaction times of finding a target (defined in a region of interest/ROI) among a set 385 of distractors are dependent on set size as well as target-distractor feature contrast. In 386 May 6, 2019 13/32 order to evaluate performance on visual search, we utilised two metrics that account for 387 the ground truth mask of specific regions for search and the saliency map (in this 388 context, it could be considered as a "relevance" map) or predicted saccade coordinates 389 (from locations with highest neuronal activity). The Saliency Index (SI) [20,77]   Based on the shuffled metric scores, traditional saliency models such as AIM overall 402 score higher on real scene images (Fig 7), scoring sAU C AIM =.663, and    better performance than other biologically-inspired ones accounting for these basis. prediction errors might arise from not correctly predicting focal fixations. 475 We simulated the inhibition factor for all datasets by substracting the inhibition 476 factor I {IoR} to our models' saliency maps (NSWAM+IoR). After computing prediction 477 errors in SA and SL for a single sample (Fig. 15-Top

491
Our model predictions on SA correlate better (i.e. obtain higher ρSA values) than other 492 scanpath models (in terms of how SA evolves over fixations), however, prediction errors 493 are higher in both SL and SA. We believe that these errors are caused by incorrectly 494 predicting locations of fixations, but not for failing on predictions of the saccade 495 sequence per se. These locations are mainly influenced by systematic tendencies in 496 free-viewing (derived by center biases and/or focal fixations in a particular region of the 497 image). Cortical magnification mechanisms might be responsible for processing higher 498 saliency at regions outside the fovea, generating tendencies of uniquely capturing large 499 saccades. These can be solved by processing high-level feature computations near the 500 fovea, which would increase the probability of fixations at lower SA. We have to hesitate 501 that first fixations are long known for being determinants of bottom-up 502 attention [20,80]. Instead, higher inter-participant differences [79] and center biases [83] 503 increase as functions of fixation number, suggested as worse candidates for predicting 504 attention. These parameters appear to specifically affect each stimuli differently (and 505 accounting that each stimulus may convey specific semantic importance between each 506 contextual element), which may relate to top-down attention but not to the image 507 characteristics per se. We also want to stress the importance of foveation in our model. 508 This is a major procedure for determining saccade characteristics (including oculomotor 509 tendencies) and saliency computations, as it determines current human actions during 510 scene visualization. The decrease of spatial resolution at increasing eccentricity provides 511 the aforementioned properties, innate in human vision and invariant to scene semantics. 512 Adding an IoR mechanism has been seen to affect model activity and therefore 513 scanpath predictions. In Fig. 14  We illustrated results of PFI and SI (Fig. 18) in relation to relative target-distractor 549 feature contrast for cases of Brigthness, Color and Size differences, as well as the Set  implicit when processing image ROI average characteristics but not when using maximal 565 activations, qualitatively shown in Fig. 17. The fact that SI scores are lower for 566 free-viewing tasks in pop-out stimuli might be caused from influences of the center bias, 567 presenting more fixations near the center in free-viewing [20]. Search in psychophysical 568 image patterns is significatively more efficient in SI when selecting maximal feature 569 activations (NSWAM+V S M ). Regarding that aspect, exemplar and categorical search 570 for objects in real image scenes would require computations with a higher number of 571 features [84,85] (which would represent in more detail each cortical cell sensitivity).
572 Fig 19. Performance on visual search evaluated on each distinct low-level feature, stimulus instances are from SID4VAM's dataset [20].

573
Current implementation of our V1 model is based on Li's excitatory-inhibitory firing 574 rate network [29], following previous hypotheses of pyramidal and interneuron 575 connectivity for orientation selectivity in V1 [54,55]. To support and extend this 576 hypothesis, distinct connectivity schemas (following up V1 cell subtype 577 characterization) [86,87] could be tested (e.g. adding dysynaptic connections between 578 inhibitory interneurons) to better understand V1 intra-cortical computations.

579
Furthermore, modeling intra-layer interactions of V1 cells [43] could explain how visual 580 information is parallely processed and integrated by simple and complex cells [84], how 581 distinct chromatic opponencies (P-,K-and M-) are computed at each layer [88], and 582 how V1 responses affect SC activity (i.e. from layer 5) [89]. Testing contributions of 583 each of these chromatic pathways (at distinct single/double opponencies and polarities), 584 as well as distinct fusion mechanisms regarding feature integration, would define a more 585 detailed description of how visual features affect saliency map predictions.

586
Previous and current scanpath model predictions could be considered to be 587 insufficient due to the scene complexity and numerous factors (such as the task 588 specificity, scene semantics, etc.) simultaneously involved in saccade programming.

589
These factors increase overall errors on scanpath predictions, as systematic tendencies 590 increase over time [20,22,79,83] We suggest that not all fixations should have the same importance when evaluating 605 saliency predictions. Nature and synthetic scene images lack of semantic (man-made) 606 information, which might contribute to the aforementioned voluntary (top-down guided) 607 eye movements [91]. Acknowledging that objects are usually composed by the Log-Gabor filters [94]. In that regard, angle configuration pop-out effects and contour 630 detection computations [95,96] can be done by changing neuron connectivity and 631 orientation tuning modulations. 632 We aim in future implementations to model the impact of feedback in 633 cortico-cortical interactions with respect striate and extrastriate areas in the HVS.

634
Some of these regions project directly to SC, including the intermediate areas (pulvinar 635 and medial dorsal) and basal ganglia [23,63,67]. Our current implementation can be 636 extended with a large scale network of spiking neurons [97], also being able to learn 637 certain image patterns through spike-timing dependent plasticity mechanisms [98]. inhibition of return mechanisms). We have shown how scanpath predictions improve by 659 parametrizing the inhibition of return, with highest performance at a size of 2 deg and a 660 decay time between 1 and 5 fixations. By processing low-level feature representations of 661 real images (considering statistics of wavelet coefficients for each object or feature 662 exemplar) and using them as top-down cues, we have been able to perform feature and 663 object search using the same computational architecture. Two search strategies are 664 presented, and we show that both the probability to gaze inside a ROI and the amount 665 of fixations inside that ROI increase with respect saliency. In previous studies, the same 666 model has been able to reproduce brightness [2] and chromatic [3] induction, as well as 667 explaining V1 cortical hyperexcitability as a indicator of visual discomfort [1]. With the 668 same parameters and without any type of training or optimization, NSWAM is also able 669 predict bottom-up and top-down attention for free-viewing and visual search tasks. 670 Model characteristics has been constrained (in both architecture and parametrization) 671 with human physiology and visual psychophysics, and can be considered as a simplified 672 and unified simulation of how low-level visual processes occur in the HVS.