{'fraction_non_alphanumeric': 0.03521909744931328, 'fraction_numerical': 0.01788750817527796, 'mean_word_length': 4.782000378143317, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction

Many online action prediction models observe complete frames to locate and attend to informative subregions in the frames called glimpses and recognize an ongoing action based on global and local information. However, in applications with constrained resources, an agent may not be able to observe the complete frame, yet must still locate useful glimpses to predict an incomplete action based on local information only. In this paper, we develop Glimpse Transformers (GliTr), which observe only narrow glimpses at all times, thus predicting an ongoing action and the following most informative glimpse location based on the partial spatiotemporal information collected so far. In the absence of a ground truth for the optimal glimpse locations for action recognition, we train GliTr using a novel spatiotemporal consistency objective: We require GliTr to attend to the glimpses with features similar to the corresponding complete frames (i.e. spatial consistency) and the resultant class logits at time t equivalent to the ones predicted using whole frames up to t (i.e. temporal consistency). Inclusion of our proposed consistency objective yields ~10% higher accuracy on the Something-Something-v2 (SSv2) dataset than the baseline cross-entropy objective. Overall, despite observing only ~33% of the total area per frame, GliTr achieves 53.02%and 93.91% accuracy on the SSv2 and Jester datasets, respectively.


Introduction
Recent models such as TSM [37], Swin-B [38], or VideoMAE [53] have achieved impressive performance on video action recognition benchmarks, but they often make several assumptions that limit their use for certain applications. For example, the aforementioned models operate in an offline manner, assuming the full clip (i.e. after the action has concluded) is available to make a decision. Offline models are often inefficient in online settings, where action recognition must be performed based on the incomplete clip seen up until the current time. For example, the performance of Swin-B drops by ∼30% on Figure 1: We propose Glimpse Transformers (GliTr), an online action prediction model that only attends to the most informative glimpses (g t ) in the frames (x t ). While never observing frames completely, GliTr predicts labelŷ t (i.e. an estimate of ongoing action at time t) and the next glimpse locationl t+1 based solely on the glimpses observed up to t.
Another common assumption is the requirement of complete spatial information over time. But, due to spatial redundancy, it is enough to observe only small but informative subregions of the full frames to make an accurate prediction. Several approaches [6,25,40,62,64] primarily process narrow regions called "glimpses". However, these approaches still require the entire frame to determine informative glimpses. While using a lightweight model for this "global" view reduces the overall computational cost, it still requires having a wide field of view initially, which does not come free. High-resolution, large FOV cameras are expensive, require more power, and consume more bandwidth to transmit data. It is essential to minimize such costs in certain high-risk time-sensitive applications, such as mobile drones for disaster recovery, monitoring at-risk animals in the wild, or real-time translation of sign language.
We thus develop an inexpensive model that predicts informative glimpse locations without observing whole frames, therefore obviating the need for high-resolution, large FOV cameras. Starting from a glimpse at a given location, our model decides which location to attend to in subsequent frames solely based on previously observed glimpses. Consequently, our model predicts an action using only the local information and in an online fashion. We choose transformers [57] to learn glimpse-based attention mechanism and action prediction, as they can efficiently encode the relations between spatially and temporally distant glimpses. We thus call our model Glimpse Transformers (GliTr). Following a factorized encoder architecture [2], we use a) a spatial encoder that solely models relations between the patches from a single glimpse to predict spatial features, and b) two temporal encoders that model interactions between various glimpse features across time to predict the class label and the next glimpse location, respectively.
Since the ground truth for optimal glimpse locations is unavailable, we propose a novel spatiotemporal teacherstudent consistency objective to incentivize GliTr to learn glimpse location in a weakly supervised manner. With only glimpses, GliTr (as the student model) is trained to reproduce the spatial features and class distribution of a teacher model ingesting the complete frames of the video. As the teacher learns to produce predictive features and logits for the downstream task of online action recognition from the full frames, enforcing this consistency loss on the student model implicitly requires focusing attention on the most informative regions, leading to learning a glimpse mechanism. We demonstrate GliTr's effectiveness on Something-Something-v2 [22] and Jester [41] datasets. Our main contributions are as follows.
• We develop GliTr: an online action prediction model that observes only glimpses and predicts ongoing action based on partial spatiotemporal observations. While previous works locate glimpses by first observing full frames, GliTr predicts the next informative glimpse location solely based on the past glimpses.
• We propose a novel spatiotemporal consistency objective to train GliTr without the ground truth for glimpse location. Under this objective, GliTr must select glimpses that summarize features and class distribution predicted from the entire frames. Our proposed consistency yields ∼ 10% gain in accuracy on SSv2 compared to the baseline cross-entropy objective.
• Our model that never observes complete frames and recognizes action solely based on local information gathered through glimpses achieves nearly 53% and 94% accuracy on SSv2 and Jester dataset, respectively, while reducing the total area observed per frame by nearly 67% (with the glimpses of size 128×128 extracted from frames of size 224×224).

Related Works
Online Action Recognition. Many state-of-the-art methods perform offline action recognition once the entire video is available [17,19,10,27,55,56,59,60]. However, these methods are not optimized for the case where the entire video is not yet available, and the model has to predict the action based on a preliminary, incomplete video. Performing an online or early action recognition based on a partially observed video is a challenging task. A partially observed video may associate with multiple possible actions, leading to the inherent uncertainty in the prediction task. Several methods focus on predicting actions from partial videos. Zhao et al. [69], Wu et al. [65], and Pang et al. [43] anticipate future actions based on the motion and object relations in the past frames. Many analyze micro-motions in the available early frames [52,34,30,28]. Other approaches such as dynamic bag-of-words [49], global-local saliency [32], memorizing hard-to-predict samples [29], soft regression with multiple soft labels [24], and probabilistic modeling [35,9] are also used. While the existing online action recognition methods focus on partial observation in the temporal dimension, we focus on partial information in the temporal as well as the spatial dimension.
Spatial Selection for Action Recognition. Spatial selection is typically performed using hard attention [42]. As opposed to soft attention models [68] that observe all regions of the scene with varying attention level, hard attention models sequentially attend to the most informative glimpses. Hard attention is widely used for image classification [3,16,42,68,44,48,47,63].
Recently, hard attention has also been applied to video action recognition. Wang et al. propose online action recognition model called Adafocus [62,64]. Chen et al. [11], Huang et al. [25] and Wang et al. [58] present offline models that first observe the entire video in order to predict attention-worthy glimpse locations. Mac et al. [40] and Baradel et al. [6] also present offline models but locate and observe multiple informative glimpses per-frame. Another line of approach leverages pose information and focuses only on the relevant body parts [5,13]. The previous approaches, irrespective of their online or offline nature, access full frames to locate informative glimpses. In contrast, our model never observes complete frames; it only observes a narrow glimpse from each frame.
Consistency Learning. Consistency is widely used for the problem of semi-supervised learning [50,51,66,7,33]. The idea is to force the output of the model to be invariant to different augmentations of the same input [51,66,7,36], or variations in the internal representations [4,50], or the model parameters at different training epochs [33]. Another related approach is pseudo-labeling [67,45], where a separate teacher model generates pseudo-labels for unlabeled samples under no perturbations, and the student model is trained to predict the pseudo-labels under some perturbations. This approach is similar to Knowledge Distillation Figure 2: An overview of our GliTr. GliTr consists of a frame-level spatial transformer T f and causal temporal transformers T c and T l . One training iteration requires T forward passes through our model. Above, we show two consecutive forward passes at time t ≤ T − 1 and t + 1 ≤ T . Forward pass t (blue path): Given a new glimpse g t , T f extracts glimpse-featureŝ f t . We appendf t tof 1:t−1 , i.e. features extracted from g 1:t−1 during previous passes. Next, T c predicts labelŷ t fromf 1:t . Simultaneously, T l predicts next glimpse locationl t+1 fromf 1:t . Forward pass t+1 (orange path): Given a predicted location l t+1 , we extract a glimpse g t+1 atl t+1 from a frame x t+1 . Then, we follow the same steps as the blue path. After T forward passes, we compute the losses shown in the right. To find targetsỹ 1:T andf 1:T for spatial and temporal consistency, we use a separate pre-trained and fixed teacher model (shown on the left and explained in Figure 3) that observes complete frames x 1:T . To maintain stability, we stop gradients from T l to T f . [23], where the student is trained to reconstruct the output or internal representation [1] of the teacher.
Many early action recognition models learn to predict the class distribution consistent with the complete video using only a subset of early frames [8,20,31,46,61]. Others have also leveraged spatiotemporal consistency for complete frames [53,18]. Inspired by previous work, we use a teacher model that predicts features from complete frames and predicts class distribution in an online fashion. Our student model observes partial spatiotemporal information and tries to predict features and class distribution consistent with the teacher model.

Models
We use a teacher model to i) initialize our GliTr -a student model and ii) compute targets for the spatiotemporal consistency objective used for training GliTr. We discuss our teacher model in Sec 3.1 followed by GliTr in Sec 3.2. We crown the quantities computed by our models using complete frames and glimpses with (˜) and (ˆ), respectively.

Teacher
Given spatially complete frames x 1:t from a preliminary video at time t ≤ T , our online teacher model predicts y t , an early estimate of true action y GT . We adapt factorized transformers encoder architecture [2] for our teacher model, and aggregate spatial and temporal information sequentially. It includes the following components.
Feature Extraction (T f ). We use a spatial transformer T f to extract featuresf t from each individual frame x t for all t. We use the ViT architecture [57,54] without the final classification head and collect features from the output corresponding to the input class token.
Early Action Prediction (T c ). We use a temporal transformer T c to aggregate featuresf 1:t and predict labelỹ t . Since transformers are permutation invariant, we enforce order in the input sequence using temporal position embeddings. Moreover, we do not use a separate class token and pass the output corresponding tof t to the linear classifier to predictỹ t . Further, to reduce training time, we use causal attention masking [21,12]. Hence, during training, T c observesf 1:T and producesỹ 1:T in a single forward pass while aggregating features in an online progressive manner, referencing onlyf 1:t to produce outputỹ t at index t.
Glimpse Location Prediction (T l ). We include temporal transformer T l to predict glimpse locationl t+1 fromf 1:t . T l has the same architecture as T c , except the final linear classifier is replaced by a linear regression head to predict coordinatesl t+1 . Though not required for online action prediction from full frames, we train T l to initialize the corresponding module in our student model. Once the student model is initialized, we discard T l from the teacher model. Saveft. 8: end for

Glimpse Transformer (GliTr) -Student
Our Glimpse Transformer (GliTr) is derived and initialized from the teacher model discussed in Sec 3.1. It is an iterative model that actively locates and attends to narrow glimpses in a scene and predicts an ongoing action early based on spatially and temporally incomplete observations. At time t, GliTr senses a new glimpse g t at locationl t from frame x t . Using glimpses g 1:t , it predicts i)ŷ t , an early approximation of label y GT and ii)l t+1 , location of the next glimpse. We display schematics of GliTr in Figure 1. We illustrate GliTr's operation in Algorithm 1 and Figure 2. It consists of the following components.
Glimpse Extraction. Given a locationl t = (i, j), we crop a glimpse g t centered at locationl t in frame x t . To maintain differentiability through the cropping operation, we use a spatial transformer network (STN) [26] 1 .
Feature Extraction (T f ). Similar to the teacher model, we use T f to extract featuresf t from glimpse g t . We derive position embeddings for patches in g t using STN.
Early Action Prediction (T c ). We input glimpse featureŝ f 1:t to T c which in turn predicts class labelŷ t .
Glimpse Location Prediction (T l ). Similarly, we pass the featuresf 1:t to T l which predicts next glimpse locationl t+1 .

Training Objectives
We discuss training objectives for GliTr in Sec 4.1. Considering GliTr as the downstream model, we design training objectives suitable for our teacher model in Sec 4.2. We crown training objectives of GliTr and the teacher model with ( ) and ( ), respectively.

Glimpse Transformer (GliTr) -Student
Classification Loss. Since our goal is to predict action label y GT early using the spatially and temporally incomplete video, we minimize the cross-entropy loss given by 1 Not to be confused with (spatial) Vision Transformers (ViT) [15].
Spatial Consistency Loss. We require GliTr to attend to the glimpses that produce features as predictive of the action as the ones predicted using complete frames by our teacher model. Hence, we minimize the mean squared error (MSE) between the glimpse featuresf t predicted by GliTr and the frame featuresf t predicted by our teacher model, which is Temporal Consistency Loss. While the teacher model has all instantaneous spatial information available in a complete frame, GliTr must rely on past glimpses to reason about the unobserved yet informative regions in the current frame. To incentivize GliTr to aggregate spatial information from the past to mitigate partial observability, we minimize the KLdivergence between the class logits predicted by GliTr using glimpses (ŷ t ) and the teacher using complete frames (ỹ t ), yielding Our final training objective for GliTr is the following:

Teacher
Classification loss. For all t, we minimize cross-entropy loss between the predictionỹ t and the ground-truth label y GT of the action, L cls = CCE(ỹ 1:T , y GT )/T.
Distillation loss. When available, we also use a more powerful offline action recognition model such as Video-MAE [53] to predict action y of f line T from a complete video, i.e. x 1:T . Then, we minimize the KL-divergence between the final predictionỹ T and the above y of f line T given by Spatiotemporal Consistency losses. Note that the above two losses train only T f and T c . We use the following strategy to train T l . First, we use the locationsl 1 (learnable parameter) andl 2:T predicted by T l , to extract glimpses g 1:T from frames x 1:T . Next, we create copies of T f and T c denoted as T ′ f and T ′ c . We input g 1:T and the corresponding position embeddings to T ′ f and predict glimpse featureŝ L temporal = KLD(ŷ 1:T ,ỹ 1:T )/T. Step 1 (blue path): Given complete video frames x 1:T , T f extracts frame featuresf 1:T . Next, T c and T l predict class labelsỹ 1:T and glimpse locationsl 2:T +1 fromf 1:T , respectively. We discardl T +1 .
Step 2 (orange path): Givenl 1 (learnable parameter) and l 2:T (predicted in step 1), we extract glimpses g 1:T from x 1:T . Then, we create non-learnable copies of T f and T c denoted as T ′ f and T ′ c . T ′ f extracts glimpse-featuresf 1:T from g 1:T and T ′ c predicts labelsŷ 1:T fromf 1:T . We compute losses shown on the right and update model parameters. To achieve stability during training, we stop gradients from T l to T f . We use the above two losses to update parameters of T l only. We design these consistency objectives based on the spatiotemporal consistency objectives of GliTr (equations 2 and 3). As discussed in Sec 4.1, they encourage T l to locate glimpses covering the most useful task-relevant regions in the frames, but based on complete frames observed in the past. We demonstrate the training procedure in Figure 3.
The final objective for our teacher model is as follows.

Experiments
Datasets. We experiment with two publicly available largescale real-world datasets, namely, Something-Something-v2 (SSv2) [22] and Jester [41]. We adopt the official training-validation splits. SSv2 dataset contains videos recording 174 human actions using everyday objects. There are ∼170K videos for training and ∼25K for validation. Jester dataset is a collection of videos capturing 27 basic hand gestures, consisting of ∼120K videos for training and ∼15K videos for validation.
Implementation. We sample a sequence of 16 and 8 frames per video from SSv2 and Jester, respectively. We resize each frame to size 224 × 224 and use glimpses of size 96 × 96, unless stated otherwise. We use ViT-Small [54] architecture for T f . For T c and T l , we use a custom transformers architecture with 768 embedding dimensions, 6 heads, and a depth of 4.
Optimization. First, we discuss the common setting followed by a model-specific setting. For all models and datasets, we use the same data augmentation scheme as the one used for VideoMAE [53]. Similar to Wang et al. [64], we stop gradients from T l to T f to maintain stability during training. We use AdamW optimizer [39] with weight decay of 5e-2 and cosine learning rate schedule with no warmup unless stated otherwise. We run experiments for SSv2 and Jester on 4 A100 GPUs with 40 GB of memory and 4 V100-SXM2 GPUs with 32 GB of memory, respectively.
To train a teacher model on SSv2 dataset, we initialize T f using an open-source ViT-S model [71] pretrained on the ImageNet dataset [14], and initialize T c and T l randomly. We form a mini-batch using b = 60 videos and use an initial learning rate of αb 128 , with base learning rate α being 1e-5, 1e-4 and 1e-4 for T f , T c and T l , respectively. We train the teacher model for 40 epochs with a warmup of 15 epochs for T l . For the Jester dataset, we initialize the teacher model with the teacher model trained on the SSv2 dataset. We do not use distillation loss L dist for Jester dataset. We use a batch size b of 100 and α of 1e-5 for all modules. The model is trained for 50 epochs.
Each student model (GliTr) is initialized from a teacher model trained on the corresponding dataset. We use base learning rate α = 1e-5 for all modules and train them for 100 and 150 epochs with a batch-size b of 360 and 800 videos from SSv2 and Jester, respectively.

Empirical Comparisons Glimpse Mechanisms Under Partial Observability
We compare the glimpse attention strategy learned by GliTr with four baselines and an approximate upper bound: • Uniform random: Glimpse locations are independently To isolate the glimpse strategy's effect on performance, we evaluate the glimpses selected by various strategies using the same model i.e. GliTr. While assessing the baselines and the upper bound, we ignore predictions from T l and instead use locations given by the specific strategies described above. We show results in Figure 5, plotting online action prediction accuracy after each t. As expected, the prediction accuracy for all strategies increases as the model observes more glimpses. The Center and the Bottom Left strategies outperform other baselines on SSv2 and Jester datasets, respectively. We suspect this is because the object of interest frequently appears in the center in SSv2; while in most examples from Jester, hand movements begin and end in the region near the bottom left corner of the frames. On the other hand, GliTr outperforms all baselines and achieves performance closest to the upper bound (i.e. the Teacher strategy). We plot a histogram of glimpse regions selected by GliTr in Figure 6. We observe that not only does GliTr successfully capture different biases (center vs. bottom left) in the two datasets, but it also ignores the bias if necessary. Notice the spread in the histograms for t > 1, suggesting GliTr observes various regions in different videos. Consequently, GliTr achieves better accuracy faster than the baselines, and at time T , outperforms the best performing baselines with the respective margins of nearly 5% and 11% on (a) (b) Figure 6: Histograms of the glimpse regions selected by GliTr with increasing time (raster scan order) on (a) SSv2 and (b) Jester. Recall that GliTr observes the first glimpse at a predetermined location followed by active selection.
SSv2 and Jester. We visualize glimpses selected by GliTr on example videos from SSv2 and Jester in Figure 4.

Models with Complete Spatial Observability
Glimpse-based offline models. We compare our GliTr with previous glimpse-based offline action recognition models in Table 1. We note that a direct comparison between these approaches is unfair since previous models  Table 1: Comparison with glimpse-based action recognition models. We count the number of pixels sensed by different approaches to perform recognition. Previous approaches are offline and use complete frames to locate informative glimpses and to recognize actions. GliTr is an online model and only observes glimpses, not complete frames. GliTr achieves competitive performance with a significant saving in the total area observed. ⋄ AdaFocus [62] and AdaFocusV2 [64] first observe 8 frames to locate useful glimpses and then sample additional 12 frames to extract glimpses, which requires sensing total 20 frames in advance due to their offline nature. § Results are based on Figure 13 from [25]. ⋆ GFNet observes two glimpses per frame. For comparison with online methods, refer to Figure 7.
also observe complete frames. Further, unlike offline approaches that initially observe a complete video and select an informative glimpse at t based on the current, past, and future frames, our GliTr -an online model -relies only on the past information to locate glimpses in the current frame. Moreover, previous methods use global information gathered from complete frames to locate glimpses and predict actions; however, GliTr only uses local information. Nevertheless, we include this analysis to highlight the savings achieved by GliTr in terms of the amount of area observed for recognition while still achieving competitive performance with partial observations.
We calculate and compare the number of pixels sensed by various methods to perform action recognition. AdaFocus [62] and AdaFocusV2 [64] uniformly sample 8 frames from a complete video to predict glimpse locations, followed by uniform sampling of another 12 frames to extract glimpses. In total, they require sensing 20 complete frames (20×(224×224)≈1M pixels) in advance due to their offline nature. GFNet [25], on the other hand, locates and extracts glimpses from the same set of complete frames. When compared to AdaFocusV2 with glimpses of size 128 × 128, our GliTr reduces the amount of sensing by nearly 74% and 87% while compromising only around 6% and 3% accuracy on SSv2 and Jester, respectively. Further, while GFNet outperforms GliTr by nearly 14.4% and 4.7% with glimpses of size 96×96 on SSv2 and Jester, GliTr (with 16 and 8 glimpses, respectively) reduces the amount of sensing by nearly 82% and 88% compared to GFNet (with 16 and 12 frames, respectively) on these datasets. We emphasize that GFNet observes full frames and two glimpses per frame in an offline manner, while GliTr observes only one glimpse per frame in an online fashion.  [38], TemPr [52] and TRN [70] predict action early based on complete frames, GliTr predicts action based on early glimpses.
Early action prediction models. We additionally compare GliTr with early action prediction models in Figure 7. We emphasize that these approaches observe entire frames (i.e. global information) from a preliminary video; whereas, GliTr observes frames only partially through glimpses (i.e. local information). For SSv2 dataset, we consider Swin-B [38] and TemPr [52]. We cite Swin-B results from [52], who evaluate Swin-B for early action prediction before (i.e. direct inference with pretrained model) and after finetuning on preliminary videos. Notice that, with glimpses of size 96 × 96 and higher, GliTr outperforms Swin-B finetuned for early action prediction. Further, GliTr also outperforms TemPr with the glimpses of size 128 × 128 when both have observed early 70% video. For the Jester dataset, GliTr outperforms TRN [70] for early action prediction with glimpses of size 96×96 and higher. The results demonstrate the efficiency of GliTr for early action prediction using only local information. We display mean±5×std from five independent runs.

Ablation on Spatiotemporal Consistency
To demonstrate the value of the proposed spatiotemporal training objectives, we perform an ablation study for each on the SSv2 dataset. We train four variants of GliTr using the following combinations of the training objectives: i) GliTr baseline using L cls , ii) GliTr spatial using L cls + L spatial , iii) GliTr temporal using L cls + L temporal , and iv) our default variant GliTr spatiotemporal using L cls + L spatial + L temporal . Note that the above variants have the same architecture and operation; only their training objectives are different. Figure 8(a) shows results. We observe that including only spatial or only temporal consistency in the training objectives boosts GliTr's accuracy by nearly 6% at t=16. Moreover, including both spatial and temporal consistency provides the highest improvement of around 10%.
To understand the sources of improvements provided by the two consistency losses, we perform two more experiments. First, we evaluate glimpse selection strategies learnt by the above versions of GliTr using an impartial teacher model in Figure 8(b). We observe better performance for GliTr when spatial consistency is included in the training objectives, indicating that spatial consistency helps GliTr learn better glimpse selection strategy and in turn improves its performance. Second, we evaluate above four versions of GliTr using an impartial Uniform random strategy in Figure 8(c). We observe that GliTr provides the highest performance for the Uniform random strategy when we include temporal consistency in the training objective, suggesting that temporal consistency improves GliTr's performance by learning a better classifier under partial observability. We experiment with different training procedures for the teacher model in Appendix A.  GliTr -64×64 GliTr -96×96 GliTr -128×128 (b) Figure 9: GliTr with early exit. We display accuracy vs an average number of glimpses seen by GliTr per video to predict a class with probability > γ. (a) SSv2 and (b) Jester.

Early Exit
We extend GliTr for applications that require timely decision-making. We terminate sensing and conclude a class when GliTr makes a sufficiently confident prediction. We evaluate confidence using the maximum value in the predicted class logits, C t = max(p(ŷ t )) and exit when GliTr achieves confidence C t > γ. We show the performance of GliTr for varying γ in Figure 9. We observe a trade-off between the glimpse size and the average number of glimpses required for confident prediction. GliTr achieves higher confidence early with larger glimpse sizes and thus requires fewer glimpses to achieve certain performance. While continued sensing improves GliTr's performance on SSv2, the performance saturates on Jester after the initial 50% of the glimpses, rendering further sensing unnecessary.

Conclusions
We develop a novel online action prediction model called Glimpse Transformer (GliTr) that observes video frames only partially through glimpses and predicts an ongoing action solely based on spatially and temporally incomplete observations. It predicts an informative glimpse location for a current frame based on the glimpses observed in the past. Without any ground truth for the glimpse locations, we train GliTr using a novel spatiotemporal consistency objective. On the Something-Something-v2 (SSv2) dataset, the proposed consistency objective yields around 10% higher accuracy than the cross-entropy-based baseline objective. Further, we establish that spatial consistency helps GliTr learn a better glimpse selection strategy, whereas temporal consistency improves classification performance under partial observability. While never observing frames completely, GliTr achieves 53.02% and 93.91% accuracy on SSv2 and Jester datasets and reduces the sensing area per frame by ∼ 67%. Finally, we also showcase a trade-off between the glimpse size and the number of glimpses required for early action prediction. GliTr is useful for lightweight, low-cost devices with small field-of-view cameras.