{'fraction_non_alphanumeric': 0.03332092330603127, 'fraction_numerical': 0.01690866219905684, 'mean_word_length': 4.747681883024251, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Zero-Shot Action Recognition with Transformer-based Video Semantic Embedding

While video action recognition has been an active area of research for several years, zero-shot action recognition has only recently started gaining traction. However, there is a lack of a formal definition for the zero-shot learning paradigm leading to uncertainty about classes that can be considered as previously unseen. In this work, we take a new comprehensive look at the inductive zero-shot action recognition problem from a realistic standpoint. Specifically, we advocate for a concrete formulation for zero-shot action recognition that avoids an exact overlap between the training and testing classes and also limits the intra-class variance; and propose a novel end-to-end trained transformer model which is capable of capturing long range spatiotemporal dependencies efficiently, contrary to existing approaches which use 3D-CNNs. The proposed approach outperforms the existing state-of-the-art algorithms in many settings on all benchmark datasets by a wide margin.


Introduction
Several visual recognition tasks, such as image classification and video action recognition, have made tremendous progress in recent years, thanks to the availability of extensively annotated datasets and enhanced deep learning architectures. However, collecting and annotating video samples for every possible interaction between objects is impractical, therefore recognizing previously unseen events remains a challenging task. On the other hand, humans are exceptionally good at recognizing new categories without seeing any visual samples. For example, if a person is familiar with ice skating and understands the concept of dancing, (s)he will have no trouble recognizing the action of ice dancing. In recent literature, this problem has drawn considerable attention and is known as Zero-Shot Learning (ZSL) for video action recognition. While several approaches have shown promising results in the image domain, zero-shot video ac-tion recognition remains largely unexplored.
In the existing research, ZSL is characterized as a classification problem in which a model is trained on a collection of known classes and then uses semantic attributes to identify unknown classes. Most existing approaches employ a 3D Convolutional Neural Network (3D-CNN) to extract visual features from videos. Because the 3D-CNNs are pretrained on a variety of large-scale datasets, there is no obvious demarcation between what defines seen and unseen classes. Several recent techniques, for example, are pretrained on the Kinetics-400/600/700 [6,25,35] datasets and evaluated on the UCF-101 [36], HMDB-51 [26], Olympics [37] and ActivityNet [12] datasets. However, as shown in Fig. 1, several classes that are considered as "unseen" are already present in the Kinetics dataset, which clearly violates the zero-shot paradigm. On the other hand, even a human is incapable of recognizing archery as an activity if he has never seen a bow and arrow before and is oblivious to the concept of shooting. Thus, it can be argued that it is redundant to include activities in the test set that are significantly dissimilar to the activities present in the training set. Furthermore, the existing evaluation setup comprises of randomly splitting the test dataset's classes into half and evaluating the proposed approach on that set. However, this leads to an unfair comparison since modifying the split can significantly alter the result. To this end, we propose a concrete formulation that addresses these issues and thus would provide guidance for future algorithm design.
Since the majority of existing works employ 2D or 3D convolutions as the principal operator for learning spatiotemporal features from videos, they suffer from several challenges [3]. First, inductive biases such as local connectivity, translation invariance and a locally restricted receptive field severely limits the learning capability of convolutional models on large datasets. Moreover, convolutional kernels are incapable of capturing spatiotemporal relations that span multiple time instances. Finally, even with advances in hardware acceleration, training and evaluating deep CNNs on large video datasets remain computationally Figure 1. There is a significant overlap between the training dataset used by recent state-of-the-art approaches CLASTER [19], E2E-664 [5], OD [27] and the benchmark datasets. Such overlap violates the ZSL premise.

expensive.
Motivated by these observations, we propose leveraging self-attention architectures, in particular a spatiotemporal Transformer model, for extracting semantic embedding from videos. The self-attention approach, in contrast to convolutional kernels, can capture long-range dependencies and is permutation invariant while being computationally efficient during training and inference. Our end-to-end trained transformer based approach is able to learn semantically more separable features as compared to 3D-CNNs. Our contributions can be summarized as follows: • A novel end-to-end transformer based approach for zero-shot action recognition. To the best of our knowledge, we are the first to propose using transformers for zero-shot video action recognition.
• A thorough analysis of the shortcomings in the existing problem definition. We propose a fair setup and formulation for zero-shot action recognition.
• An extensive evaluation of the proposed approach on several benchmark datasets. The proposed method outperforms several state-of-the-art approaches by a wide margin under different experimental setups.

Related Works
Video action recognition has been extensively studied over the past several years [7, 13-15, 34, 43]. In contrast, ZSL for video action recognition has only recently started gaining attention. Broadly, ZSL can be classified into the inductive setting [4,21,31,48,50], where the test data is completely unknown during training, and the transductive setting [1,29,41,42,[45][46][47], where the test data without class labels is available. In this work, we only focus on the inductive setting.
Existing approaches have been predominantly dependent on word embeddings to tackle the problem of ZSL. Specifically, these approaches use a pretrained model to extract visual features from training videos and map them to a semantic space and hypothesize that a good robust generalization on the semantic space can lead to improved performance on unseen classes [4, 5, 16-19, 21, 45-48]. For extracting the visual features, most recent approaches propose using a 3D convolutional models, which takes 16 frames sampled from a video as input. In [5], Brattoli et al. propose training a C3D [38] and a R(2+1)D model [39] in an end-to-end fashion for ZSL. On the other hand, Gowda et al. [19] propose a reinforcement learning based clustering approach, which uses a two-stream I3D [7] model for learning visual features. However, we argue that for realistic applications, sampling 16 frames from a video might not always be sufficient, especially when it comes to largescale datasets. In [7,22], it is shown that increasing the number of input frames only marginally increases the performance of convolutional models. On the contrary, recent transformer based approaches [3,49] have shown considerable performance gains when number of input frames are increased from 8 to 96, especially on tasks that require longer temporal reasoning. While earlier approaches use handcrafted semantic features [24], recent works have primarily use Word2Vec [28] for generating the semantic embeddings from the class labels. However, such approaches are prone to suffer from the domain shift problem, which occurs when a model trained on the seen semantic labels is unable to generalize well to the unseen class labels [21].
Recently, Brattoli et al. [5] propose an extension on the work of Roitberg et al. [33] and formulate a novel evaluation protocol for maintaining the ZSL paradigm, which involves removing certain classes from the training set which overlap with the test set by using semantic embedding matching. However, we show that such an approach fails to remove a significant portion of overlapping classes, thus violating the ZSL premise. Alternatively, Gowda et al. [20] propose a deterministic "TruZe" split for the UCF-101 [36] and HMDB-51 [26] datasets, by manually removing all classes which overlap with the Kinetics-400 dataset. While it is a promising approach, we show that it quickly becomes obsolete since several recent approaches use Kinetics-600/700 for training, which includes a majority of the classes from the "TruZe" test set. Furthermore, neither of the proposed approaches remove classes which are significantly different from any of the classes in the training set. Hence, we propose a novel test set (Section 3.3) which addresses all of the above mentioned issues.

Proposed Approach
In this section, we present the proposed approach for zero-shot video action recognition. We begin by carefully defining the problem formulation. Next, we introduce a novel end-to-end trained transformer model, ViSET (Video Semantic Embedding via Transformer), which leverages divided space-time attention to learn improved semantically differentiable visual features (Fig. 2). Then, we present a new data split for more suitable ZSL experimentation. Finally, we propose to enhance the available semantic embeddings using descriptions instead of labels.

Problem Definition
Traditionally, zero-shot video recognition has been defined as a classification problem where given a training set of videos X s and labels S from seen classes {(x s 1 , s 1 ), . . . , (x s N , s N )}, we aim to accurately classify a set of videos X u = {(x u 1 ), . . . , (x u M )} from previously unseen classes U = {u 1 , . . . , u M }, where N and M are the number of training and testing videos respectively. To satisfy the ZSL premise, there should be no overlap between the seen and unseen classes (S ∩ U = ∅).
A broad generalization capability is needed to succeed in the ZSL action recognition problem. The challenge compared to the regular action recognition task lies in the fact that no direct mapping from the input videos to the output unseen class labels can be learned during training. Typically, semantic embeddings are used to bridge the input videos and the output unseen class labels, which consist of words. The idea behind this mainstream ZSL approach is to learn a semantic embedding model f (x) for the input videos and choose the class that is semantically most similar.

Video Semantic Embedding via Transformer (ViSET)
We consider the inductive ZSL approach for training our model, i.e., during training the model only has access to the videos and corresponding class labels from the seen classes. Most existing approaches extract visual features using pretrained 3D-CNNs. Recently, it was shown in [5] that endto-end training a 3D-CNN performs significantly better than directly using a pretrained 3D-CNN model. However, the inability of 3D-CNNs to capture spatiotemporal information over a long time span make them unsuitable for large scale datasets such as Kinetics and ActivityNet. Hence, we propose a novel end-to-end trained Transformer model, called ViSET (Video Semantic Embedding via Transformer). The overall structure of ViSET is shown in Fig. 2. The proposed model is designed after the Vision Transformer (ViT) [11] and TimeSformer [3] models since they can be easily scaled to large datasets.
Input: Even with a computationally efficient architecture, end-to-end training on the entire video is impractical due to GPU limitations. Following the existing video classification approaches, we first sample a clip y of F frames and size H × W from the input video x. The standard NLP Space Attention MLP Input Video Snippet Multi-Head Self Attention Time Attention Transformer architecture [40] requires a 1D sequence of token embeddings as an input. Hence, as shown in Fig. 2, we first breakdown the entire video clip y into a sequence of 2D patches, e p t ∈ R 3×P 2 , where p = 1, . . . , N represents the spatial locations (i.e., patch index), t = 1, . . . , F denotes the temporal index, 3 is the number of color channels, and P × P is the patch size. Next, we flatten each patch e p t into v p t ∈ R 3P 2 and linearly map it into a score vector using a trainable linear projection E ∈ R q×3P 2 : where µ p t ∈ R q is a latent vector learned to encode the spatiotemporal position of each (p, t) pair. Following the NLP transformer BERT [10] we add a latent vector z 0 0 (0) ∈ R q for an additional fictional patch which will be learned to represent the score vector for the entire video by interacting with all patches through time and space self-attention. {z 0 0 (0), z p t (0)} p,t is the input to the Transformer model.
Overall Structure: The Transformer model consists of L sequential encoding blocks, each of which includes A parallel self-attention heads. The first encoding block processes the input {z 0 0 (0), z p t (0)} in parallel in its self-attention heads, as explained next, and outputs Finally, the output of the last block z 0 0 (L) is used to obtain the semantic embedding for the video. The role of z 0 0 (L) is to learn an effective summary score for the entire video, as an alternative to the straightforward approach of simply averaging z p t (L) over all patches and frames. z 0 0 (L) is passed through an MLP with three hidden layers and ReLU activation function to obtain the semantic embedding f (x) for video x.
Training and Inference: The whole Transformer model is trained end-to-end by minimizing the loss function where φ(s i ) is the semantic embedding of the class description s i for the training video x s i from a Sent2Vec model [30]. After the model is trained, for ZSL inference, the semantic embedding of the test video f (x u i ) is found using the Transformer model, and the class closest to the video in the semantic space is chosen, where D cos is the cosine distance and φ(u j ) is the Sent2Vec semantic embedding of the class description u j . Class Descriptions: Due to the unavailability of class descriptions or attributes, existing methods in the literature alternatively use the class labels and Word2Vec to extract the semantic embeddings φ(u j ). However, we argue that such an approach would severely limit the performance of a model. Specifically, several class labels in all the datasets are not sufficiently distinctive and could refer to various different activities. For example, diving in UCF-101 could either refer to scuba diving, cliff diving, sky diving or springboard diving. Furthermore, using Word2Vec on multi-word labels is not an efficient technique since averaging the semantic embedding over several words might lose the context. This can be also seen in Fig. 3, where the semantic embedding of several multi-word class labels is mapped incorrectly. To circumvent these issues, we first manually annotate all the training and testing datasets with one line class descriptions. We then propose using a Sent2Vec [30] model to better capture the semantic information with respect to the context of the sentence from class descriptions. The annotated class descriptions are available in the supplementary material.
Spatiotemporal Self-Attention: In contrast to NLP models which require only temporal attention, vision based transformers need to capture both spatial and temporal information. However, it is shown in [3] that joint spatial and temporal attention significantly increases the computational cost. Hence, we follow the more efficient divided spacetime architecture [3], in which the temporal and spatial attentions are computed sequentially.
When the input z p t (l − 1) enters the encoding block l ∈ {1, . . . , L}, each time self-attention head a ∈ {1, . . . , A} computes in parallel three vectors, called query, key, and value, for each patch (p, t), including the fictional (0, 0) patch which represents the video summary: LN denotes LayerNorm [2], and the matrices W time The temporal query and key vectors are then used to compute the self-attention weights with respect to all the frames t = 1, . . . , F for a given patch p: which is the input to the space self-attention module. Similar to the time self-attention module, the query, key, and value vectors are computed for space self-attention:

Fair Zero-Shot Action Recognition
While several recent works have shown promising results on the benchmark datasets, there are still several shortcomings. In contrast to earlier works, recent approaches [5,19,20,32] use large-scale external datasets such as the Kinetics 400/600/700 dataset to pretrain the visual feature extractors. However, as shown in Fig. 1, there is a significant overlap between the seen and unseen classes, S ∩ U = ∅, which clearly violates the ZSL paradigm. Recently, Brattoli et al. [5] proposed a novel training protocol which involves removing classes from the training set if where τ is set as 0.05. However, in Fig. 3, we show that even after applying such a constraint, there are several overlapping classes which are not removed. This can be partly attributed to semantic mismatch due to domain shift, where two very similar classes are called by slightly different names. For example, blowing out candles and blowing candles refer to the same class and yet the cosine distance between them in the semantic space is much greater than τ . Moreover, removing overlapping classes from the training set is a cumbersome approach since it requires retraining computationally expensive models for every individual test set. On the contrary, it would be far easier to remove overlapping classes from the test set as the class information is already available in the training data. Furthermore, it is also worth considering whether a given class belongs in the test set if it has a high intra-class variance with respect to all the seen classes. Intuitively, even humans cannot comprehend a new activity if it involves interactions and objects that are never seen before. Hence, we also propose removing classes which are significantly different from the activities present in the training dataset. While keeping such classes does not necessarily violate the ZSL premise, it introduces a source of randomness since no reasonable algorithm can be expected to recognize such activities. Clearly, a better approach is required to ensure a fair ZSL setup.
In the current literature, most approaches randomly split a single dataset and evaluate on it over several trials. Such an evaluation setup is not practical in a real-world scenario since most of the videos would have disjoint sources. Hence, to ensure a realistic ZSL setting, we extend the setup discussed in [5], which suggests using independent datasets for training and testing. However, they do not fully account for overlapping classes between the train and test sets, as seen in Fig. 3. For each dataset, completely removing classes that either overlap or have a high intra-class variance with respect to the train set leaves us with a very small number of test classes per test set (as shown for UCF-101 in Fig. 3)

Implementation Details
The proposed ViSET model is built upon the PySlow-Fast [9] and TimeSformer packages [3]. For training the model, the shorter side of the input video is first resized to 256 pixels and then randomly cropped to form a 224 × 224 (H ×W ) video snippet. The patch size is chosen as 16×16, resulting in N = 196 patches in a frame. The size of the score vectors (z p t (l)) at each encoding block for each patch is q = 768, and the number of self-attention heads is A = 12. The size of the learned semantic embedding f (x) and the Sent2Vec embedding for class descriptions is 600. We train two versions of the proposed Transformer, ViSET-8 and ViSET-96 for F = 8 and F = 96 frames in the input video snippet, respectively. In all our experiments, we use the ViSET-96 model unless explicitly stated otherwise. We train the models on 4 NVIDIA A40 GPUs with a batch size of 16 for ViSET-8 and 4 for ViSET-96. The loss function is minimized via synchronized SGD with a learning rate of 0.002. To extract semantic embeddings, we use the Sent2Vec algorithm proposed in [30].

Datasets
Most of the recent works evaluate their performance on three publicly available benchmark datasets, namely the UCF-101, HMDB-51 and Olympics dataset. The UCF-101 dataset consists of 13,320 videos from 101 classes, primarily focusing on five types of actions. The HMDB-51 dataset consists of 6767 videos from 51 classes based on daily human actions. The Olympics dataset consists of 16 categories, related to an Olympic sport. Recently, Brattoli et al. [5] evaluated the performance of their approach on the ActivityNet dataset by extracting labelled frames from ev- Figure 3. Visualization of the overlapping classes between UCF-101 and Kinetics-600/700. The classes in green are considered as overlapping by the metric given by Eq. (9) and removed in [5]. The classes in red are given by Word2Vec as the semantically nearest classes to the corresponding UCF-101 class, but not removed according to the criterion in Eq. (9). For several cases, shown in blue, the actual closest classes, which are almost identical to the corresponding UCF-101 classes and thus violate the ZSL idea, are missed by Word2Vec. ery video. As compared to the other benchmark datasets, ActivityNet is considerably more comprehensive, consisting of 27,801 videos from 200 classes related to daily activities. The Kinetics-700 dataset is the largest dataset available for video action recognition, with more than 500K videos in 700 categories sourced from YouTube. Since several classes from Kinetics-700 were not available or had files corrupted, we use the Kinetics-600 dataset in our experimental setup. Due to its small size and significant overlap with Kinetics, we do not consider the Olympics dataset in our evaluations.

Experimental Setup
To analyze the performance of our proposed ViSET model and provide a fair comparison with benchmark algorithms, we first follow the training and evaluation protocols used in the existing papers. The existing training protocols can be broadly classified into two categories, openended and restrictive. Most existing works are based on the open-ended formulation, whereas a few recent approaches [5,20,33] use the restrictive one.
Open-Ended: In this setup, a model is first trained on a large-scale visual dataset such as the Kinetics dataset without removing any classes, and then evaluated on a smaller application specific dataset such as UCF or HMDB. A few approaches also further fine-tune their models on the smaller dataset by randomly splitting it into train and test sets; however doing so does not necessarily improve performance. Moreover, as also suggested in [5], a model ideally should have separate video sources for training and testing to evaluate its generalization capability.
Restrictive: In the restrictive training approach proposed in [5], we remove all classes from Kinetics-600 whose distance to any class in UCF ∪ HMDB is smaller than τ when testing on UCF or HMDB, which results in a subset of Kinetics with 564 classes. For testing on Activi-tyNet, an even more restrictive approach is proposed which involves removing all classes whose distance to any class in ActivityNet ∪ UCF ∪ HMDB is smaller than τ . This setting leads to an even smaller subset of Kinetics with 505 classes.
For the open-ended formulation, we train our model on the entire Kinetics-600 dataset and choose to forgo finetuning on UCF or HMDB since Transformer based models require significant amount of data to learn meaningful representations. To make our evaluation comparable to the existing approaches, we first randomly split the test dataset in half and evaluate our proposed approach on it over 10 trials. For the restrictive approach, in addition to the random split, we also evaluate our model on the entire UCF and HMDB datasets (Table 3) since it allows for a more robust evaluation due to lack of randomness.
Fair ZSL: Since neither of the existing training protocols honor the true essence of ZSL, we propose a novel setup where a model is trained on the entire Kinetics-600/700 dataset and evaluated on the proposed test set discussed in Section 3.3. For evaluation, we compare our approach in Table 4 with the recently proposed E2E model [5] and our implementation of the CLASTER model [19]. Recently, [8] proposed an elaborate rehearsal approach which, in addition to a 3D-CNN, explicitly used an object classifier trained on ImageNet to learn objects detected in videos. Since their approach and problem setup is significantly different from the existing approaches, we do not compare our approach with them.

Results
In the model name "ViSET-X(Y) + Z", X refers to the number of frames used in the analysis of each video, Y denotes the number of the classes from Kinetics-600 used in training, and Z represents the data type used for class semantic embedding (CD for class descriptions and CL for class labels). The impacts of frame number and class descriptions are analyzed in Section 5.
We first compare the proposed method with the state-ofthe art approaches under the dominant random split setup in Table 2. The test datasets are randomly split into half using the seed 10 as in [5]. It is seen that the proposed spatiotemporal Transformer based ZSL approach consistently outperforms all other state-of-the-art approaches under all settings. On the UCF-101 dataset, we notice a significant improvement of 21.9% and 5.2% over the next best results, for the OE and R protocols, respectively. On the HMDB-51 dataset, the improvements of the proposed method are 3.4% and 1.8% for the OE and R protocols, respectively. None of the existing approaches present their results on the ActivityNet dataset for the OE protocol, hence we could not compare our result on it. However, for the R protocol, we outperform the E2E approach by a wide margin of 9.2%.
A disjoint training and testing split allows us to evaluate our model on the entire UCF-101, HMDB-51 and Activi-tyNet datasets without any random split. As shown in Table 3, the performances drop compared to the random split case in Table 2. However, we still considerably outperform the E2E based approach under the R protocol while using the class descriptions and class labels. Since the rest of the existing state-of-the-art approaches use some part of UCF-101 or HMDB-51 for fine tuning their models, we could not compare our results with them in this setup. Finally, we also consider the Fair ZSL protocol proposed in Section 4.2. Here, we compare our model with the E2E [5] method and our implementation of the CLASTER [19] method. We again notice a significant improvement of 5.9% over E2E, which shows the robustness and generalization capability of the proposed method. It should be noted that we use the E2E model trained on 664 classes for comparison since the model trained on all 700 classes is not available. However, since we train on Kinetics-600, which has even less number of classes, we believe it is a fair comparison.

Ablation Study
In this section, we analyze the contributions of different components of the proposed approach by performing empirical studies.
Impact of the Spatiotemporal Transformer: Here, we study how well the proposed ViSET model is able to learn the spatiotemporal visual features. For comparison, we consider the a state-of-the-art 3D-CNN model called I3D [7], which has been a popular choice for video action recognition, and has been used by several existing approaches [19,27,44] for zero-shot action recognition. For ViSET, we extract the output of the last block z 0 0 (L), which serves as a visual embedding for the entire video. To extract I3D features, we follow the approach employed by [19,27,44] Table 5. Comparison between the ViSET and I3D features in terms of clustering and classification performance using different metrics. The classification accuracy is for the k-NN classifier. and average the output from the Mixed 5c layer across the temporal dimension followed by pooling by four in the spatial dimension and finally flattening it to a vector of size 4096. We use both RGB and flow features and concatenate them to form a vector of size 8192. In Fig. 4, we present the t-SNE visualization for the ViSET features on UCF-101, where each point represents a video in the UCF dataset. It is clearly seen that, as compared to the I3D features, ViSET learns more semantically separable features with more welldefined clusters. We also quantitatively compare them in Table 5 by using various statistical metrics. The average silhouette score measures how tightly grouped all the points in the cluster are. The adjusted rand index computes a similarity measure between the clusterings and the ground truth. The homogeneity score checks if a cluster contains samples belonging to a single class. Finally, we also apply a simple k-NN classification algorithm on the extracted features to compute the accuracy for traditional video classification. In all metrics, the ViSET features provide a better separation. Impact of Class Descriptions: In Table 3, we compare the performance of our approach when class labels (CL) are used instead of class descriptions (CD). Here, we train the ViSET model on the semantic embeddings extracted using the Sent2Vec model on class labels, and test on the entire UCF, HMDB and ActivityNet datasets using the restrictive (R) setting. We see that there is a noticeable improvement in the performance when class descriptions are used, which ascertains our conjecture that descriptive semantic embed-

Model
Protocol UCF HMDB ViSET-8(564) + CD R 37.6 27.9 ViSET-96(564) + CD R 40.2 30.5 Table 6. Comparison between the performance of ViSET on the UCF and HMDB datasets when the number of input frames are increased from 8 to 96.
dings are crucial in improving the learned class representations. Moreover, our performance using class label embeddings is still better than existing state-of-the-art, which shows the efficacy of our proposed ViSET model. While generating class descriptions require some human involvement, we argue that the cost is still significantly less as compared to video level annotations required for supervised learning. Furthermore, manually defined attributes are already being used for zero-shot image classification [23] since class labels are not always discriminative enough to distinguish the context. Impact of Input Video Length: Thanks to the scalability of the proposed ViSET model, we are able to vary the length of the input video snippet (i.e., number of frames F ), which also leads to an increase in the number of input tokens. In Table 6, we see a significant increase in the performance when the number of input frames are increased from 8 to 96. Increasing the number of video frames is intuitive since it allows a model to better capture the spatiotemporal activities that span several frames. However, due to the current GPU limitations, we are unable to further increase the input length. On the other hand, even after increasing our model complexity to accommodate for 96 input frames, our model is still more computationally efficient as compared to the I3D model with 8 input frames, which requires 10.8 TFLOPS for inference, in contrast to the proposed ViSET-8 model, which only requires 0.79 TFLOPS, and ViSET-96, which requires 7.57 TFLOPS.

Conclusion
In this work, we introduce a spatiotemporal Transformer architecture for zero-shot video action recognition, called ViSET. Moreover, we highlight several areas where the existing approaches either violate the zero-shot learning (ZSL) premise or are unable to perform well due to the limited capabilities of 3D-CNN based visual extractors. We propose a comprehensive evaluation protocol that strictly adheres to the ZSL premise. Through several experiments, we show that the proposed approach consistently outperforms existing approaches over various various experimental setups, including the existing ones in the literature and the proposed one.