{'fraction_non_alphanumeric': 0.02771168243282456, 'fraction_numerical': 0.007747664093784458, 'mean_word_length': 4.962456747404844, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
VIDEO SEMANTICS EXPLORATION FOR INDEXING AND RETRIEVAL

Video semantic concepts exploration is a fundamental problem in a video indexing and retrieval. It has a long history of investigation since early days till recent achievements. The challenges lie in bridging the gap between low level features and semantic level ones. To stand on the thoroughly situation, video semantic concepts exploration for indexing and retrieval purposes evolution from conventional methods to the state-ofthe-art ones will be reviewed. The main contribution is to unify concepts involved and evolution in this interesting topic.


INTRODUCTION
There is a dramatic explosion in the amount of raw multimedia data produced including text, image, audio and video which are growing up rapidly since the last few years. This tremendous increase is due to social networks such as Facebook, Twitter, You-tube and also with the advancement of technology in smart handheld devices, cameras and storage devices. One of the most interesting data types is the video due to its huge amount of raw data it contains from both audio and visual channels. This led to an urgent need to semantically understand the video for indexing and retrieval purposes. Of course, this is a challenging task in computer vision and attracted researchers' attention around the world.
Perceptual understanding of the visual based data is one of the most interesting tasks due to the challenges it contains. The human visual system is confronted with a massive amount of visual information that need to be understood and interpreted. Our brains are capable of remembering from previous experiences then achieve excellently perception of visual surroundings [2].
To simulate the human visual system behavior, researchers work on making the computers interpret and understand the visual data from a human perspective.
The richest type of the multimedia is video. Video is rich of raw data that comes from visual and audio channels [1]. The video consists of a set of consecutive frames which are chronologically ordered with an audio, in addition to, speeches or user-defined metadata. Due to its richness of raw data and no prior structure, the video processing is considered a challengeable task. The challenge is to find efficient solutions with less processing and accurate results. The video will be addressed here in terms of its visual content.
One of the most beneficial applications of video is the indexing and retrieving. Indexing of video segments is about assigning it to a specific set of categories and labels. The video is indexed at different levels and contains many subtasks. Some of these tasks index the video at a high level manner while others are specialized and application-based. The most basic tasks of video indexing include object detection, action recognition, sequences classification, etc. Hence, organizing the resulted concepts in an effective way for ease of retrieval.
The investigation in this area has a long history and there is a tremendous progress in the number of research produced recently. These methods can be roughly divided into two approaches: 1) handcrafted (conventional) approaches 2) deep learning approaches.
Understanding the image and video contents lies in how the pixels interpreted. At early days, researchers interpreted image and video by extracting low-level features and tried to assign it specific semantics concepts. Therefore, there existed a semantic gap between these low-level features and semantic concepts. These approaches are usually referred to as handcrafted or conventional. Weiming et al. [1], introduced one of the most considerable surveys that review this approach. Recently, deep learning models specifically Convolutional Neural Network (CNN) architectures achieved terrific results which has attracted researchers' attention world widely. Since 2010, deep learning has strongly emerged again after the Large Scale Visual Recognition Challenge (ILSVRC) has been launched. A considerable achievement has been introduced to CNN architectures with AlexNet (the winner in 2012) and since though there is an explosion in the research introduced in CNN architectures. CNN is deep learning model that preserves spatial information for processing visual imagery data. It will be reviewed in upcoming sections.
The motivation behind this study is the lack of studies introduced in this topic. Fortunately, this topic will be addressed in this paper from the handcrafted approaches to the state-of-theart deep learning ones.

Contributions:
1-Introduce a review study to find out the directions in video indexing and retrieval. 2-Consolidate a basic structure of existing work in video indexing and retrieval for further work. 3-Reviewing of conventional methods and peers state-of-the-art deep learning ones.

Video Concepts:
Semantics concepts exploration is the essential step for many applications in the domain of video. There are several levels of representation of a video data. The lowest representation level of a video data is the pixel level which indicates color information. At a higher levels of abstractions, the features are being revealed to explore acquaintance behind. Semantic concepts are achieved at the final stage of the exploration levels.
Video semantic concepts exploration is about discovering hidden patterns and perceptual meaning of video hoping to realize a structured representation of semantic concepts.
The video concepts can be laid out in a set of layers such that in the first stage of semantic concepts there are a set of objects, motions and actions, and at successive layer there are objects relationships and a video events and other layers that are based on the application to finally extract semantic concepts which are useful for indexing.
During the era of research in computer vision, there are a going efforts in video semantic concepts exploration until today with achieved an immense success. Generally, this paradigms can be divided into two, the handcrafted (conventional) paradigms and the deep learning based ones. The handcrafted ones usually take multiple steps to achieve semantically understanding since it usually finds out low-level features from video pixels then goes up to a higher abstraction levels until realizing semantic concepts at the final stage which would be used in indexing. These lowlevel features include color, appearance, texture, shapes, etc.
The tasks which are reported in the literature for exploring video semantic concepts are basically the following: 1) Object Classification/Detection 2) Object Action Recognition 3) Event/Activity Recognition 4) Temporal Sequence Classification.
We'll review them in the upcoming sections. But before diving in the heart of the matter, there is a preliminary step which is "Video Structure Decomposition".
Some works considered this step as the first step in indexing of video before exploring the video semantic concepts. Therefore, it'll be reviewed it in the next section.

Video Structure Decomposition:
Video structure decomposition aims at dividing the problem of processing of video into sub ones. Instead of dealing with the whole video, it is efficient (time complexity) to deal with small manageable components. Most of the recent approaches neglected this step as it causes loss of the context and temporal information which are considered important for achieving better accuracy when detecting concepts of video. Therefore, some researchers considered the processing unit to be some of the representative frames. Usually the steps involved to decomposing the video structure are: 1-Shot/Scene boundary detection 2-Key-frame extraction

Shot Boundary Detection:
The video consists of a set consecutive frames which are chronologically ordered with an audio. Hence, the frame is the basic building unit of video. At a higher stage there is a shot which is a set of consecutive inter correlated set of frames. Shot boundaries are determined by a start and end of a camera operation such as camera angle change. Generally, boundaries due to cut or gradual transitions such fed, dissolve, wipe, etc. Cut transitions are easier to detect than the gradual transitions since the gradual transition distributed among multiple frames.
In shot boundary detection usually begin by extracting features from adjacent frames, then similarity distance between the frames are identified based on the extracted feature vectors and the closest frames are grouped together in one shot or one scene. The conventional methods used low level features as color histogram [42], SIFT [43] (Scale Invariant Feature Transform), motion vectors [44], bag of feature (BOF). Hence, researchers divided into those that uses threshold based approach, others treated shot boundary as a classification task and used statistical methods such as SVM for classification and others treated this as a clustering task in which frames in the same shot are clustered together in one cluster.
With deep learning success, CNN has been used as the source of features instead of low level features used by conventional methods [2], [3], [4].

Key-frame Extraction:
The redundant content nature of the video needs to be handled in an efficient manner for reduction of processing complexity. For these purpose, it is natural to process only some of the frames and neglect the duplicated ones. These frames should be selected efficiently to best reflect the shot content [66], [45], [46]. These descriptive and representative frames are referred to as Key-frames. We can categorize the methods used to extract key-frames into:

Reference Based Extraction:
These methods work in extracting a frame which is used as a reference for the upcoming ones. The shot frames are compared to this reference frame to eventually extract the key-frames. This method is sub-divided into two sub categories, sequential comparison and global comparison.
i. Sequential comparison: In these methods, an initial key-frame is selected and sequentially compares it to the subsequent until it obtains one that is different from the previously selected one. Zhang [61] computes an energy function from two consecutive frames to measure the distance between these two frames. Hang et al. [60] used histogram difference between two consecutive frames to extract key-frames.
ii. Global comparison: In these methods, a reference frame is selected. All the shot frames are compared with this frame to extract keyframes. Sun et al [59], extracts the most occurred frame as a reference frame. The distance between the frames and this frame is measured. The frames at the peaks of the curve are selected as key-frames. Ferman and Teklap [58], extracts alpha trimmed average histogram to describe colors distribution in the shot frames. This alpha trimmed histogram is compared to the histogram of each of the frames.

Objective Based Extraction:
The extraction process in these methods are done based on a predefined objective function. These objective function can be one of the following: i. Temporal variance: These methods consider that the extracted key-frames are made from a set of segments with equal temporal variance. Divakaran et al. [41], extracts keyframes from shot segments which are having equal MPEG-7 motion activity. The frame at the midpoint is selected as a key-frame.
ii. Key-frame representativeness: These methods act to scaling up the number of frames that the key-frame represents. It can sub categorized into two cases, if the number of key-frames is fixed or not. If the number of keyframes is fixed, then these methods work to maximize the number of the number of the frames that represents the key-frame. In contrast if the number of key-frames is not fixed then these methods working to minimize the number of key-frames based on a predefined fidelity criterion [62], [63].

Structure Based Extraction:
These methods aim initially at structuring the shot into structured representation. These methods can be sub divided into those use graphs and others that use curve.
i. Graph: These methods assume that the key-frames should be less correlated. From this point, they aiming at reducing the correlation between the key-frames. Porter et al. [64], represents the shot by a directed weighted graph. The shortest path is the path that represents minimum correlation between frames that represented by the key-frame.
ii. Curve: These methods represent the shot frames into points in the feature domain and linking each of the frames to form a trajectory curve, then the set of points that best represent the curve are being searched. Calic and Izquierdo [65], use the macroblocks of MPEG compression algorithm as a source of features, then they statistically analysis these macroblocks to find out the dissimilarity between frames to be linked.

Features Based Extraction:
These methods are based on extracted features and semantics of a key-frame. In these methods, the frames that lay in the same shot are considered to contain similar semantics. Calic and Thomus [56], use the regions resulted of segmentation to extract key-frames. Liu et al. [55], use color histogram to extract key-frames. Fan et al. [57], use an object segmentation to extract keyframes.

Object Classification/Detection:
Object detection is one of the most fundamental challenges of computer vision. The object detection revolves around recognizing the existence of specific objects and locates these objects in an image or a video. Object detection combines two subtasks, classification and localization. Image classification is to allocate object instances to a predefined object categories in digital image and video. Object detection is the quintessence of many applications like autonomous driving, identity recognition, video surveillance, medical, etc.
The lowest level of object features there are the object's color, brightness, appearance, texture, size, etc. In conventional methods usually features are extracted from the image and pre-crafted to the specific set of categories based on human perspectives.
Usually these methods doing well in video retrieval engines when queries are by an example, but it is more interesting to retrieve video by a text as well. This led to a gap between human perspective features and lowlevel feature.
The conventional computer vision paradigms utilize low level features and worked to find high level ones. Histogram of Oriented Gradients (HOG) detector [3] is an important one of the conventional methods that is bridging low-level features to high-level ones by counting the occurrence of gradient orientation of an image. HOG well describe the image object appearance and shape by intensity gradient and edge direction. Scale-invariant feature transform (SIFT) [4] describes highlevel features of an object by object interesting points regardless image scale, noise and illumination. SIFT features are calculated for a reference image and stored and when there is a new example, SIFT is calculated and compared to the stored ones and then some key points in the example is identified to filters out best matches. ElDahshan et Al. [67], proposed a Global Dominant SIFT (GD-SIFT) descriptor for indexing and retrieval. Viola-Jones [5] introduced real-time object detector which motivated for face detection based on Haar features which consist of different types for different face locations.
In contrast, deep learning paradigms act as features extractors and classifiers simultaneously which shown high accurate results with superior performance. Precisely since 2012 a bang of researches in this area has been achieved in different CNN architectures thanks to deep learning. Alex Krizhevsky, proposed AlexNet (ILSVRC-2012 winner) which is the effectual re-emergence of CNN models. ZFNet (ILSVRC-2013 winner) improved AlexNet and achieved better accuracy. Visual Geometry Group from University of Oxford invented VGGNet (ILSVRC-2014 1st runner) which outperformed GoogleNet (ILSVRC-2014 winner) and won the localization task. The idea beyond VGGNet is that to reduce the number of parameters that the network uses. While GoogleNet introduced the term "Inception" which to use filters with different sizes and achieved significant performance than AlexNet and ZFNet. ResNet (ILSVRC-2015 winner) introduced Residual Networks which solved vanishing gradient problem. Huang et al., introduced DenseNet which outperformed ResNet. Sara Sabour and Hinton proposed the idea of capsule networks (CapsNet) which radically changed CNN architecture by addressing spatial correlation of object parts. Continuously, a lot of CNN architectures are being introduced that realize considerable achievements until recent days. Subsequently, this motivated object detection progress and considerable achievements introduced to this task. Object detection architectures basically divided into region-based or two stage detectors and one stage one.
In contrast, deep learning paradigms act as features extractors and classifiers simultaneously which shown high accurate results with superior performance. Precisely since 2012 a bang of researches in this area has been achieved in different CNN architectures thanks to deep learning. There are a lot of considerable surveys in semantic object detection that discuss object detection in deep learning [51], [52], [53], [54]. It could be basically divided into region-based or two stage detectors and one stage one.

i. Two Stage Detector (Region Based):
In these methods, category independent regions are being proposed, then each of the regions are fed to a CNN architecture to classify each of them to a specific-category. Girshick et al. [39], proposed RCNN which uses selective search as a regions proposal. Selective search proposes roughly 2000 of interest regions that may contain objects and then each of them is rescaled to 227x227, thereafter a pre-trained CNN architecture (e.g. VGG) is then fine-tuned to train a SVM to classifies the extracted regions and another regression model for tighten bounding boxes. RCCN yielded a boost performance to object detection with mean Average Precision (mAP) with 58.5%. But RCNN has the following drawbacks: 1) Highly consuming load on file system during training 2) Superfluous redundant computations because of extracting features for overlapped regions 3) Need of rescaling of regions before feeding to CNN.

Instead of the methods that rely on selective search and edge box, S. Ren et al., proposed Faster RCNN which integrated Region Proposal Network (RPN) into CNN. Faster RCNN is a unified model that is composed of Region
Proposal Network (RPN) and Fast RCNN that each share the same features. Therefore, Faster RCNN is working out to fine-tuning RPN in end-to-end to find regions by sliding window and use it with different scales and aspect ratios and the Fast RCNN one also is fine-tuned greatly improved both precision and detection efficiency. Faster R-CNN achieved mAP of 69.9% on PASCAL VOC 2007 compared to Fast R-CNN of 66.9% with shared convolutional computations and the running time of Faster R-CNN was roughly 10 times lower than Fast R-CNN with VGG backbone.
ii. One Stage Detector: These methods use one model for both detection and bounding box regression.
Szegedy et al. [30], was first of the contributors that contribute in single stage detectors and proposed a DetectorNet which treated an object detection as a regression problem. They used AlexNet and replaced the Softmax layer with a regression one.
Sermanet et al. [31], proposed OverFeat which is a single stage object detection that simultaneously do classification, localization and object detection. The main contribution of OverFeat is do multi-scale classification at different regions, and predict bounding box with a regressor on top of features extractors.
Redmon et al. [32], proposed You Only Look Once (YOLO) network which is the first attempt to build a real-time object detector. It works by dividing the image into mxm cells, and each cell is responsible for detecting and object that centered in it. Each of the grid cells and predicts bounding boxes and their corresponding confidence scores. The confidence score is defined by Pr(object) * IoU(Predicted,Ground Truth) and the bounding box is identified with four values of the bounding box coordinates.
Liu et al., [33] proposed Single Shot MultiBox Detector which take a pyramidal hierarchy manner when extracting features with CNN. YOLO faced a difficulty while detecting small objects, instead SSD used a different manner by extracting features of each location with different sizes and aspect ratios. SSD uses VGG-16 as a backbone network for extracting features. On top of VGG-16, SSD added several Conv layers of decried sizes to extract features at multi-level scales in a pyramidal representation. This manner actually very useful when dealing with different object of different sizes since large-grained feature levels are good at capturing tinny objects while coarse-grained feature levels are good at large ones. Sabbeh et al. [50], uses VGG as a backbone for extracting semantic features from a video frame. They divide a frame into 224 x 224 overlapping segments then pass it through VGG to extract semantic objects.
A lot of efforts has been done till recent days by the researchers to investigate and further enhance this task. In the context of video, the context, and temporal information should be considered when during detection. Therefore, some researchers work on boundingbox level while some of them work on features level and others are utilized boundary box level and feature level together. The methods that work on bounding box level, such as, Kang et al. [34], who introduced a T-CNN model that uses precomputed optical flow fields and object tracking to propagate bounding boxes to nearby frames.
Han et al. [35], built Seq-NMS that boosted detection by utilizing high score detection from nearby frames. B. Hatem et al., proposed Seq-Bbox that build tubelets by linking bboxes across frames to infer missed and improve detection while the methods work on features level are achieved better improvement. Zhu et al. [36], introduced Flow-Guided Feature Aggregation (FGFA) model that used optical flow and features extracted from nearby frames for improving detection. Feichtenhoter et al. [37], proposed Detect-to-Track and Track-to-Detect (D&T) model that is jointly used for detection and tracking. It'd better to take a flavor of the bounding box and features level during detection. Yuning Chai [38], proposed PatchWork model for detecting objects from a video by utilizing a specialized memory that recovers lost context. Also, Patchwork adopted Q-learning based policy that intelligently selects sub windows to be processed in subsequent frame.

Action / Activity Recognition:
Action / Event Recognition: Video actions and events are an important concept that assist in building robust index. Action is the act which is been done by an object in a certain period of time, while the event is an occurrence of one or more actions in a consolidated. Action and event are two closely concepts and can be inferred from each other. Recognizing of actions and events of a video is a taugh task because of the temporal and context of video should be utilized in effective manner without loss of information as possible for best prediction. This task still challengeable and has a lot of investigation till now. We begin by reviewing conventional methods and peers of deep learning methods.

Conventional Action Recognition:
Our brain treat and action by first representing it with a symbolic system then treated it with a set of processors. Therefore, an action can be represented by an easy to compute structured system, representative of an action and differentiable of different actions [48]. The Earliest models of action recognition are make use of 3D of video to represent an action. Hogg et al. (1983), represents action by the Walker model for human action recognition. Rohr et al. (1994), make use of cylinders to represents pedestrian recognition. The use of 3D model to describe action is very expensive in the context of videos instead we'd discuss other methods that are more efficient.
Basically these methods divide an action recognition into two main parts: 1) Action Representation (Feature Extraction) 2) Action Classification [47].
Action Representation: By action representation, it meant to extract the action features. The extracted features need to be representative and discriminative for the action. The is no doubt being a challenging problem due to the variations in objects appearance and their motion speed, camera view (left, right, top, down or zooming), pose variations and many more. An action features exist in two levels, holistic level and local level. Holistic representation is to represent body with a thorough representation while local representation is to reflect local features.
Holistic Representation: The action information contains both spatial and temporal information which form a 3D shape. These methods aim to represent the object motion entirely. It extracts information about the motion in a certain cubic region contains the object. One of the considerable work introduced by Bobick et. al, who suggests to represent the action by Motion Energy Image (MEI) and Motion History Image (MHI). MEI shows where the action happens and MHI shows how the motion changes. This work success in representing the action into good manner although there is a problem that MEI and MHI are sensitive to different views. Laptev and Lindeberg [10] addresses this problem by proposing 3D Motion History Volume (MHV) from different camera views. Alper Yilmaz and Mubarak Shah, encoded the action into multiple 2D images represent the contours of the object with time change and generating 3D Spatio-Temporal Volume (STV) and analyzing the STV by using the differential geometric surface properties, such as peaks, pits, valleys, and ridges. There also works [21], [22] that encoded motion information by optical flow which estimates the motion by taking the moving object pixels' intensities over time by assuming that the illumination does not change.
2) Local Representations: These methods take the local regions that have salient motion information. Space-Time Interest Point (STIP) [18], [19], [23], [24] is the first emerge and most important work of local representations. It shows their effectiveness detecting actions with different translations and appearance variations of the motion. These describe motion by local features. Laptev [11], [24] extends Harries corner detector [26] in space-time space. Bregonzio et al.
[24] detected spatial-temporal interest points using Gabor filters. [26] detect interest points by using the spatiotemporal Hessian matrix. Other algorithms detect interest points by extending 2D detectors to 3D ones, 3D SIFT [20], HOG3D [19], local trinary patterns [27], etc. Other optical flow features based approaches [19], [28] are combined Histogram Optical Flow (HOF) with HOG features to represent human actions. The aforementioned approaches keep track of spatiotemporal information in short-term duration. Feature trajectory is an approach for detecting information in long-term durations [25] [25], [29] concatenated HOG, HOF and MBH features for describing trajectories.

Action Classification:
After representing an action with a feature representation, their features are learned to assign to specific class label. According to [47], action classification can be divided into:

1) Direct classification:
In these methods, an action features are being extracted and represented in feature representation, and thereafter these features are trained to assign class label to it. Classification is being done by the one of the off-the-shelf classifiers, such as support vector machine [9], [10], [11], k-nearest neighbor (k-NN) [12], [13], etc.

3) Feature Approaches:
There is a spectral of researchers who use low-level features and bridged to high-level ones because of the semantic gap between low level and semantic features. Bag-of-words was used as low-level features, and thereafter bridged it to high level one by histogram of words to finally feed to the classifier. Another approach is to fuse features from different extractors, it was discussed in [17], which learns global Gaussian mixture model (GMM), and uses multiple GMMs to characterize local regions at multiple scales.

Deep action recognition:
There are three categories of deep models used for action recognition: • Spatiotemporal networks • Multiple stream networks • Temporal coherency networks

Spatio-Temporal Networks:
When it related to sequences and videos, then the temporal information should be considered (i.e. keeping track of both spatial and temporal information) for action recognition. By adding the temporal dimension, the parameters get increased and this will affect the 3D convolution operation performance. Sun et al. (2015) investigated this issue and suggests factorizing the 3D filter into 2D and 1D ones and this proved to be efficient.

Temporal coherency networks:
The concept of temporal coherency of a video is that each consecutive set of frames are semantically and dynamically coherent.
A video is said to be coherent if: 1. The video frames are in its appropriate temporal order.
2. The video events semantics are correlated.
3. There are no abrupt changes in event semantics, or motions. Another related work investigated by Yingwei Li et al., who proposes an approach for action recognition for long-range dynamics (which is supposed to be usually inhomogeneous) by grouping short-term homogeneous ones.

Multiple Stream Networks:
This type of networks is inspired by our visual cortex. The visual cortex has two streams, Ventral stream and Dosaral stream. The Ventral stream identifies the object identity, color and appearance and the Dosaral stream recognizing the motion of the object.
Simonyan and Zisserman devised an architecture that exploits both appearance and motion (spatial and temporal) information. They built spatial stream network trained by video frames and temporal stream network trained by optical flow fields. Feichtenhofer et al.(2016) , study that the early fusion of the spatial and temporal information at the convolution layer rather than the softmax layer. They achieved the same performance as in Simonyan and Zisserman with half of the parameters.

Video Indexing:
A video indexing is the final stage after the PRE aforementioned stages. After extracting a set of semantic concepts, thereafter indexing can be done into levels based on the extracted semantic concepts. At the first level, there is the video and sub scenes and shots classification results, at a higher level there are a set of semantic objects per each of the frames, motions, and events, and at the final level there is an inverted index that best describe the video. A video Indexing is about organizing and storing these extracted semantic concepts in a structured efficient manner for and ease of retrieval. There are different methods of indexing representation, and they can be divided into a hierarchical representation and graph representation.

Hierarchical
representation: these methods represent video index at different levels of semantic levels. Gang et al., introduced a hierarchical representation of video semantics at two levels, labels, and semantics.
Graph representation: these methods depicted a video as a graph of semantic features, and the nodes represent the set of semantics concepts, and the edges are the dissimilarities between these between each node semantics, and the video shot frames are recognized by the path that offers a minimum value of weights. Porter et al. [64], depicted a shot by a directed weighted graph. Motivated by ImageNet, Podlesnaya et al. [8], introduced a way of building an index that is oriented by a graph that the nodes are the salient objects and edges are linked to other objects or the WordNet lexical database and used a Neo4j graph-oriented database.

Video Querying and Retrieval:
By obtaining a video index, we're ready to query and retrieve video segments. Video querying is to request about video or just a small segment of video. Once a query has been send, a measure of difference between query and the indices are being calculated for searching the database for best candidates.

Video Querying Types:
Querying a video can be of different types, and it should be handled in any video retrieval engine. The query may be by an example, set of specific keywords, or even by natural language.
Query by an example: this type can be done by an example segment or an image. It is somewhat a harsh retrieval one because it requires extracting of features from the provided example and then measuring the similarity difference between it and the database of video indices.

Query by semantics:
This type of querying enables the user to provide a set of objects, motions or some event to retrieve based on. The retrieval engine must be intelligently retrieving the most relevant segments of video that are most likely contains the semantics provided in the query.

Query by keywords or labels:
In this type the user can type some of the specific set of descriptive words to retrieve segments based on it. The retrieval model analysis these models to get a set of semantics and match them with the database of video indices. The most relevant segments are returned to the user. Truong et al. [49], provides ad-hoc querying in text format in a video search model.

Query by natural language:
In this type, the user can write what in his mind with just a set of natural language words and the retrieval model intelligently selects the best candidates that fit the query semantics. It is the most practical type of video retrieval engines. Aytar et al. [40], used a method of matching semantic words to retrieve relevant segments.

CONCLUSION:
Semantic concepts exploration for indexing and retrieval was explained in each of the conventional methods and their analogues of deep learning methods. The different tasks involved in semantic concepts exploration for video indexing were linked and consolidated from its origin till recently achieved works. Research trends in each of the steps was analyzed to unify concepts in this topic for further work.