#
Hiera

##
Overview

Hiera was proposed in Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles by Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, Jitendra Malik, Yanghao Li, Christoph Feichtenhofer

The paper introduces “Hiera,” a hierarchical Vision Transformer that simplifies the architecture of modern hierarchical vision transformers by removing unnecessary components without compromising on accuracy or efficiency. Unlike traditional transformers that add complex vision-specific components to improve supervised classification performance, Hiera demonstrates that such additions, often termed “bells-and-whistles,” are not essential for high accuracy. By leveraging a strong visual pretext task (MAE) for pretraining, Hiera retains simplicity and achieves superior accuracy and speed both in inference and training across various image and video recognition tasks. The approach suggests that spatial biases required for vision tasks can be effectively learned through proper pretraining, eliminating the need for added architectural complexity.

The abstract from the paper is the following:

Modern hierarchical vision transformers have added several vision-specific components in the pursuit of supervised classification performance. While these components lead to effective accuracies and attractive FLOP counts, the added complexity actually makes these transformers slower than their vanilla ViT counterparts. In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer without losing accuracy. In the process, we create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models while being significantly faster both at inference and during training. We evaluate Hiera on a variety of tasks for image and video recognition. Our code and models are available at https://github.com/facebookresearch/hiera.

This model was a joint contibution by EduardoPacheco and namangarg110. The original code can be found [here] (https://github.com/facebookresearch/hiera).

##
HieraConfig

###
class transformers.HieraConfig

< source >

( embed_dim = 96 image_size = [224, 224] patch_size = [7, 7] patch_stride = [4, 4] patch_padding = [3, 3] mlp_ratio = 4.0 depths = [2, 3, 16, 3] num_heads = [1, 2, 4, 8] embed_dim_multiplier = 2.0 num_query_pool = 3 query_stride = [2, 2] masked_unit_size = [8, 8] masked_unit_attention = [True, True, False, False] drop_path_rate = 0.0 num_channels = 3 hidden_act = 'gelu' initializer_range = 0.02 layer_norm_init = 1.0 layer_norm_eps = 1e-06 decoder_hidden_size = None decoder_depth = None decoder_num_heads = None normalize_pixel_loss = True mask_ratio = 0.6 out_features = None out_indices = None **kwargs )

Parameters

- embed_dim (`int`, optional, defaults to 96) — Dimensionality of patch embedding.
- image_size (`list(int)`, optional, defaults to `[224, 224]`) — The size (resolution) of input in the format (height, width) for images and (frames, height, width) for videos.
- patch_size (`list(int)`, optional, defaults to `[7, 7]`) — The size (resolution) of each patch.
- patch_stride (`list(int)`, optional, defaults to `[4, 4]`) — The stride of the patch.
- patch_padding (`list(int)`, optional, defaults to `[3, 3]`) — The padding of the patch.
- mlp_ratio (`float`, optional, defaults to 4.0) — The ratio of mlp hidden dim to embedding dim.
- depths (`list(int)`, optional, defaults to `[2, 3, 16, 3]`) — Depth of each layer in the Transformer encoder.
- num_heads (`list(int)`, optional, defaults to `[1, 2, 4, 8]`) — Number of attention heads in each layer of the Transformer encoder.
- embed_dim_multiplier (`float`, optional, defaults to 2.0) — The multiplier to the dimensionality of patch embedding in each layer of the Transformer encoder.
- num_query_pool (`int`, optional, defaults to 3) — The number of query pool stages.
- query_stride (`list(int)`, optional, defaults to `[2, 2]`) — The stride of the query pool.
- masked_unit_size (`list(int)`, optional, defaults to `[8, 8]`) — The size of the masked unit.
- masked_unit_attention (`list(bool)`, optional, defaults to `[True, True, False, False]`) — Whether to use masked unit attention in each layer of the Transformer encoder.
- drop_path_rate (`float`, optional, defaults to 0.0) — The drop path rate.
- num_channels (`int`, optional, defaults to 3) — The number of input channels.
- hidden_act (`str`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices and the zero_initializer for initializing all bias vectors.
- layer_norm_init (`float`, optional, defaults to 1.0) — The initial weight value for layer normalization layers.
- layer_norm_eps (`float`, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers.
- decoder_hidden_size (`int`, optional) — Dimensionality of decoder embeddings for MAE pretraining.
- decoder_depth (`int`, optional) — Depth of the decoder for MAE pretraining.
- decoder_num_heads (`int`, optional) — Number of attention heads in each layer of the decoder for MAE pretraining.
- normalize_pixel_loss (`bool`, optional, defaults to `True`) — Whether to normalize the pixel loss by the number of pixels.
- mask_ratio (`float`, optional, defaults to 0.6) — The ratio of masked tokens in the input.
- out_features (`List[str]`, optional) — If used as backbone, list of features to output. Can be any of `"stem"`, `"stage1"`, `"stage2"`, etc. (depending on how many stages the model has). If unset and `out_indices` is set, will default to the corresponding stages. If unset and `out_indices` is unset, will default to the last stage. Must be in the same order as defined in the `stage_names` attribute.
- out_indices (`List[int]`, optional) — If used as backbone, list of indices of features to output. Can be any of 0, 1, 2, etc. (depending on how many stages the model has). If unset and `out_features` is set, will default to the corresponding stages. If unset and `out_features` is unset, will default to the last stage. Must be in the same order as defined in the `stage_names` attribute.

This is the configuration class to store the configuration of a HieraModel. It is used to instantiate a Hiera model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Hiera facebook/hiera-base-224 architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import HieraConfig, HieraModel

    >>> # Initializing a Hiera hiera-base-patch16-224 style configuration
    >>> configuration = HieraConfig()

    >>> # Initializing a model (with random weights) from the hiera-base-patch16-224 style configuration
    >>> model = HieraModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
HieraModel

###
class transformers.HieraModel

< source >

( config: HieraConfig add_pooling_layer: bool = True is_mae: bool = False )

Parameters

- config (HieraConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.
- add_pooling_layer (`bool`, optional, defaults to `True`) — Whether or not to apply pooling layer.
- is_mae (`bool`, optional, defaults to `False`) — Whether or not to run the model on MAE mode.

The bare Hiera Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: Optional = None noise: Optional = None head_mask: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None interpolate_pos_encoding: Optional = None return_dict: Optional = None ) →
`transformers.models.hiera.modeling_hiera.HieraModelOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See BitImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- noise (`torch.FloatTensor` of shape `(batch_size, num_mask_units)`, optional) which is — mainly used for testing purposes to control randomness and maintain the reproducibility when is_mae is set to True.

Returns

`transformers.models.hiera.modeling_hiera.HieraModelOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.hiera.modeling_hiera.HieraModelOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (HieraConfig) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, optional, returned when `add_pooling_layer=True` is passed) — Average pooling of the last layer hidden-state.

- bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`) — Tensor indicating which patches are masked (0) and which are not (1).

- ids_restore (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Tensor containing the original index of the (shuffled) masked patches.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. These are the unrolled hidden states of the model.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- reshaped_hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, height, width, hidden_size)`. These are the reshaped and re-rolled hidden states of the model.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions.

The HieraModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoImageProcessor, HieraModel
    >>> import torch
    >>> from datasets import load_dataset

    >>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
    >>> image = dataset["test"]["image"][0]

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/hiera-tiny-224-hf")
    >>> model = HieraModel.from_pretrained("facebook/hiera-tiny-224-hf")

    >>> inputs = image_processor(image, return_tensors="pt")

    >>> with torch.no_grad():
    ...     outputs = model(**inputs)

    >>> last_hidden_states = outputs.last_hidden_state
    >>> list(last_hidden_states.shape)
    [1, 49, 768]

##
HieraForPreTraining

###
class transformers.HieraForPreTraining

< source >

( config: HieraConfig )

Parameters

- config (HieraConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The Hiera Model transformer with the decoder on top for self-supervised pre-training.

Note that we provide a script to pre-train this model on custom data in our examples directory.

This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: Optional = None noise: Optional = None head_mask: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None interpolate_pos_encoding: Optional = None return_dict: Optional = None ) →
`transformers.models.hiera.modeling_hiera.HieraForPreTrainingOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See BitImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- noise (`torch.FloatTensor` of shape `(batch_size, num_mask_units)`, optional) which is — mainly used for testing purposes to control randomness and maintain the reproducibility when is_mae is set to True.

Returns

`transformers.models.hiera.modeling_hiera.HieraForPreTrainingOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.hiera.modeling_hiera.HieraForPreTrainingOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (HieraConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`) — Pixel reconstruction loss.
- logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, patch_size ** 2 * num_channels)`) — Pixel reconstruction logits.
- bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, sequence_length)`) — Tensor indicating which patches are masked (0) and which are not (1).
- ids_restore (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Tensor containing the original index of the (shuffled) masked patches.
- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs.
- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.
- reshaped_hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each layer) of shape `(batch_size, height, width, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions.

The HieraForPreTraining forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoImageProcessor, HieraForPreTraining
    >>> import torch
    >>> from PIL import Image
    >>> import requests

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/hiera-tiny-224-mae-hf")
    >>> model = HieraForPreTraining.from_pretrained("facebook/hiera-tiny-224-mae-hf")

    >>> inputs = image_processor(images=image, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> logits = outputs.logits
    >>> loss = outputs.loss
    >>> print(list(logits.shape))
    [1, 196, 768]

##
HieraForImageClassification

###
class transformers.HieraForImageClassification

< source >

( config: HieraConfig )

Parameters

- config (HieraConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

Hiera Model transformer with an image classification head on top (a linear layer on top of the final hidden state with average pooling) e.g. for ImageNet.

Note that it’s possible to fine-tune Hiera on higher resolution images than the ones it has been trained on, by setting `interpolate_pos_encoding` to `True` in the forward of the model. This will interpolate the pre-trained position embeddings to the higher resolution.

This model is a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values head_mask: Optional = None labels: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None interpolate_pos_encoding: Optional = None return_dict: Optional = None ) →
`transformers.models.hiera.modeling_hiera.HieraForImageClassificationOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Pixel values can be obtained using AutoImageProcessor. See BitImageProcessor.call() for details.
- head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, optional) — Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`:
  - 1 indicates the head is not masked,
  - 0 indicates the head is masked.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- interpolate_pos_encoding (`bool`, optional) — Whether to interpolate the pre-trained position encodings.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.
- labels (`torch.LongTensor` of shape `(batch_size,)`, optional) — Labels for computing the image classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).

Returns

`transformers.models.hiera.modeling_hiera.HieraForImageClassificationOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.hiera.modeling_hiera.HieraForImageClassificationOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (HieraConfig) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, `optional`) — Loss value for the training task.

- logits (`torch.FloatTensor` of shape `(batch_size, num_labels)`) — Prediction scores of the classification head (logits of the output layer).

- hidden_states (`tuple(torch.FloatTensor)`, `optional`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. These are the unrolled hidden states of the model.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, `optional`) — Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- reshaped_hidden_states (`tuple(torch.FloatTensor)`, `optional`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, height, width, hidden_size)`. These are the reshaped and re-rolled hidden states of the model.

  Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions.

The HieraForImageClassification forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

    >>> from transformers import AutoImageProcessor, HieraForImageClassification
    >>> import torch
    >>> from datasets import load_dataset

    >>> dataset = load_dataset("huggingface/cats-image", trust_remote_code=True)
    >>> image = dataset["test"]["image"][0]

    >>> image_processor = AutoImageProcessor.from_pretrained("facebook/hiera-tiny-224-in1k-hf")
    >>> model = HieraForImageClassification.from_pretrained("facebook/hiera-tiny-224-in1k-hf")

    >>> inputs = image_processor(image, return_tensors="pt")

    >>> with torch.no_grad():
    ...     logits = model(**inputs).logits

    >>> # model predicts one of the 1000 ImageNet classes
    >>> predicted_label = logits.argmax(-1).item()
    >>> print(model.config.id2label[predicted_label])
    tabby, tabby cat

< > Update on GitHub