#
ImageBind

##
Overview

The ImageBind model was proposed in ImageBind: One Embedding Space To Bind Them All by Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, Ishan Misra. ImageBind is a multimodal joint embedding model for image/video, text, audio, depth, IMU, and thermal images. For any input from these six modalities, it outputs the same-sized embedding that can be used for cross-modal and multimodal tasks.

The abstract from the paper is the following:

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications ‘out-of-the-box’ including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.

This model was contributed by EduardoPacheco and ruffy369 and dg845 and shehan97. The original code can be found here.

##
Usage tips

- ImageBind can be used for multi-modality similarity and zero-shot tasks.
- Currently only Vision (image and video), Audio and Text are supported.
- One can use ImageBindProcessor to prepare all or pairs of the available modalities.
- ImageBindModel `forward` expects only one pair of modalities where one of those MUST be vision modality.
- If interest only on the modalities embeddings one can use ImageBindModel `get_xxx_features` method or the appropriate `ImageBindXxxModelWithProjection`
- As ImageBind vision and text encoders were frozen during training and are initialized with OpenCLIP ViT-H if one has an application using this model the addition of other modalities by including other encoders would be possible.

Here’s one example of how to get the embeddings for images, text and audios (this example requires `torchaudio`!)

    import torch
    import torchaudio
    from datasets import load_dataset
    from transformers import ImageBindModel, ImageBindProcessor

    ds = load_dataset("EduardoPacheco/imagebind-example-data", split="train")
    images = ds["image"]
    text = ds["text"]
    audios = ds["audio"] # It's a dict with keys -> array and sampling_rate
    audios = [
        torchaudio.functional.resample(
            torch.from_numpy(audio["array"]),
            orig_freq=audio["sampling_rate"],
            new_freq=16000
        ).numpy()
        for audio in audios
    ]

    model = ImageBindModel.from_pretrained("EduardoPacheco/imagebind-huge")
    processor = ImageBindProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    inputs = processor(text=text, images=images, audios=audios, padding=True, return_tensors="pt")

    with torch.no_grad():
        audio_embeds = model.get_audio_features(input_features=inputs.input_features)
        image_embeds = model.get_image_features(pixel_values=inputs.pixel_values)
        text_embeds = model.get_text_features(input_ids=inputs.input_ids, attention_mask=inputs.attention_mask)

    # we can compute probs to use for retrieval or zero-shot workflows.
    probs_image_text = (image_embeds @ text_embeds.T).softmax(dim=-1)
    probs_text_audio = (text_embeds @ audio_embeds.T).softmax(dim=-1)
    probs_image_audio = (image_embeds @ audio_embeds.T).softmax(dim=-1)

##
ImageBindConfig

###
class transformers.ImageBindConfig

< source >

( text_config: Union = None vision_config: Union = None audio_config: Union = None projection_dim: int = 1024 **kwargs )

Parameters

- text_config (`dict` or `ImageBindTextConfig`, optional) — Dictionary or an instance of `ImageBindTextConfig` that defines the text model configuration.
- vision_config (`dict` or `ImageBindVisionConfig`, optional) — Dictionary or an instance of `ImageBindVisionConfig` that defines the vision model configuration.
- audio_config (`dict` or `ImageBindAudioConfig`, optional) — Dictionary or an instance of `ImageBindAudioConfig` that defines the audio model configuration.
- projection_dim (`int`, optional, defaults to 1024) — Dimentionality of text and vision projection layers.
- kwargs (optional) — Dictionary of keyword arguments.

ImageBindConfig is the configuration class to store the configuration of a ImageBindModel. It is used to instantiate a ImageBind model according to the specified arguments, defining the text model and vision model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the ImageBind facebook/imagebind-huge architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import ImageBindConfig, ImageBindModel

    >>> # Initializing a ImageBindConfig with facebook/imagebind-huge style configuration
    >>> configuration = ImageBindConfig()

    >>> # Initializing a ImageBindModel (with random weights) from the facebook/imagebind-huge style configuration
    >>> model = ImageBindModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

    >>> # We can also initialize a ImageBindConfig from a ImageBindTextConfig and a ImageBindVisionConfig
    >>> from transformers import ImageBindTextConfig, ImageBindVisionConfig

    >>> # Initializing a ImageBindText and ImageBindVision configuration
    >>> config_text = ImageBindTextConfig()
    >>> config_vision = ImageBindVisionConfig()

    >>> config = ImageBindConfig.from_text_vision_configs(config_text, config_vision)

####
from_text_vision_configs

< source >

( text_config: ImageBindTextConfig vision_config: ImageBindVisionConfig **kwargs ) →
ImageBindConfig

Returns

ImageBindConfig

An instance of a configuration object

Instantiate a ImageBindConfig (or a derived class) from imagebind text model configuration and imagebind vision model configuration.

##
ImageBindTextConfig

###
class transformers.ImageBindTextConfig

< source >

( vocab_size = 49408 hidden_size = 1024 mlp_ratio = 4.0 projection_dim = 1024 num_hidden_layers = 24 num_attention_heads = 16 max_position_embeddings = 77 hidden_act = 'gelu' layer_norm_eps = 1e-06 add_kv_bias = False attention_dropout = 0.0 drop_path_rate = 0.0 initializer_range = 0.02 initializer_factor = 1.0 logit_scale_init_value = 14.2857 learnable_logit_scale = True pad_token_id = 0 bos_token_id = 49406 eos_token_id = 49407 **kwargs )

Parameters

- vocab_size (`int`, optional, defaults to 49408) — Vocabulary size of the ImageBind text model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling ImageBindModel.
- hidden_size (`int`, optional, defaults to 1024) — Dimensionality of the encoder layers and the pooler layer.
- mlp_ratio (`float`, optional, defaults to 4.0) — The ratio of the hidden size in the feedforward network to the hidden size in the encoder layers.
- projection_dim (`int`, optional, defaults to 1024) — If the ImageBind text model has an output projection layer, the dimension to which that projection layer maps to.
- num_hidden_layers (`int`, optional, defaults to 24) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 16) — Number of attention heads for each attention layer in the Transformer encoder.
- max_position_embeddings (`int`, optional, defaults to 77) — The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
- hidden_act (`str` or `function`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` `"gelu"` are supported.
- layer_norm_eps (`float`, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers.
- add_kv_bias (`bool`, optional, defaults to `False`) — Whether to add an extra learnable bias token to the attention key and value sequences. This is based on the `add_kv_bias` argument to `torch.nn.MultiHeadAttention`.
- attention_dropout (`float`, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
- drop_path_rate (`float`, optional, defaults to 0.0) — The dropout probability for the DropPath (stochastic) regularization layers.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- initializer_factor (`float`, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- logit_scale_init_value (`float`, optional, defaults to 14.2857) — The initial value of the `logit_scale` parameter for the text component. If `None`, the logits will not be scaled.
- learnable_logit_scale (`bool`, optional, defaults to `True`) — Whether the `logit_scale` is learnable or fixed.
- pad_token_id (`int`, optional, defaults to 0) — Padding token id.
- bos_token_id (`int`, optional, defaults to 49406) — Beginning of stream token id.
- eos_token_id (`int`, optional, defaults to 49407) — End of stream token id.

This is the configuration class to store the configuration of a ImageBindTextModel. It is used to instantiate a ImageBind text encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the text encoder of the ImageBind facebook/imagebind-huge architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import ImageBindTextConfig, ImageBindTextModel

    >>> # Initializing a ImageBindTextConfig with facebook/imagebind-huge style configuration
    >>> configuration = ImageBindTextConfig()

    >>> # Initializing a ImageBindTextModel (with random weights) from the facebook/imagebind-huge style configuration
    >>> model = ImageBindTextModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
ImageBindVisionConfig

###
class transformers.ImageBindVisionConfig

< source >

( hidden_size = 1280 mlp_ratio = 4.0 projection_dim = 1024 num_hidden_layers = 32 num_attention_heads = 16 num_channels = 3 num_frames = 2 image_size = 224 patch_size = 14 hidden_act = 'gelu' layer_norm_eps = 1e-06 add_kv_bias = False attention_dropout = 0.0 drop_path_rate = 0.0 initializer_range = 0.02 initializer_factor = 1.0 logit_scale_init_value = None learnable_logit_scale = False **kwargs )

Parameters

- hidden_size (`int`, optional, defaults to 1280) — Dimensionality of the encoder layers and the pooler layer.
- mlp_ratio (`float`, optional, defaults to 4.0) — The ratio of the hidden size in the feedforward network to the hidden size in the encoder layers.
- projection_dim (`int`, optional, defaults to 1024) — If the ImageBind vision model has an output projection layer, the dimension to which that projection layer maps to.
- num_hidden_layers (`int`, optional, defaults to 32) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 16) — Number of attention heads for each attention layer in the Transformer encoder.
- num_channels (`int`, optional, defaults to 3) — The number of channels in the input images.
- num_frames (`int`, optional, defaults to 2) — If using video (spatiotemporal) input, the number of video frames in the spatiotemporal data.
- image_size (`int`, optional, defaults to 224) — The size (resolution) of each image.
- patch_size (`int`, optional, defaults to 14) — The size (resolution) of each patch.
- hidden_act (`str` or `function`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported.
- layer_norm_eps (`float`, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers.
- add_kv_bias (`bool`, optional, defaults to `False`) — Whether to add an extra learnable bias token to the attention key and value sequences. This is based on the `add_kv_bias` argument to `torch.nn.MultiHeadAttention`.
- attention_dropout (`float`, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
- drop_path_rate (`float`, optional, defaults to 0.0) — The dropout probability for the DropPath (stochastic) regularization layers.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- initializer_factor (`float`, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- logit_scale_init_value (`float`, optional) — The initial value of the `logit_scale` parameter for the vision component. If `None`, the logits will not be scaled.
- learnable_logit_scale (`bool`, optional, defaults to `False`) — Whether the `logit_scale` is learnable or fixed.

This is the configuration class to store the configuration of a ImageBindVisionModel. It is used to instantiate a ImageBind vision encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the vision encoder of the ImageBind facebook/imagebind-huge architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import ImageBindVisionConfig, ImageBindVisionModel

    >>> # Initializing a ImageBindVisionConfig with facebook/imagebind-huge style configuration
    >>> configuration = ImageBindVisionConfig()

    >>> # Initializing a ImageBindVisionModel (with random weights) from the facebook/imagebind-huge style configuration
    >>> model = ImageBindVisionModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
ImageBindAudioConfig

###
class transformers.ImageBindAudioConfig

< source >

( hidden_size = 768 mlp_ratio = 4.0 projection_dim = 1024 num_hidden_layers = 12 num_attention_heads = 12 num_mel_bins = 128 target_len = 204 num_channels = 1 patch_size = 16 stride = 10 hidden_act = 'gelu' layer_norm_eps = 1e-06 add_kv_bias = True attention_dropout = 0.0 drop_path_rate = 0.1 initializer_range = 0.02 initializer_factor = 1.0 logit_scale_init_value = 20.0 learnable_logit_scale = False **kwargs )

Parameters

- hidden_size (`int`, optional, defaults to 768) — Dimensionality of the encoder layers and the pooler layer.
- mlp_ratio (`float`, optional, defaults to 4.0) — The ratio of the hidden size in the feedforward network to the hidden size in the encoder layers.
- projection_dim (`int`, optional, defaults to 1024) — If the ImageBind audio model has an output projection layer, the dimension to which that projection layer maps to.
- num_hidden_layers (`int`, optional, defaults to 12) — Number of hidden layers in the Transformer encoder.
- num_attention_heads (`int`, optional, defaults to 12) — Number of attention heads for each attention layer in the Transformer encoder.
- num_mel_bins (`int`, optional, defaults to 128) — The number of frequency bins in the log-mel spectrogram.
- target_len (`int`, optional, defaults to 204) — The length of the target sequence.
- num_channels (`int`, optional, defaults to 1) — The number of channels in the input audio data.
- patch_size (`int`, optional, defaults to 16) — The kernel size of the patch embedding 2D convolution layer.
- stride (`int`, optional, defaults to 10) — The stride of the patch embedding 2D convolution layer.
- hidden_act (`str` or `function`, optional, defaults to `"gelu"`) — The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` ``"gelu"` are supported.
- layer_norm_eps (`float`, optional, defaults to 1e-06) — The epsilon used by the layer normalization layers.
- add_kv_bias (`bool`, optional, defaults to `True`) — Whether to add an extra learnable bias token to the attention key and value sequences. This is based on the `add_kv_bias` argument to `torch.nn.MultiHeadAttention`.
- attention_dropout (`float`, optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
- drop_path_rate (`float`, optional, defaults to 0.1) — The dropout probability for the DropPath (stochastic) regularization layers.
- initializer_range (`float`, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
- initializer_factor (`float`, optional, defaults to 1.0) — A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing).
- logit_scale_init_value (`float`, optional, defaults to 20.0) — The initial value of the `logit_scale` parameter for the audio component. If `None`, the logits will not be scaled.
- learnable_logit_scale (`bool`, optional, defaults to `False`) — Whether the `logit_scale` is learnable or fixed.

This is the configuration class to store the configuration of a ImageBindAudioModel. It is used to instantiate a ImageBind audio encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the audio encoder of the ImageBind facebook/imagebind-huge architecture.

Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. Read the documentation from PretrainedConfig for more information.

Example:

    >>> from transformers import ImageBindAudioConfig, ImageBindAudioModel

    >>> # Initializing a ImageBindAudioConfig with facebook/imagebind-huge style configuration
    >>> configuration = ImageBindAudioConfig()

    >>> # Initializing a ImageBindAudioModel (with random weights) from the facebook/imagebind-huge style configuration
    >>> model = ImageBindAudioModel(configuration)

    >>> # Accessing the model configuration
    >>> configuration = model.config

##
ImageBindImageProcessor

###
class transformers.ImageBindImageProcessor

< source >

( do_resize: bool = True size: Dict = None resample: Resampling = <Resampling.BICUBIC: 3> do_center_crop: bool = True crop_size: Dict = None do_rescale: bool = True rescale_factor: Union = 0.00392156862745098 do_normalize: bool = True image_mean: Union = None image_std: Union = None do_convert_rgb: bool = True do_chunk: bool = True chunk_duration: float = 2.0 num_chunks: int = 5 num_frames_per_chunk: int = 2 fps: int = 30 **kwargs )

Parameters

- do_resize (`bool`, optional, defaults to `True`) — Whether to resize the image’s (height, width) dimensions to the specified `size`. Can be overridden by `do_resize` in the `preprocess` method.
- size (`Dict[str, int]` optional, defaults to `{"shortest_edge" -- 224}`): Size of the image after resizing. The shortest edge of the image is resized to size[“shortest_edge”], with the longest edge resized to keep the input aspect ratio. Can be overridden by `size` in the `preprocess` method.
- resample (`PILImageResampling`, optional, defaults to `Resampling.BICUBIC`) — Resampling filter to use if resizing the image. Can be overridden by `resample` in the `preprocess` method.
- do_center_crop (`bool`, optional, defaults to `True`) — Whether to center crop the image to the specified `crop_size`. Can be overridden by `do_center_crop` in the `preprocess` method.
- crop_size (`Dict[str, int]` optional, defaults to 224) — Size of the output image after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess` method.
- do_rescale (`bool`, optional, defaults to `True`) — Whether to rescale the image by the specified scale `rescale_factor`. Can be overridden by `do_rescale` in the `preprocess` method.
- rescale_factor (`int` or `float`, optional, defaults to `1/255`) — Scale factor to use if rescaling the image. Can be overridden by `rescale_factor` in the `preprocess` method.
- do_normalize (`bool`, optional, defaults to `True`) — Whether to normalize the image. Can be overridden by `do_normalize` in the `preprocess` method.
- image_mean (`float` or `List[float]`, optional, defaults to `[0.48145466, 0.4578275, 0.40821073]`) — Mean to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_mean` parameter in the `preprocess` method.
- image_std (`float` or `List[float]`, optional, defaults to `[0.26862954, 0.26130258, 0.27577711]`) — Standard deviation to use if normalizing the image. This is a float or list of floats the length of the number of channels in the image. Can be overridden by the `image_std` parameter in the `preprocess` method. Can be overridden by the `image_std` parameter in the `preprocess` method.
- do_convert_rgb (`bool`, optional, defaults to `True`) — Whether to convert the image to RGB.
- do_chunk (`bool`, optional, defaults to `True`) — Whether to chunk the video into multiple clips.
- chunk_duration (`float`, optional, defaults to 2.0) — Duration of each chunk in seconds.
- num_chunks (`int`, optional, defaults to 5) — Number of chunks to sample.
- num_frames_per_chunk (`int`, optional, defaults to 2) — Number of frames to sample per chunk.
- fps (`int`, optional, defaults to 30) — Frame rate of the video. It’s assumed that all videos have the same frame rate.

Constructs an ImageBind image processor.

####
preprocess

< source >

( images: Union = None videos: Union = None do_resize: bool = None size: Dict = None resample: Resampling = None do_center_crop: bool = None crop_size: int = None do_rescale: bool = None rescale_factor: float = None do_normalize: bool = None image_mean: Union = None image_std: Union = None do_convert_rgb: bool = None do_chunk: bool = None chunk_duration: float = None num_chunks: int = None num_frames_per_chunk: int = None fps: int = None return_tensors: Union = None data_format: Optional = <ChannelDimension.FIRST: 'channels_first'> input_data_format: Union = None **kwargs )

Parameters

- images (`ImageInput`, optional) — Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. Either `images` or `videos` must be provided.
- videos (`VideoInput`, optional) — Video to preprocess. Expects a single or batch of videos with pixel values ranging from 0 to 255. If passing in videos with pixel values between 0 and 1, set `do_rescale=False`. Either `images` or `videos` must be provided.
- do_resize (`bool`, optional, defaults to `self.do_resize`) — Whether to resize the image.
- size (`Dict[str, int]`, optional, defaults to `self.size`) — Size of the image after resizing. Shortest edge of the image is resized to size[“shortest_edge”], with the longest edge resized to keep the input aspect ratio.
- resample (`int`, optional, defaults to `self.resample`) — Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`.
- do_center_crop (`bool`, optional, defaults to `self.do_center_crop`) — Whether to center crop the image.
- crop_size (`Dict[str, int]`, optional, defaults to `self.crop_size`) — Size of the center crop. Only has an effect if `do_center_crop` is set to `True`.
- do_rescale (`bool`, optional, defaults to `self.do_rescale`) — Whether to rescale the image.
- rescale_factor (`float`, optional, defaults to `self.rescale_factor`) — Rescale factor to rescale the image by if `do_rescale` is set to `True`.
- do_normalize (`bool`, optional, defaults to `self.do_normalize`) — Whether to normalize the image.
- image_mean (`float` or `List[float]`, optional, defaults to `self.image_mean`) — Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`.
- image_std (`float` or `List[float]`, optional, defaults to `self.image_std`) — Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`.
- do_convert_rgb (`bool`, optional, defaults to `self.do_convert_rgb`) — Whether to convert the image to RGB.
- do_chunk (`bool`, optional, defaults to `self.do_chunk`) — Whether to chunk the video into multiple clips.
- chunk_duration (`float`, optional, defaults to `self.chunk_duration`) — Duration of each chunk in seconds.
- num_chunks (`int`, optional, defaults to `self.num_chunks`) — Number of chunks to sample.
- num_frames_per_chunk (`int`, optional, defaults to `self.num_frames_per_chunk`) — Number of frames to sample per chunk.
- fps (`int`, optional, defaults to `self.fps`) — Frame rate of the video. It’s assumed that all videos have the same frame rate.
- return_tensors (`str` or `TensorType`, optional) — The type of tensors to return. Can be one of:
  - Unset: Return a list of `np.ndarray`.
  - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
  - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
  - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
  - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
- data_format (`ChannelDimension` or `str`, optional, defaults to `ChannelDimension.FIRST`) — The channel dimension format for the output image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - Unset: Use the channel dimension format of the input image.
- input_data_format (`ChannelDimension` or `str`, optional) — The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of:
  - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
  - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
  - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.

Preprocess an image or batch of images.

##
ImageBindFeatureExtractor

###
class transformers.ImageBindFeatureExtractor

< source >

( feature_size = 1 sampling_rate = 16000 num_mel_bins = 128 max_length = 204 padding_value = 0.0 do_normalize = True mean = -4.268 std = 9.138 do_chunk = True chunk_duration = 2.0 num_chunks = 3 **kwargs )

Parameters

- feature_size (`int`, optional, defaults to 1) — The feature dimension of the extracted features.
- sampling_rate (`int`, optional, defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
- num_mel_bins (`int`, optional, defaults to 128) — Number of Mel-frequency bins.
- max_length (`int`, optional, defaults to 204) — Maximum length to which to pad/truncate the extracted features.
- padding_value (`float`, optional, defaults to 0.0) — The value to pad with when applying the padding strategy defined by the `padding` argument to [ImageBindAudioFeatureExtractor.call`].
- do_normalize (`bool`, optional, defaults to `True`) — Whether or not to normalize the log-Mel features using `mean` and `std`.
- mean (`float`, optional, defaults to -4.268) — The mean value used to normalize the log-Mel features. Uses the AudioSet mean by default.
- std (`float`, optional, defaults to 9.138) — The standard deviation value used to normalize the log-Mel features. Uses the AudioSet standard deviation by default.
- do_chunk (`bool`, optional, defaults to `True`) — Whether or not to sample multiple chunks from the input audio. If `False`, the entire audio will be used.
- chunk_duration (`float`, optional, defaults to 2.0) — The duration of each chunk in seconds.
- num_chunks (`int`, optional, defaults to 3) — The number of chunks to sample from the input audio.

Constructs a Audio Spectrogram Transformer (AST) feature extractor.

This feature extractor inherits from SequenceFeatureExtractor which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

This class extracts mel-filter bank features from raw speech using TorchAudio, pads/truncates them to a fixed length and normalizes them using a mean and standard deviation.

##
ImageBindProcessor

###
class transformers.ImageBindProcessor

< source >

( image_processor tokenizer feature_extractor )

Parameters

- image_processor (ImageBindImageProcessor) — An instance of ImageBindImageProcessor to process the images. This is a required input.
- tokenizer ([`CLIPTokenizer`, `CLIPTokenizerFast`]) — An instance of [‘PreTrainedTokenizer`] or PreTrainedTokenizerFast. The tokenizer is a required input.
- feature_extractor (ImageBindFeatureExtractor) — An instance of ImageBindFeatureExtractor to extract features from the audio. This is a required input.

Constructs a ImageBind processor which wraps a ImageBind image processor and feature extracotr and a CLIP tokenizer into a single processor.

ImageBindProcessor offers all the functionalities of ImageBindImageProcessor, ImageBindFeatureExtractor and CLIPTokenizerFast. See the `__call__()` and decode() for more information.

####
batch_decode

< source >

( *args **kwargs )

This method forwards all its arguments to ImageBindTokenizerFast’s batch_decode(). Please refer to the docstring of this method for more information.

####
decode

< source >

( *args **kwargs )

This method forwards all its arguments to ImageBindTokenizerFast’s decode(). Please refer to the docstring of this method for more information.

##
ImageBindModel

###
class transformers.ImageBindModel

< source >

( config: ImageBindConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: FloatTensor input_features: Optional = None input_ids: Optional = None attention_mask: Optional = None position_ids: Optional = None return_loss: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using AutoImageProcessor. See ImageBindImageProcessor.call() for details.
- input_features (`torch.FloatTensor` of shape `(batch_size, num_mel_bins, target_len)`) — Input features. Padding will be ignored by default should you provide it. Input features can be obtained using AutoFeatureExtractor. See `ImageBindFeatureExtractor.__call__()` for details.
- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- return_loss (`bool`, optional) — Whether or not to return the contrastive loss.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindConfig'>`) and inputs.

- loss (`torch.FloatTensor` of shape `(1,)`, optional, returned when `return_loss` is `True`) — Contrastive loss for image-text similarity.
- logits_per_image:(`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`) — The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores.
- logits_per_text:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`) — The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores.
- logits_per_audio:(`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`) — The scaled dot product scores between `audio_embeds` and `image_embeds`. This represents the audio-image similarity scores.
- text_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`) — The normalized text embeddings obtained by applying the projection layer to the pooled output of ImageBindTextModel, then applying L2 normalization and logit scaling.
- image_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`) — The normalized image embeddings obtained by applying the projection layer to the pooled output of ImageBindVisionModel, then applying L2 normalization and logit scaling.
- audio_embeds(`torch.FloatTensor` of shape `(batch_size, output_dim`) — The normalized audio embeddings obtained by applying the projection layer to the pooled output of ImageBindAudioModel, then applying L2 normalization and logit scaling.
- text_model_output(`BaseModelOutputWithPooling`): The output of the ImageBindTextModel.
- vision_model_output(`BaseModelOutputWithPooling`): The output of the ImageBindVisionModel.
- audio_model_output(`BaseModelOutputWithPooling`): The output of the ImageBindAudioModel.

The ImageBindModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, ImageBindModel

    >>> model = ImageBindModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> inputs = processor(
    ...     text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True
    ... )

    >>> outputs = model(**inputs)
    >>> logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
    >>> probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities

####
get_text_features

< source >

( input_ids: Optional = None attention_mask: Optional = None position_ids: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The text embeddings obtained by applying the projection layer to the pooled output of ImageBindTextModel.

The ImageBindModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoTokenizer, ImageBindModel

    >>> model = ImageBindModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> tokenizer = AutoTokenizer.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
    >>> text_features = model.get_text_features(**inputs)

####
get_image_features

< source >

( pixel_values: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using AutoImageProcessor. See ImageBindImageProcessor.call() for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The image embeddings obtained by applying the projection layer to the pooled output of ImageBindVisionModel.

The ImageBindModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, ImageBindModel

    >>> model = ImageBindModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> inputs = processor(images=image, return_tensors="pt")

    >>> image_features = model.get_image_features(**inputs)

####
get_audio_features

< source >

( input_features: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
audio_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

Parameters

- input_features (`torch.FloatTensor` of shape `(batch_size, num_mel_bins, target_len)`) — Input features. Padding will be ignored by default should you provide it. Input features can be obtained using AutoFeatureExtractor. See `ImageBindFeatureExtractor.__call__()` for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

audio_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The audio embeddings obtained by applying the projection layer to the pooled output of ImageBindAudioModel.

The ImageBindModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> import torchaudio
    >>> from datasets import load_dataset
    >>> from transformers import AutoProcessor, ImageBindModel

    >>> ds = load_dataset("EduardoPacheco/imagebind-example-data", split="train")
    >>> audio = ds[0]["audio"]
    >>> audio = torchaudio.functional.resample(torch.from_numpy(audio["array"]), orig_freq=audio["sampling_rate"], new_freq=16000).numpy()

    >>> model = ImageBindModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> inputs = processor(audios=audio, return_tensors="pt")

    >>> audio_features = model.get_audio_features(**inputs)

##
ImageBindTextModel

###
class transformers.ImageBindTextModel

< source >

( config: ImageBindTextConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The text model from ImageBind without any head or projection on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: Optional = None attention_mask: Optional = None position_ids: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindTextConfig'>`) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The ImageBindTextModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoTokenizer, ImageBindTextModel

    >>> model = ImageBindTextModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> tokenizer = AutoTokenizer.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> last_hidden_state = outputs.last_hidden_state
    >>> pooled_output = outputs.pooler_output  # pooled (EOS token) states

##
ImageBindTextModelWithProjection

###
class transformers.ImageBindTextModelWithProjection

< source >

( config: ImageBindTextConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

ImageBind Text Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_ids: Optional = None attention_mask: Optional = None position_ids: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindTextModelOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it.

  Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

  What are input IDs?

- attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, optional) — Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:
  - 1 for tokens that are not masked,
  - 0 for tokens that are masked.

  What are attention masks?

- position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`.

  What are position IDs?

- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindTextModelOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindTextModelOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindTextConfig'>`) and inputs.

- text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` optional returned when model is initialized with `with_projection=True`) — The text embeddings obtained by applying the projection layer to the pooler_output.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- normalized_text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)`, optional, returned when model is initialized with `with_projection=True`) — The normalized text embeddings obtained by applying the projection layer to the pooler_output, then applying L2 normalization and scaling the logits.

The ImageBindTextModelWithProjection forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from transformers import AutoTokenizer, ImageBindTextModelWithProjection

    >>> model = ImageBindTextModelWithProjection.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> tokenizer = AutoTokenizer.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> text_embeds = outputs.text_embeds

##
ImageBindVisionModel

###
class transformers.ImageBindVisionModel

< source >

( config: ImageBindVisionConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The vision model from ImageBind without any head or projection on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using AutoImageProcessor. See ImageBindImageProcessor.call() for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindVisionConfig'>`) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The ImageBindVisionModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, ImageBindVisionModel

    >>> model = ImageBindVisionModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> inputs = processor(images=image, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> last_hidden_state = outputs.last_hidden_state
    >>> pooled_output = outputs.pooler_output  # pooled CLS states

##
ImageBindVisionModelWithProjection

###
class transformers.ImageBindVisionModelWithProjection

< source >

( config: ImageBindVisionConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

ImageBind Vision Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( pixel_values: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindVisionModelOutput` or `tuple(torch.FloatTensor)`

Parameters

- pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`) — Pixel values. Padding will be ignored by default should you provide it. Pixel values can be obtained using AutoImageProcessor. See ImageBindImageProcessor.call() for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindVisionModelOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindVisionModelOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindVisionConfig'>`) and inputs.

- image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` optional returned when model is initialized with `with_projection=True`) — The image embeddings obtained by applying the projection layer to the pooler_output.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- normalized_image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)`, optional, returned when model is initialized with `with_projection=True`) — The normalized image embeddings obtained by applying the projection layer to the pooler_output, then applying L2 normalization and scaling the logits.

The ImageBindVisionModelWithProjection forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> from PIL import Image
    >>> import requests
    >>> from transformers import AutoProcessor, ImageBindVisionModelWithProjection

    >>> model = ImageBindVisionModelWithProjection.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg"
    >>> image = Image.open(requests.get(url, stream=True).raw)

    >>> inputs = processor(images=image, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> image_embeds = outputs.image_embeds

##
ImageBindAudioModel

###
class transformers.ImageBindAudioModel

< source >

( config: ImageBindAudioConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

The vision model from ImageBind without any head or projection on top. This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_features: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_features (`torch.FloatTensor` of shape `(batch_size, num_mel_bins, target_len)`) — Input features. Padding will be ignored by default should you provide it. Input features can be obtained using AutoFeatureExtractor. See `ImageBindFeatureExtractor.__call__()` for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindTransformerOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindAudioConfig'>`) and inputs.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — Last layer hidden-state of the first token of the sequence (classification token) after further processing through the layers used for the auxiliary pretraining task. E.g. for BERT-family of models, this returns the classification token after processing through a linear layer and a tanh activation function. The linear layer weights are trained from the next sentence prediction (classification) objective during pretraining.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

The ImageBindAudioModel forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> import torchaudio
    >>> from datasets import load_dataset
    >>> from transformers import AutoProcessor, ImageBindAudioModel

    >>> ds = load_dataset("EduardoPacheco/imagebind-example-data", split="train")
    >>> audio = ds[0]["audio"]
    >>> audio = torchaudio.functional.resample(torch.from_numpy(audio["array"]), orig_freq=audio["sampling_rate"], new_freq=16000).numpy()

    >>> model = ImageBindAudioModel.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> inputs = processor(audios=audio, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> last_hidden_state = outputs.last_hidden_state
    >>> pooled_output = outputs.pooler_output  # pooled CLS states

##
ImageBindAudioModelWithProjection

###
class transformers.ImageBindAudioModelWithProjection

< source >

( config: ImageBindAudioConfig )

Parameters

- config (ImageBindConfig) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

ImageBind Audio Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from PreTrainedModel. Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch torch.nn.Module subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

####
forward

< source >

( input_features: Optional = None output_attentions: Optional = None output_hidden_states: Optional = None return_dict: Optional = None ) →
`transformers.models.imagebind.modeling_imagebind.ImageBindAudioModelOutput` or `tuple(torch.FloatTensor)`

Parameters

- input_features (`torch.FloatTensor` of shape `(batch_size, num_mel_bins, target_len)`) — Input features. Padding will be ignored by default should you provide it. Input features can be obtained using AutoFeatureExtractor. See `ImageBindFeatureExtractor.__call__()` for details.
- output_attentions (`bool`, optional) — Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail.
- output_hidden_states (`bool`, optional) — Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail.
- return_dict (`bool`, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

Returns

`transformers.models.imagebind.modeling_imagebind.ImageBindAudioModelOutput` or `tuple(torch.FloatTensor)`

A `transformers.models.imagebind.modeling_imagebind.ImageBindAudioModelOutput` or a tuple of `torch.FloatTensor` (if `return_dict=False` is passed or when `config.return_dict=False`) comprising various elements depending on the configuration (`<class 'transformers.models.imagebind.configuration_imagebind.ImageBindAudioConfig'>`) and inputs.

- audio_embeds (`torch.FloatTensor` of shape `(batch_size, hidden_size)`) — The Audio embeddings obtained by applying the projection layer to the pooler_output.

- last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`) — Sequence of hidden-states at the output of the last layer of the model.

- attentions (`tuple(torch.FloatTensor)`, optional, returned when `output_attentions=True` is passed or when `config.output_attentions=True`) — Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`.

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads.

- hidden_states (`tuple(torch.FloatTensor)`, optional, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`) — Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`.

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

- normalized_audio_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)`, optional, returned when model is initialized with `with_projection=True`) — The normalized audio embeddings obtained by applying the projection layer to the pooler_output, then applying L2 normalization and scaling the logits.

The ImageBindAudioModelWithProjection forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

    >>> import torch
    >>> import torchaudio
    >>> from datasets import load_dataset
    >>> from transformers import AutoProcessor, ImageBindAudioModelWithProjection

    >>> ds = load_dataset("EduardoPacheco/imagebind-example-data", split="train")
    >>> audio = ds[0]["audio"]
    >>> audio = torchaudio.functional.resample(torch.from_numpy(audio["array"]), orig_freq=audio["sampling_rate"], new_freq=16000).numpy()

    >>> model = ImageBindAudioModelWithProjection.from_pretrained("EduardoPacheco/imagebind-huge")
    >>> processor = AutoProcessor.from_pretrained("EduardoPacheco/imagebind-huge")

    >>> inputs = processor(audios=audio, return_tensors="pt")

    >>> outputs = model(**inputs)
    >>> audio_embeds = outputs.audio_embeds

< > Update on GitHub