---
library_name: transformers
tags: []
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->



## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

V-JEPA 2 is a self-supervised video backbone trained on >1 M h of internet video; Meta released checkpoints with a Something-Something v2 action head. I freeze that backbone and fine-tune only the classifier head on the HMDB-51 benchmark (6 766 clips, 51 classes) for 5 epochs. The resulting model reaches competitive top-1 accuracy (see Evaluation)

- **Developed by:** 	Sujit Shelar
- **Funded by :** 	self-funded (personal compute credits)
- **Shared by :** V-JEPA 2 ViT-Large (16 frame, 256² patch) video-encoder with a 51-way classification head
- **Model type:** 	Vision (video); no text inputs
- **Language(s) (NLP) :** [More Information Needed]
- **License :** MIT – identical to the upstream V-JEPA 2 weights
- **Finetuned from model :** facebook/vjepa2-vitl-fpc16-256-ssv2

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Rapid benchmarking or research on human-action recognition in academic settings.

Feature extractor for video retrieval or robotics perception pipelines.

### Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

Starting point for further fine-tuning on custom action datasets (e.g. UCF-101).

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

Any safety-critical decision-making (medical, legal, real-time surveillance).

Generation or captioning tasks – the model outputs only class logits.

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

HMDB-51 clips come largely from Hollywood movies and internet videos, so actions, environments and demographics are skewed towards Western-centric visual culture. The small dataset size (≈6 k clips) may lead to over-fitting and poor generalisation to unseen domains. Users should not rely on predictions for sensitive applications without additional validation. 

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

## How to Get Started with the Model

Use the code below to get started with the model.

```python
from transformers import AutoVideoProcessor, AutoModelForVideoClassification
import torch, torchvision

model_id = "SujitShelar/vjepa2-vitl-fpc16-256-hmdb51"
processor = AutoVideoProcessor.from_pretrained(model_id)
model     = AutoModelForVideoClassification.from_pretrained(model_id)

# sample one 5-sec clip with torchvision.io or torchcodec, shape (T,C,H,W)
video = torch.randn(16, 3, 256, 256)        # dummy tensor

inputs = processor(video.unsqueeze(0), return_tensors="pt")
logits = model(**inputs).logits
print(model.config.id2label[logits.argmax(-1).item()])
```


## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

HMDB-51 (CC BY-4.0, 6 766 clips across 51 classes). I stratify 70 / 15 / 15 % into train/val/test (4 736 / 1 015 / 1 015 clips). 

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

|                  | value                                                  |
| ---------------- | ------------------------------------------------------ |
| Frozen layers    | all V-JEPA 2 backbone blocks                           |
| Trainable params | 1.2 M (classification head)                            |
| Epochs           | 5                                                      |
| Effective batch  | 16 (physical 4 × grad-accum 4)                         |
| Optimiser        | Adam (lr 1 e-5)                                        |
| Augmentations    | RandomResizedCrop 256², RandomHorizontalFlip           |
| Hardware         | 1× nvidia-a100-80gb                                    |


#### Preprocessing

Clips are sampled at 16 frames per video (torchcodec.clips_at_random_indices), resized/cropped to 256², then normalised by the processor.


#### Training Hyperparameters

- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->

#### Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

[More Information Needed]

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

<!-- This should link to a Dataset Card if possible. -->

[More Information Needed]

#### Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

[More Information Needed]

#### Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

| Metric                      | Definition                                                                                                     | Why we use it                                                                                                                                                                |
| --------------------------- | -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Top-1 accuracy**          | Percentage of videos for which the *predicted* class label exactly matches the **single ground-truth action**. | HMDB-51 is a 51-way closed-set task; the community almost exclusively quotes Top-1, making our scores directly comparable to prior work.                                     |
| *(optional)* Top-5 accuracy | Video is considered correct if the ground-truth label appears in the five highest-probability classes.         | Helpful when the correct class is semantically close to others (e.g. *run* vs *walk*), but **not reported here** to keep the head-only baseline in line with earlier papers. |

Evaluation protocol

Split-1 of HMDB-51 (the canonical 70 / 15 / 15 % stratified split) is used for validation during training and for final test reporting.

We sample one 16-frame clip per video at 256 × 256 resolution and apply a single-crop evaluation, following the JEPA model card. This produces a 5-D tensor (B, T, C, H, W) that the VJEPA2VideoProcessor converts to model inputs.

Accuracy is averaged over the full validation or test set; no class weighting is applied.

### Results

| Split                           | Epochs |    Top-1 accuracy   |
| ------------------------------- | :----: | :-----------------: |
| Validation                      |  1 → 5 | 14.2 % → **41.9 %** |
| Test (single-crop, single-clip) |    —   |      **42.9 %**     |

<sub>Numbers come from the run shown in the training logs (runs/vjepa2_hmdb51).</sub>

How it compares
| Method (ViT-L backbone unless noted)  | Trainable params | Clips / crops at test | HMDB-51 Top-1                        |
| ------------------------------------- | ---------------- | --------------------- | ------------------------------------ |
| **This work – head-only JEPA-L**      | 1 M (0.3 %)      | 1 ✕ 1                 | **42.9 %**                           |
| Linear probe VideoMAE-B               | 0.1 %            | 1 ✕ 1                 | 38.9 % ([arxiv.org][1])              |
| Linear probe TimeSformer-B-IN pt      | full-frozen      | 3 ✕ 10                | 42.9 % (val) ([github.com][2])       |
| **AdaptFormer** (last-block adapters) | 1 %              | 1 ✕ 1                 | 46.1 % ([proceedings.neurips.cc][3]) |
| **CVPT** visual-prompt tuning         | <1 %             | 3 ✕ 10                | 57 %                                 |
| Full fine-tune TimeSformer-B          | 100 %            | 3 ✕ 10                | 64 % ([proceedings.neurips.cc][3])   |
| Full fine-tune VideoMAE-B             | 100 %            | 3 ✕ 10                | 73 % ([arxiv.org][1])                |
| **VideoMAE V2-G (giant)**             | 100 %            | 3 ✕ 10                | 86 % ([arxiv.org][4])                |
| InBrwSANet (CNN + SA)                 | 100 %            | 3 ✕ 10                | 77 % ([researchgate.net][5])         |

[1]: https://arxiv.org/pdf/2203.12602 "[PDF] VideoMAE: Masked Autoencoders are Data-Efficient ... - arXiv"
[2]: https://github.com/facebookresearch/TimeSformer/issues/19 "About UCF101 and HMDB51 results · Issue #19 - GitHub"
[3]: https://proceedings.neurips.cc/paper_files/paper/2022/file/69e2f49ab0837b71b0e0cb7c555990f8-Paper-Conference.pdf "[PDF] Adapting Vision Transformers for Scalable Visual Recognition"
[4]: https://arxiv.org/html/2402.08875v4 "Advancing Human Action Recognition with Foundation Models ..."
[5]: https://www.researchgate.net/publication/392129870_InBRwSANet_Self-attention_based_parallel_inverted_residual_bottleneck_architecture_for_human_action_recognition_in_smart_cities "(PDF) InBRwSANet: Self-attention based parallel inverted residual ..."


Take-away

42 – 43 % is in the upper range of published “backbone-frozen” baselines; unlocking a few transformer blocks, adding LoRA / prompt adapters, or running a full fine-tune typically raises HMDB-51 accuracy into the 55 – 70 % bracket. See the Bias, Risks & Limitations and Recommendations sections for caveats and upgrade suggestions.

#### Summary



## Model Examination [optional]

<!-- Relevant interpretability work for the model goes here -->

[More Information Needed]

## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]

## Technical Specifications 

### Model Architecture and Objective

ViT-Large (307 M params backbone) within the V-JEPA 2 framework.

16 × 16 image patches over 256² input; 16-frame temporal tube.

Classification head: two MLP layers (hidden 4 096 → 51 classes).

### Compute Infrastructure

[More Information Needed]

#### Hardware

[More Information Needed]

#### Software

[More Information Needed]

## Citation

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

@misc{shelar2025vjepa2hmdb51,
  title        = {V-JEPA2 ViT-L fine-tuned on HMDB-51},
  author       = {Sujit Shelar},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/SujitShelar/vjepa2-vitl-fpc16-256-hmdb51}},
  note         = {Fine-tuned from Assran et al. (2025) V-JEPA 2.}
}


**APA:**

[More Information Needed]

## Glossary [optional]

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

[More Information Needed]

## More Information [optional]

[More Information Needed]

## Model Card Authors [optional]

[More Information Needed]

## Model Card Contact

[More Information Needed]